Hi, Xuanwo: Thanks for pointing that out, and you're right about the version used.
However, if iceberg depends on lib A 0.1.10, but app B depends on lib A > 0.1.9, users will finally get: > - iceberg 0.3 > - lib A 0.1.10 (the biggest if all lib A 0.1.X) My point is still the same, if iceberg uses lib A 0.1.9, and the 0.1.10 version of lib A doesn't work with iceberg 0.3, the user of iceberg will face either of: 1. Forced to use the old version of lib A, without enjoying the performance / bug fix of the new version of lib A. 2. Upgrade to newer version of lib A, but surprised to fail in tests or production. On Mon, Feb 24, 2025 at 6:38 PM Xuanwo <xua...@apache.org> wrote: > Hi, renjie > > > 1. Iceberg 0.3 dependents library a 0.1 > > 2. Application b dependents on iceberg 0.3 and library a 0.2 > > 3. Library a 0.2 doesn't work with iceberg 0.3 > > 4. Application b crashed in production. > > This example can't happen in rust. In rust, every version within 0.X is > considered a breaking change. > > So if iceberg 0.3 depends on lib A 0.1, and app B depends on iceberg 0.3 > and ;ib A 0.2, this user will finally get: > > - iceberg 0.3 > - lib A 0.1 > - lib A 0.2 > > Which means the two version of lib A can coexists in the same app B. > > However, if iceberg depends on lib A 0.1.10, but app B depends on lib A > 0.1.9, users will finally get: > > - iceberg 0.3 > - lib A 0.1.10 (the biggest if all lib A 0.1.X) > > > On Mon, Feb 24, 2025, at 18:12, Renjie Liu wrote: > > Thanks to xxchan's explanation, it seems that we need to resolve the > dependency version first. > > 2.1 Always use the latest dependency version in Cargo.toml, and force > downstream applications to upgrade transitive dependencies at the same time. > Benefits: let downstream enjoy latest bug fixes and performance > improvements of deps > Drawback: downstream is forced to upgrade deps together; They will need to > audit the changes of all transitive deps, instead of upgrading 1 by 1. > > > For the dependency version, I prefer this one. Though in theory the > dependencies of iceberg are supposed to maintain compatibility if the major > version is not changed, it would still be better to test against them. > Using a dependent bot to upgrade the dependency version would help to avoid > cases like this. For example: > 1. Iceberg 0.3 dependents library a 0.1 > 2. Application b dependents on iceberg 0.3 and library a 0.2 > 3. Library a 0.2 doesn't work with iceberg 0.3 > 4. Application b crashed in production. > > I know this approach doesn't solve all problems, but at least it helps us > to detect such a problem and save our time to fix it. > > > > On Mon, Feb 24, 2025 at 1:47 PM Xuanwo <xua...@apache.org> wrote: > > > > 2.2 Do not upgrade Cargo.toml, and let the application be responsible > for the transitive dependencies' version. Instead, we upgrade in Cargo.lock > which means "we've tested the latest version". > > I prefer this approach. I believe we should disable dependabot from > upgrading our patch versions, as it doesn't make sense to me. Users should > be able to decide exactly which versions they want to use. As a library, we > specify the lowest version that we are confident works well with > iceberg-rust or includes all the APIs that iceberg-rust relies on. > > On Mon, Feb 24, 2025, at 13:28, xxchan wrote: > > Hi Renjie, > > Yes you are right that `datafusion-iceberg` will not break. I somehow > mistakenly thought that we cannot compile `datafusion` with a newer rust > toolchain... Sorry for the mistake. > > Regarding the motivations > > > 1. Use new(not latest) features in stable rust as rust is still evolving. > > In theory this can be done case by case, and we don't have a > strong motivation yet. > But +1 to upgrade periodically to avoid wasting time discussing whether a > feature is "worth upgrading". > > > 2. Use newer versions of iceberg-rust's dependencies, which may include > important bug fixes and performance improvements. I've seen dependent bot > auto upgrading failed several times due to too old msrv. > > Dependencies is a complex problem. I want to clarify some points: > > 1. iceberg-rust is a library. The source of truth of the dependencies > versions will be the consumer application's Cargo.lock. The version in > iceberg-rust's Cargo.toml is a semver lower bound. MSRV or old dep version > in Cargo.toml will not prevent applications using the latest version. > > 2. So there are different policies a library can do (ref: > https://arc.net/l/quote/xghkkpqt) > > 2.1 Always use the latest dependency version in Cargo.toml, and force > downstream applications to upgrade transitive dependencies at the same time. > Benefits: let downstream enjoy latest bug fixes and performance > improvements of deps > Drawback: downstream is forced to upgrade deps together; They will need to > audit the changes of all transitive deps, instead of upgrading 1 by 1. > > 2.2 Do not upgrade Cargo.toml, and let the application be responsible for > the transitive dependencies' version. Instead, we upgrade in Cargo.lock > which means "we've tested the latest version". > > I'm not sure which dependency upgrade policy we want to adopt. (BTW, we've > once also discussed this in avro-rs, and they prefer the first one > https://github.com/apache/avro-rs/issues/4) > Although this is a little off-topic, but it seems similar to the MSRV > problem. Dependency policy may also conflict with MSRV policy. e.g., as > mentioned by Renjie, we cannot upgrade to latest version of a dep with a > low MSRV. (So if we want to maintain low MSRV, we have to adopt the second > dependency policy.) > > xxchan > > On Mon, Feb 24, 2025 at 11:45 AM Renjie Liu <liurenjie2...@gmail.com> > wrote: > > Hi, xxchan: > > In fact, datafusion is not a consumer of iceberg-rust, in fact, > `datafusion-iceberg` is a consumer of both `iceberg` and `datafusion`, > which is maintained in `iceberg-rust` repo, so a more aggressive msrv > policy will not break `datafusion-iceberg`. > > On Sun, Feb 23, 2025 at 8:40 PM xxchan <xxchan...@gmail.com> wrote: > > Hi, > > To clarify, if we want to adopt the "at least three months" policy, we > will need to wait for 3 months before upgrading to 1.85. > And I believe we don't want to break Datafusion integration, so we will > have to use the "at least six months" policy (or more conservative). > > xxchan > > On Sun, Feb 23, 2025 at 5:50 PM NOTME ZE <st810918...@gmail.com> wrote: > > Hi, > > > As Xuanwo mentions the "at least three months," policy looks good to me. > Also, it's time to upgrade to Rust 1.85 (the first version to support > Edition 2024!). See: https://github.com/apache/iceberg-rust/pull/987 > > > Xuanwo > > https://xuanwo.io/ > > Xuanwo > > https://xuanwo.io/ > >
