Conclusion in today's meeting: - Test against latest version in CI (by updating Cargo.lock with dependabot) - Have a wider range of versions support, to allow users to choose their dep version (by not updating Cargo.toml too often). Perhaps need to add a CI to test against oldest version.
On Thu, Feb 27, 2025 at 2:03 PM Renjie Liu <liurenjie2...@gmail.com> wrote: > Hi, Xuanwo: > > If we detect that iceberg is not compatible with lib A 0.1.10, then we > could know that in advance, and we could figure out ways to fix it. > > Users have several choices, like using Cargo.lock to pin the version used. > > We could have a discussion in this week's community sync. > > On Tue, Feb 25, 2025 at 12:05 PM Xuanwo <xua...@apache.org> wrote: > >> Hi, renjie >> >> > My point is still the same, if iceberg uses lib A 0.1.9, and the 0.1.10 >> version of lib A doesn't work with iceberg 0.3, the user of iceberg will >> face either of: >> > 1. Forced to use the old version of lib A, without enjoying the >> performance / bug fix of the new version of lib A. >> > 2. Upgrade to newer version of lib A, but surprised to fail in tests or >> production. >> >> I'm not sure what we can do about this. >> >> If Iceberg uses version 0.1.9, users can choose which version of lib A to >> use—either staying at 0.1.9 or upgrading to 0.1.10 (even it's broken). >> However, if Iceberg enforces the use of 0.1.10, users will always face >> issues, with no option to revert to 0.1.9. >> >> Regardless of the actions we take, we still need the upstream to fix it. >> Alternatively, we may need to migrate to other libraries. >> >> My point is to let users make this decision themselves. If we now know >> iceberg-rust doesn't work with lib A versions lower than 0.1.11. We can set >> it to 0.1.11 to prevent users from relying on the outdated 0.1.10 version. >> >> On Tue, Feb 25, 2025, at 10:53, Renjie Liu wrote: >> >> Hi, Xuanwo: >> >> Thanks for pointing that out, and you're right about the version used. >> >> However, if iceberg depends on lib A 0.1.10, but app B depends on lib A >> 0.1.9, users will finally get: >> - iceberg 0.3 >> - lib A 0.1.10 (the biggest if all lib A 0.1.X) >> >> >> My point is still the same, if iceberg uses lib A 0.1.9, and the 0.1.10 >> version of lib A doesn't work with iceberg 0.3, the user of iceberg will >> face either of: >> 1. Forced to use the old version of lib A, without enjoying the >> performance / bug fix of the new version of lib A. >> 2. Upgrade to newer version of lib A, but surprised to fail in tests or >> production. >> >> >> >> On Mon, Feb 24, 2025 at 6:38 PM Xuanwo <xua...@apache.org> wrote: >> >> >> Hi, renjie >> >> > 1. Iceberg 0.3 dependents library a 0.1 >> > 2. Application b dependents on iceberg 0.3 and library a 0.2 >> > 3. Library a 0.2 doesn't work with iceberg 0.3 >> > 4. Application b crashed in production. >> >> This example can't happen in rust. In rust, every version within 0.X is >> considered a breaking change. >> >> So if iceberg 0.3 depends on lib A 0.1, and app B depends on iceberg 0.3 >> and ;ib A 0.2, this user will finally get: >> >> - iceberg 0.3 >> - lib A 0.1 >> - lib A 0.2 >> >> Which means the two version of lib A can coexists in the same app B. >> >> However, if iceberg depends on lib A 0.1.10, but app B depends on lib A >> 0.1.9, users will finally get: >> >> - iceberg 0.3 >> - lib A 0.1.10 (the biggest if all lib A 0.1.X) >> >> >> On Mon, Feb 24, 2025, at 18:12, Renjie Liu wrote: >> >> Thanks to xxchan's explanation, it seems that we need to resolve the >> dependency version first. >> >> 2.1 Always use the latest dependency version in Cargo.toml, and force >> downstream applications to upgrade transitive dependencies at the same time. >> Benefits: let downstream enjoy latest bug fixes and performance >> improvements of deps >> Drawback: downstream is forced to upgrade deps together; They will need >> to audit the changes of all transitive deps, instead of upgrading 1 by 1. >> >> >> For the dependency version, I prefer this one. Though in theory the >> dependencies of iceberg are supposed to maintain compatibility if the major >> version is not changed, it would still be better to test against them. >> Using a dependent bot to upgrade the dependency version would help to avoid >> cases like this. For example: >> 1. Iceberg 0.3 dependents library a 0.1 >> 2. Application b dependents on iceberg 0.3 and library a 0.2 >> 3. Library a 0.2 doesn't work with iceberg 0.3 >> 4. Application b crashed in production. >> >> I know this approach doesn't solve all problems, but at least it helps us >> to detect such a problem and save our time to fix it. >> >> >> >> On Mon, Feb 24, 2025 at 1:47 PM Xuanwo <xua...@apache.org> wrote: >> >> >> > 2.2 Do not upgrade Cargo.toml, and let the application be responsible >> for the transitive dependencies' version. Instead, we upgrade in Cargo.lock >> which means "we've tested the latest version". >> >> I prefer this approach. I believe we should disable dependabot from >> upgrading our patch versions, as it doesn't make sense to me. Users should >> be able to decide exactly which versions they want to use. As a library, we >> specify the lowest version that we are confident works well with >> iceberg-rust or includes all the APIs that iceberg-rust relies on. >> >> On Mon, Feb 24, 2025, at 13:28, xxchan wrote: >> >> Hi Renjie, >> >> Yes you are right that `datafusion-iceberg` will not break. I somehow >> mistakenly thought that we cannot compile `datafusion` with a newer rust >> toolchain... Sorry for the mistake. >> >> Regarding the motivations >> >> > 1. Use new(not latest) features in stable rust as rust is still >> evolving. >> >> In theory this can be done case by case, and we don't have a >> strong motivation yet. >> But +1 to upgrade periodically to avoid wasting time discussing whether a >> feature is "worth upgrading". >> >> > 2. Use newer versions of iceberg-rust's dependencies, which may include >> important bug fixes and performance improvements. I've seen dependent bot >> auto upgrading failed several times due to too old msrv. >> >> Dependencies is a complex problem. I want to clarify some points: >> >> 1. iceberg-rust is a library. The source of truth of the dependencies >> versions will be the consumer application's Cargo.lock. The version in >> iceberg-rust's Cargo.toml is a semver lower bound. MSRV or old dep version >> in Cargo.toml will not prevent applications using the latest version. >> >> 2. So there are different policies a library can do (ref: >> https://arc.net/l/quote/xghkkpqt) >> >> 2.1 Always use the latest dependency version in Cargo.toml, and force >> downstream applications to upgrade transitive dependencies at the same time. >> Benefits: let downstream enjoy latest bug fixes and performance >> improvements of deps >> Drawback: downstream is forced to upgrade deps together; They will need >> to audit the changes of all transitive deps, instead of upgrading 1 by 1. >> >> 2.2 Do not upgrade Cargo.toml, and let the application be responsible for >> the transitive dependencies' version. Instead, we upgrade in Cargo.lock >> which means "we've tested the latest version". >> >> I'm not sure which dependency upgrade policy we want to adopt. (BTW, >> we've once also discussed this in avro-rs, and they prefer the first one >> https://github.com/apache/avro-rs/issues/4) >> Although this is a little off-topic, but it seems similar to the MSRV >> problem. Dependency policy may also conflict with MSRV policy. e.g., as >> mentioned by Renjie, we cannot upgrade to latest version of a dep with a >> low MSRV. (So if we want to maintain low MSRV, we have to adopt the second >> dependency policy.) >> >> xxchan >> >> On Mon, Feb 24, 2025 at 11:45 AM Renjie Liu <liurenjie2...@gmail.com> >> wrote: >> >> Hi, xxchan: >> >> In fact, datafusion is not a consumer of iceberg-rust, in fact, >> `datafusion-iceberg` is a consumer of both `iceberg` and `datafusion`, >> which is maintained in `iceberg-rust` repo, so a more aggressive msrv >> policy will not break `datafusion-iceberg`. >> >> On Sun, Feb 23, 2025 at 8:40 PM xxchan <xxchan...@gmail.com> wrote: >> >> Hi, >> >> To clarify, if we want to adopt the "at least three months" policy, we >> will need to wait for 3 months before upgrading to 1.85. >> And I believe we don't want to break Datafusion integration, so we will >> have to use the "at least six months" policy (or more conservative). >> >> xxchan >> >> On Sun, Feb 23, 2025 at 5:50 PM NOTME ZE <st810918...@gmail.com> wrote: >> >> Hi, >> >> >> As Xuanwo mentions the "at least three months," policy looks good to me. >> Also, it's time to upgrade to Rust 1.85 (the first version to support >> Edition 2024!). See: https://github.com/apache/iceberg-rust/pull/987 >> >> >> Xuanwo >> >> https://xuanwo.io/ >> >> Xuanwo >> >> https://xuanwo.io/ >> >> Xuanwo >> >> https://xuanwo.io/ >> >>
