Hi Renjie, Yes you are right that `datafusion-iceberg` will not break. I somehow mistakenly thought that we cannot compile `datafusion` with a newer rust toolchain... Sorry for the mistake.
Regarding the motivations > 1. Use new(not latest) features in stable rust as rust is still evolving. In theory this can be done case by case, and we don't have a strong motivation yet. But +1 to upgrade periodically to avoid wasting time discussing whether a feature is "worth upgrading". > 2. Use newer versions of iceberg-rust's dependencies, which may include important bug fixes and performance improvements. I've seen dependent bot auto upgrading failed several times due to too old msrv. Dependencies is a complex problem. I want to clarify some points: 1. iceberg-rust is a library. The source of truth of the dependencies versions will be the consumer application's Cargo.lock. The version in iceberg-rust's Cargo.toml is a semver lower bound. MSRV or old dep version in Cargo.toml will not prevent applications using the latest version. 2. So there are different policies a library can do (ref: https://arc.net/l/quote/xghkkpqt) 2.1 Always use the latest dependency version in Cargo.toml, and force downstream applications to upgrade transitive dependencies at the same time. Benefits: let downstream enjoy latest bug fixes and performance improvements of deps Drawback: downstream is forced to upgrade deps together; They will need to audit the changes of all transitive deps, instead of upgrading 1 by 1. 2.2 Do not upgrade Cargo.toml, and let the application be responsible for the transitive dependencies' version. Instead, we upgrade in Cargo.lock which means "we've tested the latest version". I'm not sure which dependency upgrade policy we want to adopt. (BTW, we've once also discussed this in avro-rs, and they prefer the first one https://github.com/apache/avro-rs/issues/4) Although this is a little off-topic, but it seems similar to the MSRV problem. Dependency policy may also conflict with MSRV policy. e.g., as mentioned by Renjie, we cannot upgrade to latest version of a dep with a low MSRV. (So if we want to maintain low MSRV, we have to adopt the second dependency policy.) xxchan On Mon, Feb 24, 2025 at 11:45 AM Renjie Liu <liurenjie2...@gmail.com> wrote: > Hi, xxchan: > > In fact, datafusion is not a consumer of iceberg-rust, in fact, > `datafusion-iceberg` is a consumer of both `iceberg` and `datafusion`, > which is maintained in `iceberg-rust` repo, so a more aggressive msrv > policy will not break `datafusion-iceberg`. > > On Sun, Feb 23, 2025 at 8:40 PM xxchan <xxchan...@gmail.com> wrote: > >> Hi, >> >> To clarify, if we want to adopt the "at least three months" policy, we >> will need to wait for 3 months before upgrading to 1.85. >> And I believe we don't want to break Datafusion integration, so we will >> have to use the "at least six months" policy (or more conservative). >> >> xxchan >> >> On Sun, Feb 23, 2025 at 5:50 PM NOTME ZE <st810918...@gmail.com> wrote: >> >>> Hi, >>> >>> As Xuanwo mentions the "at least three months," policy looks good to me. >>> Also, it's time to upgrade to Rust 1.85 (the first version to support >>> Edition 2024!). See: https://github.com/apache/iceberg-rust/pull/987 >>> >>
