mbutrovich opened a new issue, #4871: URL: https://github.com/apache/datafusion-comet/issues/4871
With the recent addition of Iceberg 1.11 compatibility to Comet (https://github.com/apache/datafusion-comet/pull/4840) we should codify what versions we will run in CI. #4840 has a table of testing after that PR, which I duplicate below: CI coverage across Spark and Iceberg versions (rows are the Spark version; columns are the Iceberg runtime paired with it): | Spark \ Iceberg | 1.5.2 | 1.8.1 | 1.9.1 | 1.10.0 | 1.11.0 | |-----------------|-------|-------|-------|--------|--------| | **3.4** | Comet suites — every PR | Iceberg tests — main + `run-iceberg-tests` | — | — | — | | **3.5** | — | Comet suites — every PR | Iceberg tests — main + `run-iceberg-tests` | Iceberg tests — main + `run-iceberg-tests` | — | | **4.0** | — | — | — | Comet suites — every PR | — | | **4.1** | — | — | — | — | Comet suites + Iceberg tests — every PR | | **4.2** | — | — | — | Comet suites — every PR | — | **Comet suites** = `CometIcebergNativeSuite` (and the other Iceberg-touching Comet unit suites) in the `pr_build_linux` "scans" bucket; the Iceberg runtime is pulled by the active `spark-*` profile in `spark/pom.xml`. Runs on every PR across all five Spark profiles. **Iceberg tests** = Apache Iceberg's own `iceberg-spark` / `-extensions` / `-runtime` suites run with Comet enabled, via the `iceberg_1_8` / `iceberg_1_9` / `iceberg_1_10` / `iceberg_1_11` jobs. 1.11 (Spark 4.1) runs on every PR; the older three run on push-to-main or on a PR labeled `run-iceberg-tests`. We maintain diffs for Iceberg 1.8, 1.9, 1.10, and 1.11 so we can run their Iceberg Java test suites. However, it really seems like each subsequent version is a superset of the former. We should 1) look at what Iceberg's stated support window is, 2) compare to Spark's stated support window, 3) reconcile with Comet's support/compatibility goals in order to come up with a consistent plan. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
