dwsmith1983 opened a new issue, #7455: URL: https://github.com/apache/datafusion-comet/issues/7455
The `spark-4.2` profile pins `delta.spark.version` to 4.3.1, and the comment above it says no Delta release targets Spark 4.2 yet. That was true when the contrib merged, but Delta 4.4.0 (August 20) added Spark 4.2 support, and 4.4.1 is the current release. The support table in `contrib/delta-spark/README.md` and the comment on the `delta_4_1` job in `ci.yml` make the same assumption, so the contrib suites have never run on 4.2. As pinned, the pairing can't run. On Spark 4.2.0, Delta 4.3.1 fails in its own `CREATE TABLE` with `NoSuchMethodError` on `CatalogStorageFormat.copy`, before Comet is involved. With Delta 4.4.1 it works in a quick test against 1.2.0. I built the contrib with `-Pspark-4.2,delta` (the jar is byte-identical whether it compiles against 4.3.1 or 4.4.1), then ran `comet-spark-spark4.2_2.13-1.2.0` from Maven Central with delta-spark 4.4.1 on `apache/spark:4.2.0`. A table with deletion vectors read through `CometDeltaNativeScan`, and the results matched Spark's reader with `spark.comet.scan.delta.enabled=false`. That's one query, not the contrib suites. Proposed fix: set `delta.spark.version` to 4.4.1 in the `spark-4.2` profile, add a `delta_4_2` caller in `ci.yml`, and update the README table and the user guide. I can open the PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
