dwsmith1983 commented on issue #5882: URL: https://github.com/apache/datafusion-comet/issues/5882#issuecomment-6547256976
Some data from trying the contrib against 1.2.0 the way a user would, which may help whichever way this goes. I built `comet-contrib-delta-spark` from the `1.2.0-rc1` tag in a clean `maven:3.9-eclipse-temurin-17` container, with `comet-spark-spark4.1_2.13:1.2.0` resolved as an external artifact (built from the same commit, since staging wasn't reachable), and ran it on `apache/spark:4.1.3` with delta-spark 4.3.1. The native scan took a table with deletion vectors (`CometDeltaNativeScan` in the plan) and matched Spark's reader, so the released `libcomet` does carry everything the contrib needs, as the issue says. Two things a build-from-source user runs into today: 1. `./mvnw -Pspark-4.1,delta -pl contrib/delta-spark package -Dmaven.test.skip=true` fails outside the reactor. Maven resolves the test-scoped `comet-spark` `tests` jar before it skips anything, and that jar isn't published. Calling the goals directly (`resources:resources scala:compile jar:jar`) works. This ties into question 3: if the test-jar stays unpublished, the README should give that command, or the contrib's dependency on it should move behind a profile so a plain `package` works against published artifacts. 2. Loading Delta with `--packages` puts delta-spark in a child classloader. The contrib catches the resulting `NoClassDefFoundError: org/apache/spark/sql/delta/DeltaParquetFileFormat` and declines the scan, so the query quietly runs on Spark's reader, and the warning says the handler "was built against a different version of Comet's internals", which points at the wrong cause. Whatever the publishing answer, the user guide needs the one-classpath requirement (all jars on `extraClassPath`), and the warning could name the missing class instead. On question 1, I'd lean toward (b): publish the jar for Spark 3.5, 4.0 and 4.1 starting with 1.3.0, with the test-jar execution moved into the `delta` profile, so the experimental reader gets tested on real workloads without a source build. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
