dwsmith1983 opened a new issue, #7394: URL: https://github.com/apache/datafusion-comet/issues/7394
### Describe the bug When `delta-spark` is loaded with `--packages` while Comet and the Delta contrib jar are on `extraClassPath`, Delta ends up in a child classloader that Comet's classes can't see. The contrib's scan hook then throws `NoClassDefFoundError: org/apache/spark/sql/delta/DeltaParquetFileFormat`. `CometScanContrib.firstClaimFrom` catches every `LinkageError` the same way and logs: ``` Contrib scan handler ... failed with java.lang.NoClassDefFoundError, indicating it was built against a different version of Comet's internals than is on the classpath now; declining it and continuing with Comet's built-in handling ``` The query then runs on Spark's reader. The jars all match, so the message names the wrong cause, and since the fallback is silent, that log line is the only hint the user gets. Found in #5882 ([comment](https://github.com/apache/datafusion-comet/issues/5882#issuecomment-6547256976)). ### Steps to reproduce 1. Spark 4.1.3, Comet 1.2.0 (`comet-spark-spark4.1_2.13`) and `comet-contrib-delta-spark` on `spark.driver.extraClassPath` and `spark.executor.extraClassPath`. 2. Add delta-spark 4.3.1 with `--packages io.delta:delta-spark_2.13:4.3.1`, plus the usual Delta session extension and catalog settings and `spark.comet.scan.delta.enabled=true`. 3. Read any Delta table. The plan has no `CometDeltaNativeScan`, and the driver log shows the warning above. With all jars on `extraClassPath` the same read plans `CometDeltaNativeScan`. ### Expected behavior - For `NoClassDefFoundError` or `ClassNotFoundException`, the warning names the missing class and says it isn't visible from the classloader that loaded Comet, mentioning `--packages` or `--jars` as the likely cause. The version-skew wording stays for `NoSuchMethodError`, `AbstractMethodError` and the other linkage errors, where it fits. - The Delta user guide (`docs/source/user-guide/latest/delta.md`) states that Comet, the contrib jar and `delta-spark` have to be on the same classloader, for example all on `spark.{driver,executor}.extraClassPath`. It currently only says the contrib jar goes "on the classpath, alongside `delta-spark`". ### Additional context The catch is in `spark/src/main/scala/org/apache/comet/rules/CometScanContrib.scala` (`firstClaimFrom`). `ContribServices` has similar "built against a different Comet or Spark version" wording for provider loading failures, which may deserve the same split. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
