andygrove commented on PR #1920: URL: https://github.com/apache/datafusion-comet/pull/1920#issuecomment-3005415617
In my local copy of Iceberg, I updated `SparkBatchQueryScan` to implement `SupportsComet`. When using both the Comet and Iceberg jars on the classpath, Comet is unable to recognize that `SparkBatchQueryScan` implements `SupportsComet`, and I think this is because there are two copies of the `SupportsComet` interface (one in the Comet jar and one in the Iceberg jar). If I just have the Iceberg jar on the classpath then Comet does try and accelerate the scan, but now I am running into Arrow shading issues. Spark 3.4.3 uses Arrow 11.0.0 Comet uses Arrow 18.3.0 Iceberg uses Arrow 15.0.2 It is not possible to shade the JNI classes in Arrow because the Java names have to match the function names in the native code, so this is quite challenging to resolve. We may need to look into using class loader isolation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
