SemyonSinchenko opened a new issue, #190:
URL: https://github.com/apache/arrow-datafusion-comet/issues/190
### Describe the bug
I want to try Comet on Databricks.
I built Comet for Spark 3.4 (`-Prelease -Pspark3-4`), added it to the jars
via init-script, added all needed configurations (`spark conf`). When I am
running any query, it tends to the following:
```
com.databricks.backend.common.rpc.SparkDriverExceptions$SQLExecutionException:
java.lang.NoSuchMethodError:
org.apache.spark.sql.execution.FileSourceScanExec.copy$default$9()Z
at
org.apache.spark.sql.comet.CometScanExec$.apply(CometScanExec.scala:442)
at
org.apache.comet.CometSparkSessionExtensions$CometScanRule$$anonfun$apply$5.applyOrElse(CometSparkSessionExtensions.scala:127)
at
org.apache.comet.CometSparkSessionExtensions$CometScanRule$$anonfun$apply$5.applyOrElse(CometSparkSessionExtensions.scala:77)
at
org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:465)
at
org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:69)
at
org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:465)
at
org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$3(TreeNode.scala:470)
at
org.apache.spark.sql.catalyst.trees.UnaryLike.mapChildren(TreeNode.scala:1269)
at
org.apache.spark.sql.catalyst.trees.UnaryLike.mapChildren$(TreeNode.scala:1268)
at
org.apache.spark.sql.execution.FilterExec.mapChildren(basicPhysicalOperators.scala:216)
```
### Steps to reproduce
1. Built Comet for spark 3.4
2. Add init-script to Databricks cluster, that adds Comet-JAR to
`/databricks/jars`
3. Run any query
### Expected behavior
Comet is working on Databricks 13.3 LTS (Spark 3.4.1, Scala 2.12)
### Additional context
I can provide more extended stack-trace, but there is nothing special before
lines above, only catalyst-calls.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
