andygrove opened a new issue, #408:
URL: https://github.com/apache/datafusion-comet/issues/408
### Describe the bug
I am benchmarking locally with small scale factor (100GB) and saw q16 fail.
Debug logging showed some reasons we can't support this query natively:
```
WARN CometSparkSessionExtensions$CometExecRule: Comet cannot execute this
plan natively because:
- BuildRight is not supported
- Native shuffle is not enabled
```
The query failed at runtime:
```
org.apache.spark.SparkUnsupportedOperationException: ColumnarToRow does not
implement doExecuteBroadcast.
```
I used these Comet configs:
```
command.append("--jars $COMET_JAR")
command.append("--conf
spark.sql.extensions=org.apache.comet.CometSparkSessionExtensions")
command.append("--conf spark.comet.enabled=true")
command.append("--conf spark.comet.explainFallback.enabled=true")
command.append("--conf spark.comet.exec.enabled=true")
command.append("--conf spark.comet.exec.all.enabled=true")
command.append("--conf spark.comet.exec.all.expr.enabled=true")
command.append("--conf spark.comet.exec.shuffle.enabled=true")
command.append("--conf spark.comet.exec.broadcast.enabled=true")
command.append("--conf spark.comet.columnar.shuffle.enabled=true")
command.append("--conf spark.comet.cast.allowIncompatible=true")
command.append("--conf
spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager")
command.append("--conf spark.driver.extraClassPath=$COMET_JAR")
command.append("--conf spark.executor.extraClassPath=$COMET_JAR")
```
### Steps to reproduce
_No response_
### Expected behavior
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
