comphead commented on PR #5666:
URL:
https://github.com/apache/datafusion-comet/pull/5666#issuecomment-5531193980
Tested on
```
/opt/homebrew/Cellar/apache-spark/3.5.5/bin/spark-submit
--master="local[*]" --conf spark.driver.memory=8G --conf
spark.executor.instances=1 --conf spark.executor.cores=8 --conf
spark.cores.max=8 --conf spark.executor.memory=16g --conf
spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=16g
--conf spark.comet.explainFallback.enabled=true --conf
spark.comet.logFallbackReasons.enabled=true --conf spark.eventLog.enabled=false
--jars $COMET_JAR --driver-class-path $COMET_JAR --conf
spark.driver.extraClassPath=$COMET_JAR --conf
spark.executor.extraClassPath=$COMET_JAR --conf
spark.plugins=org.apache.spark.CometPlugin --conf
spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager
--conf spark.comet.enabled=true --conf
spark.comet.exec.shuffle.enableFastEncoding=true --conf
spark.comet.cast.allowIncompatible=true --conf
spark.comet.exec.memoryGuard.enabled=true --conf spark.com
et.exec.memoryPool=real_usage
../datafusion-benchmarks/runners/datafusion-comet/tpcbench.py --benchmark
tpcds --data ../datafusion-benchmarks/tpcds/data_parquet/ --queries
../datafusion-benchmarks/tpcds/queries-spark --query 67 --output .
--iterations 1
```
Before PR the Q67 failed on
```
26/09/02 09:12:13 WARN TaskSetManager: Lost task 110.0 in stage 29.0 (TID
261) (192.168.4.142 executor driver): TaskKilled (Stage cancelled: Job aborted
due to stage failure: Task 109 in stage 29.0 failed 1 times, most recent
failure: Lost task 109.0 in stage 29.0 (TID 260) (192.168.4.142 executor
driver): org.apache.comet.CometNativeException: Additional allocation failed
for ExternalSorterMerge[0] with top memory consumers (across reservations) as:
ExternalSorter[0]#1106(can spill: true) consumed 1024.0 MB, peak 1058.2 MB,
ExternalSorterMerge[0]#1107(can spill: false) consumed 456.0 B, peak 10.0
MB,
ShuffleRepartitioner[0]#1109(can spill: true) consumed 0.0 B, peak 0.0 B.
Error: Failed to acquire 228842 bytes, only got 11829 bytes. Reserved:
1073729995 bytes
```
After PR the query passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
