viirya commented on code in PR #250:
URL:
https://github.com/apache/arrow-datafusion-comet/pull/250#discussion_r1561585917
##########
dev/diffs/3.4.2.diff:
##########
@@ -1414,6 +1424,7 @@ index ed2e309fa07..4cfe0093da7 100644
+ .set("spark.shuffle.manager",
+
"org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager")
+ .set("spark.comet.exec.shuffle.enabled", "true")
++ .set("spark.comet.memoryOverhead", "10g")
Review Comment:
Observed that Comet is unable to acquire enough memory for columnar shuffle
when doing Spark SQL tests:
For example, DatasetPrimitiveSuite:
```
Cause: org.apache.spark.SparkException: Job aborted due to stage failure:
Task 1 in stage 37.0 failed 1 times, most recent failure: Lost task 1.0 in
stage 37.0 (TID 75) (e4773b5abe7e executor driver):
org.apache.spark.memory.SparkOutOfMemoryError: Unable to acquire 67108848 bytes
of memory, got 96 bytes. Available: 96
[info] at
org.apache.spark.shuffle.comet.CometShuffleMemoryAllocator.allocate(CometShuffleMemoryAllocator.java:132)
[info] at
org.apache.spark.shuffle.comet.CometShuffleMemoryAllocator.allocatePage(CometShuffleMemoryAllocator.java:119)
[info] at
org.apache.spark.sql.comet.execution.shuffle.SpillWriter.initialCurrentPage(SpillWriter.java:158)
[info] at
org.apache.spark.sql.comet.execution.shuffle.CometDiskBlockWriter.insertRow(CometDiskBlockWriter.java:284)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
