andygrove commented on PR #1063:
URL:
https://github.com/apache/datafusion-comet/pull/1063#issuecomment-2462429540
test failure:
```
[info] - Spark vectorized reader - with partition data column - select a
single complex field from a map entry and its parent map entry *** FAILED ***
(653 milliseconds)
[info] org.apache.spark.SparkException: Job aborted due to stage failure:
Task 0 in stage 215.0 failed 1 times, most recent failure: Lost task 0.0 in
stage 215.0 (TID 370) (4bf8ef4698e6 executor driver):
java.lang.IllegalArgumentException: CometShuffleMemoryAllocator should be used
with off-heap memory mode, but got ON_HEAP
[info] at
org.apache.spark.shuffle.comet.CometShuffleMemoryAllocator.getInstance(CometShuffleMemoryAllocator.java:44)
[info] at
org.apache.spark.sql.comet.execution.shuffle.CometDiskBlockWriter.<init>(CometDiskBlockWriter.java:139)
[info] at
org.apache.spark.sql.comet.execution.shuffle.CometBypassMergeSortShuffleWriter.write(CometBypassMergeSortShuffleWriter.java:181)
```
I think we need to specify `spark.memory.offHeap.enabled=true` when running
Spark tests? I need to do the same in
https://github.com/apache/datafusion-comet/pulls
For this PR we should also fall back to Spark for shuffle if
`spark.memory.offHeap.enabled=false`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
