andygrove opened a new issue, #1438:
URL: https://github.com/apache/datafusion-comet/issues/1438
### Describe the bug
I am using unified memory management:
```
--conf spark.memory.offHeap.enabled=true \
--conf spark.memory.offHeap.size=2g \
```
I deliberately allocated a small amount of memory for testing purposes.
However, the stack trace below shows that Comet is using
`CometTestShuffleMemoryAllocator` which should not be the case.
```
Caused by: org.apache.spark.memory.SparkOutOfMemoryError: Unable to acquire
16777216 bytes of memory, got 2588672 bytes. Available: 2588672
at
org.apache.spark.shuffle.comet.CometTestShuffleMemoryAllocator.allocateMemoryBlock(CometTestShuffleMemoryAllocator.java:129)
at
org.apache.spark.shuffle.comet.CometTestShuffleMemoryAllocator.allocate(CometTestShuffleMemoryAllocator.java:116)
at
org.apache.spark.sql.comet.execution.shuffle.SpillWriter.initialCurrentPage(SpillWriter.java:158)
at
org.apache.spark.shuffle.sort.CometShuffleExternalSorter.insertRecord(CometShuffleExternalSorter.java:374)
at
org.apache.spark.sql.comet.execution.shuffle.CometUnsafeShuffleWriter.insertRecordIntoSorter(CometUnsafeShuffleWriter.java:278)
at
org.apache.spark.sql.comet.execution.shuffle.CometUnsafeShuffleWriter.write(CometUnsafeShuffleWriter.java:206)
```
This is created in the following code in `CometShuffleMemoryAllocatorTrait`:
```scala
boolean useUnifiedMemAllocator =
(boolean)
CometConf$.MODULE$.COMET_COLUMNAR_SHUFFLE_UNIFIED_MEMORY_ALLOCATOR_IN_TEST().get();
if (!useUnifiedMemAllocator) {
synchronized (CometShuffleMemoryAllocator.class) {
if (INSTANCE == null) {
// CometTestShuffleMemoryAllocator handles pages by itself so it
can be a singleton.
INSTANCE = new CometTestShuffleMemoryAllocator(conf,
taskMemoryManager, pageSize);
}
}
return INSTANCE;
```
It seems that the wrong config is being checked here.
### Steps to reproduce
_No response_
### Expected behavior
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
