andygrove commented on issue #2453:
URL:
https://github.com/apache/datafusion-comet/issues/2453#issuecomment-3335514079
More info.
This starts when an allocation returns less than the request:
```java
// Called by Comet native through JNI.
// Returns the actual amount of memory (in bytes) granted.
public long acquireMemory(long size) {
long acquired = internal.acquireExecutionMemory(size,
nativeMemoryConsumer);
used.addAndGet(acquired);
if (acquired < size) {
// If memory manager is not able to acquire the requested size, log
memory usage
internal.showMemoryUsage();
}
return acquired;
}
```
This logs messages such as:
```
INFO TaskMemoryManager: Memory used in task 21129
INFO TaskMemoryManager: 3221225472 bytes of memory were used by task 21129
but are not associated with specific consumers
INFO TaskMemoryManager: 18329799984 bytes of memory are used for execution
and 1009734 bytes of memory are used for storage
```
This shows that one task is using using 3GB and 18GB is allocated in the
off-heap pool in total.
This is followed by
```
WARN ExecutionMemoryPool: Internal error: release called on 821408 bytes but
task only has 0 bytes of memory from the off-heap execution pool
ERROR Executor: Exception in task 113.0 in stage 199.3 (TID 21150)
comet-benchmark-derived-from-tpch-08872799821543de-exec-7
spark-kubernetes-executor org.apache.comet.CometNativeException: Additional
allocation failed with top memory consumers (across reservations) as:
comet-benchmark-derived-from-tpch-08872799821543de-exec-7
spark-kubernetes-executor ExternalSorter[138]#692(can spill: true) consumed
2.9 GB, peak 3.1 GB,
comet-benchmark-derived-from-tpch-08872799821543de-exec-7
spark-kubernetes-executor ExternalSorterMerge[138]#693(can spill: false)
consumed 70.2 MB, peak 70.2 MB,
comet-benchmark-derived-from-tpch-08872799821543de-exec-7
spark-kubernetes-executor SMJStream[138]#694(can spill: false) consumed 0.0
B, peak 0.0 B.
comet-benchmark-derived-from-tpch-08872799821543de-exec-7
spark-kubernetes-executor Error: Failed to acquire 409640 bytes, only got
83912. Reserved: 3222910152
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
