marin-ma commented on issue #6863:
URL:
https://github.com/apache/incubator-gluten/issues/6863#issuecomment-2290864191
Shuffle spill is triggered but it fails to allocate extra memory during spill
```
at
org.apache.gluten.memory.memtarget.ThrowOnOomMemoryTarget.borrow(ThrowOnOomMemoryTarget.java:105)
at
org.apache.gluten.memory.listener.ManagedReservationListener.reserve(ManagedReservationListener.java:49)
at
org.apache.gluten.vectorized.ShuffleWriterJniWrapper.nativeEvict(Native Method)
at
org.apache.spark.shuffle.ColumnarShuffleWriter$$anon$1.spill(ColumnarShuffleWriter.scala:173)
```
The memory allocation likely stems from compression. It can be fixed via
pre-allocating the compression buffer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
