tomvanbussel commented on pull request #29785: URL: https://github.com/apache/spark/pull/29785#issuecomment-696770689
@Ngone51 There are three arrays in the example: 1. The old pointer array. 2. A new large pointer array that's allocated in `growPointerArrayIfNecessary()` to replace the old pointer array. The allocation of this array triggered the spill. 3. A new small pointer array that's allocated while spilling to replace the old pointer array. The allocation of this array causes the OOM exception. The old pointer array has been released, but `TaskMemoryManager` is still trying to allocate the new large pointer array when spilling is triggered, and still holds onto the memory it got from the `MemoryManager`. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
