tomvanbussel commented on pull request #29785: URL: https://github.com/apache/spark/pull/29785#issuecomment-698206841
@Ngone51 Thanks for clarifying! It's both trying to allocate a new large pointer array and a new small pointer array. The allocation of the new large pointer array was what triggered the initial spill, and the new small pointer is allocated while spilling. While it's spilling (and the new small pointer array is allocated), the `TaskMemoryManager` is still holding to the memory for the partially allocated new large pointer array. Any ideas on how I can clarify this in the description? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
