Ngone51 commented on pull request #29785: URL: https://github.com/apache/spark/pull/29785#issuecomment-697173137
> The old pointer array has been released, but TaskMemoryManager is still trying to allocate the new large pointer array when spilling is triggered, and still holds onto the memory it got from the MemoryManager. But from the description of step 3,4,5, the `TaskMemoryManager` is trying to allocate the new small pointer array rather than the large one: > 3. TaskMemoryManager asks UnsafeExternalSorter to spill, which causes UnsafeExternalSorter to spill the current run to disk, to free its record pages and to reset its UnsafeInMemorySorter. > 4. UnsafeInMemorySorter frees the old pointer array, and tries to allocate a new small pointer array. > 5. TaskMemoryManager tries to allocate the memory backing the small array using MemoryManager, but MemoryManager is unwilling to give it any memory, as the TaskMemoryManager is still holding on to the memory it got for the new large array. IIUC, it should be the small pointer array... ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
