wForget commented on PR #53710:
URL: https://github.com/apache/spark/pull/53710#issuecomment-3728109188
@Ngone51 Thank you for your reply. I mistakenly thought that the spilling
operation was not releasing memory correctly. After adding some logs, I think
this might be because the Window operator cannot trigger the memory spill of
the subsequent ObjectHashAggregate sorter.
The SparkPlan like this:
```
SortExec
|
WindowExec --> ExternalAppendOnlyUnsafeRowArray.add ->
UnsafeExternalSorter.growPointerArrayIfNecessary (Throw SparkOutOfMemoryError:
unable to acquire memroy and unable to spill
ObjectAggregationIterator.inputSorter.sorter)
|
ObjectHashAggregateExec --> ObjectAggregationIterator.inputSorter.sorter
(Hold 2.1 GiB)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
