cloud-fan commented on a change in pull request #29785:
URL: https://github.com/apache/spark/pull/29785#discussion_r492175952
##########
File path:
core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java
##########
@@ -384,6 +401,28 @@ private void acquireNewPageIfNecessary(int required) {
}
}
+ /**
+ * Allocates more memory in order to insert an additional record. This will
request additional
+ * memory from the memory manager and spill if the requested memory can not
be obtained.
+ *
+ * @param required the required space in the data page, in bytes, including
space for storing
+ * the record size.
+ */
+ private void allocateMemoryForRecordIfNecessary(int required) throws
IOException {
+ // 1. Ensure that the pointer array has space for another record. This may
cause a spill.
+ growPointerArrayIfNecessary();
+ // 2. Ensure that the last page has space for another record. This may
cause a spill.
+ acquireNewPageIfNecessary(required);
+ // 3. The allocation in step 2 could have caused a spill, which would have
freed the pointer
+ // array allocated in step 1. Therefore we need to check again whether
we have to allocate
+ // a new pointer array.
+ // Note that if the allocation in this step causes a spill event then
it won't cause the page
+ // allocated in the previous step to be freed. This step will only
allocate memory if we
+ // have spilled in the previous step, and spill() will only free memory
if a record has been
Review comment:
I may need more context to understand it. If step 2 triggers a spill,
then why won't we spill the data pages here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
