viirya commented on a change in pull request #25953: [SPARK-29244][Core]
Prevent freed page in BytesToBytesMap free again
URL: https://github.com/apache/spark/pull/25953#discussion_r329330012
##########
File path: core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
##########
@@ -787,8 +783,16 @@ private void allocate(int capacity) {
assert (capacity >= 0);
capacity = Math.max((int) Math.min(MAX_CAPACITY,
ByteArrayMethods.nextPowerOf2(capacity)), 64);
assert (capacity <= MAX_CAPACITY);
- longArray = allocateArray(capacity * 2L);
- longArray.zeroOut();
+ try {
+ longArray = allocateArray(capacity * 2L);
+ longArray.zeroOut();
+ } catch (SparkOutOfMemoryError e) {
+ // When OOM, allocated page was already freed by `TaskMemoryManager`.
+ // We should not keep it in `longArray`. Otherwise it might be freed
again in task
+ // complete listeners and cause unnecessary error.
+ longArray = null;
Review comment:
@JoshRosen Thanks for your detailed review and good analysis! Yes, I think
your proposal should work. It is simpler and I think it is easier to
understand. I will simplify this patch.
As you said, once with this fix, I think an single-threaded test would be
possible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
