Ngone51 commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r539855837
##########
File path:
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -586,6 +592,7 @@ public void onData(String streamId, ByteBuffer buf) throws
IOException {
deferredBufs = null;
return;
}
+ abortIfNecessary();
Review comment:
It's OK if the further writes throw IOExceptions. I just worry, can we
handle it correctly if writes succeed? Or it may lead to some undefined
situation which could affect later shuffle fetch in turn.
So I'd like to avoid any writes in the first place (`onData`) so we ensure
that we don't go into any undefined situation. And I think, maybe, we can
update it a little, like this - we still receive data in `onData` but just not
write when the partition already reach IO threshold and only call
`abortIfNecessary` inside `onCompelte`.
WDYT? @Victsm @otterc
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
