otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r539630414
##########
File path:
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -586,6 +592,7 @@ public void onData(String streamId, ByteBuffer buf) throws
IOException {
deferredBufs = null;
return;
}
+ abortIfNecessary();
Review comment:
hmmm. I think the intention is to not have the server attempt writing if
the threshold is reached for the partition. Probably this check here makes this
behavior more accurate.
However, I don't have a strong opinion on this since the assumption is that
if these number of IOExceptions have already reached the threshold, any further
writes will result in an IOException as well. With that assumption, the write
in `onData` after this threshold is met, will very likely throw `IOException`
as well and since the threshold is already met, the server will instead throw
`IOExceptions exceeded threshold`.
I can remove it from `onData`. @Ngone51 What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
