otterc commented on PR #39725: URL: https://github.com/apache/spark/pull/39725#issuecomment-1404040435
> Sum up the total number of bytes received in `onData` - does not matter if it was written, deferred, etc. > In onComplete/onFailure, add it to ignored bytes if we are deciding not to use data from this push. > Will keep the change much more constrained, and avoid issues where a push failed/was aborted/etc after it has partially written data out. So will we account for bytes that the server failed to write as well in `ignoredBytes`? I was thinking about this while making the suggestions and decided that the bytes that the server fails to write (throws an exception) are **not** `ignored`. In the current PR, even when it aborts, only deferred bytes are included in `ignoredBytes`. The bytes during which the server encounters IOException while writing are not included in `ignoredBytes`. I am actually okay with your suggestion but I just want to be clear on what we are capturing as ignored. If we are capturing the failed bytes as ignored, then maybe we should either rename it or at least add the comment to make it clear. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
