tgravescs commented on pull request #28848:
URL: https://github.com/apache/spark/pull/28848#issuecomment-648870480
> I'm thinking we should rename the epochs to blockManagerFailureEpoch and
externalShuffleFailureEpoch
@squito Maybe I'm miss understanding what you are suggesting (over what is
in patch now) but the fileLost case right now is more than external shuffle, it
is if an executor is lost in standalone mode or if external shuffle is not
enabled, in those cases we still unregister the output for the executor. This
is why I suggested the parameter fileLost to be unregisterShuffleOutput. The
new Epoch we are tracking is whether it has already removed the output from
that shuffle so I don't think externalShuffleFailureEpoch is inclusive enough.
Unless you are suggesting the change that the remove output is in 2 places -
1 stays under blockManagerFailureEpoch in the case its not exteranal shuffle
and then you have an additional check for exteranal shuffle and only do remove
shuffle output with externalShuffleFailureEpoch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
