Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/8007#issuecomment-130863421
Hey @mccheah, good stuff. Turns out I have a similar pull request
lingering: #8093
In that pull request, I discuss the same race condition you are tackling
here (although I don't think you talk about it explicitly) whereby an
onDisconnected event may be received by the driver before a RemoveExecutor
event. The manifestation of that race, that I am most concerned about is the
correct reason showing the UI if Yarn killed the container.
I had been playing with a few ideas on how to do that. The first one I
hacked was using a new message for SparkListenerBus. If the onDisconnected()
event was received before RemoveExecutor, when the RemoveExecutor with the real
error would arrive, a new message would be sent down the listener bus with the
update reason for the killing of that executor. The code for that up at
https://github.com/markgrover/spark/compare/nm2...race just fyi.
That's just a preliminary hack I did to see if that works, and it seems
like it does work pretty well.
However, after looking at your PR, I think you solve the same problem,
albeit in a more general way. And, I think that may solve my problem too,
perhaps with some extension. That's because with your suggested change,
regardless of whether onDisconnected or RemoveExecutor is received first, they
both have a legitimate reason, in legitimate cases.
Anyways, I just wanted to let you know where my head is at, and I may pull
down some of your bits and pieces from this PR, into mine and it see if we need
to make any updates to those pieces to solve my problem.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
