Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/20781
@jerryshao
Thanks for advice. I spent some time digging to find why multiple `kill`
sent from Driver to AM, but didn't figure out a way to reproduce.
I come to find that it's possible `YarnAllocator` process same completed
container multiple
times(https://github.com/apache/spark/blob/master/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala#L573
this log is printed multiple times for same container). And I made another
change(see last commit in this pr) to propose my idea -- replace
`numExecutorsRunning` with a set.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
