I tried to access worker info from spark context but it seems spark context 
does no expose such API. The reason of doing that is: it seems spark context 
itself does not have logic to detect if its workers are in dead status. So I 
like to add such logic by myself. 

BTW, it seems spark web UI has some logic of detecting dead workers. But all 
relevant classes are declared as private for the spark package scope. 

Please let me know how to solve this issue (or if there is an alternative way 
to achieve the same purpose)

Thanks

Reply via email to