steveloughran commented on issue #26674: [SPARK-30059][CORE]Stop AsyncEventQueue when interrupted in dispatch URL: https://github.com/apache/spark/pull/26674#issuecomment-559807729 > @steveloughran do you have any idea how this interrupt from DataStreamer is getting back to the spark event log writer? errors terminating the datastreamer thread are caught and then escalated to whichever the next API call uses the instance. Usually they are IO problems considered non-recoverable I don't know HDFS internals, but a look at the code hints this happened due to failures to talk to any datanode. Or at least, that's what the code is assuming -that any interrupt is a timeout in connections, If you can show there's a problem happening on the 3.2.x libraries you should be able to persuade some (else!) to have a look @ this.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
