squito commented on issue #26674: [SPARK-30059][CORE]Stop AsyncEventQueue when interrupted in dispatch URL: https://github.com/apache/spark/pull/26674#issuecomment-559104531 do you know what version of hadoop you are on? I am trying to compare with the code -- its clearly not trunk (the `DataStreamer` class has been moved and plenty of other refactoring has happened). Still, even looking at trunk, I have a guess at what is happening. The first part of your log shows the interrupt is coming from the DataStreamer, though that is running in a separate thread and isnt' directly interrupting the event log queue thread. But my guess is that calls to `flush()` in the event log thread check the status of that data streamer, and will set the event log thread's interrupt status. (eg. something like this, though probably not these lines: https://github.com/apache/hadoop/blob/7f2ea2ac46596883fb8f110f754a0eadeb69205e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java#L730-L734) so its possible that we're actually leaving the *previous* call to `flush()` with the interrupt status set, but we're ignoring it. And then when we get to the next call to `queue.take()`, it immediately throws an interrupted exception because the interrupt status has already been set. that would probably be an hdfs bug, but seems to at least fit the pattern of what we see here, and something we could at least check for. @steveloughran do you have any idea how this interrupt from DataStreamer is getting back to the spark event log writer?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
