imback82 commented on pull request #28435: URL: https://github.com/apache/spark/pull/28435#issuecomment-623653253
> How frequently do you see this? I've never seen it in my Hadoop 3.x testing. I am using 2.9.x Hadoop and I can reproduce this consistently in a cluster - 8 physical nodes, 14 executors each with two cores - when I run the `SparkPi` example with 100,000 slices. >What do you mean by "didn't run at all"? Meaning it didn't do anything at all in the shutdown sequence? `Future<?> future = EXECUTOR.submit(entry.getHook());` is called right away, but the hook submitted is not run by `ThreadPoolExecutor` and eventually times out. (One interesting thing is the hook starts running when `future.cancel(true)` is called when timeout happens.) I am not a JVM expert, but it seems that JVM is busy during this time that it doesn't run the thread from `ThreadPoolExecutor`. What do you think? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
