Burak KÖSE created SPARK-28230:
----------------------------------
Summary: Fix Spark Streaming gracefully shutdown on Hadopp 2.8.x
Key: SPARK-28230
URL: https://issues.apache.org/jira/browse/SPARK-28230
Project: Spark
Issue Type: Bug
Components: Spark Core, Structured Streaming
Affects Versions: 2.4.3
Reporter: Burak KÖSE
Gracefully shutdown is not properly working on Hadoop 2.8.x. Hadoop introduces
10 seconds timeout by default. This is hardcoded and not something that we can
configure with Hadoop settings.
[https://github.com/apache/hadoop/blob/branch-2.8.5/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ShutdownHookManager.java#L51]
However, if we use
{code:java}
public void addShutdownHook(Runnable shutdownHook, int priority, long timeout,
TimeUnit unit){code}
in
[https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/ShutdownHookManager.scala#L180|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/ShutdownHookManager.scala#L180,]
problem might be solved.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
