Hi Reynold, Is there any way to know when an executor will no longer have any tasks? It seems to me there is no timeout which is appropriate that is long enough to ensure that no more tasks will be scheduled on the executor, and short enough to be appropriate to wait on during an interactive shell shutdown.
- Dan On Wed, Mar 16, 2016 at 2:40 PM, Reynold Xin <r...@databricks.com> wrote: > Maybe just add a watch dog thread and closed the connection upon some > timeout? > > > On Wednesday, March 16, 2016, Dan Burkert <d...@cloudera.com> wrote: > >> Hi all, >> >> I'm working on the Spark connector for Apache Kudu, and I've run into an >> issue that is a bit beyond my Spark knowledge. The Kudu connector >> internally holds an open connection to the Kudu cluster >> <https://github.com/apache/incubator-kudu/blob/master/java/kudu-spark/src/main/scala/org/kududb/spark/KuduContext.scala#L37> >> which >> internally holds a Netty context with non-daemon threads. When using the >> Spark shell with the Kudu connector, exiting the shell via <ctrl>-D causes >> the shell to hang, and a thread dump reveals it's waiting for these >> non-daemon threads. Registering a JVM shutdown hook to close the Kudu >> client does not do the trick, as it seems that the shutdown hooks are not >> fired on <ctrl>-D. >> >> I see that there is an internal Spark API for handling shutdown >> <https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/ShutdownHookManager.scala>, >> is there something similar available for cleaning up external data sources? >> >> - Dan >> >
