Since we need to use ipython 3.2.1 to gracefully shutdown sparkcontext. the only way that I know is to enter Ctrl +C in the ipython console, however, our user scenario does not allow us to access the ipython console, so what we do now is just to kill -9 the ipython process, but this will cause ipython not finish or end the sparkcontext gracefully. we figured out the way that we can stop the kernel process of ipython one by one, then sparkcontext can be gracefully shutdown, but the kernel process kept autostart，so I am just wondering if we would disable this function.
Or do you have other ways to deal with this? 2016-08-26 20:17 GMT+08:00 MinRK <[email protected]>: > I'm not sure disabling autorestart is supported by the notebook > application. What prompts your need for dead kernels? > > -MinRK > > On Wed, Aug 17, 2016 at 6:16 PM, <[email protected]> wrote: > >> Hi, all >> Currently, we are using IPython Notebook + Spark 1.6.1, and I just >> found in default situation, a kernel would autostart if it dies, I found >> this configuration parameter in ipython_notebook_config.py file, set the >> value to be False, like following: >> >> # Should we autorestart the kernel if it dies. >> c.KernelManager.autorestart = False >> >> but anytime I kill the kernel process, it still will autostart: >> >> 16/08/18 08:02:55 INFO SparkContext: Successfully stopped SparkContext >> 16/08/18 08:02:55 INFO ShutdownHookManager: Shutdown hook called >> 16/08/18 08:02:55 INFO SparkContext: Successfully stopped SparkContext >> 16/08/18 08:02:55 INFO ShutdownHookManager: Shutdown hook called >> 16/08/18 08:02:55 INFO ShutdownHookManager: Deleting directory >> /tmp/1098e87d-99ef-4f53-8f0d-67f2403a677e/spark-b12c6466-d9ca-4155-a8a2-aede2b213744 >> >> 16/08/18 08:02:55 INFO RemoteActorRefProvider$RemotingTerminator: >> Shutting down remote daemon. >> 16/08/18 08:02:55 INFO ShutdownHookManager: Deleting directory >> /tmp/1098e87d-99ef-4f53-8f0d-67f2403a677e/spark-b12c6466-d9c >> a-4155-a8a2-aede2b213744/pyspark-d5fec37d-85e8-491e-b7e4-22b95e04aa48 >> 16/08/18 08:02:55 INFO ShutdownHookManager: Deleting directory >> /tmp/1098e87d-99ef-4f53-8f0d-67f2403a677e/spark-896417f0-f56 >> 5-4c0c-a9ef-c9e81724afb7/pyspark-3754abd6-d18b-46ab-bb7c-f7a8ed6f7ae5 >> 16/08/18 08:02:55 INFO ShutdownHookManager: Deleting directory >> /tmp/1098e87d-99ef-4f53-8f0d-67f2403a677e/spark-896417f0-f565-4c0c-a9ef-c9e81724afb7 >> >> 16/08/18 08:02:55 INFO RemoteActorRefProvider$RemotingTerminator: Remote >> daemon shut down; proceeding with flushing remote transports. >> 16/08/18 08:02:55 INFO RemoteActorRefProvider$RemotingTerminator: >> Shutting down remote daemon. >> 16/08/18 08:02:55 INFO RemoteActorRefProvider$RemotingTerminator: Remote >> daemon shut down; proceeding with flushing remote transports. >> *[I 08:02:56.580 NotebookApp] KernelRestarter: restarting kernel (1/5) * >> *WARNING:root:kernel 372cdeb7-5a0b-4792-94e5-883ca4e46d40 restarted * >> *[I 08:02:56.814 NotebookApp] KernelRestarter: restarting kernel (1/5) * >> >> Could anyone tell me how to disable this feature? am I miss some other >> configuration parameters? >> Thanks in advance! >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Project Jupyter" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> To view this discussion on the web visit https://groups.google.com/d/ms >> gid/jupyter/f74448e5-7a76-476c-bf98-3176752c4190%40googlegroups.com >> <https://groups.google.com/d/msgid/jupyter/f74448e5-7a76-476c-bf98-3176752c4190%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Project Jupyter" group. > To unsubscribe from this topic, visit https://groups.google.com/d/ > topic/jupyter/DgQ-MW5x3uY/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To post to this group, send email to [email protected]. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/jupyter/CAHNn8BUswuW-KPTWCqfLUk%2B28nJaoEKYf1i2Op4jkM% > 3DAKoBabQ%40mail.gmail.com > <https://groups.google.com/d/msgid/jupyter/CAHNn8BUswuW-KPTWCqfLUk%2B28nJaoEKYf1i2Op4jkM%3DAKoBabQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- Yalan Xue ， SIT Lab of Nanjing University Address: Nanjing ,China。 -- You received this message because you are subscribed to the Google Groups "Project Jupyter" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/CACNymyvF6%2BH73X%2B2U064xx99%3D8MNeARPUx9mAHvsJQ_eumpm6g%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
