Looking through the code a bit more I found this code snippet in
jupyter_client/manager.py
<https://github.com/jupyter/jupyter_client/blob/master/jupyter_client/manager.py#L188-L193>
:
def _launch_kernel(self, kernel_cmd, **kw):
"""actually launch the kernel
override in a subclass to launch kernel subprocesses differently
"""
return launch_kernel(kernel_cmd, **kw)
I know the PID of the running kernel process. If I could somehow do a
subprocess.Popen() on that existing process and return a reference to that,
I think that would work.
Unfortunately I can't anything related to doing a subprocess.Popen() on an
*existing* process...
On Tuesday, February 13, 2018 at 1:50:04 PM UTC-8, Edward Banner wrote:
>
> *Arg! I can't seem to get formatting down on google forums. I apologize.
> Here is my response in full.*
>
> Hi Roland,
>
> Thanks for your response. Your suggestion makes sense. But does that mean
> going with a custom server extension
> <http://jupyter-notebook.readthedocs.io/en/stable/extending/handlers.html> is
> the way to go?
>
> I spent yesterday attempting to accomplish this solely though jupyter
> configuration options. For a running kernel with information
>
> {
> "shell_port": 63391,
> "iopub_port": 63394,
> "stdin_port": 63392,
> "control_port": 63393,
> "hb_port": 63396,
> "ip": "127.0.0.1",
> "key": "fff108f8-b1bf9b17aa337ce7d3d14565",
> "transport": "tcp",
> "signature_scheme": "hmac-sha256",
> "kernel_name": ""
> }
>
> I set the following variables in jupyter_notebook_config.py.
>
> c.ConnectionFileMixin.connection_file =
> '/Users/ebanner/Library/Jupyter/runtime/kernel-10962.json'
> c.ConnectionFileMixin.control_port = 63393
> c.ConnectionFileMixin.iopub_port = 63394
> c.ConnectionFileMixin.stdin_port = 63392
> c.ConnectionFileMixin.hb_port = 63396
> c.ConnectionFileMixin.ip = '127.0.0.1'
> c.ConnectionFileMixin.shell_port = 63391
> c.ConnectionFileMixin.transport = 'tcp'
>
> However when I created a new notebook I get errors like
>
> [I 12:53:44.981 NotebookApp] KernelRestarter: restarting kernel (1/5)
> [D 12:53:44.982 NotebookApp] Starting kernel: [
> '/Users/ebanner/.anaconda/envs/py36/bin/python', '-m',
> 'ipykernel_launcher', '-f',
> '/Users/ebanner/Library/Jupyter/runtime/kernel-f746726f-0d27-4d63-b520-2f1c84bf5196.json'
> ]
> [D 12:53:44.986 NotebookApp] Connecting to: tcp://127.0.0.1:63393
> Traceback (most recent call last):
> File "/Users/ebanner/.anaconda/envs/py36/lib/python3.6/runpy.py", line
> 193, in _run_module_as_main
> "__main__", mod_spec)
> File "/Users/ebanner/.anaconda/envs/py36/lib/python3.6/runpy.py", line
> 85, in _run_code
> exec(code, run_globals)
> File
> "/Users/ebanner/.anaconda/envs/py36/lib/python3.6/site-packages/ipykernel_launcher.py"
> , line 16, in <module>
> app.launch_new_instance()
> File
> "/Users/ebanner/.anaconda/envs/py36/lib/python3.6/site-packages/traitlets/config/application.py"
> , line 657, in launch_instance
> app.initialize(argv)
> File "<decorator-gen-123>", line 2, in initialize
> File
> "/Users/ebanner/.anaconda/envs/py36/lib/python3.6/site-packages/traitlets/config/application.py"
> , line 87, in catch_config_error
> return method(app, *args, **kwargs)
> File
> "/Users/ebanner/.anaconda/envs/py36/lib/python3.6/site-packages/ipykernel/kernelapp.py"
> , line 448, in initialize
> self.init_sockets()
> File
> "/Users/ebanner/.anaconda/envs/py36/lib/python3.6/site-packages/ipykernel/kernelapp.py"
> , line 238, in init_sockets
> self.shell_port = self._bind_socket(self.shell_socket, self.shell_port
> )
> File
> "/Users/ebanner/.anaconda/envs/py36/lib/python3.6/site-packages/ipykernel/kernelapp.py"
> , line 180, in _bind_socket
> s.bind("tcp://%s:%i" % (self.ip, port))
> File "zmq/backend/cython/socket.pyx", line 495, in zmq.backend.cython.
> socket.Socket.bind (zmq/backend/cython/socket.c:5653)
> File "zmq/backend/cython/checkrc.pxd", line 25, in zmq.backend.cython.
> checkrc._check_rc (zmq/backend/cython/socket.c:10014)
> zmq.error.ZMQError: Address already in use
>
> until it hits the maximum number of retries. Presumably this is because
> the ipykernel command attempts to create a *new* socket but since it
> already exists it fails to do so.
>
> Thinking that I could "trick" jupyter into not using ipykernel and just
> use the existing sockets I set
>
> c.KernelManager.kernel_cmd = [
> '/Users/ebanner/.anaconda/envs/py36/bin/python', '/Users/ebanner/bar.py']
>
> where bar.py contains
>
> while True:
> pass
>
> After creating another new notebook I get
>
> [D 12:59:37.283 NotebookApp] Opening websocket /api/kernels/9a6c1e7d-12c4-
> 4d98-a0c6-1f227627f784/channels
> [D 12:59:37.283 NotebookApp] Connecting to: tcp://127.0.0.1:63391
> [D 12:59:37.284 NotebookApp] Connecting to: tcp://127.0.0.1:63394
> [D 12:59:37.284 NotebookApp] Connecting to: tcp://127.0.0.1:63392
>
> which gave me hope! But then when I tried to execute a cell it just hangs.
> I did notice that http://localhost:8888/api/kernels has the following
> information.
>
> [
> {
> id: "9b260abf-072f-48e2-b5f0-213d934418a2",
> name: "python3",
> last_activity: "2018-02-13T21:10:41.460182Z",
> execution_state: "starting",
> connections: 0
> }
> ]
>
> However the execution_state being stuck at "starting" and connections
> being at 0 make me think that bar.py isn't quite doing enough.
>
> Digging into the code a bit I found the object returned from the call to
> ipykernel is a subprocess.Popen() object. But I couldn't find anywhere else
> in the code except KernelManager.is_alive() where Popen.poll() is called
> on it
> <https://github.com/jupyter/jupyter_client/blob/38b81cc07ddf70c2f534f11c4f4ef0103bfd5fe2/jupyter_client/multikernelmanager.py#L204-L205>
> .
>
> I also tried creating a custom kernel manager class and set
>
> c.NotebookApp.kernel_manager_class =
> 'kernelmanager.ExistingMappingKernelManager'
>
> where kernelmanager.ExistingMappingKernelManager is the class
>
> from tornado import gen, web
> from jupyter_client import KernelManager
> from notebook.services.kernels.kernelmanager import MappingKernelManager
>
> class ExistingMappingKernelManager(MappingKernelManager):
> """A KernelManager that just connects to an existing kernel."""
>
> @gen.coroutine
> def start_kernel(self, kernel_id=None, path=None, **kwargs):
> kernel_id = 1
> km = KernelManager(kernel_name='python3')
> kc = km.client()
> kc.load_connection_file(
> '/Users/ebanner/Library/Jupyter/runtime/kernel-10962.json')
> kc.start_channels()
> try:
> kc.wait_for_ready()
> except RuntimeError:
> kc.stop_channels()
> raise
> raise gen.Return(kernel_id)
>
> but when I create a new notebook I get the error
>
> [E 13:08:17.169 NotebookApp] Unhandled error in API request
>
>
> Traceback (most recent call last):
>
>
> File
> "/Users/ebanner/.anaconda/lib/python3.6/site-packages/notebook/base/handlers.py"
> , line 516, in wrapper
>
> result = yield gen.maybe_future(method(self, *args, **kwargs))
>
>
> File
> "/Users/ebanner/.anaconda/lib/python3.6/site-packages/tornado/gen.py",
> line 1055, in run
>
> value = future.result()
>
>
> File
> "/Users/ebanner/.anaconda/lib/python3.6/site-packages/tornado/concurrent.py"
> , line 238, in result
>
> raise_exc_info(self._exc_info)
>
>
> File "<string>", line 4, in raise_exc_info
>
>
> File
> "/Users/ebanner/.anaconda/lib/python3.6/site-packages/tornado/gen.py",
> line 1063, in run
> yielded = self.gen.throw(*exc_info)
>
> File
> "/Users/ebanner/.anaconda/lib/python3.6/site-packages/notebook/services/sessions/handlers.py"
> , line 75, in post
>
> type=mtype))
>
>
> File
> "/Users/ebanner/.anaconda/lib/python3.6/site-packages/tornado/gen.py",
> line 1055, in run
>
> value = future.result()
>
>
> File
> "/Users/ebanner/.anaconda/lib/python3.6/site-packages/tornado/concurrent.py"
> , line 238, in result
>
> raise_exc_info(self._exc_info)
>
> File "<string>", line 4, in raise_exc_info
>
>
> File
> "/Users/ebanner/.anaconda/lib/python3.6/site-packages/tornado/gen.py",
> line 1069, in run
> yielded = self.gen.send(value)
>
> File
> "/Users/ebanner/.anaconda/lib/python3.6/site-packages/notebook/services/sessions/sessionmanager.py"
> , line 81, in create_session
>
> self.save_session(session_id, path=path, name=name, type=type,
> kernel_id=kernel_id)
> File
> "/Users/ebanner/.anaconda/lib/python3.6/site-packages/notebook/services/sessions/sessionmanager.py"
> , line 125, in save_session
>
> return self.get_session(session_id=session_id)
>
> File
> "/Users/ebanner/.anaconda/lib/python3.6/site-packages/notebook/services/sessions/sessionmanager.py"
> , line 170, in get_session
> return self.row_to_model(row)
>
> File
> "/Users/ebanner/.anaconda/lib/python3.6/site-packages/notebook/services/sessions/sessionmanager.py"
> , line 209, in row_to_model
>
> raise KeyError
>
> KeyError
>
>
> [E 13:08:17.171 NotebookApp] {
>
> "Host": "localhost:8888",
>
> "Connection": "keep-alive",
>
> "Content-Length": "96",
>
> "Accept": "application/json, text/javascript, */*; q=0.01",
>
> "Origin": "http://localhost:8888";,
>
> "X-Requested-With": "XMLHttpRequest",
>
>
> "X-Xsrftoken":
> "2|e5e3448b|a67102a2af982e0bece9d040d46b1c87|1516914005",
>
>
> "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6)
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
> ,
>
> "Content-Type": "application/json",
>
>
> "Referer": "http://localhost:8888/notebooks/Kernel%20Client.ipynb";,
>
>
> "Accept-Encoding": "gzip, deflate, br",
>
>
> "Accept-Language": "en-US,en;q=0.9",
>
>
> "Cookie":
> "_xsrf=2|e5e3448b|a67102a2af982e0bece9d040d46b1c87|1516914005;
> username-localhost-8889=\"2|1:0|10:1517089627|23:username-localhost-8889|44:M2NiNzEzOGRjOGVhNDhlMzk0YjRlZGI2OWQ4M2RkZjk=|fcb4100a390a3d9608150659dc573ec135609a1e778f386e293b029e1fe016e3\";
>
> user
> name-localhost-8890=\"2|1:0|10:1518302103|23:username-localhost-8890|44:OTdjNGRhZGRjYTRlNGMxZjg0YzY4ZDIzZjU1NjEzYTQ=|6c9c7647d95641d6f2d7375d9d45130a96e9ed9ec690b2bca01b6652c23ccd05\";
>
> username-localhost-8888=\"2|1:0|10:1518555822|23:username-localhost-8888|44:M2M3OTdhYTh
>
> iNjQ2NDE3ZWEyZjA3Mjk2MzdjMjZlMmI=|e2c3fe789c16638c5fa4573a9d7626b87a00cb4d33ed1488122c6543b8832398\""
>
>
> }
>
> presumably because I did not call super(MappingKernelManager,
> self).start_kernel(**kwargs) which presumably does some important setup
> work.
>
> I *did* get a hacky solution "working" by the end of the day, though! I
> put the following code snippet in a jupyter notebook.
>
> from jupyter_client import BlockingKernelClient
>
> client = BlockingKernelClient()
> client.load_connection_file(
> '/Users/ebanner/Library/Jupyter/runtime/kernel-10962.json')
> client.start_channels()
>
> Then I am able to do client.execute_interactive(code) as many times as I
> want!
>
> Using this approach one way forward would be to create a magic command
> that wraps code in a call to client.execute_interactive(). However for a
> single notebook that would require having to put the magic inside every
> code cell, which I would rather avoid if possible.
>
> So my question is can I start a new jupyter notebook and attach it to a
> running IPython kernel with configurables alone? If so am I close? If not
> what would be the most promising direction to proceed? The closest example
> I can think to study is nb2kg <https://github.com/jupyter-incubator/nb2kg>
> which
> allows a jupyter notebook to attach to a remote kernel. However that kernel
> was started by a jupyter notebook so it still might not be close enough for
> my use case to solve my problem
>
> I truly appreciate any help I can get as I am running out of ideas.
>
> On Tuesday, February 13, 2018 at 1:28:46 PM UTC-8, Edward Banner wrote:
>>
>> Hi Roland,
>>
>> Thanks for your response. Your suggestion makes sense. But does that mean
>> going with a custom server extension
>> <http://jupyter-notebook.readthedocs.io/en/stable/extending/handlers.html>
>> is the way to go?
>>
>> I spent yesterday attempting to accomplish this solely though jupyter
>> configuration options. For a running kernel with information
>>
>> {
>> "shell_port": 63391,
>> "iopub_port": 63394,
>> "stdin_port": 63392,
>> "control_port": 63393,
>> "hb_port": 63396,
>> "ip": "127.0.0.1",
>> "key": "fff108f8-b1bf9b17aa337ce7d3d14565",
>> "transport": "tcp",
>> "signature_scheme": "hmac-sha256",
>> "kernel_name": ""
>> }
>>
>> I set the following variables in jupyter_notebook_config.py.
>>
>> c.ConnectionFileMixin.connection_file =
>> '/Users/ebanner/Library/Jupyter/runtime/kernel-10962.json'
>> c.ConnectionFileMixin.control_port = 63393
>> c.ConnectionFileMixin.iopub_port = 63394
>> c.ConnectionFileMixin.stdin_port = 63392
>> c.ConnectionFileMixin.hb_port = 63396
>> c.ConnectionFileMixin.ip = '127.0.0.1'
>> c.ConnectionFileMixin.shell_port = 63391
>> c.ConnectionFileMixin.transport = 'tcp'
>>
>> However when I created a new notebook I get errors like
>>
>> [I 12:53:44.981 NotebookApp] KernelRestarter: restarting kernel (1/5)
>> [D 12:53:44.982 NotebookApp] Starting kernel: [
>> '/Users/ebanner/.anaconda/envs/py36/bin/python', '-m',
>> 'ipykernel_launcher', '-f',
>> '/Users/ebanner/Library/Jupyter/runtime/kernel-f746726f-0d27-4d63-b520-2f1c84bf5196.json'
>> ]
>> [D 12:53:44.986 NotebookApp] Connecting to: tcp://127.0.0.1:63393
>> Traceback (most recent call last):
>> File "/Users/ebanner/.anaconda/envs/py36/lib/python3.6/runpy.py", line
>> 193, in _run_module_as_main
>> "__main__", mod_spec)
>> File "/Users/ebanner/.anaconda/envs/py36/lib/python3.6/runpy.py", line
>> 85, in _run_code
>> exec(code, run_globals)
>> File
>> "/Users/ebanner/.anaconda/envs/py36/lib/python3.6/site-packages/ipykernel_launcher.py"
>> , line 16, in <module>
>> app.launch_new_instance()
>> File
>> "/Users/ebanner/.anaconda/envs/py36/lib/python3.6/site-packages/traitlets/config/application.py"
>> , line 657, in launch_instance
>> app.initialize(argv)
>> File "<decorator-gen-123>", line 2, in initialize
>> File
>> "/Users/ebanner/.anaconda/envs/py36/lib/python3.6/site-packages/traitlets/config/application.py"
>> , line 87, in catch_config_error
>> return method(app, *args, **kwargs)
>> File
>> "/Users/ebanner/.anaconda/envs/py36/lib/python3.6/site-packages/ipykernel/kernelapp.py"
>> , line 448, in initialize
>> self.init_sockets()
>> File
>> "/Users/ebanner/.anaconda/envs/py36/lib/python3.6/site-packages/ipykernel/kernelapp.py"
>> , line 238, in init_sockets
>> self.shell_port = self._bind_socket(self.shell_socket, self.
>> shell_port)
>> File
>> "/Users/ebanner/.anaconda/envs/py36/lib/python3.6/site-packages/ipykernel/kernelapp.py"
>> , line 180, in _bind_socket
>> s.bind("tcp://%s:%i" % (self.ip, port))
>> File "zmq/backend/cython/socket.pyx", line 495, in zmq.backend.cython.
>> socket.Socket.bind (zmq/backend/cython/socket.c:5653)
>> File "zmq/backend/cython/checkrc.pxd", line 25, in zmq.backend.cython.
>> checkrc._check_rc (zmq/backend/cython/socket.c:10014)
>> zmq.error.ZMQError: Address already in use
>>
>> until it hits the maximum number of retries. Presumably this is because
>> the ipykernel command attempts to create a *new* socket but since it
>> already exists it fails to do so.
>>
>> Thinking that I could "trick" jupyter into not using ipykernel and just
>> use the existing sockets I set
>>
>> c.KernelManager.kernel_cmd = [
>> '/Users/ebanner/.anaconda/envs/py36/bin/python', '/Users/ebanner/bar.py']
>>
>> where bar.py contains
>>
>> while True:
>> pass
>>
>> After creating another new notebook I get
>>
>> [D 12:59:37.283 NotebookApp] Opening websocket /api/kernels/9a6c1e7d-12c4
>> -4d98-a0c6-1f227627f784/channels
>> [D 12:59:37.283 NotebookApp] Connecting to: tcp://127.0.0.1:63391
>> [D 12:59:37.284 NotebookApp] Connecting to: tcp://127.0.0.1:63394
>> [D 12:59:37.284 NotebookApp] Connecting to: tcp://127.0.0.1:63392
>>
>> which gave me hope! But then when I tried to execute a cell it just
>> hangs. I did notice that http://localhost:8888/api/kernels has the
>> following information.
>>
>> <span
>>
>
--
You received this message because you are subscribed to the Google Groups
"Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/jupyter/eb394188-0acc-4c17-ab4b-2ecd8e4dbc8a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
