This is an Engineering Notebook post. Feel free to ignore. Otoh, this very
long post documents several major milestones.
At last I have a simple, effective method for exploring jupyter code that
involves reading neither code nor docs ;-) Of course, I've done enough
reading of both by now to have some background...
The technique is to insert g.trace statements that show how the *installed
*jupyter
packages work. These traces appear in the jupyter_client, jupyter,
jupyter_core and qtconsole packages.
The rest of this post explains what is happening in detail, editing out all
the meanderings involved in the discovery process.
*Starting the demo app*
The following starts the console-based demo:
cd C:\Anaconda2\Lib\site-packages\jupyter_console
python2 __main__.py
This starts qt-based app:
cd C:\Anaconda2\Lib\site-packages\qtconsole
python2 __main__.py
At last I see that using the console demo is simpler because there is no qt
code involved.
*Preparing to trace*
I added the following lines to all files (in the python2 library)
containing code I wanted to trace:
import leo.core.leoGlobals as g
assert g
I was unsure whether stdout was being redirected as in server code. Would
I need to use logging messages? To be absolutely sure I would know when
code was being executed, I added calls to g.pdb(). It turns out that
g.trace *does* work as always, which greatly simplifies matters.
*Tracing init code*
At first, it was difficult to determine what code is being executed. There
are lots of similar classes involved. To disambiguate the traces, I added
explicit indications of what class was involved. Like this, for the
Session ctor:
g.trace('(Session)', kwargs)
And here is the result of starting the app:
>c:\Anaconda2\python.exe __main__.py
C:\Anaconda2\Lib\site-packages\jupyter_console>c:\Anaconda2\python.exe
__main__.py
init_kernel_manager (JupyterConsoleApp)
__init__ (Session) {'parent': <jupyter_console.app.ZMQTerminalIPythonApp
object at 0x03384490>}
init_kernel_client (JupyterConsoleApp)
get_connection_info (ConnectionFileMixin)
{
control_port: 53176,
hb_port: 53177,
iopub_port: 53174,
ip: u'127.0.0.1',
session: <jupyter_client.session.Session object at 0x03384590>,
shell_port: 53173,
stdin_port: 53175,
transport: u'tcp'
}
start_channels (KernelClient)
get_msg (ZMQSocketChannel block True timeout 1
__init__ (Session) {'parent': <ipykernel.kernelapp.IPKernelApp object at
0x036D1430>}
get_msg (ZMQSocketChannel block True timeout 1
Jupyter Console 4.1.1
get_msg (ZMQSocketChannel block True timeout 1
get_msg (ZMQSocketChannel block True timeout 1
In [1]:
The last line is the typical ipython/jupyter cell prompt.
*Executing commands*
A vital step was figuring out what code actually handles the key activities
of the app. After much mucking about, I realized that the "execute" method
handles code execution. There are also "history", "completion" and
"inspection" methods.
For the console app, these methods reside in the KernelClient class.* These
methods are the keys that unlock the jupyter code*. Leo will use these
methods, perhaps slightly modified, when implementing cells within Leo.
*Tracing messages*
I also added, again by trial and error, traces show how messages are sent
and replies received. Typing "2+2" into the input cell results in:
get_msg (ZMQSocketChannel block True timeout 1.0
execute (KernelClient) code: 2+2
get_msg (ZMQSocketChannel block True timeout 0.05
get_msg (ZMQSocketChannel block True timeout None
get_msg (ZMQSocketChannel block True timeout None
get_msg (ZMQSocketChannel block True timeout None
get_msg (ZMQSocketChannel block True timeout None
get_msg (ZMQSocketChannel block True timeout None
get_msg (ZMQSocketChannel block True timeout None
get_msg (ZMQSocketChannel block True timeout None
get_msg (ZMQSocketChannel block True timeout None
get_msg (ZMQSocketChannel block True timeout None
Out[1]: 4
get_msg (ZMQSocketChannel block True timeout None
get_msg (ZMQSocketChannel block False timeout 0.05
handle_execute_reply (ZMQTerminalInteractiveShell)
{
buffers: [],
content: {
execution_count: 1,
payload: [],
status: u'ok',
user_expressions: {}
},
header: {
date: datetime.datetime(2017, 11, 6, 3, 31, 54, 429000),
msg_id: u'f59b9f4c-ca27-4630-a138-5ee0e3b0a2a6',
msg_type: u'execute_reply',
session: u'187e92d1-9cbe-4126-ad32-774e371cd1d6',
username: u'username',
version: u'5.0'
},
metadata: {
dependencies_met: True,
engine: u'f2e67b09-71fd-4e00-a60b-4c556593ebc7',
started: u'2017-11-06T03:31:54.413000',
status: u'ok'
},
msg_id: u'f59b9f4c-ca27-4630-a138-5ee0e3b0a2a6',
msg_type: u'execute_reply',
parent_header: {
date: datetime.datetime(2017, 11, 6, 3, 31, 54, 413000),
msg_id: u'6035ad0f-592e-42bc-bc4d-3df628073815',
msg_type: u'execute_request',
session: u'b982860d-e2e2-414b-8562-f9a0c6ee7d73',
username: u'username',
version: u'5.0'
}
}
*Waiting for replies*
The traces above show that the app simply goes into a loop waiting for a
reply. The ZMQTerminalInteractiveShell.run_cell contains this polling loop:
while self.client.is_alive():
try:
self.handle_execute_reply(msg_id, timeout=0.05)
except Empty:
pass
else:
break
Here is the start of ZMQTerminalInteractiveShell.handle_execute_reply:
def handle_execute_reply(self, msg_id, timeout=None):
msg = self.client.shell_channel.get_msg(block=False, timeout=timeout)
g.trace('(ZMQTerminalInteractiveShell)')
g.printObj(msg)
And finally, here is all of ZMQSocketChannel.get_msg:
def get_msg(self, block=True, timeout=None):
""" Gets a message if there is one that is ready. """
g.trace('(ZMQSocketChannel', 'block', block, 'timeout', repr(timeout))
if block:
if timeout is not None:
timeout *= 1000 # seconds to ms
ready = self.socket.poll(timeout)
else:
ready = self.socket.poll(timeout=0)
if ready:
return self._recv()
else:
raise Empty
*Discussion*
There is a lot of subtle information in these traces:
- Apparently, the app only uses a single session for communicating with the
kernel.
- The dog that isn't barking: there is no server involved, only one (or
more) kernels.
- A simple polling loop suffices to wait for replies.
The manager module in the jupyter_client package handles creating kernels.
It should be straightforward to use this module.
*Summary*
Being able to trace what the jupyter_console app is doing is a huge step
forward. The low-energy phase of this project is over.
The "execute", "history", "completion" and "inspection" methods of the
KernelClient methods show how to send and receive the messages needed to
handle the key IPython/Jupyter capabilities.
It is now clear that it's possible to create kernels without interacting
with the Jupyter notebook server. In retrospect, Terry has likely been
trying to tell me this for awhile...
The next step will be to create a stand-alone demo that will, say, execute
2+2 using a jupyter (python) kernel.
The plan is to distill the jupyter_console code down to the bare minimum
needed to accomplish the goal. Distillation should be straightforward.
It's a lot easier to remove code than to write it ;-)
The result will contain no qt code, so it should be possible to execute it
as a Leo script.
Edward
--
You received this message because you are subscribed to the Google Groups
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.
