Here is something that may be of interest to various members of the Leo community. I suspect Ville knows all about this, but it seems cool enough to repeat here.
It comes via Chris Barker and the pyxides list. It was written by Fernando Perez, who is always fun to read... Subject: [IPython-dev] Qt/Curses interfaces future: results of the weekend mini-sprint (or having fun with 0mq) Date: Tue, 23 Mar 2010 14:01:37 -0700 From: Fernando Perez <[email protected]> To: IPython Development list <[email protected]> Hi all, I realize that we have a significant task ahead in sorting some thorny issues out to get 0.11 ready for release, especially with respect to GUI handling. But the big refactor of 0.11 starts to give us a clean codebase on which new/better interfaces using ipython can be built, and for that several people want to contribute right away, so it's best that we build a path forward for those contributions, even if some of us still have to finish the 0.11 struggle. Wendell is looking into curses and Gerardo and Omar in Colombia are looking at Qt, and it would be fantastic to be able to get these developments moving forward soon. So Brian and I got together over the weekend and did a design/coding sprint that turned out to be surprisingly fun and productive. This is the summary of those results. I hope this thread can serve as a design brief we'll later formalize in the docs and which can be used to plan for a possible GSOC [EKR: Google Summer of code] submission, for example (which the Qt guys have in mind). The basic issue we need to solve is the ability to have out-of-process interfaces that are efficient, simple to develop, and that support fully asynchronous operation. In today's ipython, you type code into a program that is the same tasked with executing the code, so that if your code crashes, it takes the interface down with it. So we need to have a two-process system where the user-facing client and the kernel that executes code live in separate processes (we'll retain a minimal in-process interface for embedding, no worries, but the bulk of the real-world use should be in two processes). We want the user-facing client (be it readline-, curses- or qt-based) to remain responsive when the kernel is executing code, and to survive a full kernel crash. So client/kernel need to communicate, and the communication should hopefully be possible *even when the kernel is busy*, at least to the extent that low-level messaging should continue to function even if the kernel is busy with Python code. Up until now our engines use Twisted, and the above requirements can simply not be met with Twisted (not to mention Twisted's complexity and the concern we have with it not being ported soon to py3). We recently stumbled on the 0mq messaging library: http://www.zeromq.org/ and Brian was able to quickly build a set of Python bindings for it (see link at the 0mq site, I'm writing this offline) using Cython. They are fast, we have full control over them, and since Cython is python-3 compliant, it means we can get a py3 version anytime we need. 0mq is a really amazing library: I'd only read about it recently and only used it for the first time this weekend (I started installing it at Brian's two days ago), and I was blown away by it. It does all the messaging in C++ system threads that are 100% Python-threads safe, so the library is capable of queuing messages until the Python layer is available to handle them. The api is dead-simple, it's blazingly fast, and we were able to get in two intense days a very real prototype that solves a number of problems that we were never able to make a dent into with Twisted. Furthermore, with Twisted it was only really Brian and Min who ever wrote major amounts of code for Ipython: twisted is really hard to grasp and has layers upon layers of abstraction, making it a very difficult library to pick up without a major commitment. 0mq is exactly the opposite: Brian explained the basic concepts to me in a few minutes (I haven't read a single doc yet!), we did some queuing tests interactively (by just making objects at an ipython prompt) and we then started writing a real prototype that now works. We are very much considering abandoning twisted as we move forward and using 0mq for everything, including the distributed computing support (while keeping the user-facing apis unchanged). So what's in this example? For now, you'll need to install 0mq and pyzmq from git; for 0mq, clone the repo at: git://github.com/sustrik/zeromq2.git then run ./autogen.sh ./configure --prefix=your_favorite_installation_prefix make make install This should give you a fully working 0mq. Then for the python bindings, clone Brian's repo and get the kernel branch: git clone git://github.com/ellisonbg/pyzmq.git cd pyzmq git co -b kernel origin/kernel then cp setup.cfg.template setup.cfg and edit setup.cfg to indicate where you put your libraries. This is basically the prefix above with /lib and /include appended. Then you can do the usual python setup.py install --prefix=your_favorite_installation_prefix The prototype we wrote lives in examples/kernel. To play with it, open 3 terminals: - T1: run kernel.py, just leave it there. - T2 and T2: run frontend.py Both T2 and T3 are simple interactive prompts that run python code. You can quit them and restart them, type in both of them and they both manage the same namespace from the kernel in T1. Each time you hit return, they synchronize with each other's input/output, so you can see what each client is sending to the kernel. In a terminal this is done in-line and only when you hit return, but a curses/qt client with a real event loop can actually fire events when data arrives and display output from other clients as it is produced. You can 'background' inputs by putting ';' as the last character, and you can keep typing interactively while the kernel continues to process. If you type something that's taking a long time, Ctrl-C will break out of the wait but will leave the code running in the background (like Ctrl-Z in unix). This is NOT meant to be production code, it has no ipython dependencies at all, no tab-completion yet, etc. It's meant to: - let us understand the basic design questions, - settle on the messaging api between clients and kernel, - establish what common api all clients can use: a base to be shared by readline/curses/qt clients, on top of which the frontend-specific code will go. So I'd encourage those of you who are interested in this problem to have a look and let us know how it goes. For now the code lives in pyzmq because it makes for a great zmq example, but we're almost ready to start from it putting real ipython machinery. For thinking about this design though, it's a lot easier to work with a tiny prototype that fits in 3 files than to deal with all of ipython's complexity. Cheers, f -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/leo-editor?hl=en.
