IbPy is a Python interface to the Interactive Brokers API and something I've been involved with for a while.
A piece of IbPy is a code generator which takes IB's Java client code and generates Python. I've had issues with performance... Hence the message below.. http://groups.google.com/group/ibpy-discuss/msg/1bf318a75376c64f and pasted here: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So, I had this weird thought last night about IbPy... and the java to python generator / compiler... and something occurred to me. Cython.... I've been playing around with Cython for a while. Its pretty amazing stuff really and the syntax is very much python-like with a few exceptions... those exceptions are where the Cython compiler does things to make python code fast (i.e. where you know things about the data, you can take the load off the Python interpreter and generate native code) The speedup can be dramatic... like 200x. There are a number of important Cython characteristics but its better to just read about it (links below) An important link first: This to a talk William Stein gave to/at Microsoft which has a little Cython example which gets executed in real-time (i.e. C/C++ code gets generated and linked into python dynamically through the sage workbook which is browser based). Of course, just like generating .pyc files, once this happens in real life, the .dll or .so files are there so just need to be dynamically linked at runtime unless there is a source code change. If you're not a math person, just scroll down to the Cython example http://sagenb.org/home/pub/202/ Note that if you go ahead and get an account on the Sage machines, you'll be able to run this all yourself... live. (for the more adventurous, building Sage is amazingly easy and it is completely self contained... I could expand greatly but those so inclined should check out Sage (links below) But, thats not my point. My point is that since IbPy uses the J2py compiler (called something like that) its probably "relatively" easy to generate a little extra python code in key areas to make it Cythonic (couplea type specifications)... which will make it operate at native code speeds yet still be completely python at the user level... just that instead of the library being interpreted, its native... no diff to the user at all... and I need to be clear clear cuz it might not be obvious at first glance. I'm NOT saying the user interface is Cython, just that the underlying IB handling library is Cython. The user still codes in Python (unless you want to use Cython which is another story altogether and might make a lot of sense for those of us who run into compute problems... this is sort of like using cPickle instead of Pickle. Same interface, just runs way faster cuz its compiled C code instead of Python) This effort might be amost a freebie given the existence of the code generator used for IbPy. The java code parsed to generate IbPy has the information necessary to spew the few key type definitions where IbPy spends all its cycles (Java is strongly typed), in particular, the inner loops which do the comms handling and object generation. Cython homepage: http://www.cython.org/ Sage homepage: http://www.sagemath.org/ -glenn On Jan 30, 2:22 am, glenn <gl...<http://groups.google.com/groups/unlock?_done=/group/ibpy-discuss/browse_thread/thread/0cd27f0c47248af0/7b027b93b5472100%3Fshow_docid%3D7b027b93b5472100&msg=1bf318a75376c64f> @tarbox.org> wrote: - Hide quoted text - - Show quoted text - <http://groups.google.com/group/ibpy-discuss/browse_thread/thread/0cd27f0c47248af0/7b027b93b5472100?hide_quotes=no#msg_1bf318a75376c64f> > On Jan 29, 4:55 pm, Troy Melhase > <troy.melh...<http://groups.google.com/groups/unlock?_done=/group/ibpy-discuss/browse_thread/thread/0cd27f0c47248af0/7b027b93b5472100%3Fshow_docid%3D7b027b93b5472100&msg=1bf318a75376c64f> @gmail.com> wrote: > > I once did a bit of looking. There are python interfaces of various > > > maturities although I think I remember more about interfaces in Java > > > and possibly CORBA (not something with which one dabbles in lightly, > > > particularly with Python) > > ... > > > Therefore, you build python remote access to an IbPy process which can > > > be accessed both from your gnumeric python and your PyQt process over > > > the wire (pickle is your friend here). > > What I'm hearing from this thread is that the choice for RPC between > > an arbitrary process (gnumeric, OO) and an IbPy process (your script, > > gui, app) is either non-existent or is substandard. And you really > > shouldn't have to make a choice, there should be a solid, thought-thru > > solution for you. > Wouldn't it be wonderful if everyone thought like you. :-) > > Every time I've had to construct my own RPC mechanism, I've used a > > small web server. The benefits typically outweigh the downsides > > (stateless nature) of the protocol. So I think I'll code up an "ibpy > > server" and an "ibpy command-line client" for the 0.7.7 release. > I'll write some below, but if you were going to do this, I think it > would be useful to take a step back and think broadly about the > architecture. > If you're gonna expose the interface "over the wire", the server side > language can be independent of the client if you're willing to avoid > python specific mechanisms (pickle) > I had to move away from IbPy on the server side for performance > reasons... when you pull depth and a bunch of instruments, throw some > database stuff and maybe an analysis op or two and you quickly hit > performance limits. > In my case, I went all the way: I implemented an interface to a Java > server using Google Protocol Buffers. The reasons are many and its > arguable whether its the right choice for an IB only system, but Java, > C++ and Python are supported natively. > Unfortunately, another problem quickly became apparent: Protocol > buffers are slow in python due to their implementation (exploit a lot > of reflection... makes it elegant and all, but brings the python up > again). So, its great for lots of python apps, but not data at any > real volume. > However, my system is quite a bit more performance intensive than most > need... particularly since I got a live tick feed which requires Java > or C++ anyways. > > The server would expose URLs like this: > > /ib/cancelOrder/<orderid> > > /ib/reqMktData/<tickid> > > /ib/reqHistoricalData/<tickid> > > Those above would be available via HTTP POST, and other required > > params would have to be sent as part of the request (e.g., in > > reqMktData we can't have > > /<tickid>/<contractfield1>/<contractfield2>/<etc>, so we collapse > > those and only require the key (ticker id) as part of the URL). > > These calls would be available via HTTP GET: > > /ib/order/<orderid> > > /ib/marketdata/<tickerid> > > /ib/histdata/<tickid> > > In the case of these, they are data-returning calls, meaning that > > clients can make requests to fetch the values whenever they like. > > These calls would return data encoded as CSV, RSS, or JSON. > So, the proposal is some kind of REST web services interface. I think > this is a great idea... > But, then we're into why is the server in Python? > Believe me, I do every bit of coding in Python I can (which includes > Numpy, Matplotlib, Sage) but Python has performance issues. So, if > you're gonna build a server using "web stuff", I'm guessing that a > rethink might be in order. > It would likely be possible to generate a Python client side interface > which works similarly to your current "Pythonic" API but with > enhancements to support synchronous clients (i.e. no callbacks). Of > course, the design issues here can get tricky as there are results and > the client may choose to poll which means the server side needs to > queue responses. Not nasty technically, but the use case explosion can > sometimes bite you. > > As for the command line client, it would operate in one of two modes: > > as an IbPy HTTP client or as an TWS library client. In both cases, > > the interface would be the same, and it would certainly allow for > > integration with any program that can start other processes. > > Examples: > > $ runibpy --clientid=1 --host=myserver --command=reqMktData > > --tickerid=832 --rows=50 > > $ runibpy --http --host=myserver:8080 --command=reqMktData ... > > The only question here is how to deal with compound data structures > > like the contract and order objects. > you betcha... this is where I spent a great deal of time hacking. It > was more typing really as I took the IB java code, commented it all > out, and typed the Protocol Buffer interface definitions kinda in- > line. It took a few days but there are constructs in protobuf which > help with the more complex structures. > Of course, one could also define an XSD, the schema isn't hugely > nasty, but now people who use IbPy because its simple and Python are > wrestling with a different kind of headache.... wrestling with XML can > be kinda nasty although there are certainly good tools / libraries > etc. > > > In previous posts, there are references to the Twisted IbPy > > > implementation I wrote. Also, I built / maintain the twisted Qt > > > reactor. If you're going to do asynchronous processing, I highly > > > recommend twisted although its a non-trivial elephant to swallow... > > > IMHO, its worth it as is Qt (thanks Troy) > > Does twisted have some kind of WSGI interface or support? I'd be > > willing to try that over the basic HTTP servers in the standard > > library. > Twisted has a full HTTP server implementation and would also be > helpful in that it can easily handle multiple clients and IB without > requiring threads or all the nastiness which comes with it. Its also > pretty high performance as its architected well and all... but its a > non-trivial beast to swallow... as is asynchronous programming in > general. I highly recommend it, but I'm not sure you should take my > recommendation :-) > > Thoughts? > As an aside, one of the advantages of the approach I took with a java > based server, is that I can connect to multiple IB clients all pumping > a bunch of data without issue. This is, again, not the kind of > problem one often finds "in the wild", but it helped me out quite a > bit. > Of course, I eventually gave up, wrote a check, and am getting every > tick for every exchange I trade from another vendor (NxCore). Next > step is to go FIX... > -glenn > > troy -- *** Note: New Number Glenn H. Tarbox, PhD || 206-274-6919
_______________________________________________ Cython-dev mailing list [email protected] http://codespeak.net/mailman/listinfo/cython-dev
