On Feb 9, 2009, at 10:28 AM, Dag Sverre Seljebotn wrote: >> Tyler Brough wrote: > Hello All, >> >> I am new to Cython, though I heard about it long ago. I have been >> programming in python using scipy and pytables a lot. I am >> building an >> HDF5 >> database using pytables from extremely large files containing tick >> by tick >> stock data (one single day is several GBs). As you can imagine >> this is >> very >> slow going, even trying to optimize the file io in python. What I am >> wondering is will trying to do the file io in cython functions >> speed up >> the >> process enough to make it worth while? Can I simply access c FILE >> pointers >> directly in cython and pass them back as python objects? Your >> advice is >> very much appreciated! > > A quick look on the pytables source reveal that they have plenty of C > files there, and also some Pyrex and/or Cython files. The Pytable > authors > have likely thought of all the obvious ways to speed things up > already... > > But to answer your question; from Cython you can call all the C IO > functions directly, but a C FILE pointer cannot be passed into > Python. You > might be able to access the underlying handle of Python file > objects if > that's what you mean, you should look in the Python headers and > CPython > documentation for that.
You can pass file objects around in Python, which are basically wrappers around c FILE pointers (which can easily be extracted and used directly from Cython code). As has already been mentioned, it is unclear if this will provide speed gains (especially if your application is io bound rather than cpu bound). - Robert _______________________________________________ Cython-dev mailing list [email protected] http://codespeak.net/mailman/listinfo/cython-dev
