Tyler Brough wrote: > I am new to Cython, though I heard about it long ago. I have been > programming in python using scipy and pytables a lot. I am building an > HDF5 > database using pytables from extremely large files containing tick by tick > stock data (one single day is several GBs). As you can imagine this is > very > slow going, even trying to optimize the file io in python. What I am > wondering is will trying to do the file io in cython functions speed up > the process enough to make it worth while?
Cython will not magically speed up your I/O operations for you. It does, however, allow you to work at the C level if you need to, thus avoiding I/O overhead yourself, at the cost of writing low-level-ish code. Also, if you keep your code free from Python-level operations, you can free the GIL and thus leverage full thread parallelism on I/O operations. > Can I simply access c FILE pointers > directly in cython and pass them back as python objects? Cython doesn't convert between FILE and file() automatically. However, the Python 2.x C-API allows you to access the low-level FILE* from a file() instance, so you can open a file at the Python level (also from Cython), and then work with the FILE* in your I/O operations. I never worked with pytables, so I can't say how that fits together, and what Cython can do for you here. What makes you think the I/O performance itself is your problem? What kind of operations are you performing on the database? Stefan _______________________________________________ Cython-dev mailing list [email protected] http://codespeak.net/mailman/listinfo/cython-dev
