For convenience and clarity, this is the diff in question: https://github.com/Dynetrekk/numpy-1/commit/5bde67531a2005ef80a2690a75c65cebf97c9e00
And this is my numpy fork: https://github.com/Dynetrekk/numpy-1/ Paul On Sun, Sep 30, 2012 at 4:14 PM, Paul Anton Letnes <[email protected]> wrote: > Hello everyone, > > I've modified loadtxt to make it (potentially) more memory efficient. > The idea is that if a user passes a seekable file, (s)he can also pass > the 'seekable=True' kwarg. Then, loadtxt will count the number of > lines (containing data) and allocate an array of exactly the right > size to hold the loaded data. The downside is that the line counting > more than doubles the runtime, as it loops over the file twice, and > there's a sort-of unnecessary np.array function call in the loop. The > branch is called faster-loadtxt, which is silly due to the runtime > doubling, but I'm hoping that the false advertising is acceptable :) > (I naively expected a speedup by removing some needless list > manipulation.) > > I'm pretty sure that the function can be micro-optimized quite a bit > here and there, and in particular, the main for loop is a bit > duplicated right now. However, I got the impression that someone was > working on a More Advanced (TM) C-based file reader, which will > replace loadtxt; this patch is intended as a useful thing to have > while we're waiting for that to appear. > > The patch passes all tests in the test suite, and documentation for > the kwarg has been added. I've modified all tests to include the > seekable kwarg, but that was mostly to check that all tests are passed > also with this kwarg. I guess it's bit too late for 1.7.0 though? > > Should I make a pull request? I'm happy to take any and all > suggestions before I do. > > Cheers > Paul _______________________________________________ NumPy-Discussion mailing list [email protected] http://mail.scipy.org/mailman/listinfo/numpy-discussion
