One thing to keep in mind is that there's usually a trade-off between
setup (writing and testing) and execution time. For one-off data
processing, I'd focus on implementation speed rather than execution
speed (in other words, FORTRAN might not be ideal unless you're already
fluent with it).
That said, I'd take a look at python, octave or R. Python's relatively
easy to learn, and more flexible than octave/R; but it doesn't have the
built-in statistic functions that octave and R do.
One other tip which you've probably already though of - Depending on
your runtimes (I don't think 100s MB of data is usually considered an
enormous amount, but it'll depend on what you're doing) it may be worth
getting things working on a small subset of the data first.
Pete
Jacob Keller wrote:
Dear List,
since this probably comes up a lot in manipulation of pdb/reflection files
and so on, I was curious what people thought would be the best language for
the following: I have some huge (100s MB) tables of tab-delimited data on
which I would like to do some math (averaging, sigmas, simple arithmetic,
etc) as well as some sorting and rejecting. It can be done in Excel, but
this is exceedingly slow even in 64-bit, so I am looking to do it through
some scripting. Just as an example, a "sort" which takes >10 min in Excel
takes ~10 sec max with the unix command sort (seems crazy, no?). Any
suggestions?
Thanks, and sorry for being off-topic,
Jacob
