I am replying to you out side of the forum about Jd, a new Jsoftware product that includes a fast csv loader.
On Mon, Nov 11, 2013 at 7:49 PM, Scott Locklin <[email protected]> wrote: > So, I'm having a hard time loading some very simple, but relatively large csv > files into J for processing. The problem arises because J boxes all the atoms > individually. Since in general (headers excluded), columns of the csv are all > of the same type, this seems wasteful. In this case, everything's an int > (about 100M rows of 3 columns of longints), so it seems particularly > wasteful. Having to invoke ". on all the elements to create the array also > seems wasteful for routine csv work. Is there some csv loader trick which I > am missing out on, or a particularly J-like way of doing this? > > > I was going to write a quick C function to do it with numeric arrays, but > solving the problem in general for numeric arrays seems like a good idea for > solving real world problems. One can always use tr, cut or sed to strip out > the character columns for processing later by standard tables/csv means. > > > It's a bit frustrating in that J is outrageously good at the sort of task I > needed to do on the ints (I. basically), but the overhead of loading the data > was time consuming. > > -Scott > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
