Hi - tonight's NYCJUG has a section that may be relevant to your problem. I've developed an adverb that applies an arbitrary verb across a large file. In this example, the verb I apply is one that simply breaks the large file into smaller ones but, in theory, any arbitrary verb could be applied to work its way across a file.
Take a look at http://www.jsoftware.com/jwiki/NYCJUG/2013-11-12#Working_with_Large_Files_in_Piecesto see my example. The "doSomething" adverb handles keeping track of any left-over pieces from the previous invocation for the case where we don't want to break the file arbitrarily but want to respect EOL boundaries so we only break at line-ends. Please let me know what you think. Regards, Devon On Tue, Nov 12, 2013 at 12:01 AM, bill lam <[email protected]> wrote: > Yes, the assumption for well formed can fail for too many > reasons, some represent missing data with consecutive comma, > and dyad (nor monad)". alone cannot handle them, eg > > _99 ". '' > > {.(!._99) _99 ". '' > _99 > > Вт, 12 ноя 2013, Ric Tikkanz писал(а): > > I'd probably use freads to read the text file so I wouldn't have to check > > what the line-endings were. > > > > I prefer to use dyadic ". (Numbers) rather than monadic ". (Do) to > convert > > strings to numeric because it gives me an easy way to see if something > went > > wrong (i.e. does the resulting array contain the "error" number I gave as > > the left arg). Of course if you're *sure* that the data is well-formed > then > > this is redundant. > > > > > > On Tue, Nov 12, 2013 at 5:20 PM, bill lam <[email protected]> wrote: > > > > > If data contain negatives or CR, then I would prefer > > > filter them first using charsub > > > > > > ".;._2 ('-_',CR,' ') charsub makecsv _2+ 12 4 ?.@$ 5 > > > > > > Вт, 12 ноя 2013, Ric Tikkanz писал(а): > > > > The problem with a general addon tables/csv is that it has to handle > all > > > > sorts of edge cases, this has a significant overhead and as Bill has > > > > pointed out, if you can rely on your data being well-formed then > using > > > the > > > > tables/csv is definitely overkill. Bill's suggestion will work > great if > > > > you can rely on the fact that you have a numeric csv with no > negatives. > > > If > > > > that is not the case then the following might work better: > > > > > > > > _99 ". ([: ];._2 ,&',');._2 freads datloc,'chunk1.csv' > > > > > > > > For example compare the results of > > > > require 'tables/csv' > > > > ".;._2 makecsv _2+ 12 4 ?.@$ 5 > > > > to > > > > _99 ". ([: ];._2 ,&',');._2 makecsv _2+ 12 4 ?.@$ 5 > > > > > > > > > > > > > > > > On Tue, Nov 12, 2013 at 4:12 PM, Scott Locklin < > [email protected] > > > >wrote: > > > > > > > > > "bill lam-2" wrote: > > > > > > > > > > > ".;._2 a > > > > > > 1 2 3 > > > > > > 4 5 6 > > > > > > > > > > > > > > > Oh chrikey, that is a whole lot better (and a lot easier than > writing > > > a C > > > > > gizmo to do this). Thank you for your patience in showing me that > one. > > > > > > > > > > > > > > > -SL > > > > > > ---------------------------------------------------------------------- > > > > > For information about J forums see > http://www.jsoftware.com/forums.htm > > > > > > > > > > ---------------------------------------------------------------------- > > > > For information about J forums see > http://www.jsoftware.com/forums.htm > > > > > > -- > > > regards, > > > ==================================================== > > > GPG key 1024D/4434BAB3 2008-08-24 > > > gpg --keyserver subkeys.pgp.net --recv-keys 4434BAB3 > > > gpg --keyserver subkeys.pgp.net --armor --export 4434BAB3 > > > ---------------------------------------------------------------------- > > > For information about J forums see http://www.jsoftware.com/forums.htm > > > > > ---------------------------------------------------------------------- > > For information about J forums see http://www.jsoftware.com/forums.htm > > -- > regards, > ==================================================== > GPG key 1024D/4434BAB3 2008-08-24 > gpg --keyserver subkeys.pgp.net --recv-keys 4434BAB3 > gpg --keyserver subkeys.pgp.net --armor --export 4434BAB3 > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm > -- Devon McCormick, CFA ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
