I probably was not clear. My question is not how to use mapped files, but where to go from there. Mapped files does not solve the problem directly since I can't use the same algorithm on it.
For instance, catopen would take tremendous time and space. Moreover, since the length of the lines is not fixed I can't state the number of columns when mapping the file. On s scalar language (for instance python) I would do: for line in file.readlines(): handle_line(line) this is very efficient spacewise since readlines() reads several blocks at a time. But: 1) As far as I understand, walking over the lines is not the J way. 2) Even if I want that, I didn't find the equivalent of Python's readlines() in the docs. Yoel On 5/14/06, Björn Helgason <[EMAIL PROTECTED]> wrote:
Answers to you questions are as youself point out in memory mapped files Read the labs and experiment with them 2006/5/14, Yoel Jacobsen <[EMAIL PROTECTED]>: > > Hello, > > I'm new to J so please forgive me if this is a FAQ. > > I wrote some short sentences to parse a log file. I want to retrieve all > the > unique values of some attribute. The way it shows in the log file is > <attribute name>SPACE<attribute value> such as "..... csn 92892849893284 > ..." > > My initial (brute force) program is: > > text =: 1!:1 < '/tmp/logfile' > words =: cutopen text > bv =: (<'csn') = words > srbv =: _1 |.!.0 bv > csns =: ~. srbv # words > > Now csns holds the unique values as requested. > > The program works fine for small files (few megabytes). > > My question is, what should be done to make it work for large files (say, > 1GB or more)? I guess it involves memory mapped files but I have no clue > where to continue from here. > > Further, is there any notion of 'laziness' (evaluate only when the data is > really needed) in J? can a verb be decalred as a lazy verb? > > Thanks, > > Yoel > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm > -- Björn Helgason, Verkfræðingur Fugl&Fiskur ehf, Þerneyjarsund 23, Skype: gosiminn, gsm: +3546985532 801 Grímsnes ,t-póst: [EMAIL PROTECTED] Landslags og skrúðgarðagerð, gröfuþjónusta http://groups.google.com/group/J-Programming Tæknikunnátta höndlar hið flókna, sköpunargáfa er meistari einfaldleikans góður kennari getur stigið á tær án þess að glansinn fari af skónum /|_ .-----------------------------------. ,' .\ / | Með léttri lund verður | ,--' _,' | Dagurinn í dag | / / | Enn betri en gærdagurinn | ( -. | `-----------------------------------' | ) | (`-. '--.) `. )----' ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
