You may not have understood what mapped files are You do not read them into the workarea Opening a mapped file takes a very short time catopen you mention is probably reading all the data into the workarea and that is not the way mapped files will help your case As you see in the mapped file labs the mapped files stay outside the workarea You only bring in the bits you do need when you need them
2006/5/14, Yoel Jacobsen <[EMAIL PROTECTED]>:
I probably was not clear. My question is not how to use mapped files, but where to go from there. Mapped files does not solve the problem directly since I can't use the same algorithm on it. For instance, catopen would take tremendous time and space. Moreover, since the length of the lines is not fixed I can't state the number of columns when mapping the file. On s scalar language (for instance python) I would do: for line in file.readlines(): handle_line(line) this is very efficient spacewise since readlines() reads several blocks at a time. But: 1) As far as I understand, walking over the lines is not the J way. 2) Even if I want that, I didn't find the equivalent of Python's readlines() in the docs. Yoel On 5/14/06, Björn Helgason <[EMAIL PROTECTED]> wrote: > > Answers to you questions are as youself point out in memory mapped files > Read the labs and experiment with them > > 2006/5/14, Yoel Jacobsen <[EMAIL PROTECTED]>: > > > > Hello, > > > > I'm new to J so please forgive me if this is a FAQ. > > > > I wrote some short sentences to parse a log file. I want to retrieve all > > the > > unique values of some attribute. The way it shows in the log file is > > <attribute name>SPACE<attribute value> such as "..... csn 92892849893284 > > ..." > > > > My initial (brute force) program is: > > > > text =: 1!:1 < '/tmp/logfile' > > words =: cutopen text > > bv =: (<'csn') = words > > srbv =: _1 |.!.0 bv > > csns =: ~. srbv # words > > > > Now csns holds the unique values as requested. > > > > The program works fine for small files (few megabytes). > > > > My question is, what should be done to make it work for large files > (say, > > 1GB or more)? I guess it involves memory mapped files but I have no clue > > where to continue from here. > > > > Further, is there any notion of 'laziness' (evaluate only when the data > is > > really needed) in J? can a verb be decalred as a lazy verb? > > > > Thanks, > > > > Yoel > > ---------------------------------------------------------------------- > > For information about J forums see http://www.jsoftware.com/forums.htm > > > > > > -- > Björn Helgason, Verkfræðingur > Fugl&Fiskur ehf, Þerneyjarsund 23, > Skype: gosiminn, gsm: +3546985532 > 801 Grímsnes ,t-póst: [EMAIL PROTECTED] > Landslags og skrúðgarðagerð, gröfuþjónusta > http://groups.google.com/group/J-Programming > > > Tæknikunnátta höndlar hið flókna, sköpunargáfa er meistari einfaldleikans > > góður kennari getur stigið á tær án þess að glansinn fari af skónum > /|_ .-----------------------------------. > > ,' .\ / | Með léttri lund verður | > ,--' _,' | Dagurinn í dag | > / / | Enn betri en gærdagurinn | > ( -. | `-----------------------------------' > | ) | > (`-. '--.) > `. )----' > > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm > > ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
-- Björn Helgason, Verkfræðingur Fugl&Fiskur ehf, Þerneyjarsund 23, Skype: gosiminn, gsm: +3546985532 801 Grímsnes ,t-póst: [EMAIL PROTECTED] Landslags og skrúðgarðagerð, gröfuþjónusta http://groups.google.com/group/J-Programming Tæknikunnátta höndlar hið flókna, sköpunargáfa er meistari einfaldleikans góður kennari getur stigið á tær án þess að glansinn fari af skónum /|_ .-----------------------------------. ,' .\ / | Með léttri lund verður | ,--' _,' | Dagurinn í dag | / / | Enn betri en gærdagurinn | ( -. | `-----------------------------------' | ) | (`-. '--.) `. )----'
---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
