If all you want to do is store J arrays to file, I've been doing this for many years using my own WS code: https://code.jsoftware.com/wiki/Scripts/WS_Files.
On Fri, Feb 3, 2023 at 9:18 AM Don Guinn <dongu...@gmail.com> wrote: > It's nice to not have to read and write the file using mapped files. J or > the system does it for you. The data can be shared by multiple instances of > J or whatever. And data is preserved over restarts. But things get messy if > the array gets bigger than the size of the mapped file. But jmf does not > support boxed arrays as the data for boxed arrays may not be contiguous in > memory. So complex sets of data for an app may require several mapped > files. This leads to the possibility that in the event of a crash the files > may not be in sync. > > I would suspect that 3!:1 most of the overhead is having to copy the entire > array to a contiguous area of memory and not just increment the use count > on the data. Maybe for non-boxed data it may not have to make a copy as it > is already contiguous in memory. 3!:2 overhead is probably converting > relative pointers from relative to actual memory address and checking the > little/big endian. > > I like to use 3!:1 and 3!:2 because I control when the data for an app is > written to disk, kind of like a commit in database terminology. Kind of > like a package of data. And the size of the data is not a concern as the > size of the file is not set when created. > > On Fri, Feb 3, 2023 at 5:55 AM Raul Miller <rauldmil...@gmail.com> wrote: > > > Minor nitpick: > > > > 3!:1 (and 3!:3) represent the a noun textually > > > > 3!:2 reconstitutes the original noun. > > > > Anyways, personally, I'd prefer to say that 3!:1 gives a > > representation of the noun. (Or 3!:3, but 3!:1 is more convenient, and > > efficient, when writing to a file.) > > > > This is a really minor issue, roughly equivalent to a grammar nitpick. > > However, computers tend to be fussy about such things, so I thought I > > should mention it. > > > > Thanks, > > > > -- > > Raul > > > > On Fri, Feb 3, 2023 at 7:17 AM Jan-Pieter Jacobs > > <janpieter.jac...@gmail.com> wrote: > > > > > > 3!:2 just gives a representation of the noun. JMF memory-maps a file > via > > > the OS, so you can access the file's content as if it were just normal > > > memory. It will stay only on disk as long as only in-place methods are > > used > > > to change it (doing something like othervar=: +: mapped var will copy > it > > > entirely to memory anyhow). This page > > > https://code.jsoftware.com/wiki/Mapped_Files and this page (basically > > the > > > same as the jmf Lab): > > https://code.jsoftware.com/wiki/Studio/Mapped_Files > > > explain how to use JMF, and how it works. > > > > > > I think the two mechanisms are entirely different, and both have their > > > pro's and con's: > > > > > > reading data from file and applying 3!:2 is far simpler than JMF, and > > > allows working with the data faster once read, as the data is copied to > > > memory. JMF mapped nouns remain on disk, and incur the corresponding > > > transfer speed penalty. > > > > > > On the other hand JMF allows using data that doesn't fit in memory, but > > > requires care to only apply in-place modification and care not to read > > more > > > than fits your memory. > > > > > > On Fri, 3 Feb 2023, 11:04 Ak O, <akin...@gmail.com> wrote: > > > > > > > I guess one of my questions directed at how the function works under > > the > > > > hood. > > > > > > > > Is it that 3!:2 is the mechanism jmf uses to treat the map? > > > > How do these forms differ? > > > > > > > > Ak > > > > > > > > On Thu., Feb. 2, 2023, 22:31 Raul Miller, <rauldmil...@gmail.com> > > wrote: > > > > > > > > > You can measure overhead with timespacex > > > > > > > > > > Maybe you had already been doing that? > > > > > > > > > > -- > > > > > Raul > > > > > > > > > > On Thu, Feb 2, 2023 at 11:29 PM Ak O <akin...@gmail.com> wrote: > > > > > > > > > > > > Is this less overhead than the jmf form? > > > > > > > > > > > > In your example, every case where I want to operate on a file or > an > > > > > object > > > > > > within the file requires translation through the 3!:2 operator. > > > > > > > > > > > > Maybe it is functionally the same as the map_jmf_ function, or > am I > > > > > > thinking about this incorrectly? > > > > > > Or can you please explain the difference. > > > > > > > > > > > > Thx > > > > > > > > > > > > > > > > > > Ak > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu., Feb. 2, 2023, 15:43 Don Guinn, <dongu...@gmail.com> > > wrote: > > > > > > > > > > > > > 'testfile.txt' fwrite~3!:1 'Hello World!';(i.3 4);<2 1$'Text > > Here';1 > > > > 2 > > > > > 3 4 > > > > > > > 456 > > > > > > > > > > > > > > 3!:2 fread 'testfile.txt' > > > > > > > > > > > > > > ┌────────────┬─────────┬───────────┐ > > > > > > > > > > > > > > │Hello World!│0 1 2 3│┌─────────┐│ > > > > > > > > > > > > > > │ │4 5 6 7││Text Here││ > > > > > > > > > > > > > > │ │8 9 10 11│├─────────┤│ > > > > > > > > > > > > > > │ │ ││1 2 3 4 ││ > > > > > > > > > > > > > > │ │ │└─────────┘│ > > > > > > > > > > > > > > └────────────┴─────────┴───────────┘ > > > > > > > > > > > > > > On Thu, Feb 2, 2023 at 2:19 PM Ak O <akin...@gmail.com> wrote: > > > > > > > > > > > > > > > In this case the data is an array of boxes. > > > > > > > > The datatype is 'boxe'd. > > > > > > > > The data is of some shape. > > > > > > > > > > > > > > > > These things have to be preserved in order to correctly > restore > > > > them > > > > > > > > later. (at some cost, up-front to specify them, afterwards > to > > > > > recall > > > > > > > the > > > > > > > > specifics, or something else) It might be easy to write as > > literal, > > > > > but > > > > > > > > reverting back can present some challenges. > > > > > > > > > > > > > > > > For example: > > > > > > > > ary=: 'abcd';4 5 6 7; 2 2 $ 8.9 7.6 .6.5 5.4 > > > > > > > > > > > > > > > > datatype ary > > > > > > > > boxed > > > > > > > > > > > > > > > > datatype ":ary > > > > > > > > literal > > > > > > > > ***Danger*** > > > > > > > > $ary NB. An array of structures that preserves > > operational > > > > > > > > intention. > > > > > > > > 3 > > > > > > > > $":ary NB. An array whose structure is different than > > its > > > > > > > > operational intention. > > > > > > > > 4 22 > > > > > > > > > > > > > > > > These are different and need to be accounted for. > > > > > > > > > > > > > > > > datatype (>0{ary) > > > > > > > > literal > > > > > > > > > > > > > > > > datatype (>1{ary) > > > > > > > > Integer > > > > > > > > > > > > > > > > datatype (>2{ary) > > > > > > > > floating > > > > > > > > > > > > > > > > > > > > > > > > Each unit has a shape and a type to preserve. > > > > > > > > > > > > > > > > If I want to operate on this object with the meaning intended > > for > > > > it > > > > > when > > > > > > > > it was created, many errors can be avoided when unnecessary > > > > > intermediate > > > > > > > > conversions are avoided. > > > > > > > > > > > > > > > > Jmf allows you to operate directly without having the need to > > > > > > > > revert from literal to your operation/operand, datatype (with > > the > > > > > > > > associated overhead and tracking).I am not saying that there > > is no > > > > > > > overhead > > > > > > > > with jmf, just that the cognitive load is offset or preset > > because > > > > > jmf > > > > > > > > treats it for the user. > > > > > > > > > > > > > > > > > > > > > > > > (1!2 & 1!:/, fwrite , fread) forms. > > > > > > > > 1. Convert to literal. > > > > > > > > 2. Write literal to file. > > > > > > > > 3. Read literal from file. > > > > > > > > 4. Convert literal type to operation type > > > > > > > > 5. Operate with type. > > > > > > > > End > > > > > > > > ___ > > > > > > > > > > > > > > > > jmf form. > > > > > > > > 1. Create jmf file. > > > > > > > > 2. Map jmf file. > > > > > > > > 3. Operate with file. > > > > > > > > End > > > > > > > > > > > > > > > > > > > > > > > > I am interested in understanding better how the best > > performance is > > > > > > > > achieved comparing these forms. > > > > > > > > > > > > > > > > Thoughts. > > > > > > > > > > > > > > > > > > > > > > > > Ak > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu., Feb. 2, 2023, 07:51 bill lam, <bbill....@gmail.com> > > > > wrote: > > > > > > > > > > > > > > > > > Using Jmf is overkill. Also what are the advantages of jmf > > over > > > > > > > 3!:1/3!:2 > > > > > > > > > in this case? > > > > > > > > > > > > > > > > > > On Thu, 2 Feb 2023 at 7:17 PM Ak O <akin...@gmail.com> > > wrote: > > > > > > > > > > > > > > > > > > > load 'jmf' > > > > > > > > > > NB. Loads jmf facilities. > > > > > > > > > > > > > > > > > > > > testfile =: {2,\?50#75 > > > > > > > > > > > > > > > > > > > > fn =: jpath > > > > > 'C:\Users\skip\J904-user\temp\foo\testfile_name.jmf' > > > > > > > > > > NB. Links a covername to the file path. > > > > > > > > > > createjmf_jmf_ fn;(([:(*&8)#);testfile) > > > > > > > > > > NB. Create the container for your file. > > > > > > > > > > NB. -fn is your reference name > > > > > > > > > > NB. -(([:(*&8)#);testfile) allocates the size of yor fike > > in > > > > > bytes > > > > > > > > > > > > > > > > > > > > map_jmf_ 'testfile_disk';fn > > > > > > > > > > NB. Maps the file to a noun 'testfile_disk' > > > > > > > > > > ] testfile_disk_jmf_ =: testfile > > > > > > > > > > > > > > > > > > > > unmap_jmf 'testfile_disk_jmf_' > > > > > > > > > > NB. Release mappings > > > > > > > > > > Or > > > > > > > > > > Exit 0 > > > > > > > > > > Close session. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > New session. > > > > > > > > > > > > > > > > > > > > load 'jmf' > > > > > > > > > > fn =: jpath > > > > > > > 'C:\Users\skip\J904-user\temp\foo\testfile_name.jmf' > > > > > > > > > > map_jmf_ 'testfile1';fn > > > > > > > > > > NB. Map file > > > > > > > > > > ] testfile1_jmf_ > > > > > > > > > > NB. File loaded as noun 'testfile1_jmf_' > > > > > > > > > > > > > > > > > > > > Some potential benefits of the .jmf datatype: > > > > > > > > > > -Preserves header and shape information. > > > > > > > > > > -If you want to map you file as read only, use the > > following > > > > > syntax > > > > > > > > > > map_jmf_ 'testfile1';fn;'';1 > > > > > > > > > > -If you would like it to be copy-on-write use the > following > > > > > syntax > > > > > > > > > > map_jmf_ 'testfile1';fn;'';2 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ak > > > > > > > > > > > > > > > > > > > > On Wed., Feb. 1, 2023, 21:48 'Skip Cave' via > Programming, < > > > > > > > > > > programm...@jsoftware.com> wrote: > > > > > > > > > > > > > > > > > > > > > I have a boxed noun: > > > > > > > > > > > > > > > > > > > > > > ] testfile =: {2,\?15#50 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ┌────┬─────┬─────┬─────┬─────┬─────┬────┬────┬────┬────┬─────┬────┬───┬────┐ > > > > > > > > > > > > > > > > > > > > > > │9 21│21 47│47 37│37 13│13 33│33 20│20 4│4 49│49 6│6 > > 25│25 > > > > > 33│33 > > > > > > > 9│9 > > > > > > > > > 6│6 > > > > > > > > > > > 43│ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > └────┴─────┴─────┴─────┴─────┴─────┴────┴────┴────┴────┴─────┴────┴───┴────┘ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I want to store it in the following location on my > > machine: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ]fn =. < 'C:\Users\skip\J904-user\temp' > > > > > > > > > > > > > > > > > > > > > > ┌────────────────────────────┐ > > > > > > > > > > > > > > > > > > > > > > │C:\Users\skip\J904-user\temp│ > > > > > > > > > > > > > > > > > > > > > > └────────────────────────────┘ > > > > > > > > > > > > > > > > > > > > > > Then I will close that J session. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Later, I will start a new J session, and I want to read > > that > > > > > saved > > > > > > > > file > > > > > > > > > > > into a noun called 'test1' > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > What is the J code for writing the noun into a file in > > the > > > > > first > > > > > > > > > session? > > > > > > > > > > > > > > > > > > > > > > What is the J code to read the file into a noun in the > > second > > > > > > > > session? > > > > > > > > > > > > > > > > > > > > > > Should I use 1!2 & 1!:1, or fwrite & fread, or > something > > > > else? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Where in the J doc are these file operations and their > > > > > tradeoffs > > > > > > > > > > described? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Skip > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Skip Cave > > > > > > > > > > > Cave Consulting LLC > > > > > > > > > > > > > > > > > > > > > > > > > > ---------------------------------------------------------------------- > > > > > > > > > > > For information about J forums see > > > > > > > > http://www.jsoftware.com/forums.htm > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ---------------------------------------------------------------------- > > > > > > > > > > For information about J forums see > > > > > > > http://www.jsoftware.com/forums.htm > > > > > > > > > > > > > > > > > > > > > > > > > > ---------------------------------------------------------------------- > > > > > > > > > For information about J forums see > > > > > http://www.jsoftware.com/forums.htm > > > > > > > > > > > > > > > > > > > > > > > > ---------------------------------------------------------------------- > > > > > > > > For information about J forums see > > > > > http://www.jsoftware.com/forums.htm > > > > > > > > > > > > > > > > > > > > ---------------------------------------------------------------------- > > > > > > > For information about J forums see > > > > http://www.jsoftware.com/forums.htm > > > > > > > > > > > > > > > ---------------------------------------------------------------------- > > > > > > For information about J forums see > > http://www.jsoftware.com/forums.htm > > > > > > > ---------------------------------------------------------------------- > > > > > For information about J forums see > > http://www.jsoftware.com/forums.htm > > > > > > > > > > ---------------------------------------------------------------------- > > > > For information about J forums see > http://www.jsoftware.com/forums.htm > > > > > > > ---------------------------------------------------------------------- > > > For information about J forums see http://www.jsoftware.com/forums.htm > > ---------------------------------------------------------------------- > > For information about J forums see http://www.jsoftware.com/forums.htm > > > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm > -- Devon McCormick, CFA Quantitative Consultant ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
