There's also memory limits imposed from outside of J. For example, there's the `ulimit` unix command, and there can be hard coded OS limitations (for example on a 32 bit machine, the OS might limit you to 2GB or 3 GB of memory). And, of course, there's the amount of memory which you actually have (sometimes with this limit softened through the use of swap space and virtual memory).
Also... a rough rule of thumb (that works reasonably often) would be to allow 5 times as much memory for intermediate results (when compared to what you need for arguments or final results). -- Raul On Sat, Feb 4, 2023 at 2:56 AM 'robert therriault' via Programming <programm...@jsoftware.com> wrote: > > The only memory settings that I am aware of are the 9!:20 and 9!:21 foreigns > https://www.jsoftware.com/help/dictionary/dx009.htm > > Cheers, bob > > > On Feb 3, 2023, at 23:47, Ak O <akin...@gmail.com> wrote: > > > > On Fri., Feb. 3, 2023, 05:17 Jan-Pieter Jacobs, <janpieter.jac...@gmail.com> > > wrote: > > > >> 3!:2 just gives a representation of the noun. JMF memory-maps a file via > >> the OS, so you can access the file's content as if it were just normal > >> memory. It will stay only on disk as long as only in-place methods are used > >> to change it (doing something like othervar=: +: mapped var will copy it > >> entirely to memory anyhow). This page > >> https://code.jsoftware.com/wiki/Mapped_Files and this page (basically the > >> same as the jmf Lab): https://code.jsoftware.com/wiki/Studio/Mapped_Files > >> explain how to use JMF, and how it works. > >> > >> I think the two mechanisms are entirely different, and both have their > >> pro's and con's: > >> > >> reading data from file and applying 3!:2 is far simpler than JMF, and > >> allows working with the data faster once read, as the data is copied to > >> memory. JMF mapped nouns remain on disk, and incur the corresponding > >> transfer speed penalty. > > > > Oh right, size is one factor. > > How do I determine where the memory limit is (is this part of the 4gb > > allocated to the j instance by the operating system)? > > Can it be increased (or is this only for the jmf domain)? > > > > > >> > >> On the other hand JMF allows using data that doesn't fit in memory, but > >> requires care to only apply in-place modification and care not to read more > >> than fits your memory. > >> > > > > Thx > > > > Ak. > > > > > >> On Fri, 3 Feb 2023, 11:04 Ak O, <akin...@gmail.com> wrote: > >> > >>> I guess one of my questions directed at how the function works under the > >>> hood. > >>> > >>> Is it that 3!:2 is the mechanism jmf uses to treat the map? > >>> How do these forms differ? > >>> > >>> Ak > >>> > >>> On Thu., Feb. 2, 2023, 22:31 Raul Miller, <rauldmil...@gmail.com> wrote: > >>> > >>>> You can measure overhead with timespacex > >>>> > >>>> Maybe you had already been doing that? > >>>> > >>>> -- > >>>> Raul > >>>> > >>>> On Thu, Feb 2, 2023 at 11:29 PM Ak O <akin...@gmail.com> wrote: > >>>>> > >>>>> Is this less overhead than the jmf form? > >>>>> > >>>>> In your example, every case where I want to operate on a file or an > >>>> object > >>>>> within the file requires translation through the 3!:2 operator. > >>>>> > >>>>> Maybe it is functionally the same as the map_jmf_ function, or am I > >>>>> thinking about this incorrectly? > >>>>> Or can you please explain the difference. > >>>>> > >>>>> Thx > >>>>> > >>>>> > >>>>> Ak > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> On Thu., Feb. 2, 2023, 15:43 Don Guinn, <dongu...@gmail.com> wrote: > >>>>> > >>>>>> 'testfile.txt' fwrite~3!:1 'Hello World!';(i.3 4);<2 1$'Text > >> Here';1 > >>> 2 > >>>> 3 4 > >>>>>> 456 > >>>>>> > >>>>>> 3!:2 fread 'testfile.txt' > >>>>>> > >>>>>> ┌────────────┬─────────┬───────────┐ > >>>>>> > >>>>>> │Hello World!│0 1 2 3│┌─────────┐│ > >>>>>> > >>>>>> │ │4 5 6 7││Text Here││ > >>>>>> > >>>>>> │ │8 9 10 11│├─────────┤│ > >>>>>> > >>>>>> │ │ ││1 2 3 4 ││ > >>>>>> > >>>>>> │ │ │└─────────┘│ > >>>>>> > >>>>>> └────────────┴─────────┴───────────┘ > >>>>>> > >>>>>> On Thu, Feb 2, 2023 at 2:19 PM Ak O <akin...@gmail.com> wrote: > >>>>>> > >>>>>>> In this case the data is an array of boxes. > >>>>>>> The datatype is 'boxe'd. > >>>>>>> The data is of some shape. > >>>>>>> > >>>>>>> These things have to be preserved in order to correctly restore > >>> them > >>>>>>> later. (at some cost, up-front to specify them, afterwards to > >>>> recall > >>>>>> the > >>>>>>> specifics, or something else) It might be easy to write as > >> literal, > >>>> but > >>>>>>> reverting back can present some challenges. > >>>>>>> > >>>>>>> For example: > >>>>>>> ary=: 'abcd';4 5 6 7; 2 2 $ 8.9 7.6 .6.5 5.4 > >>>>>>> > >>>>>>> datatype ary > >>>>>>> boxed > >>>>>>> > >>>>>>> datatype ":ary > >>>>>>> literal > >>>>>>> ***Danger*** > >>>>>>> $ary NB. An array of structures that preserves > >> operational > >>>>>>> intention. > >>>>>>> 3 > >>>>>>> $":ary NB. An array whose structure is different than its > >>>>>>> operational intention. > >>>>>>> 4 22 > >>>>>>> > >>>>>>> These are different and need to be accounted for. > >>>>>>> > >>>>>>> datatype (>0{ary) > >>>>>>> literal > >>>>>>> > >>>>>>> datatype (>1{ary) > >>>>>>> Integer > >>>>>>> > >>>>>>> datatype (>2{ary) > >>>>>>> floating > >>>>>>> > >>>>>>> > >>>>>>> Each unit has a shape and a type to preserve. > >>>>>>> > >>>>>>> If I want to operate on this object with the meaning intended for > >>> it > >>>> when > >>>>>>> it was created, many errors can be avoided when unnecessary > >>>> intermediate > >>>>>>> conversions are avoided. > >>>>>>> > >>>>>>> Jmf allows you to operate directly without having the need to > >>>>>>> revert from literal to your operation/operand, datatype (with the > >>>>>>> associated overhead and tracking).I am not saying that there is > >> no > >>>>>> overhead > >>>>>>> with jmf, just that the cognitive load is offset or preset > >> because > >>>> jmf > >>>>>>> treats it for the user. > >>>>>>> > >>>>>>> > >>>>>>> (1!2 & 1!:/, fwrite , fread) forms. > >>>>>>> 1. Convert to literal. > >>>>>>> 2. Write literal to file. > >>>>>>> 3. Read literal from file. > >>>>>>> 4. Convert literal type to operation type > >>>>>>> 5. Operate with type. > >>>>>>> End > >>>>>>> ___ > >>>>>>> > >>>>>>> jmf form. > >>>>>>> 1. Create jmf file. > >>>>>>> 2. Map jmf file. > >>>>>>> 3. Operate with file. > >>>>>>> End > >>>>>>> > >>>>>>> > >>>>>>> I am interested in understanding better how the best performance > >> is > >>>>>>> achieved comparing these forms. > >>>>>>> > >>>>>>> Thoughts. > >>>>>>> > >>>>>>> > >>>>>>> Ak > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Thu., Feb. 2, 2023, 07:51 bill lam, <bbill....@gmail.com> > >>> wrote: > >>>>>>> > >>>>>>>> Using Jmf is overkill. Also what are the advantages of jmf over > >>>>>> 3!:1/3!:2 > >>>>>>>> in this case? > >>>>>>>> > >>>>>>>> On Thu, 2 Feb 2023 at 7:17 PM Ak O <akin...@gmail.com> wrote: > >>>>>>>> > >>>>>>>>> load 'jmf' > >>>>>>>>> NB. Loads jmf facilities. > >>>>>>>>> > >>>>>>>>> testfile =: {2,\?50#75 > >>>>>>>>> > >>>>>>>>> fn =: jpath > >>>> 'C:\Users\skip\J904-user\temp\foo\testfile_name.jmf' > >>>>>>>>> NB. Links a covername to the file path. > >>>>>>>>> createjmf_jmf_ fn;(([:(*&8)#);testfile) > >>>>>>>>> NB. Create the container for your file. > >>>>>>>>> NB. -fn is your reference name > >>>>>>>>> NB. -(([:(*&8)#);testfile) allocates the size of yor fike in > >>>> bytes > >>>>>>>>> > >>>>>>>>> map_jmf_ 'testfile_disk';fn > >>>>>>>>> NB. Maps the file to a noun 'testfile_disk' > >>>>>>>>> ] testfile_disk_jmf_ =: testfile > >>>>>>>>> > >>>>>>>>> unmap_jmf 'testfile_disk_jmf_' > >>>>>>>>> NB. Release mappings > >>>>>>>>> Or > >>>>>>>>> Exit 0 > >>>>>>>>> Close session. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> New session. > >>>>>>>>> > >>>>>>>>> load 'jmf' > >>>>>>>>> fn =: jpath > >>>>>> 'C:\Users\skip\J904-user\temp\foo\testfile_name.jmf' > >>>>>>>>> map_jmf_ 'testfile1';fn > >>>>>>>>> NB. Map file > >>>>>>>>> ] testfile1_jmf_ > >>>>>>>>> NB. File loaded as noun 'testfile1_jmf_' > >>>>>>>>> > >>>>>>>>> Some potential benefits of the .jmf datatype: > >>>>>>>>> -Preserves header and shape information. > >>>>>>>>> -If you want to map you file as read only, use the following > >>>> syntax > >>>>>>>>> map_jmf_ 'testfile1';fn;'';1 > >>>>>>>>> -If you would like it to be copy-on-write use the following > >>>> syntax > >>>>>>>>> map_jmf_ 'testfile1';fn;'';2 > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Ak > >>>>>>>>> > >>>>>>>>> On Wed., Feb. 1, 2023, 21:48 'Skip Cave' via Programming, < > >>>>>>>>> programm...@jsoftware.com> wrote: > >>>>>>>>> > >>>>>>>>>> I have a boxed noun: > >>>>>>>>>> > >>>>>>>>>> ] testfile =: {2,\?15#50 > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>> > >>> > >> ┌────┬─────┬─────┬─────┬─────┬─────┬────┬────┬────┬────┬─────┬────┬───┬────┐ > >>>>>>>>>> > >>>>>>>>>> │9 21│21 47│47 37│37 13│13 33│33 20│20 4│4 49│49 6│6 25│25 > >>>> 33│33 > >>>>>> 9│9 > >>>>>>>> 6│6 > >>>>>>>>>> 43│ > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>> > >>> > >> └────┴─────┴─────┴─────┴─────┴─────┴────┴────┴────┴────┴─────┴────┴───┴────┘ > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> I want to store it in the following location on my machine: > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> ]fn =. < 'C:\Users\skip\J904-user\temp' > >>>>>>>>>> > >>>>>>>>>> ┌────────────────────────────┐ > >>>>>>>>>> > >>>>>>>>>> │C:\Users\skip\J904-user\temp│ > >>>>>>>>>> > >>>>>>>>>> └────────────────────────────┘ > >>>>>>>>>> > >>>>>>>>>> Then I will close that J session. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Later, I will start a new J session, and I want to read > >> that > >>>> saved > >>>>>>> file > >>>>>>>>>> into a noun called 'test1' > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> What is the J code for writing the noun into a file in the > >>>> first > >>>>>>>> session? > >>>>>>>>>> > >>>>>>>>>> What is the J code to read the file into a noun in the > >> second > >>>>>>> session? > >>>>>>>>>> > >>>>>>>>>> Should I use 1!2 & 1!:1, or fwrite & fread, or something > >>> else? > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Where in the J doc are these file operations and their > >>>> tradeoffs > >>>>>>>>> described? > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Skip > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Skip Cave > >>>>>>>>>> Cave Consulting LLC > >>>>>>>>>> > >>>>>>> > >>>> ---------------------------------------------------------------------- > >>>>>>>>>> For information about J forums see > >>>>>>> http://www.jsoftware.com/forums.htm > >>>>>>>>>> > >>>>>>>>> > >>>>>> > >>> ---------------------------------------------------------------------- > >>>>>>>>> For information about J forums see > >>>>>> http://www.jsoftware.com/forums.htm > >>>>>>>>> > >>>>>>>> > >>>> ---------------------------------------------------------------------- > >>>>>>>> For information about J forums see > >>>> http://www.jsoftware.com/forums.htm > >>>>>>>> > >>>>>>> > >>>> ---------------------------------------------------------------------- > >>>>>>> For information about J forums see > >>>> http://www.jsoftware.com/forums.htm > >>>>>>> > >>>>>> > >>> ---------------------------------------------------------------------- > >>>>>> For information about J forums see > >>> http://www.jsoftware.com/forums.htm > >>>>>> > >>>>> > >> ---------------------------------------------------------------------- > >>>>> For information about J forums see > >> http://www.jsoftware.com/forums.htm > >>>> ---------------------------------------------------------------------- > >>>> For information about J forums see http://www.jsoftware.com/forums.htm > >>>> > >>> ---------------------------------------------------------------------- > >>> For information about J forums see http://www.jsoftware.com/forums.htm > >>> > >> ---------------------------------------------------------------------- > >> For information about J forums see http://www.jsoftware.com/forums.htm > >> > > ---------------------------------------------------------------------- > > For information about J forums see http://www.jsoftware.com/forums.htm > > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
