If I recall correctly (and I really should go look up the definition of ma() in the source and explore it and related words), the memory allocation routine adds an extra byte of padding to the character type, so that J's internals can use C api mechanisms which require null termination without needing any extra padding.
I believe that this design decision predates the implementation of efficient concatenation loops. Anyways, if I recall correctly, booleans get the same padding -- the memory allocator was just looking at the width of the data elements (because type was handled at a different level of abstraction). FYI, -- Raul On Fri, Dec 14, 2012 at 2:20 PM, Devon McCormick <[email protected]> wrote: > FYI - here's something I wrote on looking at J internal data structures: > http://www.jsoftware.com/jwiki/DevonMcCormick/Data/JInternalRepresentation . > > > On Fri, Dec 14, 2012 at 8:00 AM, Raul Miller <[email protected]> wrote: > >> On Fri, Dec 14, 2012 at 7:24 AM, Konrad Hinsen >> <[email protected]> wrote: >> > Raul Miller writes: >> > >> > > Even then your dominant cost is reading the data off of disk. If the >> > > data is converted on the way in from the disk to RAM, and most of your >> > > CPU time is spent waiting for the disk, your conversion time might not >> > > even be measurable. >> > >> > My situation would be that the data is read to memory, ending up in a >> > NumPy array, and then I have to copy it to a J array in a subsequent >> > step. That's why I am interested in exploring the shared data approach. >> >> Yes... I was thinking that this aspect might be redesigned. >> >> > > Note that if your data structure occupies half of your physical >> > > memory, J intermediate results might be expensive -- my rule of thumb >> > > (just a guess based on raw data size, until I get real measurements) >> > > is to expect 5x memory overhead from J. >> > >> > That's also a problem with NumPy, but there I know how to get around >> > it. With J, that remains to be explored. >> >> Typically, I get my algorithms working with smaller data sets and then >> introduce some kind of mechanism to break the content up in blocks. >> Initially, I start with very small blocks (like 3 elements or >> something) to make sure that I have my edge conditions set up >> properly, and then I bump the blocks up to something useful (like >> something in the 10s of megabytes). >> >> Thanks, >> >> -- >> Raul >> ---------------------------------------------------------------------- >> For information about J forums see http://www.jsoftware.com/forums.htm >> > > > > -- > Devon McCormick, CFA > ^me^ at acm. > org is my > preferred e-mail > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
