Yeah, that's an ouch. Big boxed arrays are a sore spot.
Marshall Lochbaum has a good solution, which involves flagging blocks to
avoid traversing the whole tree. This is the wholesale approach. It
hasn't been debugged yet.
I have taken the retail approach, getting rid of needless ra() calls as
I find them. Maybe there will be some way to solve this particular
problem too.
Henry
On 1/12/2017 9:29 PM, Joe Bogner wrote:
Henry, thanks for the explanation, it was really helpful.
It looks like the issue in this routine is the the traverse of x, which is
a box of length n -- where n = ^:(<n)
In the usage of:
f1 =: 3 : '(>: each)^:(<y) (<0)'
With f1 100000, it will increase the reference count of all 100,000
preallocated boxes each 10th iteration (10,000 times). Ouch!.
static DF1(jtply1){PROLOG(0040);DECLFG;A b,hs,j,x,*xv,y,z;B*bv,q;I
i,k,m,n,*nv,old,p=0;
...
GATV(x,BOX,m,1,0); xv=AAV(x); //allocate for a box equal to n ^:(<n)
...
for(i=1;i<=n;++i){
...
if(!(i%10))gc3(x,z,0L,old);
}
Since x is fixed for the loop, does it need to even need to touch the ref
counts of x?
I tried doing this if(!(i%10))gc3(0L,z,0L,old); //skip x in gc3, but that
created problems with the old
Possibly moving ra(x) out of the loop but not sure exactly how all this
works
You might be wondering why I'm so interested in ^:(<n) ... I see it as a
"guarded" version of converge to some degree. Converge is really difficult
to work with because if you mess it up then it will cause J to run
indefinitely (maybe there is a workaround). It would be really nice to
incorporate a break out of (<n) too... for an early exit. I recall some
discussion along those lines for ^: in general -- maybe to check some flag
and do an early exit from the loop
On Thu, Jan 12, 2017 at 7:19 PM, Henry Rich <henryhr...@gmail.com> wrote:
No, ra() raises the usecount on a block and all the blocks it contains.
Garbage collection (coalescing free blocks) takes way less than 1% of time
in my traces, while allocate/free take more, so I don't think it's a winner
to add anything to alloc/free.
I was very surprised at how well garbage collection performs. Blocks above
about 16K are handled by the OS; smaller blocks are handled by asking the
OS for 16K blocks which are then split into smaller blocks as needed. A
16K block is returned to the OS only when all the blocks it was split into
are all free. I thought this would fragment the 16K blocks, but somehow
the fragmentation seems to be transitory.
As part of polishing the code for memory allocation, I did rewrite the
garbage collector to save the time of sorting the block addresses, by using
a different way of matching the subblocks.
Henry Rich
On 1/12/2017 7:07 PM, Raul Miller wrote:
ra() is the thing that merges contiguous free blocks?
Anyways, would it be worthwhile, when freeing a block, to check if
there is an adjacent free block to merge with (before or after)?
This would imply keeping freed blocks in sorted lists, which (because
sorting can be expensive) would in turn imply keeping multiple sorted
lists (each order of magnitude larger being sorted with an order of
magnitude longer interval and merging in the shorter lists at that
time), and having the check walk through each of those lists. (Perhaps
a pre-allocated space for holding the list of those sorted lists.)
Er... unless there's some other constraint that makes this proposal be
nonsense?
Thanks,
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
