I think issues might be (a) we don't know how much memory is being allocated in each iteration, and (b) memory allocation time might be an awkward spot for a garbage collect.
That said, logically speaking, we want to reclaim memory when we have a lot of unused space. And that tends to mean after a lot has been allocated. So, anyways, it should be the case that we're tracking how much memory has been added and we attempt to reclaim after we've added a lot. (There's a potential exception here when we're right at the threshold, but that issue will tend to cause performance to take a dive, and it can become really difficult to interrupt J at that point - I'm not sure there's any good approach for that situation. Probably worth neglecting initially but eventually we'll have to deal with resource thresholds. On a related note, the current jbrk implementation tends to lead first to states where you have to reboot the machine to regain control and then after that to being reluctant to experiment with J. Personally, I'd prefer if jqt had jbrk functionality built it (with the dreaded 'are you sure'). But this tangent is getting way too far off topic now...) Anyways, there's no perfect solutions but maybe we can do better. Thanks, -- Raul On Thu, Jan 12, 2017 at 3:41 PM, Joe Bogner <joebog...@gmail.com> wrote: > It looks like there was some discussion along these lines previously: > > http://www.jsoftware.com/pipermail/source/2016-May/000785.html > > I also don't understand why gc3 is needed. It's also interesting that it > changed some time between j v7 and the gpl source in j7 > > https://github.com/zeotrope/j7-src/blob/0fba52cac4035de6442c2716b614d7b232e91693/cp.c#L31 > > > while(k<m&&i==nv[jv[k]]){xv[jv[k]]=z; ++k;} > gc3(x,z,0L,old); > > > https://github.com/jsoftware/jsource/blob/master/jsrc/cp.c#L97 > > while(k<m&&i==nv[k]){xv[k]=z; ++k; q=k<m?bv[k]:0;} > if(!(i%10))gc3(x,z,0L,old); > > > It looks like the (i%10) was added to make the gc3 run less frequently - > every 10th iteration. If there was a loop of 11, then it wouldn't execute > on the 11th iteration, which makes me wonder why it's even there in the > first place -- maybe to free up temporary memory? That seems like an > expensive operation to run every 10 iterations. I could see it making sense > on memory constrained systems but seems less relevant now and actually is > very costly from a CPU perspective > > > > > > On Tue, Jan 10, 2017 at 9:51 PM, Joe Bogner <joebog...@gmail.com> wrote: > >> I was investigating the slowness of this and identified a call to gc3 as >> being the culprit >> >> >> With the gc3: >> >> f1 =: 3 : '(>: each)^:(<y) (<0)' >> >> (6!:2) 'f1 100000' >> 33.0624 >> >> Without it >> >> (6!:2) 'f1 100000' >> 0.272726 >> >> >> It still works: >> f1 5 >> +-+-+-+-+-+ >> |0|1|2|3|4| >> +-+-+-+-+-+ >> >> >> Why is it doing this operation (garbage collection?) every 10 loops >> >> https://github.com/jsoftware/jsource/blob/e68277f408433fe34fd31f0161a31c >> 96d9ccb44e/jsrc/cp.c#L97 >> >> >> Can anyone elaborate on this? Is the 10 count too frequent? >> >> >> Repeatedly running f1 100000 does not result in a memory leak as far as >> I can tell. >> cp.c >> >> static DF1(jtply1){PROLOG(0040);DECLFG;A b,hs,j,x,*xv,y,z;B*bv,q;I >> i,k,m,n,*nv,old,p=0; >> hs=sv->h; m=AN(hs); >> RZ(x=ravel(hs)); RZ(y=from(j=grade1(x),x)); nv=AV(y); >> GATV(x,BOX,m,1,0); xv=AAV(x); >> while(p<m&&0>nv[p])p++; >> if(p<m){ >> RZ(z=ca(w)); >> n=nv[m-1]; k=p; >> while(k<m&&!nv[k]){xv[k]=z; ++k;} >> RZ(b=eq(ainf,from(j,ravel(gs)))); bv=BAV(b); q=k<m?bv[k]:0; >> old=jt->tnextpushx; >> for(i=1;i<=n;++i){ >> RZ(z=CALL1(f1,y=z,fs)); >> if(q&&equ(y,z)){DO(m-k, xv[k]=z; ++k;); break;} >> while(k<m&&i==nv[k]){xv[k]=z; ++k; q=k<m?bv[k]:0;} >> if(!(i%10))gc3(x,z,0L,old); >> }} >> if(0<p){ >> RZ(fs=inv(fs)); f1=VAV(fs)->f1; >> RZ(z=ca(w)); >> n=nv[0]; k=p-1; >> RZ(b=eq(scf(-inf),from(j,ravel(gs)))); bv=BAV(b); q=bv[k]; >> old=jt->tnextpushx; >> for(i=-1;i>=n;--i){ >> RZ(z=CALL1(f1,y=z,fs)); >> if(q&&equ(y,z)){DO(1+k, xv[k]=z; --k;); break;} >> while(0<=k&&i==nv[k]){xv[k]=z; --k; q=0<=k?bv[k]:0;} >> # ----------------> if(!(i%10))gc3(x,z,0L,old); >> }} >> z=ope(reshape(shape(hs),from(grade1(j),x))); EPILOG(z); >> } >> >> >> >> void jtgc3(J jt,A x,A y,A z,I old){ >> if(x)ra(x); if(y)ra(y); if(z)ra(z); >> tpop(old); >> if(x)tpush(x); if(y)tpush(y); if(z)tpush(z); >> } >> >> >> >> ---------- Forwarded message ---------- >> From: Joe Bogner <joebog...@gmail.com> >> Date: Tue, Jan 10, 2017 at 6:26 PM >> Subject: Re: ^: slowness with boxes >> To: programm...@jsoftware.com >> >> >> For a little more color: previous discussion: http://www.jsoftwa >> re.com/pipermail/programming/2014-March/035994.html >> >> I'm going to switch back to using i. for this >> >> >> >> On Tue, Jan 10, 2017 at 6:15 PM, Joe Bogner <joebog...@gmail.com> wrote: >> >>> Looks like it is due to the boxing. Please disregard >>> >>> http://code.jsoftware.com/wiki/Vocabulary/comma#dyadic mentions that >>> append in place special code does not work for boxes >>> >>> f4 =: 3 : 0 >>> >>> a=:a: >>> >>> for_n. (i. y) do. a=:a,(<n) end. >>> >>> ) >>> >>> >>> (6!:2) 'f4 100' >>> >>> 0.000336555 >>> >>> (6!:2) 'f4 1000' >>> >>> 0.0149772 >>> >>> (6!:2) 'f4 10000' >>> >>> 0.667025 >>> >>> (6!:2) 'f4 20000' >>> >>> 2.69115 >>> >>> (6!:2) 'f4 100000' >>> >>> 170.261 >>> >>> >>> >>> >>> On Tue, Jan 10, 2017 at 5:26 PM, Joe Bogner <joebog...@gmail.com> wrote: >>> >>>> Why are these so different in execution speed? I'm surprised assembling >>>> that linking boxes makes that much of a difference over an number. I need >>>> it to be boxed because I'm going to something more complex with each >>>> iteration. >>>> >>>> f1 =: 3 : '(>: each)^:(<y) (<0)' >>>> >>>> (6!:2) 'f1 100' >>>> 6.01622e_5 >>>> (6!:2) 'f1 1000' >>>> 0.0012719 >>>> (6!:2) 'f1 10000' >>>> 0.145702 >>>> (6!:2) 'f1 100000' >>>> 33.0624 >>>> >>>> f2 =: 3 : '(>:)^:(<y) (0)' >>>> >>>> (6!:2) 'f2 100' >>>> 4.24675e_5 >>>> (6!:2) 'f2 1000' >>>> 0.000189688 >>>> (6!:2) 'f2 10000' >>>> 0.00180487 >>>> (6!:2) 'f2 100000' >>>> 0.0256836 >>>> >>>> >>>> I'm thinking of using ^: to execute something N times with an increasing >>>> argument. >>>> >>>> I could use i. but wanted to try something different >>>> >>>> >>>> JVERSION >>>> >>>> Engine: j805/j64/windows >>>> >>>> Release: commercial/2016-12-11T08:02:16 >>>> >>>> Library: 8.05.11 >>>> >>>> Qt IDE: 1.5.3s/5.6.2 >>>> >>>> Platform: Win 64 >>>> >>>> Installer: J805 install >>>> >>>> InstallPath: c:/program files/j64-805 >>>> >>>> Contact: www.jsoftware.com >>>> >>> >>> >> >> > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
