Hi Yury, (and all) Thank you again for reporting this! I have now found and fixed the bug (and pushed the fix to GitHub), which I will describe for posterity / morbid curiosity. Your supplied script now stays at the same memory-usage.
Slices::rangeb takes OtherPars which are “SV*”. PP takes copies of those where it can figure how to do so, which includes “SV*”. It also tries to free those copies when done, when it knows how to do so. If an explicit “Comp” is supplied, as with “rangeb”, there was a bug for a while where the free-ing would only happen for that and not the “OtherPars”. There was an additional bug where it wouldn’t actually do the free-ing of the copies it had made. Since interpND uses rangeb and passes in the indices as an SV, that was getting copied and never freed. Best regards, Ed From: Ed .<mailto:[email protected]> Sent: 28 March 2023 05:38 To: Yury Pakhomov<mailto:[email protected]>; [email protected]<mailto:[email protected]> Subject: RE: [Pdl-general] PDL 2.082 released Hi Yury, Thank you for reporting this, and for the script that helps reproduce it, which is incredibly valuable. I have found a great module (https://metacpan.org/pod/Test::LeakTrace) that is helping track down the problems this highlights, and have just pushed a fix for the first one, a memory-leak in “sclr”, detectable using this script: use PDL; use Test::LeakTrace; no_leaks_ok { pdl(3)->sclr }; Similarly, your script can be reduced to this, since a single iteration will still leak: use PDL; use Test::LeakTrace; no_leaks_ok { zeroes(1,1,1)->interpND(pdl(1,1,1)) }; When I’ve squashed all of the problems revealed, there is likely to be a new version of PDL out soon. Best regards, Ed From: Yury Pakhomov<mailto:[email protected]> Sent: 23 March 2023 22:19 To: [email protected]<mailto:[email protected]> Subject: Re: [Pdl-general] PDL 2.082 released Thanks!!! But today I found a memory hole for v2.081 in interpND function. Within a several minutes one-two gigabytes are spend. If I run 20 processes in parallel then the calculations are interrupted. You can check this code: use strict; use PDL; use Memory::Usage; my $mu = Memory::Usage->new(); my $data=zeroes(100,100,100); $mu->record('starting work'); for(my $i=0; $i<100; $i++){ for(my $j=0;$j<100;$j++){ for(my $k=0; $k<100; $k++){ my $ind=pdl($i,$j,$k); my $res=$data->interpND($ind); } } $mu->record('step by i'); my $mem=$mu->state(); printf "%3i: RES=%.3f Mb\n",$i,($mem->[-1]->[3])/1024; } _________________________ 0: RES=33.750 Mb 1: RES=40.375 Mb 2: RES=47.000 Mb 3: RES=53.625 Mb 4: RES=60.250 Mb 5: RES=66.875 Mb 6: RES=73.375 Mb 7: RES=80.000 Mb .... 6.6 Mb was spend for each step by i (10^4 calls of interpND). Best wishes Yuri 23.03.2023 06:31, Ed. пишет: Dear PDL folks, PDL 2.082 has just been released. Notable changes since 2.081: 1. allow OtherPars “nsize => n” to set index-size of not-otherwise-used index 2. fix [o] OtherPars to work if PMCode provided 3. fix nested loop() bug for Pars with e.g. (n,n) 4. add pp_add_typemaps 5. bug fixes: MatrixOps::det now always returns an ndarray, fixed an intersect bug Future plans, in something like intended order: 1. Finish PDL::OpenCV 2. Restructure the TriD stuff so there is a consistent API between OpenGL and X3D/VRML 3. fix more open GitHub issues 4. add event-handling hooks for ndarrays 5. “loop fusion” techniques to maximise locality of computation, minimising data’s trips through the “straw” between CPU and main RAM 6. finish the independent C interface for making PDL usable from e.g. Python 7. use OpenCL or other means to also utilise GPUs if available The IRC channel (#pdl on irc.perl.org) is a great virtual place to come and ask questions, or just watch the GitHub messages flow by. As usual, please give the new release a try and report problems. Best regards, Ed _______________________________________________ pdl-general mailing list [email protected]<mailto:[email protected]> https://lists.sourceforge.net/lists/listinfo/pdl-general
_______________________________________________ pdl-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/pdl-general
