Re: [Pdl-general] Changes I noted PDL2.025 -> PDL2.084 - rcols issue

[Karl Glazebrook via pdl-general]

Thanks Ed.

Yes I can confirm the code snippet bugs as you describe on PDL-2.084

Good detective work and impressive use of git! Hope it is just a bug. I can 
also confirm it works on 2.077 and bugs on 2.078

Karl


> On 14 Jan 2024, at 1:16 am, Ed .  wrote:
> 
> Hi Karl,
>  
> Thank you for both reporting this issue, then doing this deep investigation 
> as well.
>  
> In order to track down exactly when this changed, I stripped down your repro 
> code to:
> use PDL::LiteF; # not full PDL so can just build “make core”
> $x = sequence(4)+1;
> $x = $x->mv(-1,0)->slice("0:2");
> $x *= 100;
> $y=&PDL::_clump_int($x,-1);
> print "x=$x y=$y\n";
> $wrong = "$y" eq "[0 0 0]";
> exit $wrong;
>  
> As you can see in that comment, if one comments out the multiplication, it 
> all still works correctly. The “sequence” can be replaced with hardcoded “pdl 
> [1..5]”. But if one changes either the “mv”, or the “slice”, or the inplace 
> “*=” (including replacing with “$x = $x * 100” ), or the “_clump_int”, it 
> doesn’t misbehave.
>  
> Anyway, using the above code to tell me whether I’d found where it started 
> failing, I did “git bisect” as follows (noted here to help anyone who wants 
> to do this themselves; I’d forgotten and had to look up how):
> git bisect start
> git bisect bad # current “master” is bad
> git bisect good 2.025 # tell it it was working as of 2.025
> perl Makefile.PL && time make core && perl -Mblib repro-script; echo $? # 
> kept running this, then:
> git bisect bad # if failing
> git bisect good # if working correctly
> git bisect reset # when finished, to close down the bisect
> Note the use of “make core” which takes about 2 mins from scratch on my 
> system, vs about 6 to “make” everything, saving lots of time.
>  
> It turns out it was this, which was released with 2.078:
> commit a4678091acf7e450c02a7b0feaf3c7578f37e53f
> Author: Ed J  >
> Date:   Sun Apr 3 22:27:28 2022 +0100
>  
> parents of non-flowing trans also track trans_children so can de-register 
> on destroy
>  
> Basic/Core/pdlapi.c | 46 +++---
> 1 file changed, 19 insertions(+), 27 deletions(-)
>  
> I am now investigating a fix, which given how specific it is to trigger, will 
> probably be small, and is surely related to book-keeping of parents vs 
> children, and flowing transformations.
>  
> Best regards,
> Ed
>  
> From: Karl Glazebrook via pdl-general 
> 
> Sent: 07 January 2024 06:30
> To: perldl 
> Subject: Re: [Pdl-general] Changes I noted PDL2.025 -> PDL2.084 - rcols issue
>  
> Seems I can’t help myself! I have now found the offending line in rcols() and 
> can now reproduce this without rcols. That should at least make it easier to 
> track down.
>  
> use PDL;
> $x = sequence(100)+1; # This works
> $x = $x->mv(-1,0)->slice("0:3");
> print $x, "\n";
> $x *= 100;
> print $x, "\n";
> $y=&PDL::_clump_int($x,-1);
> print $y, "\n"; # prints [0 0 0 0]
> 1;
>  
> It seems to be the particular combination of ->mv and ->slice (that was on 
> line 693 of misc.pd). This is a 1D ndarray so the ->mv(-1,0) should do 
> nothing. Removing it however makes the problem go away.
>  
> Works fine in PDL-2.025. Some bug that has been introduced in ->mv ?
>  
> Sorry to the stream of consciousness series of emails. I will stop looking 
> now...
>  
> Karl
>  
>  
>  
> 
> 
> On 7 Jan 2024, at 4:32 pm, Karl Glazebrook  > wrote:
>  
> OK here is some deeper diving in to the problem
>  
>  
> use PDL;
> $x = rcols 'tmp.dat'; # This does causes the error
> #$x = sequence(4)+1; # This works
> print $x, "\n";
> $x *= 100;
> print $x, "\n";
> $y=&PDL::_clump_int($x,-1);
> print $y, "\n"; # prints [0 0 0 0]
> ;
>  
>  
>  
>  looks like the error is happening in the internal pp routine _clump_int in 
> slices.pd.
>  
>  
> I also found ->sever() had the same behaviour with rcols:
>  
> pdl> $x = rcols 'tmp.dat' 
>   
> Reading data into ndarrays of type: [ Double ]
> Read in  4  elements.
>  
> pdl> $x *= 100
>   
>  
> pdl> p $x 
>   
> [100 200 300 400]
> pdl> p $x->sever  
> 
> [0 0 0 0]
> 
> 
>  
> Note this is not a general problem with dataflow, if I make a sequence and 
> slice or index is then the ops work fine. It is just something weird on the 
> ndarray produced by rcols.
>  
> Karl
>  
>  
>  
>  
> 
> 
> On 7 Jan 2024, at 3:41 pm, Karl Glazebrook via pdl-gener

Re: [Pdl-general] Changes I noted PDL2.025 -> PDL2.084 - scalars vs piddles

[Karl Glazebrook via pdl-general]

Hi Ed

I believe the original logic here (likely mine) was that if the 0D PDLs turned 
automatically in to scalars they could be fed directly, without casting, in to 
perl-to-C calls and PDL did a lot of that. e.g. I used to do a lot of direct 
calling of the PGPLOT module commands.

I guess if that sort of thing has now been extirpated from the PDL source 
(which seems likely given all tests pass) itself all is good. Probably not many 
end user scripts need fixing. I think you are right it is better these days if 
objects do not magically change type.

Happy for the name change :) It was not my joke...

Karl

> On 14 Jan 2024, at 12:46 am, Ed .  wrote:
> 
> Hi Karl,
>  
> Good to hear from you! And I hope that apart from the issues you’ve raised, 
> that your experience with the new PDL is good. Have a look at the updated 
> demo system, including the updated 3D demos, and see if you can tell the 
> pthreading is done automatically on large-enough data by detecting the number 
> of cores available by default.
>  
> The change in the aggregate functions was introduced in 2.056. I would now 
> say I didn’t take sufficient account of backwards compatibility on that one. 
> Sorry for the inconvenience. My reasoning was in how shocked I was that those 
> functions were returning Perl scalars.
>  
> By the way, you (of all people!) are welcome to call these data objects what 
> you like, but another change made (this time in 2.040) was to rename 
> “piddles” (which always struck me as faintly juvenile, and I felt would 
> undermine PDL’s credibility for no good reason) to “ndarrays”, which is a 
> widely-used term. The “piddle” function was retained for back-compatibility.
>  
> Best regards,
> Ed
>  
> From: Karl Glazebrook via pdl-general 
> 
> Sent: 07 January 2024 00:29
> To: perldl 
> Subject: [Pdl-general] Changes I noted PDL2.025 -> PDL2.084 - scalars vs 
> piddles
>  
> Hi all,
> 
> This dinosaur just upgraded from PDL v2.025 to v.2.084 (yes, I know that is 
> lame)
> 
> I noticed a few things when running one of my complicated codes, I will start 
> seperate email threads
> 
> Next I think this one is a design choice change I missed.
> 
> - Functions like median() max() etc now return a 0D piddle and no longer a 
> scalar. This broke some gnarly code I had.
> 
> I expect this change was sensible, I am just curious as the reasons and when 
> it happened?
> 
> 
> best
> 
> Karl
> 
> 
> 
> 
> 
> ___
> pdl-general mailing list
> pdl-general@lists.sourceforge.net 
> https://lists.sourceforge.net/lists/listinfo/pdl-general 
> 
___
pdl-general mailing list
pdl-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pdl-general

Re: [Pdl-general] Changes I noted PDL2.025 -> PDL2.084 - PDL::NiceSlice issue

[Karl Glazebrook via pdl-general]

Thanks Ed & Luis. I will chalk that one as being down to the limitations of 
NiceSlice and update that particular code not to use it.

Karl


> On 14 Jan 2024, at 4:42 am, Ed .  wrote:
> 
> Hi Luis,
>  
> Thank you for your help on both this and the “clump” issue.
>  
> NiceSlice got changed (as a default) from the simpler but non-string-aware 
> Filter::Util::Call implementation (which would mangle eg SQL strings in the 
> code) to the string-aware but imperfect in other ways Filter::Simple 
> implementation in 2.063_04. You can set the environment variable 
> “PDL_NICESLICE_ENGINE” to Filter::Util::Call to use that, the previous 
> default.
>  
> Filter::Simple uses Text::Balanced, which got some sizable updates from me to 
> improve its parsing of Perl code, but still makes some mistakes, as you can 
> see. It works by finding all strings, comments, here-docs, and regexes, and 
> removes them from the code before filtering, then puts them back after. The 
> mistake here is that it misidentifies the text “$x/2/3” as having a regex in 
> the middle, which gets removed from the “code” as you see.
>  
> Similarly, Karl’s code which has “$y / (multi lines of stuff) / (other 
> stuff)” is being misidentified as the “y /(multi lines)/” being a regex. 
> Therefore it doesn’t go through the filter, therefore it retains its 
> non-correct Perl status, and Perl gives a syntax error.
>  
> A workaround is to avoid $y as a variable name, and possibly $x as well 
> (since “x” is an operator and may throw off the parsing, for now). I will 
> look at updating T:B to make it behave correctly in these cases, but that 
> won’t happen today I’m afraid. NiceSlice continues to cause me agony out of 
> all proportion to how “nice” it actually is. It’s a pity Perl doesn’t have a 
> properly hookable parser yet (at least to my knowledge).
>  
> Best regards,
> Ed
>  
> From: Luis Mochan 
> Sent: 07 January 2024 02:46
> To: Karl Glazebrook 
> Cc: perldl 
> Subject: Re: [Pdl-general] Changes I noted PDL2.025 -> PDL2.084 - 
> PDL::NiceSlice issue
>  
> I found some time back problems with NiceSlice. Actually, with
> Filter::Simple (used by default by NiceSlice, though it can use other
> options, controlled with environment variables). I vaguely recall that
> two slashes, as in two divisions consecutive, confuse it completely,
> as they seem as regular expression.
> 
> I found this in my notes of 2022-11-05:
> ---
> Make a filter:
>   package MyFilter;
>   use v5.36;
>   use Filter::Simple;
>   FILTER_ONLY
>   code_no_comments => sub { say "Code:\n$_" },
>   all => sub { say "All:\n$_" };
>   1;
> And a program:
>   use v5.36;
>   use lib ".";
>   use MyFilter;
>   my $x=1;
>   say $x/2/3;
> 
> Execution yields:
> 
>   Code:
>   my $x=1;
>   say $x3;
> 
>   All:
>   my $x=1;
>   say $x/2/3;
> 
>   0.167
> 
> This means that /2/ is removed incorrectly. The name of the variable
> ($x and not $y) seems important.
> ---
> 
> Regards,
> Luis
> 
> 
> 
> 
> On Sun, Jan 07, 2024 at 12:01:27PM +1100, Karl Glazebrook via pdl-general 
> wrote:
> >
> >
> > > On 7 Jan 2024, at 11:55 am, Karl Glazebrook via pdl-general 
> > >  > > > wrote:
> > >
> > > (The mismatch of line numbers seems to be related to the use of hereto 
> > > text earlier in the code, the debugger shows different line numbers)
> > >
> >
> >
> > PS just to prove this point given the line numbers issue:
> >
> >
> >
> >
> > syntax error at /Users/karl/Software/perl/KGB/Redshift.pm line 251, near 
> > "$FOO("
> > Execution of /Users/karl/Software/perl/KGB/Redshift.pm aborted due to 
> > compilation errors.
> >
> >
> >
> > You can see FOO triggers a compilation error and BAR does not, use 
> > PDL::NiceSlice is way at the top of the module
> >
> > Karl
> >
> 
> 
> > ___
> > pdl-general mailing list
> > pdl-general@lists.sourceforge.net 
> > https://lists.sourceforge.net/lists/listinfo/pdl-general 
> > 
> 
> 
> -- 
> 
>   o
> W. Luis Mochán,  | tel:(52)(777)329-1734 /<(*)
> Instituto de Ciencias Físicas, UNAM  | fax:(52)(777)317-5388 `>/   /\
> Av. Universidad s/n CP 62210 |   (*)/\/  \
> Cuernavaca, Morelos, México  | moc...@fis.unam.mx 
>    /\_/\__/
> GPG: 791EB9EB, C949 3F81 6D9B 1191 9A16  C2DF 5F0A C52B 791E B9EB
> 
> 
> ___
> pdl-general mailing list
> pdl-general@lists.sourceforge.net