Re: [Plplot-devel] A mysterious pyc file corruption issue

[Alan W. Irwin]

On 2017-04-17 16:30-0700 Alan W. Irwin wrote:

> I feel it is important to get back to Python 2 PostScript difference
> perfection and to also achieve that perfection for Python 3.

I now (commit 9185cce) have achieved perfect PostScript difference
reports for both Python 2 and Python 3.  I had never heard of floor
division before, but that turned out to be the answer for the Python 3
versus Python 2 numerical differences we still had after my previous
commit.  See the commit 9185cce message for details concerning floor
division.

> Note also that there are additional non-standard Python examples in
> the examples/python subdirectory so those should all be checked to
> make sure they work with both Python 2 and Python 3.  And similarly
> for the pyqt4 and pyqt5 standard examples.

This part still needs to be done, but getting the standard examples
working perfectly for both Python 2 and Python 3 is a big step in the
right direction.

Alan
__
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__

Linux-powered Science
__

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Plplot-devel mailing list
Plplot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/plplot-devel

Re: [Plplot-devel] A mysterious pyc file corruption issue

[Alan W. Irwin]

On 2017-04-17 16:30-0700 Alan W. Irwin wrote:

> [T]here are now PostScript consistency issues _both_ for Python 2 and
> Python 3 results that were not there before for Python 2 before your
> push.
>
> python (2)
>   Missing examples:
>   Differing graphical output  :  33
>   Missing stdout  :
>   Differing stdout:  23
>
> python (3)
>   Missing examples:
>   Differing graphical output  :  02 09 11 14a 15 16
>   Missing stdout  :
>   Differing stdout:

I have now (commit 2bbdcb0) fixed the above two Python 2 issues. Those
fixes did not affect the above Python 3 issues which I plan to tackle
(likely quite slowly because of my current unfamiliarity with Python
3) in ascending order.

Alan
__
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__

Linux-powered Science
__

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Plplot-devel mailing list
Plplot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/plplot-devel

Re: [Plplot-devel] A mysterious pyc file corruption issue

[Alan W. Irwin]

On 2017-04-16 16:55-0400 Hazen Babcock wrote:

> On 02/21/2017 09:26 PM, Alan W. Irwin wrote:
>>
>> I have now been in contact with the OP, Barry Warsaw of python.org, of
>> that thread who was quite helpful.  For example, Barry told me that
>> Python is designed so it is frankly impossible for
>>
>> import Plframe
>> from Plframe import *
>>
>> to race (i.e., the first import completely finishes before the second
>> one starts).  And I cannot find any other cases where Plframe is
>> imported.  So I think the best bet for explaining this *.pyc
>> Python-generated file corruption is some unknown Python 2 bug that
>> does not have anything to do with races.  I got the sense from
>> what Barry said that he feels Python 3 is now much more reliable than
>> Python 2.  So this may be another instance of that general idea.
>>
>> Anyhow, I think the next step is to test whether this corruption
>> occurs for Python 3.  (And if it does I get the sense that Barry would
>> be anxious to figure out what that Python 3 bug was.)
>>
>> @Hazen:
>>
>> This issue lends lots of additional motivation for making PLplot work
>> correctly with Python 3.  So please go ahead and push your Python 3
>> topic as soon as it is in reasonable state, and we can mature it
>> further (if necessary) from there.
>
> Pushed.

Hi Hazen:

Thanks for pushing your work.

> One possibly important thing to note is that Python3 does not allow a
> mix of tabs and spaces in a file. So these changes are less extensive
> then they might appear as a lot of it was converting the examples to be
> all spaces. And hopefully our file formatting utility will not introduce
> regressions.

I agree we should use only blanks for Python identation (just as in
our styled C code) since using a mixture of tabs and blanks for this
purpose is just an accident waiting to happen.

I have just checked, and scripts/style_source.sh, and
scripts/remove_trailing_whitespace.sh do not affect Python indentation
so your changed files should retain blank indentation from now on
unless someone deliberately introduces tab indentation by mistake when
editing these files.  In addition, I noticed there was still some tab
indentation left in our Python source files.  I found those using

find . -name '*.py*' |grep -vE '(pyc|~)$' |xargs grep -l $'\t'

and fixed them in commit dd9d258.

Both your commit and that commit build and run here for both Python 2
and Python 3 confirming your test results on Lubuntu.  However, see
below for PostScript difference issues that your changes
introduced.

For the record, the following Debian Jessie packages were installed
for my tests:

python2.7
libpython2.7-dev:amd64
python-numpy

python3.4
libpython3.4-dev:amd64
python3-numpy

Debian Jessie does not configure its etc-alternatives system for Python
so by default CMake finds /usr/bin/python which is an (indirect) symbolic link 
to
/usr/bin/python2.7 and python library and numpy that are consistent
with that version.  To try the Python 3 alternative, I used the
CMake option DPython_ADDITIONAL_VERSIONS:STRING=3 which found
/usr/bin/python3.4 and python library and numpy that was consistent with
that version.

So all appears well with the build system and the python build.
Furthermore, the standard python examples execute without any obvious
run-time error such as segfaults for both Python 2 and 3.  However,
there are now PostScript consistency issues _both_ for Python 2 and
Python 3 results that were not there before for Python 2 before your
push.

python (2)
   Missing examples:
   Differing graphical output  :  33
   Missing stdout  :
   Differing stdout:  23

python (3)
   Missing examples:
   Differing graphical output  :  02 09 11 14a 15 16
   Missing stdout  :
   Differing stdout:

I obtained these results both before and after my changes so I assume
the python 2 issues were introduced by your commit, and the python 3
issues are due to some remaining python 2 to python 3 conversion
issues.

I feel it is important to get back to Python 2 PostScript difference
perfection and to also achieve that perfection for Python 3 so I am
willing to work on the above issues starting with the Python 2 case
and then following up with the Python 3 case in ascending order of the
examples.  Assuming you want to help with this effort without
duplicating my work, please start with the Python 3 issues in
descending order by example until we meet in the middle.  :-)

Note also that there are additional non-standard Python examples in
the examples/python subdirectory so those should all be checked to
make sure they work with both Python 2 and Python 3.  And similarly
for the pyqt4 and pyqt5 standard examples.  So there is obviously a
lot more checking/debugging that needs to be done, but you have
certainly made a good Python 3 start with this push.

Alan
__
Alan W. Irwin

Astronomical research affiliation with Department of 

Re: [Plplot-devel] A mysterious pyc file corruption issue

[Hazen Babcock]

On 02/21/2017 09:26 PM, Alan W. Irwin wrote:
>
> I have now been in contact with the OP, Barry Warsaw of python.org, of
> that thread who was quite helpful.  For example, Barry told me that
> Python is designed so it is frankly impossible for
>
> import Plframe
> from Plframe import *
>
> to race (i.e., the first import completely finishes before the second
> one starts).  And I cannot find any other cases where Plframe is
> imported.  So I think the best bet for explaining this *.pyc
> Python-generated file corruption is some unknown Python 2 bug that
> does not have anything to do with races.  I got the sense from
> what Barry said that he feels Python 3 is now much more reliable than
> Python 2.  So this may be another instance of that general idea.
>
> Anyhow, I think the next step is to test whether this corruption
> occurs for Python 3.  (And if it does I get the sense that Barry would
> be anxious to figure out what that Python 3 bug was.)
>
> @Hazen:
>
> This issue lends lots of additional motivation for making PLplot work
> correctly with Python 3.  So please go ahead and push your Python 3
> topic as soon as it is in reasonable state, and we can mature it
> further (if necessary) from there.

Pushed.

One possibly important thing to note is that Python3 does not allow a 
mix of tabs and spaces in a file. So these changes are less extensive 
then they might appear as a lot of it was converting the examples to be 
all spaces. And hopefully our file formatting utility will not introduce 
regressions.

-Hazen


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Plplot-devel mailing list
Plplot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/plplot-devel

Re: [Plplot-devel] A mysterious pyc file corruption issue

[Alan W. Irwin]

On 2017-02-21 11:01-0800 Alan W. Irwin wrote:

> Yesterday for the second time in two months an interactive
> comprehensive test failed with a "ValueError: bad marshal data
> (unknown type code)" error for bindings/python/Plframe.py.
[ These errors were so common]
> that the python developers list in 2013 became
> concerned that python would be subject to race conditions when
> generating these files and thus was the author of at least some of
> these corruptions (see discussion thread at
> 
> with the subject line "[Python-Dev] Mysterious Python pyc file
> corruption problems".

I have now been in contact with the OP, Barry Warsaw of python.org, of
that thread who was quite helpful.  For example, Barry told me that
Python is designed so it is frankly impossible for

import Plframe
from Plframe import *

to race (i.e., the first import completely finishes before the second
one starts).  And I cannot find any other cases where Plframe is
imported.  So I think the best bet for explaining this *.pyc
Python-generated file corruption is some unknown Python 2 bug that
does not have anything to do with races.  I got the sense from
what Barry said that he feels Python 3 is now much more reliable than
Python 2.  So this may be another instance of that general idea.

Anyhow, I think the next step is to test whether this corruption
occurs for Python 3.  (And if it does I get the sense that Barry would
be anxious to figure out what that Python 3 bug was.)

@Hazen:

This issue lends lots of additional motivation for making PLplot work
correctly with Python 3.  So please go ahead and push your Python 3
topic as soon as it is in reasonable state, and we can mature it
further (if necessary) from there.

Alan
__
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__

Linux-powered Science
__

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
___
Plplot-devel mailing list
Plplot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/plplot-devel