Re: [Plplot-devel] A mysterious pyc file corruption issue

2017-04-18 Thread Alan W. Irwin 
On 2017-04-17 16:30-0700 Alan W. Irwin wrote:

> I feel it is important to get back to Python 2 PostScript difference
> perfection and to also achieve that perfection for Python 3.

I now (commit 9185cce) have achieved perfect PostScript difference
reports for both Python 2 and Python 3.  I had never heard of floor
division before, but that turned out to be the answer for the Python 3
versus Python 2 numerical differences we still had after my previous
commit.  See the commit 9185cce message for details concerning floor
division.

> Note also that there are additional non-standard Python examples in
> the examples/python subdirectory so those should all be checked to
> make sure they work with both Python 2 and Python 3.  And similarly
> for the pyqt4 and pyqt5 standard examples.

This part still needs to be done, but getting the standard examples
working perfectly for both Python 2 and Python 3 is a big step in the
right direction.

Alan
__
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__

Linux-powered Science
__

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Plplot-devel mailing list
Plplot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/plplot-devel

Re: [Plplot-devel] A mysterious pyc file corruption issue

2017-04-17 Thread Alan W. Irwin 
On 2017-04-17 16:30-0700 Alan W. Irwin wrote:

> [T]here are now PostScript consistency issues _both_ for Python 2 and
> Python 3 results that were not there before for Python 2 before your
> push.
>
> python (2)
>   Missing examples:
>   Differing graphical output  :  33
>   Missing stdout  :
>   Differing stdout:  23
>
> python (3)
>   Missing examples:
>   Differing graphical output  :  02 09 11 14a 15 16
>   Missing stdout  :
>   Differing stdout:

I have now (commit 2bbdcb0) fixed the above two Python 2 issues. Those
fixes did not affect the above Python 3 issues which I plan to tackle
(likely quite slowly because of my current unfamiliarity with Python
3) in ascending order.

Alan
__
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__

Linux-powered Science
__

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Plplot-devel mailing list
Plplot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/plplot-devel

Re: [Plplot-devel] A mysterious pyc file corruption issue

2017-04-17 Thread Alan W. Irwin 
On 2017-04-16 16:55-0400 Hazen Babcock wrote:

> On 02/21/2017 09:26 PM, Alan W. Irwin wrote:
>>
>> I have now been in contact with the OP, Barry Warsaw of python.org, of
>> that thread who was quite helpful.  For example, Barry told me that
>> Python is designed so it is frankly impossible for
>>
>> import Plframe
>> from Plframe import *
>>
>> to race (i.e., the first import completely finishes before the second
>> one starts).  And I cannot find any other cases where Plframe is
>> imported.  So I think the best bet for explaining this *.pyc
>> Python-generated file corruption is some unknown Python 2 bug that
>> does not have anything to do with races.  I got the sense from
>> what Barry said that he feels Python 3 is now much more reliable than
>> Python 2.  So this may be another instance of that general idea.
>>
>> Anyhow, I think the next step is to test whether this corruption
>> occurs for Python 3.  (And if it does I get the sense that Barry would
>> be anxious to figure out what that Python 3 bug was.)
>>
>> @Hazen:
>>
>> This issue lends lots of additional motivation for making PLplot work
>> correctly with Python 3.  So please go ahead and push your Python 3
>> topic as soon as it is in reasonable state, and we can mature it
>> further (if necessary) from there.
>
> Pushed.

Hi Hazen:

Thanks for pushing your work.

> One possibly important thing to note is that Python3 does not allow a
> mix of tabs and spaces in a file. So these changes are less extensive
> then they might appear as a lot of it was converting the examples to be
> all spaces. And hopefully our file formatting utility will not introduce
> regressions.

I agree we should use only blanks for Python identation (just as in
our styled C code) since using a mixture of tabs and blanks for this
purpose is just an accident waiting to happen.

I have just checked, and scripts/style_source.sh, and
scripts/remove_trailing_whitespace.sh do not affect Python indentation
so your changed files should retain blank indentation from now on
unless someone deliberately introduces tab indentation by mistake when
editing these files.  In addition, I noticed there was still some tab
indentation left in our Python source files.  I found those using

find . -name '*.py*' |grep -vE '(pyc|~)$' |xargs grep -l $'\t'

and fixed them in commit dd9d258.

Both your commit and that commit build and run here for both Python 2
and Python 3 confirming your test results on Lubuntu.  However, see
below for PostScript difference issues that your changes
introduced.

For the record, the following Debian Jessie packages were installed
for my tests:

python2.7
libpython2.7-dev:amd64
python-numpy

python3.4
libpython3.4-dev:amd64
python3-numpy

Debian Jessie does not configure its etc-alternatives system for Python
so by default CMake finds /usr/bin/python which is an (indirect) symbolic link 
to
/usr/bin/python2.7 and python library and numpy that are consistent
with that version.  To try the Python 3 alternative, I used the
CMake option DPython_ADDITIONAL_VERSIONS:STRING=3 which found
/usr/bin/python3.4 and python library and numpy that was consistent with
that version.

So all appears well with the build system and the python build.
Furthermore, the standard python examples execute without any obvious
run-time error such as segfaults for both Python 2 and 3.  However,
there are now PostScript consistency issues _both_ for Python 2 and
Python 3 results that were not there before for Python 2 before your
push.

python (2)
   Missing examples:
   Differing graphical output  :  33
   Missing stdout  :
   Differing stdout:  23

python (3)
   Missing examples:
   Differing graphical output  :  02 09 11 14a 15 16
   Missing stdout  :
   Differing stdout:

I obtained these results both before and after my changes so I assume
the python 2 issues were introduced by your commit, and the python 3
issues are due to some remaining python 2 to python 3 conversion
issues.

I feel it is important to get back to Python 2 PostScript difference
perfection and to also achieve that perfection for Python 3 so I am
willing to work on the above issues starting with the Python 2 case
and then following up with the Python 3 case in ascending order of the
examples.  Assuming you want to help with this effort without
duplicating my work, please start with the Python 3 issues in
descending order by example until we meet in the middle.  :-)

Note also that there are additional non-standard Python examples in
the examples/python subdirectory so those should all be checked to
make sure they work with both Python 2 and Python 3.  And similarly
for the pyqt4 and pyqt5 standard examples.  So there is obviously a
lot more checking/debugging that needs to be done, but you have
certainly made a good Python 3 start with this push.

Alan
__
Alan W. Irwin

Astronomical research affiliation with Department of

Re: [Plplot-devel] A mysterious pyc file corruption issue

2017-04-16 Thread Hazen Babcock 
On 02/21/2017 09:26 PM, Alan W. Irwin wrote:
>
> I have now been in contact with the OP, Barry Warsaw of python.org, of
> that thread who was quite helpful.  For example, Barry told me that
> Python is designed so it is frankly impossible for
>
> import Plframe
> from Plframe import *
>
> to race (i.e., the first import completely finishes before the second
> one starts).  And I cannot find any other cases where Plframe is
> imported.  So I think the best bet for explaining this *.pyc
> Python-generated file corruption is some unknown Python 2 bug that
> does not have anything to do with races.  I got the sense from
> what Barry said that he feels Python 3 is now much more reliable than
> Python 2.  So this may be another instance of that general idea.
>
> Anyhow, I think the next step is to test whether this corruption
> occurs for Python 3.  (And if it does I get the sense that Barry would
> be anxious to figure out what that Python 3 bug was.)
>
> @Hazen:
>
> This issue lends lots of additional motivation for making PLplot work
> correctly with Python 3.  So please go ahead and push your Python 3
> topic as soon as it is in reasonable state, and we can mature it
> further (if necessary) from there.

Pushed.

One possibly important thing to note is that Python3 does not allow a 
mix of tabs and spaces in a file. So these changes are less extensive 
then they might appear as a lot of it was converting the examples to be 
all spaces. And hopefully our file formatting utility will not introduce 
regressions.

-Hazen


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Plplot-devel mailing list
Plplot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/plplot-devel

Re: [Plplot-devel] A mysterious pyc file corruption issue

2017-02-21 Thread Alan W. Irwin 
On 2017-02-21 11:01-0800 Alan W. Irwin wrote:

> Yesterday for the second time in two months an interactive
> comprehensive test failed with a "ValueError: bad marshal data
> (unknown type code)" error for bindings/python/Plframe.py.
[ These errors were so common]
> that the python developers list in 2013 became
> concerned that python would be subject to race conditions when
> generating these files and thus was the author of at least some of
> these corruptions (see discussion thread at
> 
> with the subject line "[Python-Dev] Mysterious Python pyc file
> corruption problems".

I have now been in contact with the OP, Barry Warsaw of python.org, of
that thread who was quite helpful.  For example, Barry told me that
Python is designed so it is frankly impossible for

import Plframe
from Plframe import *

to race (i.e., the first import completely finishes before the second
one starts).  And I cannot find any other cases where Plframe is
imported.  So I think the best bet for explaining this *.pyc
Python-generated file corruption is some unknown Python 2 bug that
does not have anything to do with races.  I got the sense from
what Barry said that he feels Python 3 is now much more reliable than
Python 2.  So this may be another instance of that general idea.

Anyhow, I think the next step is to test whether this corruption
occurs for Python 3.  (And if it does I get the sense that Barry would
be anxious to figure out what that Python 3 bug was.)

@Hazen:

This issue lends lots of additional motivation for making PLplot work
correctly with Python 3.  So please go ahead and push your Python 3
topic as soon as it is in reasonable state, and we can mature it
further (if necessary) from there.

Alan
__
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__

Linux-powered Science
__

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
___
Plplot-devel mailing list
Plplot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/plplot-devel

[Plplot-devel] A mysterious pyc file corruption issue

2017-02-21 Thread Alan W. Irwin 
Yesterday for the second time in two months an interactive
comprehensive test failed with a "ValueError: bad marshal data
(unknown type code)" error for bindings/python/Plframe.py.  This is a
rather common error with python and typically means the associated
*.pyc that is generated by python has been corrupted.  I moved that
corresponding *.pyc file out of the way, and the comprehensive test
(with *.pyc regenerated by python) sailed through afterward without
issues.  For the record, this issue occurred on my Debian Jessie
platform with python version string of

irwin@raven> python --version
Python 2.7.9

There are lots of potential reasons for such *.pyc corruption issues
such as a change in python version and hardware issues, but these
errors are so common that the python developers list in 2013 became
concerned that python would be subject to race conditions when
generating these files and thus was the author of at least some of
these corruptions (see discussion thread at

with the subject line "[Python-Dev] Mysterious Python pyc file
corruption problems".

I did an octal dump of the corrupted file versus the uncorrupted
regenerated one, and as far as I can tell the only difference is
a missing byte in the corrupted file.  (If anyone is interested
I can send those files to you for inspection.)

Yesterday I did do some obvious tests (with memtest, fsck, and git
fsck) of my PC hardware (which is 9 [!] years old, but still going
strong), and all was well.  Furthermore, the above octal dumps showed
no i/o issue with the corrupted file, and the problem always occurs
(so far) with just this particular file.  And these rare errors only
started when I started enabling testing of examples/python/pytkdemo
(our only file that imports PLframe which would generate the *.pyc as
a byproduct of that import) with the test_pytkdemo target. So I am
pretty sure this evidence largely rules out any hardware issue.  And I
have not been fiddling with my python versions, and in any case those
changes should just change a version stamp (at least two bytes) in the
file and not simply remove one byte.

So by a process of elimination, I think this is likely one more
candidate for the mysterious python pyc corruption issue.  However, if
the source of this corruption is a race condition in the python
generation of these files, I believe that would only be an issue if
there are simultaneous attempts to generate this file.  The tests I
run do use parallel builds but the test_pytkdemo target is implemented
with a CMake custom target where there should be no build race
conditions (attempting to build that target twice) unless there is a
bug in either CMake or make.  But if that were the case, we would be
seeing similar errors for our other python test targets, and we don't.
However, if you look at examples/python/pytkdemo, it is interesting
that it imports PLframe in two ways, i.e.

import Plframe
from Plframe import *

This is a fairly common (but sloppy) python idiom for importing both a
namespaced and unnamespaced version of PLframe (because some of our
code uses the namespaced version and some of our code does not).
However, the only way you get a race out of that is if python looks
ahead and starts doing the second import (which would attempt to also
generate a PLframe.pyc file) before the first import was finished, and
I have no idea whether that is a possibility or not.

Anyhow, in the near future I plan to track down all our references to
the version of PLframe that is not namespaced and convert it to the
namespaced version so that second import can be eliminated.  And it
will be interesting to see if that makes this corruption issue
disappear.

Meanwhile, if anyone else can replicate this issue that would even be
stronger evidence it is not due to my hardware.  So if you want to
help out with that, you should run the test_pytkdemo target but only
after touching examples/python/Plframe.py (which would force python to
regenerate the *.pyc file when the test_pytkdemo target is run).  And
you should do this test from time to time under a variety of load
conditions so generating the above error even once may be difficult to
accomplish.

Alan
__
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__

Linux-powered Science
__

--
Check out the vibrant tech community on one of the