[issue15822] installed python may fail incorrectly trying to rebuild lib2to3 grammar pickles

[Bun Hin]

Bun Hin  added the comment:

I have similar problem with python 3.7.0 or 3.7.2,
how to fix my installation to get it work?

please help

Thanks

--
nosy: +bunhin

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15822] installed python may fail incorrectly trying to rebuild lib2to3 grammar pickles

[Georg Brandl]

Georg Brandl added the comment:

Picked to release clone as 411d500e56a1.

--
priority: release blocker - high

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15822
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15822] installed python may fail incorrectly trying to rebuild lib2to3 grammar pickles

[Ned Deily]

Changes by Ned Deily n...@acm.org:


--
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15822
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15822] installed python may fail incorrectly trying to rebuild lib2to3 grammar pickles

[Roundup Robot]

Roundup Robot added the comment:

New changeset 411d500e56a1 by Ned Deily in branch 'default':
Issue #15822: Fix installation of lib2to3 grammar pickles to ensure
http://hg.python.org/cpython/rev/411d500e56a1

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15822
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15822] installed python may fail incorrectly trying to rebuild lib2to3 grammar pickles

[Martin v . Löwis]

Martin v. Löwis added the comment:

I think PYTHON_FOR_BUILD includes -E to make sure that if PYTHONPATH (or 
PYTHONHOME) is set that it then won't affect the build.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15822
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15822] installed python may fail incorrectly trying to rebuild lib2to3 grammar pickles

[Roundup Robot]

Roundup Robot added the comment:

New changeset 2def9ef683da by Ned Deily in branch '2.7':
Issue #15822: Fix installation of lib2to3 grammar pickles to ensure
http://hg.python.org/cpython/rev/2def9ef683da

New changeset adc5e6aca251 by Ned Deily in branch '3.2':
Issue #15822: Fix installation of lib2to3 grammar pickles to ensure
http://hg.python.org/cpython/rev/adc5e6aca251

New changeset a829889f96b0 by Ned Deily in branch 'default':
Issue #15822: Fix installation of lib2to3 grammar pickles to ensure
http://hg.python.org/cpython/rev/a829889f96b0

--
nosy: +python-dev

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15822
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15822] installed python may fail incorrectly trying to rebuild lib2to3 grammar pickles

[Ned Deily]

Ned Deily added the comment:

I like Martin's suggested approach.  It is explicit and much cleaner than the 
changes we introduced for Issue15645 as well as the original.  The applied 
changes revert the previous changes and instead implement an explicit build of 
the two pickles directly in the installed lib2to3 during make install 
regardless of the presence of pickles in the build directory.  This also 
resolves Issue15838.

Georg, I think a829889f96b0 should be considered for inclusion in 3.3.0.

Ronald, if you want to pursue the $PYTHON_FOR_BUILD -E issue, it would be best 
to use another specific issue for that.

--
resolution:  - fixed
stage: patch review - committed/rejected
status: open - pending

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15822
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15822] installed python may fail incorrectly trying to rebuild lib2to3 grammar pickles

[Ned Deily]

Changes by Ned Deily n...@acm.org:


--
status: pending - open
Removed file: http://bugs.python.org/file27084/issue15822_33.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15822
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15822] installed python may fail incorrectly trying to rebuild lib2to3 grammar pickles

[Martin v . Löwis]

Martin v. Löwis added the comment:

I think the patch for issue 15645 was incorrect. Instead of generating the 
pickles in the source tree and copying them, it should have arranged lib2to3 to 
generate them in the target directory instead (just as all the compileall 
invocations also generate files in LIBDEST).

There are multiple ways to achieve this. The least intrusive (perhaps) is to 
insert $(DESTDIR)$(LIBDEST) into sys.path before importing lib2to3 in the 
Makefile.

The approach that I suggest is to make lib2to3.pgen2.driver a script which 
expects a .txt path, and generates a pickle next to it, so the installation 
would do

$(PYTHON_FOR_BUILD) -m lib2to3.pgen2.driver 
$(DESTDIR)$(LIBDEST)/lib2to3/Grammar.txt
$(PYTHON_FOR_BUILD) -m lib2to3.pgen2.driver 
$(DESTDIR)$(LIBDEST)/lib2to3/PatternGrammar.txt

--
nosy: +loewis

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15822
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15822] installed python may fail incorrectly trying to rebuild lib2to3 grammar pickles

[Ronald Oussoren]

Ronald Oussoren added the comment:

See als Issue15838.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15822
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15822] installed python may fail incorrectly trying to rebuild lib2to3 grammar pickles

[Ronald Oussoren]

Ronald Oussoren added the comment:

The other alternative to fix this is to revert the patch in issue 15645  and 
then remove the -E flag from PYTHON_FOR_BUILD.

PYTHON_FOR_BUILD was introduced by a, possibly incomplete, attempt to introduce 
support for cross compiling. In particular, the variable was introduced in 
issue 14330. Neither the patch, nor the tracker discussion explains why the -E 
is necessary. It might be because all uses of $(BUILDPYTHON) that were replaced 
by PYTHON_FOR_BUILD had the -E flag at the time.

BTW. There also doesn't seem to documentation for cross building.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15822
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15822] installed python may fail incorrectly trying to rebuild lib2to3 grammar pickles

[Ned Deily]

Ned Deily added the comment:

The problem here is in the libinstall Makefile target and that hidden 
dependency within lib2to3 that, as Tomi notes, causes lib2to3 to try to 
recreate the pickles anytime it runs if it thinks they are out of date.  The 
fixes for Issue15645 cause the recipes for that target to create the two 
grammar pickles in the source Lib directory before files are copied and 
installed into the destination lib directories. But there is no guarantee that 
the source files, Grammar.txt and PatternGrammar.txt, are installed into the 
destination lib before the corresponding pickle files are.  
lib2to3/pgen2/driver compares the value of os.path.getmtime() for the txt and 
pickle files (assuming both exist).  On many (most) file systems, the copies 
happen fast enough and the resolution of getmtime is coarse enough that the 
files are created with the same time value so post-installation use of lib2to3 
works OK. But on file systems like ext4 with high-resolution time values, it 
may happen that one or both o
 f the pickle files is copied first and ends up with a slightly older time 
value than its corresponding txt file.  In that case the first use of lib2to3 
will try to recreate the pickle which will most likely fail with a file 
permission error since the install directories are typically read-only to 
non-root users.

The attached patch tries to work around around the problem by touching the two 
pickle files after they and all the other relevant lib files have been 
installed.

Since any user could run into this problem on the right configuration, I think 
the fix (assuming reviewers concur with it) should be cherry-picked for 
3.3.0rc2.

--
components: +Installation
keywords: +patch
nosy: +ronaldoussoren
priority: high - release blocker
stage:  - patch review
title: Python 3.3 creates lib2to3 grammar in wrong order - installed python 
may fail incorrectly trying to rebuild lib2to3 grammar pickles
versions: +Python 2.7, Python 3.2
Added file: http://bugs.python.org/file27084/issue15822_33.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15822
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15822] installed python may fail incorrectly trying to rebuild lib2to3 grammar pickles

[Arfrever Frehtes Taifersar Arahesis]

Arfrever Frehtes Taifersar Arahesis added the comment:

 - done
 + done; \

Not needed.

 + touch $(DESTDIR)$(LIBDEST)/lib2to3/*Grammar*.pickle || true

- prefix can be used to ignore failure of command:

+   -touch $(DESTDIR)$(LIBDEST)/lib2to3/*Grammar*.pickle

However maybe failure of touch should not be ignored here.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15822
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15822] installed python may fail incorrectly trying to rebuild lib2to3 grammar pickles

[Ned Deily]

Ned Deily added the comment:

I did it that way to ensure that the touch is executed after the copies.  And 
the touch needs to be optional because the creation of the pickles can 
optionally fail.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15822
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15822] installed python may fail incorrectly trying to rebuild lib2to3 grammar pickles

[Arfrever Frehtes Taifersar Arahesis]

Arfrever Frehtes Taifersar Arahesis added the comment:

 And the touch needs to be optional because the creation of the pickles can 
 optionally fail.

If creation of pickles failed, then touch will create 1 file literally named 
*Grammar*.pickle.

...
done; \
for i in $(DESTDIR)$(LIBDEST)/lib2to3/*Grammar*.pickle; do \
if test -f $$i; then \
touch $$i; \
fi; \
done

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15822
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15822] installed python may fail incorrectly trying to rebuild lib2to3 grammar pickles

[Ned Deily]

Ned Deily added the comment:

Right you are, thanks!  I had that in an earlier version.  Are you able to 
verify the fix?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15822
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com