[issue34022] 6 tests fail using SOURCE_DATE_EPOCH env var

[STINNER Victor]

STINNER Victor  added the comment:

> With this change, only 3 tests fail on 3.7, and all these 3 fail because of 
> zipimport: test_cmd_line_script, test_multiprocessing_main_handling and 
> test_runpy.

These tests were fixed by:

commit a6e956bcb0edbfe7f18af9be2215a5326ea6bf05
Author: Elvis Pranskevichus 
Date:   Wed Nov 7 13:34:59 2018 -0500

bpo-34726: Fix handling of hash-based pycs in zipimport. (GH-10327)

Current support for hash-based bytecode files in `zipimport` is rather
sparse, which leads to test failures when the test suite is ran with
the ``SOURCE_DATE_EPOCH`` environment variable set.

This teaches zipimport to handle hash-based pycs properly.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34022] 6 tests fail using SOURCE_DATE_EPOCH env var

[STINNER Victor]

STINNER Victor  added the comment:

> New changeset 24b51b1a4919e310d338629cc60371387f475a32 by Miss Islington 
> (bot) in branch '3.7':

With this change, only 3 tests fail on 3.7, and all these 3 fail because of 
zipimport: test_cmd_line_script, test_multiprocessing_main_handling and 
test_runpy.

Chih-Hsuan Yen:
> Looks like in https://github.com/python/cpython/pull/10327, Serhiy Storchaka 
> considers fixing these issues for 3.8 only - "It is easier to skip the test 
> on 3.7." and then remove the "needs backport to 3.7" label. How about 
> changing the target version to 3.8 only and close this issue?

Now I agree to close the issue :-)

I'm ok to not fix zipimport in 3.7.

Anyway, it's not like Python is perfect in the master branch: frozenset in 
marshal is still not reproducible. See bpo-29708.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed
versions: +Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34022] 6 tests fail using SOURCE_DATE_EPOCH env var

[miss-islington]

miss-islington  added the comment:


New changeset 24b51b1a4919e310d338629cc60371387f475a32 by Miss Islington (bot) 
in branch '3.7':
bpo-34022: Stop forcing of hash-based invalidation with SOURCE_DATE_EPOCH 
(GH-9607)
https://github.com/python/cpython/commit/24b51b1a4919e310d338629cc60371387f475a32


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34022] 6 tests fail using SOURCE_DATE_EPOCH env var

[STINNER Victor]

STINNER Victor  added the comment:

test_compileall, test_importlib, test_py_compile: I created 
https://github.com/python/cpython/pull/10775 backport.

test_cmd_line_script, test_runpy and test_multiprocessing_main_handling 
failures are related to zipimport. They have been fixed in master by commit 
a6e956bcb0edbfe7f18af9be2215a5326ea6bf05 which modifies Lib/zipimport.py... but 
in 3.7, zipimport is written in C. It seems non-trivial to fix zipimport, and 
I'm not sure that it's worth it.

Note: test_posix now pass, no idea why it failed a few days ago.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34022] 6 tests fail using SOURCE_DATE_EPOCH env var

[miss-islington]

Change by miss-islington :


--
pull_requests: +10018

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34022] 6 tests fail using SOURCE_DATE_EPOCH env var

[Chih-Hsuan Yen]

Chih-Hsuan Yen  added the comment:

Got it. Thanks for the pointer.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34022] 6 tests fail using SOURCE_DATE_EPOCH env var

[STINNER Victor]

STINNER Victor  added the comment:

"What’s New In Python 3.7" mentions twice "build reproducibility":
https://docs.python.org/dev/whatsnew/3.7.html

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34022] 6 tests fail using SOURCE_DATE_EPOCH env var

[Chih-Hsuan Yen]

Chih-Hsuan Yen  added the comment:

Looks like in https://github.com/python/cpython/pull/10327, Serhiy Storchaka 
considers fixing these issues for 3.8 only - "It is easier to skip the test on 
3.7." and then remove the "needs backport to 3.7" label. How about changing the 
target version to 3.8 only and close this issue?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34022] 6 tests fail using SOURCE_DATE_EPOCH env var

[STINNER Victor]

STINNER Victor  added the comment:

> As https://github.com/python/cpython/pull/10327 has been merged, I think this 
> issue can be closed?

The Versions field of this issue is set to Python 3.7, but 7 tests are failing 
with:

$ SOURCE_DATE_EPOCH=0 ./python -m test -j0 -r -u all,-gui
(...)
7 tests failed:
test_cmd_line_script test_compileall test_importlib
test_multiprocessing_main_handling test_posix test_py_compile
test_runpy

Note: the full test suite pass with SOURCE_DATE_EPOCH=0 on master.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34022] 6 tests fail using SOURCE_DATE_EPOCH env var

[Chih-Hsuan Yen]

Chih-Hsuan Yen  added the comment:

As https://github.com/python/cpython/pull/10327 has been merged, I think this 
issue can be closed?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34022] 6 tests fail using SOURCE_DATE_EPOCH env var

[Elvis Pranskevichus]

Elvis Pranskevichus  added the comment:

https://github.com/python/cpython/pull/10327 fixes the remaining test failures.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34022] 6 tests fail using SOURCE_DATE_EPOCH env var

[Elvis Pranskevichus]

Change by Elvis Pranskevichus :


--
pull_requests: +9630

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34022] 6 tests fail using SOURCE_DATE_EPOCH env var

[STINNER Victor]

STINNER Victor  added the comment:

There are still tests which fail when SOURCE_DATE_EPOCH env var is defined:

$ SOURCE_DATE_EPOCH=0 ./python -m test -j0 -r
(...)
3 tests failed:
test_cmd_line_script test_multiprocessing_main_handling test_runpy
(...)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34022] 6 tests fail using SOURCE_DATE_EPOCH env var

[STINNER Victor]

STINNER Victor  added the comment:


New changeset a6b3ec5b6d4f6387820fccc570eea08b9615620d by Victor Stinner (Elvis 
Pranskevichus) in branch 'master':
bpo-34022: Stop forcing of hash-based invalidation with SOURCE_DATE_EPOCH 
(GH-9607)
https://github.com/python/cpython/commit/a6b3ec5b6d4f6387820fccc570eea08b9615620d


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34022] 6 tests fail using SOURCE_DATE_EPOCH env var

[STINNER Victor]

STINNER Victor  added the comment:

> https://github.com/python/cpython/pull/9607 turns SOURCE_DATE_EPOCH into a 
> *default* rather than absolute override.

Once we decided to merge or reject this PR, we should maybe update the PEP 552 
to clarify the interaction with the SOURCE_DATE_EPOCH environment variable.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34022] 6 tests fail using SOURCE_DATE_EPOCH env var

[Elvis Pranskevichus]

Elvis Pranskevichus  added the comment:

https://github.com/python/cpython/pull/9607 turns SOURCE_DATE_EPOCH into a 
*default* rather than absolute override.

test_cmd_line_script and test_runpy fail due to the lack of support for 
hash-based .pycs in zipimport, which needs fixing also.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34022] 6 tests fail using SOURCE_DATE_EPOCH env var

[Elvis Pranskevichus]

Change by Elvis Pranskevichus :


--
pull_requests: +9004

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34022] 6 tests fail using SOURCE_DATE_EPOCH env var

[Eli Schwartz]

Change by Eli Schwartz :


--
nosy: +eschwartz

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34022] 6 tests fail using SOURCE_DATE_EPOCH env var

[Elvis Pranskevichus]

Elvis Pranskevichus  added the comment:

Perhaps SOURCE_DATE_EPOCH should only be consulted to determine the default 
behavior if invalidation_mode was not passed explicitly to py_compile.compile().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34022] 6 tests fail using SOURCE_DATE_EPOCH env var

[Benjamin Peterson]

Benjamin Peterson  added the comment:

I dislike that having SOURCE_DATE_EPOCH set magically changes command line 
tools behavior. In my view, this problem should be fixed by reverting 
ccbe5818af2.

--
nosy: +benjamin.peterson

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34022] 6 tests fail using SOURCE_DATE_EPOCH env var

[Elvis Pranskevichus]

Elvis Pranskevichus  added the comment:

> In particular, if a build system sets SOURCE_DATE_EPOCH without
specifying a pyc format for py_compile or compileall, Python 3.7 will
give you checked hashes by default

Actually, py_compile will _force_ the CHECKED_HASH mode if SOURCE_DATE_EPOCH is 
set, regardless of what you specified for py_compile or compileall.  I'm not 
sure if that's actually intended or a bug.

The fix for the tests is to actually be aware of py_compile behavior and 
control for SOURCE_DATE_EPOCH.  I submitted a PR with a fix.

--
nosy: +Elvis.Pranskevichus

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34022] 6 tests fail using SOURCE_DATE_EPOCH env var

[Elvis Pranskevichus]

Change by Elvis Pranskevichus :


--
keywords: +patch
pull_requests: +8894
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34022] 6 tests fail using SOURCE_DATE_EPOCH env var

[Chih-Hsuan Yen]

Change by Chih-Hsuan Yen :


--
nosy: +yan12125

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34022] 6 tests fail using SOURCE_DATE_EPOCH env var

[Alexander Mohr]

Alexander Mohr  added the comment:

we're compiling similar to debian and getting the same issues now only with 3.7 
so interested in what the resolution is.  I tried both w/ and w/o setting 
SOURCE_DATE_EPOCH and still get failures.

--
nosy: +thehesiod

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34022] 6 tests fail using SOURCE_DATE_EPOCH env var

[Matej Cepl]

Matej Cepl  added the comment:

>From https://mail.python.org/pipermail/python-dev/2018-July/154273.html
(by Nick Coghlan):

In particular, if a build system sets SOURCE_DATE_EPOCH without
specifying a pyc format for py_compile or compileall, Python 3.7 will
give you checked hashes by default:
https://docs.python.org/3/library/py_compile.html?highlight=source_date_epoch#py_compile.compile

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34022] 6 tests fail using SOURCE_DATE_EPOCH env var

[STINNER Victor]

STINNER Victor  added the comment:

See bpo-29708 and PEP 552 for more info about reproducible builds.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34022] 6 tests fail using SOURCE_DATE_EPOCH env var

[STINNER Victor]

STINNER Victor  added the comment:

FAIL: test_invalidation_mode (test.test_py_compile.PyCompileTests)
--
Traceback (most recent call last):
  File "/home/abuild/rpmbuild/BUILD/Python-3.7.0/Lib/test/test_py_compile.py", 
line 153, in test_invalidation_mode
self.assertEqual(flags, 0b1)
AssertionError: 3 != 1


This failure looks suspicious. I looked at the code, and the tested function 
behaves differently depending of the SOURCE_DATE_EPOCH environment variable is 
set.

Aha! I found something:

$ SOURCE_DATE_EPOCH=1530605018 ./python -m test -j0
...
6 tests failed:
test_cmd_line_script test_compileall test_importlib
test_multiprocessing_main_handling test_py_compile test_runpy
...

--
title: Python 3.7: 6 tests failed on OpenSUSE and Debian -> 6 tests fail using 
SOURCE_DATE_EPOCH env var

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com