[issue23911] Move path-based bootstrap code to a separate frozen file.

[Roundup Robot]

Roundup Robot added the comment:

New changeset cc2e52878393 by Zachary Ware in branch 'default':
Issue #23911: Fix ctypes test on Windows.
https://hg.python.org/cpython/rev/cc2e52878393

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23911] Move path-based bootstrap code to a separate frozen file.

[Eric Snow]

Eric Snow added the comment:

changeset:   95887:3bea670c9830
user:Eric Snow 
date:Tue May 05 21:29:31 2015 -0600
summary: Remove an unnecessary flag.

--
resolution:  -> fixed
stage: needs patch -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23911] Move path-based bootstrap code to a separate frozen file.

[Eric Snow]

Eric Snow added the comment:

Thanks for pointing this out, Ned.  Early on I ran into a problem when running 
_freeze_importlib without the flag set.  However, I expect that it was not 
necessary after a certain point (e.g. once I had a valid 
_importlib_external.h).  I'll remove the flag as suggested.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23911] Move path-based bootstrap code to a separate frozen file.

[Ned Deily]

Ned Deily added the comment:

This checkin also breaks OS X framework builds.  For some reason, framework 
builds are compiled with the gcc -fno-common option.  The code in configure.ac 
to add that option dates back to the initial OS X framework support 
(c3c87ce4afdc from 2001).  It's not clear to me why we use that option just in 
the case of framework builds (I suspect it may no longer be necessary) but the 
fact is that we have been and currently still do.  And, in any case, the GCC 
description of -fno-common 
(https://gcc.gnu.org/onlinedocs/gcc/Code-Gen-Options.html) suggests that there 
may be valid reasons to use it on some platforms or that other compilers might 
behave this way by default.  A simple way to produce the error on Linux is:

./configure OPT="-fno-common" && make

[...]
gcc -pthread   -o Programs/_freeze_importlib Programs/_freeze_importlib.o 
Modules/getbuildinfo.o Parser/acceler.o Parser/grammar1.o Parser/listnode.o 
Parser/node.o Parser/parser.o Parser/bitset.o Parser/metagrammar.o 
Parser/firstsets.o Parser/grammar.o Parser/pgen.o Parser/myreadline.o 
Parser/parsetok.o Parser/tokenizer.o Objects/abstract.o Objects/accu.o 
Objects/boolobject.o Objects/bytes_methods.o Objects/bytearrayobject.o 
Objects/bytesobject.o Objects/cellobject.o Objects/classobject.o 
Objects/codeobject.o Objects/complexobject.o Objects/descrobject.o 
Objects/enumobject.o Objects/exceptions.o Objects/genobject.o 
Objects/fileobject.o Objects/floatobject.o Objects/frameobject.o 
Objects/funcobject.o Objects/iterobject.o Objects/listobject.o 
Objects/longobject.o Objects/dictobject.o Objects/memoryobject.o 
Objects/methodobject.o Objects/moduleobject.o Objects/namespaceobject.o 
Objects/object.o Objects/obmalloc.o Objects/capsule.o Objects/rangeobject.o 
Objects/setobject.o Objects/sliceo
 bject.o Objects/structseq.o Objects/tupleobject.o Objects/typeobject.o 
Objects/unicodeobject.o Objects/unicodectype.o Objects/weakrefobject.o 
Python/_warnings.o Python/Python-ast.o Python/asdl.o Python/ast.o 
Python/bltinmodule.o Python/ceval.o Python/compile.o Python/codecs.o 
Python/dynamic_annotations.o Python/errors.o Python/frozenmain.o 
Python/future.o Python/getargs.o Python/getcompiler.o Python/getcopyright.o 
Python/getplatform.o Python/getversion.o Python/graminit.o Python/import.o 
Python/importdl.o Python/marshal.o Python/modsupport.o Python/mystrtoul.o 
Python/mysnprintf.o Python/peephole.o Python/pyarena.o Python/pyctype.o 
Python/pyfpe.o Python/pyhash.o Python/pylifecycle.o Python/pymath.o 
Python/pystate.o Python/pythonrun.o Python/pytime.o Python/random.o 
Python/structmember.o Python/symtable.o Python/sysmodule.o Python/traceback.o 
Python/getopt.o Python/pystrcmp.o Python/pystrtod.o Python/pystrhex.o 
Python/dtoa.o Python/formatter_unicode.o Python/fileutils.o Python/dynload
 _shlib.o   Python/thread.o Modules/config.o Modules/getpath.o Modules/main.o 
Modules/gcmodule.o  Modules/_threadmodule.o  Modules/signalmodule.o  
Modules/posixmodule.o  Modules/errnomodule.o  Modules/pwdmodule.o  
Modules/_sre.o  Modules/_codecsmodule.o  Modules/_weakref.o  
Modules/_functoolsmodule.o  Modules/_operator.o  Modules/_collectionsmodule.o  
Modules/itertoolsmodule.o  Modules/atexitmodule.o  Modules/_stat.o  
Modules/timemodule.o  Modules/_localemodule.o  Modules/_iomodule.o 
Modules/iobase.o Modules/fileio.o Modules/bytesio.o Modules/bufferedio.o 
Modules/textio.o Modules/stringio.o  Modules/zipimport.o  
Modules/faulthandler.o  Modules/_tracemalloc.o Modules/hashtable.o  
Modules/symtablemodule.o  Modules/xxsubtype.o -lpthread -ldl  -lutil   -lm
Python/pylifecycle.o:(.bss+0x20): multiple definition of `Py_FrozenFlag'
Programs/_freeze_importlib.o:(.data+0x0): first defined here
collect2: error: ld returned 1 exit status
Makefile:710: recipe for target 'Programs/_freeze_importlib' failed
make: *** [Programs/_freeze_importlib] Error 1

The OS X version of ld helpfully lists the modules:

duplicate symbol _Py_FrozenFlag in:
Programs/_freeze_importlib.o
Python/pylifecycle.o

I'm also not sure why the Py_FrozenFlag definition was added to 
_freeze_importlib.c in the first place.  Reverting it with the following patch 
seems to solve the problem in the few configurations I tried (OS X framework, 
OS X non-shared, Linux non-shared, Linux shared).

diff Programs/_freeze_importlib.c
--- a/Programs/_freeze_importlib.c  Tue May 05 12:04:35 2015 -0700
+++ b/Programs/_freeze_importlib.c  Tue May 05 17:10:39 2015 -0700
@@ -12,8 +12,6 @@
 #include 
 #endif

-int Py_FrozenFlag = 1;  /* Suppress errors from getpath.c */
-
 /* To avoid a circular dependency on frozen.o, we create our own structure
of frozen modules instead, left deliberately blank so as to avoid
unintentional import of a stale version of _frozen_importlib. */

--
nosy: +ned.deily
resolution: fixed -> 
stage: resolved -> needs patch
status: closed -> open

___
Python tracker 

[issue23911] Move path-based bootstrap code to a separate frozen file.

[Eric Snow]

Changes by Eric Snow :


--
status: pending -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23911] Move path-based bootstrap code to a separate frozen file.

[Eric Snow]

Changes by Eric Snow :


--
resolution:  -> fixed
status: open -> pending

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23911] Move path-based bootstrap code to a separate frozen file.

[Roundup Robot]

Roundup Robot added the comment:

New changeset 36b902bbc992 by Eric Snow in branch 'default':
Issue #23911: Fix mixed bytes/strings.
https://hg.python.org/cpython/rev/36b902bbc992

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23911] Move path-based bootstrap code to a separate frozen file.

[Larry Hastings]

Larry Hastings added the comment:

This checkin broke the buildbots.  If you build trunk then run

./python -bb -m test test_site

the test fails.  "-bb" is used by the normal test runner ("make test").

The problem is in the lines

self.assertTrue(os.path.isabs(os__file__),
"expected absolute path, got {}".format(os__file__))
self.assertTrue(os.path.isabs(os__cached__),
"expected absolute path, got {}".format(os__cached__))

os__file__ and os__cached__ are bytes but you're passing them into .format() on 
a str.

--
nosy: +larry
resolution: fixed -> 
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23911] Move path-based bootstrap code to a separate frozen file.

[Roundup Robot]

Roundup Robot added the comment:

New changeset 02e3bf65b2f8 by Eric Snow in branch 'default':
Issue #23911: Move path-based bootstrap code to a separate frozen module.
https://hg.python.org/cpython/rev/02e3bf65b2f8

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23911] Move path-based bootstrap code to a separate frozen file.

[Eric Snow]

Changes by Eric Snow :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23911] Move path-based bootstrap code to a separate frozen file.

[Eric Snow]

Changes by Eric Snow :


Removed file: http://bugs.python.org/file39274/path-based-importlib.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23911] Move path-based bootstrap code to a separate frozen file.

[Eric Snow]

Changes by Eric Snow :


Added file: http://bugs.python.org/file39275/path-based-importlib.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23911] Move path-based bootstrap code to a separate frozen file.

[Eric Snow]

Changes by Eric Snow :


Added file: http://bugs.python.org/file39274/path-based-importlib.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23911] Move path-based bootstrap code to a separate frozen file.

[Eric Snow]

Changes by Eric Snow :


Removed file: http://bugs.python.org/file39273/path-based-importlib.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23911] Move path-based bootstrap code to a separate frozen file.

[Eric Snow]

Eric Snow added the comment:

Gah.  I had tried exactly that but did it in the wrong spot.  Here's an updated 
patch which fixes the test.

--
Added file: http://bugs.python.org/file39273/path-based-importlib.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23911] Move path-based bootstrap code to a separate frozen file.

[Nick Coghlan]

Nick Coghlan added the comment:

As a compatibility hack for setuptools versions with the issue, I'd suggest
making importlib._bootstrap.setup alias SourceFileLoader back into
importlib._bootstrap, with an explanation and link to back to this issue in
a comment.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23911] Move path-based bootstrap code to a separate frozen file.

[Eric Snow]

Changes by Eric Snow :


Removed file: http://bugs.python.org/file39271/path-based-importlib.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23911] Move path-based bootstrap code to a separate frozen file.

[Eric Snow]

Eric Snow added the comment:

Here's the correct patch.

--
Added file: http://bugs.python.org/file39272/path-based-importlib.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23911] Move path-based bootstrap code to a separate frozen file.

[Eric Snow]

Eric Snow added the comment:

Looks like setuptool's pkg_resources is directly importing 
importlib._bootstrap.  I've filed a bug: 
https://bitbucket.org/pypa/setuptools/issue/378.  In the meantime, what are our 
options for getting that test passing?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23911] Move path-based bootstrap code to a separate frozen file.

[Eric Snow]

Eric Snow added the comment:

As I mentioned, I'm pretty sure that the failing venv test is due to the 
bundled pip.  Here's the test output:

test test_venv failed -- Traceback (most recent call last):
  File "/home/esnow/projects/cpython/Lib/test/test_venv.py", line 356, in 
test_with_pip
with_pip=True)
subprocess.CalledProcessError: Command '['/tmp/tmphxh1zztf/bin/python', '-Im', 
'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/esnow/projects/cpython/Lib/test/test_venv.py", line 362, in 
test_with_pip
self.fail(msg.format(exc, details))
AssertionError: Command '['/tmp/tmphxh1zztf/bin/python', '-Im', 'ensurepip', 
'--upgrade', '--default-pip']' returned non-zero exit status 1

**Subprocess Output**
Traceback (most recent call last):
  File "/home/esnow/projects/cpython/Lib/runpy.py", line 170, in 
_run_module_as_main
"__main__", mod_spec)
  File "/home/esnow/projects/cpython/Lib/runpy.py", line 85, in _run_code
exec(code, run_globals)
  File "/home/esnow/projects/cpython/Lib/ensurepip/__main__.py", line 4, in 

ensurepip._main()
  File "/home/esnow/projects/cpython/Lib/ensurepip/__init__.py", line 209, in 
_main
default_pip=args.default_pip,
  File "/home/esnow/projects/cpython/Lib/ensurepip/__init__.py", line 116, in 
bootstrap
_run_pip(args + [p[0] for p in _PROJECTS], additional_paths)
  File "/home/esnow/projects/cpython/Lib/ensurepip/__init__.py", line 40, in 
_run_pip
import pip
  File "", line 958, in _find_and_load
  File "", line 947, in _find_and_load_unlocked
  File "", line 655, in _load_unlocked
  File "", line 625, in _load_backward_compatible
  File "/tmp/tmp57j9vkrt/pip-6.1.1-py2.py3-none-any.whl/pip/__init__.py", line 
13, in 
  File "", line 958, in _find_and_load
  File "", line 947, in _find_and_load_unlocked
  File "", line 655, in _load_unlocked
  File "", line 625, in _load_backward_compatible
  File "/tmp/tmp57j9vkrt/pip-6.1.1-py2.py3-none-any.whl/pip/utils/__init__.py", 
line 22, in 
  File "", line 958, in _find_and_load
  File "", line 947, in _find_and_load_unlocked
  File "", line 655, in _load_unlocked
  File "", line 625, in _load_backward_compatible
  File 
"/tmp/tmp57j9vkrt/pip-6.1.1-py2.py3-none-any.whl/pip/_vendor/pkg_resources/__init__.py",
 line 1712, in 
AttributeError: module 'importlib._bootstrap' has no attribute 
'SourceFileLoader'

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23911] Move path-based bootstrap code to a separate frozen file.

[Eric Snow]

Eric Snow added the comment:

Here's an updated patch with "_pathy.py" changed to "_bootstrap_external.py" 
(and similar changes with freezing).  The patch does not include fixing the 
venv test (i.e. the bundled pip).

Also, I'll be adding a note to NEWS.

--
Added file: http://bugs.python.org/file39271/path-based-importlib.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23911] Move path-based bootstrap code to a separate frozen file.

[Eric Snow]

Eric Snow added the comment:

Glad to hear the patch is conceptually consistent with other components. :)  
And the "internal"/"external" suggestion is a good one.  I'll update the patch 
when I have a minute.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23911] Move path-based bootstrap code to a separate frozen file.

[Nick Coghlan]

Nick Coghlan added the comment:

Nice, this ties directly into one of the thornier problems in the PEP 432 
bootstrapping work, which needs to set up the core import system in 
Py_BeginInitialization, but delay setting up the rest of it until 
Py_EndInitialization.

Your patch achieves this by moving everything that belongs in the latter part 
of the operation to the "import _frozen_path_importer" step.

As far as naming goes, I'd suggest referring to the two import subsystems as 
"internal imports" and "external imports". The key aspect of builtin and frozen 
modules is that they're an inherent part of the interpreter itself - if you 
have an interpreter, you have all of its buitin and frozen modules natively 
available. By contrast, setting up the external import machinery requires that 
the interpreter first be configured to appropriately communicate with the host 
system.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23911] Move path-based bootstrap code to a separate frozen file.

[Eric Snow]

Eric Snow added the comment:

s/PEP 489/PEP 488/

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23911] Move path-based bootstrap code to a separate frozen file.

[Eric Snow]

Eric Snow added the comment:

Here's an updated patch, with the PEP 489 changes merged in.  Only one test 
isn't passing and it is due to something in the pip that is bundled into 
ensurepip.  I'll work on fixing that when I have some time.  I'm sure there's 
documentation near the bundle that explains how to update it.

--
Added file: http://bugs.python.org/file39138/path-based-importlib.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23911] Move path-based bootstrap code to a separate frozen file.

[Raúl Cumplido]

Changes by Raúl Cumplido :


--
nosy: +raulcd

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23911] Move path-based bootstrap code to a separate frozen file.

[Eric Snow]

New submission from Eric Snow:

The bootstrap code has a clear division between the core import functionality 
and the path-based import machinery.  The attached patch makes that division 
explicit by moving the latter into its own module.  The module is also frozen, 
necessarily.  In addition to clearly distinguishing the two parts, this 
division will help with some later work that I'd like to do later with an 
encapsulated import system abstraction.

The patch uses the name "pathy" throughout, which I'll change to something more 
descriptive later.  I'll also add the news entry then.

--
assignee: eric.snow
components: Interpreter Core
files: path-based-importlib.diff
keywords: patch
messages: 240457
nosy: brett.cannon, eric.snow, ncoghlan
priority: normal
severity: normal
stage: patch review
status: open
title: Move path-based bootstrap code to a separate frozen file.
type: enhancement
versions: Python 3.5
Added file: http://bugs.python.org/file38893/path-based-importlib.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com