Brett Cannon br...@python.org added the comment:
Do people feel the need to change the importlib tests to run against both the
frozen code and importlib/_bootstrap.py? If so either the test_warnings
approach (or a new one using class decorators or something) should be used to
run the tests
Nick Coghlan ncogh...@gmail.com added the comment:
Given Antoine's other change to the build process to fix the bootstrapping
problem, I'm OK with leaving it up to anyone hacking on _bootstrap.py to
remember that they need to run make if they want the interpreter to see any of
their changes.
Roundup Robot devn...@psf.upfronthosting.co.za added the comment:
New changeset e3a984076837 by Antoine Pitrou in branch 'default':
Issue #14657: The frozen instance of importlib used for bootstrap is now also
the module imported as importlib._bootstrap.
Antoine Pitrou pit...@free.fr added the comment:
Ok, I've committed the patch. I'm closing this issue, but of course potential
improvements can be posted under a new issue.
--
dependencies: -test.support.DirsOnSysPath should be replaced by
importlib.test.util.import_state
resolution:
Antoine Pitrou pit...@free.fr added the comment:
Unless someone plans to do further work on this, I'd like to commit
unique_importlib3.patch. A working solution is better than nothing.
--
___
Python tracker rep...@bugs.python.org
Eric Snow ericsnowcurren...@gmail.com added the comment:
It's actually at the top of my list, but may still be a week or two until I can
get to it.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14657
Nick Coghlan ncogh...@gmail.com added the comment:
+1 Antoine - your patch is better than the status quo, so it makes sense to
apply it.
If Eric can make the other way work in time for 3.3, great, but at least we'll
have a solid fallback position without it.
--
Brett Cannon br...@python.org added the comment:
Realized that any decorator or context manager that is created for swapping
_frozen_importlib/importlib._bootstrap should also verify no module is left in
sys.modules with a bad loader and that sys.path_importer_cache doesn't have a
bad finder
Nick Coghlan ncogh...@gmail.com added the comment:
In that case, how about we go with:
1. By default, importlib._bootstrap is never imported. Instead, it is set to be
a reference to _frozen_importlib. However, _frozen_importlib does *not* lie
about where it came from (and doesn't assume the
Brett Cannon br...@python.org added the comment:
Should we have a separate context manager for this, or just make it a flag for
a unified import_state() decorator? Or do we want to *always* force the use of
the Python code instead of the frozen code?
--
Antoine Pitrou pit...@free.fr added the comment:
Should we have a separate context manager for this, or just make it a
flag for a unified import_state() decorator? Or do we want to *always*
force the use of the Python code instead of the frozen code?
Ideally, we would want to test both
Antoine Pitrou pit...@free.fr added the comment:
Ideally, we would want to test both versions, so that any oddity in the
freezing mechanism gets exercised and diagnosed properly.
(not to mention the speedups in import.c)
--
___
Python tracker
Eric Snow ericsnowcurren...@gmail.com added the comment:
I'm +1 on Nick's recommendation.
@Antoine
Ideally, we would want to test both versions, so that any oddity in
the freezing mechanism gets exercised and diagnosed properly.
+1
Does this mean that the whole test suite should be run
Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com added the comment:
It might be sufficient to only run tests from the following files with both
importlib._bootstrap and _frozen_importlib:
test_imp.py
test_import.py
test_importhooks.py
test_importlib.py
test_pkgimport.py
Antoine Pitrou pit...@free.fr added the comment:
It might be sufficient to only run tests from the following files with
both importlib._bootstrap and _frozen_importlib:
I was only thinking about test_importlib myself.
--
___
Python tracker
Brett Cannon br...@python.org added the comment:
I would say test_importlib and test_imp (test_import really should just get
folded into test_importlib).
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14657
Nick Coghlan ncogh...@gmail.com added the comment:
The piece you're missing is that the interpreter state holds a direct reference
to the import machinery in interp-importlib, and *that's* what gets used by
the builtin __import__ implementation.
I'm beginning to think the thing to do is to
Nick Coghlan ncogh...@gmail.com added the comment:
Forgot to add: in our own tests, we should ensure that both the frozen and
on-disk versions get executed.
I believe that's already the case, since I don't recall anyone removing the
test infrastructure that ensured both import.c and importlib
Antoine Pitrou pit...@free.fr added the comment:
I believe that's already the case, since I don't recall anyone
removing the test infrastructure that ensured both import.c and
importlib are tested for correct behaviour.
What do you mean? I think test_importlib only tests the on-disk version.
Antoine Pitrou pit...@free.fr added the comment:
Here's my take. No one will care about _frozen_importlib vs.
importlib._bootstrap normally, right? If __module__/__file__ says
_frozen_importlib, it's no big deal.
The reason I'd prefer __file__ to point to the actual Python file is so
that
Brett Cannon br...@python.org added the comment:
To respond to Nick's yes, there are two copies of importlib._bootstrap
leanings, distutils2 has actually run into issues with this because they
initially made some assumptions about consistency in what importlib returned
vs. what import does
Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com added the comment:
It was distribute (fork of setuptools, with added support for Python 3), not
distutils2.
distribute has been changed to directly use _frozen_importlib:
https://bitbucket.org/tarek/distribute/changeset/a2685f3af854
Antoine Pitrou pit...@free.fr added the comment:
It was distribute (fork of setuptools, with added support for Python 3), not
distutils2.
distribute has been changed to directly use _frozen_importlib:
This sounds like a rather good hint we need to avoid duplicate copies.
--
Brett Cannon br...@python.org added the comment:
I think it's beyond a hint and says we need to find a solution or else other
people will run into similar issues.
And while I'm thinking about it, there is precedent for exposing modules under
a different name than they are actually installed
Eric Snow ericsnowcurren...@gmail.com added the comment:
Here's my take. No one will care about _frozen_importlib vs.
importlib._bootstrap normally, right? If __module__/__file__ says
_frozen_importlib, it's no big deal. The only time you care about the
distiction for importlib._bootstrap
Roundup Robot devn...@psf.upfronthosting.co.za added the comment:
New changeset 257cbd2fac38 by Brett Cannon in branch 'default':
Issue #13959: Re-implement imp.get_suffixes() in Lib/imp.py.
http://hg.python.org/cpython/rev/257cbd2fac38
--
nosy: +python-dev
Nick Coghlan ncogh...@gmail.com added the comment:
Uploaded new bootstrapping patch that handles the fully explicit import
machinery.
I also tweaked a couple of things so it plays nicely in terms of building an
initial version with the checked in importlib.h. Specifically: pythonrun still
Marc-Andre Lemburg m...@egenix.com added the comment:
Antoine Pitrou wrote:
Antoine Pitrou pit...@free.fr added the comment:
Code to detect whether you're running off a checkout vs. a normal
installation by looking at even more directories ? I don't
see any in getpath.c (and that's
Antoine Pitrou pit...@free.fr added the comment:
Look for pybuilddir.txt.
Oh dear. Another one of those hacks... why wasn't this done using
constants passed in by the configure script and simple string
comparison ?
How would that help distinguish between an installed Python and a
Marc-Andre Lemburg m...@egenix.com added the comment:
Antoine Pitrou wrote:
Antoine Pitrou pit...@free.fr added the comment:
Look for pybuilddir.txt.
Oh dear. Another one of those hacks... why wasn't this done using
constants passed in by the configure script and simple string
Antoine Pitrou pit...@free.fr added the comment:
The question pybuildir.txt apparently tries to solve is whether Python
is running from the build dir or not. It's not whether Python was
installed or not.
That's the same, for all we're concerned.
But pybuilddir.txt does not only solve that
Marc-Andre Lemburg m...@egenix.com added the comment:
Antoine Pitrou wrote:
Antoine Pitrou pit...@free.fr added the comment:
The question pybuildir.txt apparently tries to solve is whether Python
is running from the build dir or not. It's not whether Python was
installed or not.
Nick Coghlan ncogh...@gmail.com added the comment:
Still no patch from me, but I did create the rudiments of a shared script for
poking around at the import internals (Tools/scripts/import_diagnostics.py)
Looking at Antoine's patch, I'd be happier with it if it *didn't* mutate the
attributes
Antoine Pitrou pit...@free.fr added the comment:
Would be easier to tell distutils to install the extensions
in a fixed name dir (instead of using a platform and version
in the name) and then use that getpath.c. distutils is pretty
flexible at that :-)
Look, this is becoming very off-topic
Antoine Pitrou pit...@free.fr added the comment:
Looking at Antoine's patch, I'd be happier with it if it *didn't*
mutate the attributes of _frozen_importlib, but instead just added
importlib._bootstrap as an alias for accessing it.
I thought it would be nicer for __file__, __name__ and
Brett Cannon br...@python.org added the comment:
To answer MAL's question about startup, I benchmarked on my machine using the
normal_startup benchmark from hg.python.org/benchmarks and the bootstrap work
only caused a 5-6% slowdown in a non-debug build. If you do it in a debug build
it's
Nick Coghlan ncogh...@gmail.com added the comment:
OK, I'm leaning back towards my original preference of getting
_frozen_importlib out of the way as quickly as we can.
Specifically, I'm thinking of separating out the entry point used by
importlib.__init__ from that used by pythonrun.c, such
Antoine Pitrou pit...@free.fr added the comment:
Attached patch is an initial attempt (the reference counting on the
two modules is likely still a bit dodgy - this is my first version
that didn't segfault as I got used to the mechanics of dealing with a
frozen module, so it errs on the side
Nick Coghlan ncogh...@gmail.com added the comment:
Yes, in that you'll be able to pick up changes in _bootstrap.py *without*
having to rebuild Python.
With this in place, we could then get rid of the automatic regeneration of
importlib.h which is a complete nightmare if you ever break your
Nick Coghlan ncogh...@gmail.com added the comment:
Actually, rather than a test in test suite, we would just change the current
automatic rebuild to a Modules/Setup style 'Lib/importlib._bootstrap.py' is
newer than 'Python/importlib.h', you may need to run 'make freeze_importlib'
--
Éric Araujo mer...@netwok.org added the comment:
How do we currently tell that the interpreter is running in a checkout?
sysconfig.is_python_build()
Someone has to confirm that this works on Windows too, as I’ve been told that
not installed vs. installed is less clear on that OS.
--
Antoine Pitrou pit...@free.fr added the comment:
Actually, rather than a test in test suite, we would just change the
current automatic rebuild to a Modules/Setup style
'Lib/importlib._bootstrap.py' is newer than 'Python/importlib.h', you
may need to run 'make freeze_importlib'
-1 from me.
Nick Coghlan ncogh...@gmail.com added the comment:
The other advantage of splitting the entry points is that we can tweak Brett's
plan to make the import machinery explicit such that it happens in a separate
function that's only called from __init__.py.
That way the published hooks will
Nick Coghlan ncogh...@gmail.com added the comment:
At the very least, failing to regenerate importlib.h shouldn't be a fatal build
error. It should just run with what its got, and hopefully you will get a
working interpreter out the other end, such that you can regenerate the frozen
module on
Brett Cannon br...@python.org added the comment:
So how would you tweak the explicit work I'm doing? The code is going to rely
on sys.path_hooks and sys.meta_path being populated. I guess the frozen code
can set up initially, and then importlib simply substitutes out classes from
the frozen
Marc-Andre Lemburg m...@egenix.com added the comment:
Nick Coghlan wrote:
Nick Coghlan ncogh...@gmail.com added the comment:
At the very least, failing to regenerate importlib.h shouldn't be a fatal
build error. It should just run with what its got, and hopefully you will get
a working
Marc-Andre Lemburg m...@egenix.com added the comment:
Marc-Andre Lemburg wrote:
Marc-Andre Lemburg m...@egenix.com added the comment:
Nick Coghlan wrote:
Nick Coghlan ncogh...@gmail.com added the comment:
At the very least, failing to regenerate importlib.h shouldn't be a fatal
build
Nick Coghlan ncogh...@gmail.com added the comment:
My plan would be for the frozen version to be entirely implicit, and have only
the subsequent import of the version from disk actually modify the public hooks.
However, I realised today that my current patch would break
stdlib-from-zipfile
Antoine Pitrou pit...@free.fr added the comment:
So why the mutation? Are you that worried someone is going to import
importlib._bootstrap directly?
Well, importing importlib *does* import importlib._bootstrap, and
creates another copy of the module. importlib.__import__ is then wired
to
Antoine Pitrou pit...@free.fr added the comment:
This would also mean that changes to importlib._bootstrap would
actually take effect for user code almost immediately, *without*
rebuilding Python, as the frozen version would *only* be used to get
hold of the pure Python version.
Actually,
Marc-Andre Lemburg m...@egenix.com added the comment:
Antoine Pitrou wrote:
Antoine Pitrou pit...@free.fr added the comment:
This would also mean that changes to importlib._bootstrap would
actually take effect for user code almost immediately, *without*
rebuilding Python, as the frozen
Brett Cannon br...@python.org added the comment:
To start, I'm *not* going to make the final call on this issue's solution. I'm
inches away from importlib burnout and general integration frustration with
trying to clean up the implicit behaviour. So to prevent me from making a bad
decision I
Antoine Pitrou pit...@free.fr added the comment:
Le mardi 24 avril 2012 à 15:10 +, Brett Cannon a écrit :
Both get the same outcome but with different approaches, it's just a
question of which one is easiest to maintain.
I don't have any strong preference. Nick's proposal sounds slightly
Marc-Andre Lemburg m...@egenix.com added the comment:
test me
thod. Another option is we hide the source as _importlib or something to allow
direct importation w/o any tricks under a protected name.
Using the freeze everything approach you make things easier for the
implementation, since you
Eric Snow ericsnowcurren...@gmail.com added the comment:
would be great if we had a
command to stop module execution or code execution for a block to
make that more elegant, e.g. break at module scope :-)
I floated that proposal on python-list a while back and the reaction was mixed.
[1]
Brett Cannon br...@python.org added the comment:
I don't quite follow what you are suggesting, MAL. Are you saying to freeze
importlib.__init__ and importlib._bootstrap and somehow have improtlib.__init__
choose what to load, frozen or source?
--
Marc-Andre Lemburg m...@egenix.com added the comment:
Brett Cannon wrote:
Brett Cannon br...@python.org added the comment:
I don't quite follow what you are suggesting, MAL. Are you saying to freeze
importlib.__init__ and importlib._bootstrap and somehow have
improtlib.__init__ choose
Brett Cannon br...@python.org added the comment:
So basically if you are running in a checkout, grab the source file and compile
it manually since its location is essentially hard-coded and thus you don't
need to care about sys.path and all the other stuff required to do an import,
while
Antoine Pitrou pit...@free.fr added the comment:
That's an interesting idea. How do we currently tell that the
interpreter is running in a checkout? Is that exposed in any way to
Python code?
Look for _BUILDDIR_COOKIE in setup.py. But that's only for non-Windows
platforms (I don't think
Marc-Andre Lemburg m...@egenix.com added the comment:
Brett Cannon wrote:
Brett Cannon br...@python.org added the comment:
So basically if you are running in a checkout, grab the source file and
compile it manually since its location is essentially hard-coded and thus you
don't need to
Brett Cannon br...@python.org added the comment:
Modules/getpath.c seems to be where the C code does it when getting paths for
sys.path. So it would be possible to use that same algorithm to set some sys
attribute (e.g. in_checkout or something) much like sys.gettotalrefcount is
optional and
Antoine Pitrou pit...@free.fr added the comment:
Otherwise some directory structure check could be done (e.g. find
importlib/_bootstrap.py off of sys.path, and then see
if ../Modules/Setup or something also exists that would never show up
in an installed CPython).
Well, the directory
Brett Cannon br...@python.org added the comment:
That's why I was thinking of tying into Modules/getpath.c because I assume that
would work cross-platform. Is that incorrect?
--
___
Python tracker rep...@bugs.python.org
Antoine Pitrou pit...@free.fr added the comment:
That's why I was thinking of tying into Modules/getpath.c because I
assume that would work cross-platform. Is that incorrect?
Windows uses PC/getpathp.c, not Modules/getpath.c (with tons of
duplicate code)... So you would have to tie into both
Marc-Andre Lemburg m...@egenix.com added the comment:
Brett Cannon wrote:
Modules/getpath.c seems to be where the C code does it when getting paths for
sys.path. So it would be possible to use that same algorithm to set some sys
attribute (e.g. in_checkout or something) much like
Antoine Pitrou pit...@free.fr added the comment:
Adding more cruft to getpath.c or similar routines is just going to
slow down startup time even more...
The code is already there.
--
___
Python tracker rep...@bugs.python.org
Marc-Andre Lemburg m...@egenix.com added the comment:
Antoine Pitrou wrote:
Adding more cruft to getpath.c or similar routines is just going to
slow down startup time even more...
The code is already there.
Code to detect whether you're running off a checkout vs. a normal
installation by
Antoine Pitrou pit...@free.fr added the comment:
Code to detect whether you're running off a checkout vs. a normal
installation by looking at even more directories ? I don't
see any in getpath.c (and that's good).
Look for pybuilddir.txt.
--
___
Brett Cannon br...@python.org added the comment:
That solves the I'm in a checkout problem but it doesn't tell you necessarily
where the Lib directory is if you e.g. build from within another directory like
Python/, which places the executable and pybuilddir.txt in the current
directory.
Now
New submission from Antoine Pitrou pit...@free.fr:
This patch avoids creating a second copy of importlib._bootstrap when a first
one exists as _frozen_importlib.
This isn't perfect as it mutates the module when importlib is imported for the
first time, but I think it's better than the status
Changes by Antoine Pitrou pit...@free.fr:
Removed file: http://bugs.python.org/file25328/unique_importlib.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14657
___
Changes by Eric V. Smith e...@trueblade.com:
--
nosy: +eric.smith
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14657
___
___
Python-bugs-list
Changes by Antoine Pitrou pit...@free.fr:
Added file: http://bugs.python.org/file25329/unique_importlib.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14657
___
Antoine Pitrou pit...@free.fr added the comment:
New patch with tests.
--
Added file: http://bugs.python.org/file25330/unique_importlib2.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14657
Antoine Pitrou pit...@free.fr added the comment:
New patch also avoids calling _setup() a second time (which can be annoying
since _setup() has a list.append() call somewhere).
--
Added file: http://bugs.python.org/file25331/unique_importlib3.patch
Changes by Eric Snow ericsnowcurren...@gmail.com:
--
nosy: +eric.snow
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14657
___
___
Python-bugs-list
Brett Cannon br...@python.org added the comment:
So why the mutation? Are you that worried someone is going to import
importlib._bootstrap directly?
This also costs in development complexity because not only do you have to run
'make' to get changes to be testable, but it also leads to
Brett Cannon br...@python.org added the comment:
I should also mention that all of this becomes much less important once issue
#14605 is finished because at that point sys.meta_path and sys.path_hooks will
have _frozen_importlib objects and that will be what importlib works off of
directly.
Nick Coghlan ncogh...@gmail.com added the comment:
My preference would also be for _frozen_importlib._bootstrap to overwrite as
much evidence of itself as it can with the real one.
This would also mean that changes to importlib._bootstrap would actually take
effect for user code almost
Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com:
--
nosy: +Arfrever
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14657
___
80 matches
Mail list logo
