On Fri, 19 Aug 2016 at 09:47 Chris Angelico <ros...@gmail.com> wrote:
> On Sat, Aug 20, 2016 at 2:25 AM, Steve Dower <steve.do...@python.org>
> wrote:
> > On 19Aug2016 0910, Chris Angelico wrote:
> >>
> >> On Sat, Aug 20, 2016 at 1:26 AM, Steve Dower <steve.do...@python.org>
> >> wrote:
> >>>
> >>> Check any .pth files you can find. I suspect mpl_toolkits has some
> magic
> >>> in
> >>> it to make the namespace package work on 2.7.
> >>
> >>
> >> $ cat
> >> /usr/local/lib/python3.6/site-packages/matplotlib-1.5.1-py3.6-nspkg.pth
> >> import sys, types, os;p =
> >> os.path.join(sys._getframe(1).f_locals['sitedir'],
> >> *('mpl_toolkits',));ie =
> >> os.path.exists(os.path.join(p,'__init__.py'));m = not ie and
> >> sys.modules.setdefault('mpl_toolkits',
> >> types.ModuleType('mpl_toolkits'));mp = (m or []) and
> >> m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)
> >>
> >> Is it possible that that's the cause? And if so, should "python -I"
> >> remove /usr/local/lib/python3.6/site-packages from sys.path?
> >
> >
> > That explains the extra modules - types.py imports collections and
> > collections.abc, which would seem to account for the rest.
> >
> > I'm not sure whether that's causing the problem or not, but temporarily
> > renaming that pth file so it isn't picked up may help you track down the
> > test issue.
>
> Sure enough, that did fix the problem - that is to say, renaming it
> did allow test_site to pass. Thanks!
>
> > "python -S" is the only way to avoid .pth files, but that's a pretty
> heavy
> > hammer.
>
> Hmmmmmmm. So the question is, what is this test testing?
It's making sure people who work on the modules that are imported during
startup don't accidentally add another module dependency to the startup
sequence. Since module imports impact startup time we have historically
worked hard to minimize import dependencies for startup (hence things like
certain functions in the os module doing imports within a function even
though we discourage that practice in general).
> For
> reference, here's the code:
>
> class StartupImportTests(unittest.TestCase):
>
> def test_startup_imports(self):
> # This tests checks which modules are loaded by Python when it
> # initially starts upon startup.
> popen = subprocess.Popen([sys.executable, '-I', '-v', '-c',
> 'import sys; print(set(sys.modules))'],
> stdout=subprocess.PIPE,
> stderr=subprocess.PIPE)
> stdout, stderr = popen.communicate()
> stdout = stdout.decode('utf-8')
> stderr = stderr.decode('utf-8')
> modules = eval(stdout)
>
> self.assertIn('site', modules)
>
> # http://bugs.python.org/issue19205
> re_mods = {'re', '_sre', 'sre_compile', 'sre_constants',
> 'sre_parse'}
> # _osx_support uses the re module in many placs
> if sys.platform != 'darwin':
> self.assertFalse(modules.intersection(re_mods), stderr)
> # http://bugs.python.org/issue9548
> self.assertNotIn('locale', modules, stderr)
> if sys.platform != 'darwin':
> # http://bugs.python.org/issue19209
> self.assertNotIn('copyreg', modules, stderr)
> # http://bugs.python.org/issue19218>
> collection_mods = {'_collections', 'collections', 'functools',
> 'heapq', 'itertools', 'keyword', 'operator',
> 'reprlib', 'types', 'weakref'
> }.difference(sys.builtin_module_names)
> self.assertFalse(modules.intersection(collection_mods), stderr)
>
>
> It asserts that 'site' is in the list of loaded modules, so simply
> adding '-S' to the subpython parameters doesn't work. But this test,
> as written, depends on the behaviour of arbitrary third-party modules
> installed with pip, which seems wrong for a standard library test.
> AIUI this is trying to catch changes that accidentally cause large
> modules to be loaded on startup, correct? Which is all well and good,
> but Python shouldn't have test failures due to matplotlib being
> installed.
>
> I see two (or three) ways to deal with this.
>
> 1) Declare it a bug in matplotlib, and demand that .pth files not
> import anything that wouldn't otherwise be imported.
>
> 2) Change the test to add '-S' and remove the assertion about 'site'
> being present; this means that site.py itself is no longer covered by
> the test.
>
> 3) Accept the status quo: this test can fail if .pth files are present.
>
> Incidentally, where's it documented that .pth files can be executed?
It's not in the site.py docs [1] or docstring, only in the addpackage
> function within that (which I found by browsing the source).
Actually it is documented in the site documentation: "Lines starting with
import (followed by space or tab) are executed". It's subtle, but
"executed" literally means execution in this context; the line is passed
into exec():
https://github.com/python/cpython/blob/b505c47044fb5b8f03b9708ee1f210a4c935baf5/Lib/site.py#L167
.
It's an abuse that e.g. setuptools relies on to make eggs work.
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
