[issue29592] abs_paths() in site.py is slow

[INADA Naoki]

INADA Naoki added the comment:


New changeset 2e4e011795d26cab1a3843383d0539c12fea2458 by INADA Naoki in branch 
'master':
bpo-29592: site: skip abs_paths() when it's redundant (GH-167)
https://github.com/python/cpython/commit/2e4e011795d26cab1a3843383d0539c12fea2458


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29592] abs_paths() in site.py is slow

[INADA Naoki]

Changes by INADA Naoki :


--
resolution:  -> fixed
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29592] abs_paths() in site.py is slow

[Gregory P. Smith]

Changes by Gregory P. Smith :


--
nosy: +gregory.p.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29592] abs_paths() in site.py is slow

[INADA Naoki]

INADA Naoki added the comment:

I got it.
removeduppaths() may change relpath in sys.path to absolute path.
abs_paths() changes __file__ and __cached__ for consistency with the changed 
sys.path.

I updated PR 167 to call abs_paths() only if removeduppaths() modified sys.path.
Strictly speaking, abs_paths() is required only when removeduppaths() converted 
relpath to absolute path.

But because duplicated paths are rare too, I think this approach is practical 
enough.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29592] abs_paths() in site.py is slow

[Nick Coghlan]

Nick Coghlan added the comment:

Note that "os" doesn't get imported normally, it gets injected as part of the 
importlib bootstrapping process (since _bootstrap_external.py needs the os 
module in order to work).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29592] abs_paths() in site.py is slow

[Nick Coghlan]

Nick Coghlan added the comment:

CI failure indicating it isn't redundant, but could still potentially be made 
faster since non-absolute paths should be relatively rare now:

==

FAIL: test_s_option (test.test_site.HelperFunctionsTests)

--

Traceback (most recent call last):

  File "/home/travis/build/python/cpython/Lib/test/test_site.py", line 173, in 
test_s_option

self.assertIn(usersite, sys.path)

AssertionError: '/home/travis/.local/lib/python3.7/site-packages' not found in 
['', '/usr/local/lib/python37.zip', '/home/travis/build/python/cpython/Lib', 
'/home/travis/build/python/cpython/build/lib.linux-x86_64-3.7-pydebug']

==

FAIL: test_abs_paths (test.test_site.ImportSideEffectTests)

--

Traceback (most recent call last):

  File "/home/travis/build/python/cpython/Lib/test/test_site.py", line 365, in 
test_abs_paths

.format(os__file__.decode('ascii')))

AssertionError: False is not true : expected absolute path, got ../../Lib/os.py

--

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29592] abs_paths() in site.py is slow

[Nick Coghlan]

Nick Coghlan added the comment:

Aye, I agree it should be redundant now - import system should always be making 
__file__ and __cached__ absolutely at import time these days.

So +1 for dropping this from 3.7 and getting a bit of startup time back.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29592] abs_paths() in site.py is slow

[INADA Naoki]

Changes by INADA Naoki :


--
pull_requests: +131

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29592] abs_paths() in site.py is slow

[INADA Naoki]

INADA Naoki added the comment:

$ python2 -S
Python 2.7.12+ (default, Sep 17 2016, 12:08:02) 
[GCC 6.2.0 20160914] on linux2
>>> import x
>>> x.__file__
'x.py'

$ python3 -S
Python 3.6.0 (default, Dec 30 2016, 20:49:54) 
[GCC 6.2.0 20161005] on linux
>>> import x
>>> x.__file__
'/home/inada-n/x.py'

I think we can remove `abs_paths()` in site.py, thanks to 
_frozen_importlib_external.

I added all import experts to nosy list.
Please give me advice.

--
nosy: +brett.cannon, eric.snow, fdrake, ncoghlan

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29592] abs_paths() in site.py is slow

[INADA Naoki]

INADA Naoki added the comment:

https://github.com/python/cpython/blob/master/Lib/importlib/_bootstrap_external.py#L1089-L1095

While '' in sys.path, it is converted to `getcwd()` before calling PathHook.
Is there any chance relative / not normalized path is in sys.path before site 
is loaded?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29592] abs_paths() in site.py is slow

[INADA Naoki]

INADA Naoki added the comment:

path normalization was added by
https://github.com/python/cpython/commit/38cb9f1f174415d3b37fbaeb5d152d65525839d2

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29592] abs_paths() in site.py is slow

[INADA Naoki]

New submission from INADA Naoki:

abs_paths() is the another most slow parts of site module.
We can speedup it if we have C implementation of os.path.abspath() or 
abs_paths().

But before trying it, is abs_paths() really needed for now?
It only rewrite `__file__` and `__cached__` of modules imported before site is 
imported.
What is difference between modules loaded befere / after site module?

Here is profile output. sysconfig dependency is removed by #29585 patch, and 
pstat is modified
to show time in ms instead of sec.

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  3/10.0040.0012.5252.525 {built-in method builtins.exec}
10.0030.0032.5242.524 x.py:1()
10.0100.0101.8061.806 site.py:555(main)
  4/30.0220.0051.1790.393 :958(_find_and_load)
  4/30.0170.0041.1100.370 :931(_find_and_load_unlocked)
10.1950.1950.9280.928 site.py:99(abs_paths)
  1080.0980.0010.7760.007 posixpath.py:367(abspath)
40.0350.0090.6470.162 :861(_find_spec)
40.0050.0010.5890.147 :1150(find_spec)
40.0430.0110.5840.146 :1118(_get_spec)
  2/10.0120.0060.5570.557 :641(_load_unlocked)
  2/10.0060.0030.5110.511 :673(exec_module)
  1080.4610.0040.4930.005 posixpath.py:329(normpath)
   160.1500.0090.4530.028 :1234(find_spec)

--
components: Library (Lib)
messages: 288019
nosy: inada.naoki
priority: normal
severity: normal
status: open
title: abs_paths() in site.py is slow
type: performance
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com