[issue26587] Possible duplicate entries in sys.path if .pth files are used with zip's

[Brett Cannon]

Brett Cannon added the comment:

I simplified Wolfgang's patch by simply using os.path.exists(). That eliminated 
the one place where os.path.isdir() was in use that was too specific to 
directories where files were reasonable to expect.

I also quickly tweaked the docs for the site  module in 3.5 to not promise that 
files would work.

If you think there is still an issue with keeping things tied together, 
SilentGhost, feel free to open another issue to track it.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26587] Possible duplicate entries in sys.path if .pth files are used with zip's

[Roundup Robot]

Roundup Robot added the comment:

New changeset 09dc97edf454 by Brett Cannon in branch '3.5':
Issue #26587: Remove an incorrect statement from the docs
https://hg.python.org/cpython/rev/09dc97edf454

New changeset 94d5c57ee835 by Brett Cannon in branch 'default':
Merge w/ 3.5 for issue #26587
https://hg.python.org/cpython/rev/94d5c57ee835

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26587] Possible duplicate entries in sys.path if .pth files are used with zip's

[Roundup Robot]

Roundup Robot added the comment:

New changeset bd1af1a97c2e by Brett Cannon in branch 'default':
Issue #26587: Allow .pth files to specify file paths as well as
https://hg.python.org/cpython/rev/bd1af1a97c2e

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26587] Possible duplicate entries in sys.path if .pth files are used with zip's

[Brett Cannon]

Changes by Brett Cannon :


--
assignee:  -> brett.cannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26587] Possible duplicate entries in sys.path if .pth files are used with zip's

[Wolfgang Langner]

Wolfgang Langner added the comment:

Ok, I implemented point 3.
Check if it is a dir or file and makepath only in this case.
All other entries are added unmodified to the set.

Added a test case also for an URL path.

I think duplicate detection is now improved and it should break nothing.

--
Added file: http://bugs.python.org/file42296/site2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26587] Possible duplicate entries in sys.path if .pth files are used with zip's

[SilentGhost]

SilentGhost added the comment:

I still think my fix is more appropriate as it ensures that known_paths and 
sys.path stay connected somehow.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26587] Possible duplicate entries in sys.path if .pth files are used with zip's

[Brett Cannon]

Brett Cannon added the comment:

And the code under discussion can be found at 
https://hg.python.org/cpython/file/default/Lib/site.py#l133

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26587] Possible duplicate entries in sys.path if .pth files are used with zip's

[Brett Cannon]

Brett Cannon added the comment:

Unfortunately you can't simply remove that directory check because you don't 
want to blindly normalize case. If someone put some token value on sys.path for 
their use that was case-sensitive even on a case-insensitive OS then the 
proposed change would break those semantics (e.g. if someone put a URL in 
sys.path for a REST-based importer).

The possibilities I see are:

1. Don't change anything; duplicate entries don't really hurt anything
2. Remove duplicate entries, but only normalize case for directories
3. Remove duplicate entries, but normalize case for anything that points to 
something on the filesystem (i.e. both directories and files)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26587] Possible duplicate entries in sys.path if .pth files are used with zip's

[Wolfgang Langner]

Wolfgang Langner added the comment:

I think a fix for 3.6 only is ok, because it changes behaviour.
But this is only an internal function with a "_".

Should I add a test with a temporary created pth file?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26587] Possible duplicate entries in sys.path if .pth files are used with zip's

[Wolfgang Langner]

Wolfgang Langner added the comment:

Extended unit test for the issue and patch for site.py.

--
Added file: http://bugs.python.org/file42225/site.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26587] Possible duplicate entries in sys.path if .pth files are used with zip's

[SilentGhost]

SilentGhost added the comment:

Thanks for this, Mandeep. I don't think it is entirely the same issue that 
Wolfgang is describing. He's particularly concerned about the clash of .zip 
files from the sys.path with ones coming from .pth files for example. And while 
the proposed fix would resolve the issue, I don't feel it would be entirely 
correct. For the record, I'm not even sure sys.path is guaranteed to be contain 
no duplicates, at least I wasn't able to find a statement to that effect in the 
docs. In any case, I'm adding Brett here who was seemingly the last one to 
touch that bit of code.

I'm also attaching a path with what I think is a more appropriate and smaller 
fix.

--
keywords: +patch
nosy: +brett.cannon
stage: test needed -> patch review
Added file: http://bugs.python.org/file4/issue26587.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26587] Possible duplicate entries in sys.path if .pth files are used with zip's

[Mandeep Singh Grang]

Mandeep Singh Grang added the comment:

Here is a testcase to reproduce the issue:

> cat test.py
import site, sys
site.addsitedir('/foo/bar')
print (sys.path)

This prints just a single instance of '/foo/bar':

['/local/mnt/workspace/python/tst', '/foo/bar', '/usr/local/lib/python36.zip', 
'/local/mnt/workspace/python/src/Lib', 
'/local/mnt/workspace/python/src/Lib/plat-linux', 
'/local/mnt/workspace/python/build/build/lib.linux-x86_64-3.6-pydebug']

Now if we explicitly set PYTHONPATH to include '/foo/bar'
> export PYTHONPATH=/foo/bar

and then run test.py here is the output:

['/local/mnt/workspace/python/tst', '/foo/bar', '/usr/local/lib/python36.zip', 
'/local/mnt/workspace/python/src/Lib', 
'/local/mnt/workspace/python/src/Lib/plat-linux', 
'/local/mnt/workspace/python/build/build/lib.linux-x86_64-3.6-pydebug', 
'/foo/bar']

We see that there are duplicate entries for '/foo/bar'.

As Wolfgang rightly said the issue comes from the check for
"if os.path.isdir(dir)" inside _init_pathinfo() in site.py.
On removing this check I no longer see the duplicate entry for '/foo/bar'.

But since this is the first bug I am looking at I am not sure of the 
implications of removing this check. Can someone please confirm that what I see 
is indeed a failing test case, or is this the intended behavior?

Thanks,
Mandeep

--
nosy: +Mandeep Singh Grang

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26587] Possible duplicate entries in sys.path if .pth files are used with zip's

[Wolfgang Langner]

New submission from Wolfgang Langner:

In site.py there is the internal function _init_pathinfo() This function builds 
a set of path entries from sys.path. This is used to avoid duplicate entries in 
sys.path.
But this function has a check if it is a directory with os.path.isdir(...). All 
this is fine as long as someone has a .zip file in sys.path or a zipfile 
subpath. Then the path entry is not part of the set. With this duplicate 
detection with none directories does not work.

The fix is as simple as removing the os.path.isdir(...) line and fixing the 
indent. Also the docstring should be modified.

Detected by using this function in a project reusing addsitedir(...) 
functionality to add another path with .pth processing.

--
components: Library (Lib)
messages: 261958
nosy: tds333
priority: normal
severity: normal
status: open
title: Possible duplicate entries in sys.path if .pth files are used with zip's
type: enhancement
versions: Python 2.7, Python 3.4, Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26587] Possible duplicate entries in sys.path if .pth files are used with zip's

[SilentGhost]

SilentGhost added the comment:

Could you provide a code example of your using addsitedir that results in 
duplicates?

--
nosy: +SilentGhost
stage:  -> test needed
versions:  -Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com