[issue34395] memory leaks in error handling in csv and pickle modules

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset ae1f0127298b1193618062fd0c8a3b434656e780 by Serhiy Storchaka 
(Sergey Fedoseev) in branch '2.7':
bpo-34395: Fix memory leaks caused by incautious usage of PyMem_Realloc(). 
(GH-8785)
https://github.com/python/cpython/commit/ae1f0127298b1193618062fd0c8a3b434656e780


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6700] inspect.getsource() returns incorrect source lines at the module level

[Aivar Annamaa]

Change by Aivar Annamaa :


--
title: inspect.getsource() returns incorrect source lines -> 
inspect.getsource() returns incorrect source lines at the module level
versions: +Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6700] inspect.getsource() returns incorrect source lines

[Aivar Annamaa]

Change by Aivar Annamaa :


--
nosy: +Aivar.Annamaa

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24218] Also support SMTPUTF8 in smtplib's send_message method.

[Jens Troeger]

Jens Troeger  added the comment:

Thanks David: PR on Github (which is R/O) or where should I submit to?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34410] Segfault/TimeoutError: itertools.tee of multiprocessing.pool.imap_unordered

[Carlo Rosati]

Carlo Rosati  added the comment:

`for i in itertools.count()` in the first implementation I posted should be 
`while True`. I was using that for debugging.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34410] Segfault/TimeoutError: itertools.tee of multiprocessing.pool.imap_unordered

[Carlo Rosati]

Carlo Rosati  added the comment:

I've actually written a few workarounds that should be considered a 
multiprocessing specific tee function. I need feedback/critique on these. 
Hopefully we can all agree on one solution that's the best. It is unfortunate 
that the multiprocessing manager does not provide a dequeue.

The first one I wrote uses a managed list. 

def multiprocessing_tee(iterable, n=2):
"""Write a multiprocessing safe itertools.tee"""
it = iter(iterable)
m = multiprocessing.Manager()
mylock = m.Lock()
lists = [m.list() for i in range(n)]
def gen(local_list):
for i in itertools.count():
with mylock:
if not local_list: # when the local list is empty
newval = next(it)  # fetch a new value and
for l in lists:# load it to all the lists
l.append(newval)
yield local_list.pop(0)
return tuple(gen(l) for l in lists)

The second two implementations use queues. 

def multiprocessing_tee_q(iterable, n=2):
"""Write a multiprocessing safe itertools.tee"""
it = iter(iterable)
m = multiprocessing.Manager()
lock = m.Lock()
queues = [m.Queue(-1) for _ in range(n)] # -1 means infinite maxsize (so 
puts won't block)
def gen(myqueue):
while True:
with lock: # no one else touches anything
try:
newval = myqueue.get_nowait()
except Queue.Empty:
newval = next(it)
for q in queues:
q.put(newval)
newval = myqueue.get()
yield newval
return tuple(gen(q) for q in queues)

class Sentinel(object):
"""used as Queue Sentinel"""

def multiprocessing_tee_q2(iterable, n=2):
"""Write a multiprocessing safe itertools.tee"""
it = iter(iterable)
m = multiprocessing.Manager()
lock = m.Lock()
queues = [m.Queue(-1) for _ in range(n)] # -1 means infinite maxsize (so 
puts won't block)
def gen(myqueue):
while True:
try:
retval = myqueue.get_nowait()
except Queue.Empty:
# what happens if the other process puts last item in my queue 
before i get lock?
with lock: # no one else touches anything
try:
newval = next(it)
except StopIteration:
newval = Sentinel
for q in queues:
q.put(newval)
retval = myqueue.get()
if retval is Sentinel:
raise StopIteration
yield retval
return tuple(gen(q) for q in queues)

I'm just throwing out my sketches here. I'm hoping the more experienced here 
can weigh in on these implementations.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34416] PyCapsule_Import seems to release the GIL without acquiring it

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

Summary of how I see this dump:

1) Thread 2 tries to get the GIL.
2) Thread 3 tries to get the GIL.
3 Gil acquired! (0)
4) Thread 2 drops the GIL, which means that thread 2 managed to get it before.
5) Gil acquired! (1)
6) Thread 2 tries to get the GIL.
7) Thread 3 drops the GIL which means that thread 2 managed to get the GIL.

... thread 2 and 3 battle for the GIL

8) Gil released! (1)


... thread 2 and 3 battle for the GIL


9) Gil released! (0)

It does not seem to me that two threads have the GIL at the same time.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34414] Absolute imports conflict with local files

[Eric V. Smith]

Eric V. Smith  added the comment:

See https://docs.python.org/3/tutorial/modules.html#the-module-search-path

Calling this a "hack", and "crazy", is not the way to get volunteers to help 
you with your issue. Please show a little more tolerance, please.

--
nosy: +eric.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34217] windows: cross compilation fails due to headers with uppercase

[Steve Dower]

Steve Dower  added the comment:

> The current situation is that 100% of the windows includes are lowercase and 
> allow cross compilation.

Please be precise with what you are saying - the "current situation" is only so 
because you proposed changes and got them merged without proper discussion, and 
"allow cross compilation" with only one alternate toolset. Pretending this is 
the status quo is dishonest, so please don't insult me like that.

> if mingw-w64 would change to camelcase headers, it would break backwards 
> compatibility

I'm not recommending camel case - I'm recommending matching the casing used for 
each file in the official SDK (which I agree is a weird mix, but there are 
certainly compatibility reasons involved in keeping them this way). And yes, it 
would be a compatibility break for the clones of the SDK, but to a large extent 
that is their fault for choosing different casing in the first place, so I'm 
only a little bit sympathetic to that argument.


What I'm mostly opposed to is the very weak bug report and the random selection 
of changes that came with it. Failing to compile on an unsupported toolset is 
not an immediate "must-fix" issue, and it really ought to go through the 
affected platform experts so they can confirm acceptable impact on developers 
who are using the supported tools. There also needs to be this central point so 
that, assuming we decide to keep them this way, the next person who comes in 
and complains that the casing doesn't match the actual files is given the 
correct explanation - otherwise we'll keep switching them back and forth 
forever.

You're also likely to face a maintenance burden for this change, since there is 
currently no rule or validation that will prevent people from adding properly 
cased includes in the future (as I hinted, most IDEs on Windows will 
autocorrect filename casing for headers). If you want one, propose a change to 
PEP 7 on python-dev, and if it's approved then you can add a build step to 
validate. Otherwise, you'll have to track changes that are made and fix them as 
necessary. Without an explicit rule, our default position is "whatever the 
native target platform/tools prefer".

I hope that explains the position a bit better, and why I push back against 
changes with insufficient justification being provided. At this point, I'm not 
going to revert these particular changes, as the cases where they will affect 
native developers are startlingly few these days, but I'm also explicitly 
saying that this does not mean open-season for all the fixes required for mingw 
to be happy. Each one will be taken on its merits (primarily compatibility and 
maintainability), at least until someone has committed to fully support the 
toolset.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34416] PyCapsule_Import seems to release the GIL without acquiring it

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
Removed message: https://bugs.python.org/msg323627

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34416] PyCapsule_Import seems to release the GIL without acquiring it

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

Breaking in take_gil and drop_gil I get this trace:

Thread 2 "gil" hit Breakpoint 1, 0x5559563d in acquire_gil ()
Thread 3 "gil" hit Breakpoint 1, 0x5559563d in acquire_gil ()
Waiting for GIL (0)
Waiting for GIL (1)
Thread 2 "gil" hit Breakpoint 2, take_gil (tstate=tstate@entry=0x7f40) 
at Python/ceval_gil.h:192
Thread 3 "gil" hit Breakpoint 2, take_gil (tstate=tstate@entry=0x7fffe8000b30) 
at Python/ceval_gil.h:192
Gil acquired! (0)
Thread 2 "gil" hit Breakpoint 3, drop_gil (tstate=tstate@entry=0x7f40) 
at Python/ceval_gil.h:150
Gil acquired! (1)
Thread 2 "gil" hit Breakpoint 2, take_gil (tstate=tstate@entry=0x7f40) 
at Python/ceval_gil.h:192
Thread 3 "gil" hit Breakpoint 3, drop_gil (tstate=tstate@entry=0x7fffe8000b30) 
at Python/ceval_gil.h:150
Thread 2 "gil" hit Breakpoint 3, drop_gil (tstate=tstate@entry=0x7f40) 
at Python/ceval_gil.h:150
Thread 3 "gil" hit Breakpoint 2, take_gil (tstate=tstate@entry=0x7fffe8000b30) 
at Python/ceval_gil.h:192
Thread 3 "gil" hit Breakpoint 3, drop_gil (tstate=tstate@entry=0x7fffe8000b30) 
at Python/ceval_gil.h:150
Thread 3 "gil" hit Breakpoint 2, take_gil (tstate=tstate@entry=0x7fffe8000b30) 
at Python/ceval_gil.h:192
Thread 3 "gil" hit Breakpoint 3, drop_gil (tstate=tstate@entry=0x7fffe8000b30) 
at Python/ceval_gil.h:150
Thread 3 "gil" hit Breakpoint 2, take_gil (tstate=tstate@entry=0x7fffe8000b30) 
at Python/ceval_gil.h:192
Thread 3 "gil" hit Breakpoint 3, drop_gil (tstate=tstate@entry=0x7fffe8000b30) 
at Python/ceval_gil.h:150
Thread 3 "gil" hit Breakpoint 2, take_gil (tstate=tstate@entry=0x7fffe8000b30) 
at Python/ceval_gil.h:192
Thread 2 "gil" hit Breakpoint 2, take_gil (tstate=tstate@entry=0x7f40) 
at Python/ceval_gil.h:192
Thread 3 "gil" hit Breakpoint 3, drop_gil (tstate=tstate@entry=0x7fffe8000b30) 
at Python/ceval_gil.h:150
Thread 3 "gil" hit Breakpoint 2, take_gil (tstate=tstate@entry=0x7fffe8000b30) 
at Python/ceval_gil.h:192
Thread 2 "gil" hit Breakpoint 3, drop_gil (tstate=tstate@entry=0x7f40) 
at Python/ceval_gil.h:150
Thread 2 "gil" hit Breakpoint 2, take_gil (tstate=tstate@entry=0x7f40) 
at Python/ceval_gil.h:192
Thread 3 "gil" hit Breakpoint 3, drop_gil (tstate=0x0) at Python/ceval_gil.h:150
Thread 2 "gil" hit Breakpoint 3, drop_gil (tstate=tstate@entry=0x7f40) 
at Python/ceval_gil.h:150
Thread 2 "gil" hit Breakpoint 2, take_gil (tstate=tstate@entry=0x7f40) 
at Python/ceval_gil.h:192
Thread 2 "gil" hit Breakpoint 3, drop_gil (tstate=tstate@entry=0x7f40) 
at Python/ceval_gil.h:150
Thread 2 "gil" hit Breakpoint 2, take_gil (tstate=tstate@entry=0x7f40) 
at Python/ceval_gil.h:192
Thread 2 "gil" hit Breakpoint 3, drop_gil (tstate=tstate@entry=0x7f40) 
at Python/ceval_gil.h:150
Thread 2 "gil" hit Breakpoint 2, take_gil (tstate=tstate@entry=0x7f40) 
at Python/ceval_gil.h:192
Thread 2 "gil" hit Breakpoint 3, drop_gil (tstate=tstate@entry=0x7f40) 
at Python/ceval_gil.h:150
Thread 2 "gil" hit Breakpoint 2, take_gil (tstate=tstate@entry=0x7f40) 
at Python/ceval_gil.h:192
Thread 2 "gil" hit Breakpoint 3, drop_gil (tstate=0x0) at Python/ceval_gil.h:150
Gil released! (0)


which seems normal to me. This is tested on current master.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34416] PyCapsule_Import seems to release the GIL without acquiring it

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

Breaking in take_gil and drop_gil I get this trace:

Thread 2 "gil" hit Breakpoint 1, 0x5559563d in acquire_gil ()
Thread 3 "gil" hit Breakpoint 1, 0x5559563d in acquire_gil ()
Waiting for GIL (0)
Waiting for GIL (1)
Thread 2 "gil" hit Breakpoint 2, take_gil (tstate=tstate@entry=0x7f40) 
at Python/ceval_gil.h:192
Thread 3 "gil" hit Breakpoint 2, take_gil (tstate=tstate@entry=0x7fffe8000b30) 
at Python/ceval_gil.h:192
Gil acquired! (0)
Thread 2 "gil" hit Breakpoint 3, drop_gil (tstate=tstate@entry=0x7f40) 
at Python/ceval_gil.h:150
Gil acquired! (1)
Thread 2 "gil" hit Breakpoint 2, take_gil (tstate=tstate@entry=0x7f40) 
at Python/ceval_gil.h:192
Thread 3 "gil" hit Breakpoint 3, drop_gil (tstate=tstate@entry=0x7fffe8000b30) 
at Python/ceval_gil.h:150
Thread 2 "gil" hit Breakpoint 3, drop_gil (tstate=tstate@entry=0x7f40) 
at Python/ceval_gil.h:150
Thread 3 "gil" hit Breakpoint 2, take_gil (tstate=tstate@entry=0x7fffe8000b30) 
at Python/ceval_gil.h:192
Thread 3 "gil" hit Breakpoint 3, drop_gil (tstate=tstate@entry=0x7fffe8000b30) 
at Python/ceval_gil.h:150
Thread 3 "gil" hit Breakpoint 2, take_gil (tstate=tstate@entry=0x7fffe8000b30) 
at Python/ceval_gil.h:192
Thread 3 "gil" hit Breakpoint 3, drop_gil (tstate=tstate@entry=0x7fffe8000b30) 
at Python/ceval_gil.h:150
Thread 3 "gil" hit Breakpoint 2, take_gil (tstate=tstate@entry=0x7fffe8000b30) 
at Python/ceval_gil.h:192
Thread 3 "gil" hit Breakpoint 3, drop_gil (tstate=tstate@entry=0x7fffe8000b30) 
at Python/ceval_gil.h:150
Thread 3 "gil" hit Breakpoint 2, take_gil (tstate=tstate@entry=0x7fffe8000b30) 
at Python/ceval_gil.h:192
Thread 2 "gil" hit Breakpoint 2, take_gil (tstate=tstate@entry=0x7f40) 
at Python/ceval_gil.h:192
Thread 3 "gil" hit Breakpoint 3, drop_gil (tstate=tstate@entry=0x7fffe8000b30) 
at Python/ceval_gil.h:150
Thread 3 "gil" hit Breakpoint 2, take_gil (tstate=tstate@entry=0x7fffe8000b30) 
at Python/ceval_gil.h:192
Thread 2 "gil" hit Breakpoint 3, drop_gil (tstate=tstate@entry=0x7f40) 
at Python/ceval_gil.h:150
Thread 2 "gil" hit Breakpoint 2, take_gil (tstate=tstate@entry=0x7f40) 
at Python/ceval_gil.h:192
Thread 3 "gil" hit Breakpoint 3, drop_gil (tstate=0x0) at Python/ceval_gil.h:150
Thread 2 "gil" hit Breakpoint 3, drop_gil (tstate=tstate@entry=0x7f40) 
at Python/ceval_gil.h:150
 Gil released! (1)
Thread 2 "gil" hit Breakpoint 2, take_gil (tstate=tstate@entry=0x7f40) 
at Python/ceval_gil.h:192
Thread 2 "gil" hit Breakpoint 3, drop_gil (tstate=tstate@entry=0x7f40) 
at Python/ceval_gil.h:150
Thread 2 "gil" hit Breakpoint 2, take_gil (tstate=tstate@entry=0x7f40) 
at Python/ceval_gil.h:192
Thread 2 "gil" hit Breakpoint 3, drop_gil (tstate=tstate@entry=0x7f40) 
at Python/ceval_gil.h:150
Thread 2 "gil" hit Breakpoint 2, take_gil (tstate=tstate@entry=0x7f40) 
at Python/ceval_gil.h:192
Thread 2 "gil" hit Breakpoint 3, drop_gil (tstate=tstate@entry=0x7f40) 
at Python/ceval_gil.h:150
Thread 2 "gil" hit Breakpoint 2, take_gil (tstate=tstate@entry=0x7f40) 
at Python/ceval_gil.h:192
Thread 2 "gil" hit Breakpoint 3, drop_gil (tstate=0x0) at Python/ceval_gil.h:150
Gil released! (0)


which seems normal to me. This is tested on current master.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20849] add exist_ok to shutil.copytree

[Giampaolo Rodola']

Giampaolo Rodola'  added the comment:

I think this feature request is reasonable for 2 reasons:

1) As it stands if dst directory exists copytree() cannot be used. The only 
possible workaround is using rmtree(dst) first, but that doesn't seem to make 
much sense. The change may look extremely easy but the fact that copytree() 
cannot be used in a specific circumstance justifies the cost of introducing a 
new argument, basically because there is no other way around it.

2) "cp -r" does this by default, which is probably indicative (I don't think 
the default should be True though).

I think the new argument should be called "dirs_exists_ok" instead of 
"exists_ok" though, so that it's clear that the behavior does not affect files.

[ R. David Murray ]
> I do not think that *just* making it OK for the destination directory to 
> exist would be a good API.

I don't think that refusing (or allowing) to copy existing files is a common 
enough use case to justify the addition of a new parameter (note: also "cp" 
does not provide this use case). If such a  behavior is desired the user can 
simply provide its own custom "copy_function".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34217] windows: cross compilation fails due to headers with uppercase

[Erik Janssens]

Erik Janssens  added the comment:

As I understand it, if mingw-w64 would change to camelcase headers, it would 
break backwards compatibility, since a lot of windows source code uses file 
names in includes that differ from those on this, which works in MSVC, so it 
goes unnoticed.  There is no good solution.

Even if camelcase headers would be available, the CPython source code would 
break in much more places.

I agree that it would be strange to have a style rule for a specific toolset, 
however mixing different styles is even stranger.

The current situation is that 100% of the windows includes are lowercase and 
allow cross compilation.  I would tend to see that as an advantage.

If a more comprehensive solution would be available, that would be ideal of 
course.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34416] PyCapsule_Import seems to release the GIL without acquiring it

[Paul Ganssle]

Paul Ganssle  added the comment:

Using a modified version of Python 3.7.0 that prints "Releasing GIL" whenever 
PyGILState_Release, I get this:

Waiting for GIL (0)
Gil acquired! (0)
Waiting for GIL (1)
Gil acquired! (1)
Releasing GIL
Gil released! (0)
Releasing GIL
Gil released! (1)

So whatever is allowing the GIL to be acquired twice is not calling 
PyGILState_Release.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8478] tokenize.untokenize first token missing failure case

[Berker Peksag]

Change by Berker Peksag :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34417] imp.find_module reacts badly to iterator

[Phillip M. Feldman]

New submission from Phillip M. Feldman :

`imp.find_module` goes down in flames if one tries to pass an iterator rather 
than a list of folders.  Firstly, the message that it produces is somewhat 
misleading:

   RuntimeError: sys.path must be a list of directory names

Secondly, it would be helpful if one could pass an iterator. I'm thinking in 
particular of the situation where one wants to import something from a large 
folder tree, and the module in question is likely to be found early in the 
search process, so that it is more efficient to explore the folder tree 
incrementally.

--
components: Library (Lib)
messages: 323623
nosy: phillip.m.feld...@gmail.com
priority: normal
severity: normal
status: open
title: imp.find_module reacts badly to iterator
type: behavior
versions: Python 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34217] windows: cross compilation fails due to headers with uppercase

[Steve Dower]

Steve Dower  added the comment:

Have you posted to their mailing list to ask why they use different casing from 
the official SDK and whether they would consider supporting both (via symlink, 
presumably)?

I think it would be strange for CPython to add a style rule for a non-supported 
toolset. The supported tooling will default to using the correct case for the 
header files.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34416] PyCapsule_Import seems to release the GIL without acquiring it

[Eric N. Vander Weele]

Change by Eric N. Vander Weele :


--
nosy: +ericvw

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34217] windows: cross compilation fails due to headers with uppercase

[Erik Janssens]

Erik Janssens  added the comment:

The lowercase headers are indeed part of the mingw-w64 SDK, which is the most 
convenient option to cross compile on a linux host for a windows target.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34416] PyCapsule_Import seems to release the GIL without acquiring it

[Paul Ganssle]

New submission from Paul Ganssle :

I was recently debugging some multithreaded Rust code that was deadlocking, and 
I tracked it down to what I'm fairly certain is a bug somewhere in 
PyCapsule_Import, where it seems that PyCapsule_Import releases the GIL without 
first acquiring it.

I've attached a MWE of a multi-threaded application that is able to 
simultaneously acquire the GIL twice. The relevant portion is here:

void *acquire_gil(void *arg) {
bool import = ((arg_t *)arg)->import;
int n = ((arg_t *)arg)->id;

printf("Waiting for GIL (%d)\n", n);
int gstate = PyGILState_Ensure();
printf("Gil acquired! (%d)\n", n);
usleep(125000);
if (import) {
PyCapsule_Import(CAPSULE_NAME, 0);
}
usleep(125000);
PyGILState_Release(gstate);
printf("Gil released! (%d)\n", n);
return NULL;
}

If you run it with `./gil` such that the PyCapsule_Import call is never 
reached, you get:

Waiting for GIL (0)
Gil acquired! (0)
Waiting for GIL (1)
Gil released! (0)
Gil acquired! (1)
Gil released! (1)

However, if you run with `./gil import` such that PyCapsule_Import is reached, 
you get (emphasis mine):

Waiting for GIL (0)
Gil acquired! (0)
Waiting for GIL (1)
**Gil acquired! (1)**
**Gil released! (1)**
Gil released! (0)

For convenience sake, I have created a small repo with a make file for the PoC: 
https://github.com/pganssle/capsule-gil-poc

I have tested this on version 3.6.6 and 3.7.0. The makefile works in a 
virtualenv, but you have to manually tweak the PY_MAJOR and PY_MINOR variables 
in Makefile because I didn't get too fancy with it.

--
files: main.c
messages: 323620
nosy: p-ganssle, pablogsal
priority: normal
severity: normal
status: open
title: PyCapsule_Import seems to release the GIL without acquiring it
versions: Python 3.6, Python 3.7
Added file: https://bugs.python.org/file47753/main.c

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34217] windows: cross compilation fails due to headers with uppercase

[Eryk Sun]

Eryk Sun  added the comment:

> New windows will support case sensitive filesystem too.

NTFS in Windows 10 supports a flag to mark a directory as case sensitive [*] 
via NtSetInformationFile: FileCaseSensitiveInformation, which can be set from 
the command line via fsutil.exe. In recent builds this flag was made 
inheritable by child directories, so it doesn't have to be set individually on 
every directory in a tree. This feature was added for inter-operation with 
(WSL) Linux programs. It's not intended for system directories such as "Windows 
Kits".

> can you tell me where you got your lowercased named files from?

IIRC, MinGW lower-cases the filenames in its modified version of the SDK.

---

[*] Prior to Windows XP, case-sensitive file access was possible via 
FILE_FLAG_POSIX_SEMANTICS (CreateFile) and FIND_FIRST_EX_CASE_SENSITIVE 
(FindFirstFileEx). Windows XP (the first consumer version of NT) broke those 
flags by adding a (default-enabled) global flag in the kernel object manager 
that makes all device and file access case-insensitive -- regardless of the 
system call's OBJ_CASE_INSENSITIVE flag. Now in Windows 10 they're bringing 
back selective case sensitivity in NTFS, but this time using a per-directory 
flag that overrides the API. It seems they don't care that this breaks the 
existing OBJ_CASE_INSENSITIVE flag in the NT API. IMO, this could have been 
handled more cleanly by combining the per-directory flag with one or two new 
API flags.

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34415] Typo in logging.Formatter docstring

[Semyon]

New submission from Semyon :

There is a typo in the docstring for logging.Formatter:

> default value of "%s(message)\\n" is used.

I am sure it should be different (and in sync with the actual value):

> default value of "%(message)s\\n" is used.

The problem is in py2.7, py3.7 and most likely other versions.

--
components: Library (Lib)
messages: 323618
nosy: MarSoft
priority: normal
severity: normal
status: open
title: Typo in logging.Formatter docstring
versions: Python 2.7, Python 3.4, Python 3.5, Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34391] test_ftplib is failing with TLS 1.3

[Christian Heimes]

Christian Heimes  added the comment:


New changeset 2ec530cd5537dfda5ca0af6ac696e45013ed31d2 by Christian Heimes in 
branch '2.7':
[2.7] bpo-34391: Fix ftplib test for TLS 1.3 (GH-8787) (GH-8791)
https://github.com/python/cpython/commit/2ec530cd5537dfda5ca0af6ac696e45013ed31d2


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34391] test_ftplib is failing with TLS 1.3

[Christian Heimes]

Christian Heimes  added the comment:


New changeset 870247a57e84ccaa3f6a6ce955f4168632b967a8 by Christian Heimes 
(Miss Islington (bot)) in branch '3.7':
[3.7] bpo-34391: Fix ftplib test for TLS 1.3 (GH-8787) (GH-8789)
https://github.com/python/cpython/commit/870247a57e84ccaa3f6a6ce955f4168632b967a8


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34391] test_ftplib is failing with TLS 1.3

[Christian Heimes]

Christian Heimes  added the comment:


New changeset cabe916dc694997d4892b58986e73a713d5a2f8d by Christian Heimes 
(Miss Islington (bot)) in branch '3.6':
[3.6] bpo-34391: Fix ftplib test for TLS 1.3 (GH-8787) (#8790)
https://github.com/python/cpython/commit/cabe916dc694997d4892b58986e73a713d5a2f8d


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17180] shutil copy* unsafe on POSIX - they preserve setuid/setgit bits

[Michael Felt]

Michael Felt  added the comment:

On 16/08/2018 17:34, Ronald Oussoren wrote:
> Ronald Oussoren  added the comment:
>
> I don't understand this clarification:
>
>> Clarification: while Mac/OS falls under "posix" in python terms - maybe
>> "breakage" will need to be accepted,
>> or, for "back-ports" Mac/OS will be "as if root or super-user" and use
>> an additional (optional) argument in 3.8 and beyond
>> to keep backwards compatibility.
> AFAIK macOS should behave just like other posix-y platforms here.  In 
> particular, I've verified that cp(1) behaves the same as on other platforms: 
> the SUID bit is stripped when copying a setuid file.
Glad to hear!
>
> Do you have a reason to assume that macOS is special here?
No reason to assume that macOS is different. "They" are all called
"posix" by python (Linux, macOS, AIX, FreeBSD, and I am sure there is
something else I have forgotten). So, I was trying to neither assume
that macOS is more "posix" or more "gnu". As the comments refer to gnu
coreutils behavior, not "posix" behavior (chmod --reference...) I am
looking for responses to be able to come up with better ideas for tests
we need - and then define/design code that meets the demands of the tests.

I was expecting or hoping macOS would behave as you describe but I was
also trying to prepare myself for a discussion of macOS user experience
being a discord.
>
>
> P.S. macOS is spelled macOS, not Mac/OS
Should be clear I am not a macOS user. Corrected!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20849] add exist_ok to shutil.copytree

[Joshua Bronson]

Change by Joshua Bronson :


--
pull_requests: +8266

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34410] Segfault/TimeoutError: itertools.tee of multiprocessing.pool.imap_unordered

[Raymond Hettinger]

Raymond Hettinger  added the comment:

Davin, is there anything itertools.tee() can do about this or is this a 
multiprocessing issue?

--
nosy: +davin, rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34391] test_ftplib is failing with TLS 1.3

[Christian Heimes]

Change by Christian Heimes :


--
pull_requests: +8265

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34391] test_ftplib is failing with TLS 1.3

[miss-islington]

Change by miss-islington :


--
pull_requests: +8264

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34391] test_ftplib is failing with TLS 1.3

[miss-islington]

Change by miss-islington :


--
pull_requests: +8263

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34391] test_ftplib is failing with TLS 1.3

[Christian Heimes]

Christian Heimes  added the comment:


New changeset 1590c393360df059160145e7475754427bfc6680 by Christian Heimes in 
branch 'master':
bpo-34391: Fix ftplib test for TLS 1.3 (GH-8787)
https://github.com/python/cpython/commit/1590c393360df059160145e7475754427bfc6680


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34414] Absolute imports conflict with local files

[Jonathan Hadida]

Jonathan Hadida  added the comment:

Thank you for your quick reply.

How can this be expected behaviour? Could I please kindly ask you to point to 
an documented explanation, specifically for why the folder is PREpended to 
sys.path (put before), instead of being APpended (put after).

The fact that local files conflict with absolute imports _within dependent 
modules_ is nothing short of a hack (think from the point of view of the 
dependencies themselves), and IMO clearly conflicts with the idea of absolute 
imports in the first place. This is crazy!

--
status: pending -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34395] memory leaks in error handling in csv and pickle modules

[Sergey Fedoseev]

Change by Sergey Fedoseev :


--
pull_requests: +8262

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34414] Absolute imports conflict with local files

[Ronald Oussoren]

Ronald Oussoren  added the comment:

This is expected behaviour: When your run a script the directory containing the 
script is added to the start of sys.path. Running "python3.6 a/foo.py" 
therefore adds "a" to the start of sys.path, and "math.py" then shadows the 
stdlib module "math".

This has nothing to do with absolute imports. That is shown by "python3.6 -m 
foo.a". This runs the code in foo/a.py as module foo.a, and "math.py" is now 
only accessible as "foo.math" (it won't shadow the builtin module "math", even 
if you add "import math" to foo/a.py).

--
resolution:  -> not a bug
status: open -> pending

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34391] test_ftplib is failing with TLS 1.3

[Christian Heimes]

Change by Christian Heimes :


--
keywords: +patch
pull_requests: +8261

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34414] Absolute imports conflict with local files

[Jonathan Hadida]

New submission from Jonathan Hadida :

This submission follows a post on StackOverflow: 
https://stackoverflow.com/q/51878397/472610

I have reproduced the unexpected behaviour with multiple python versions, 
either with a Homebrew install, or using Anaconda/Miniconda. Note that comments 
to the post on StackOverflow indicate that this behaviour could only be 
reproduced on Windows and Linux using conda with late versions of Python.


THE ISSUE
-

Absolute imports seem to conflict with local files (not in any module). 
This is at odds with the documented behaviour 
(https://docs.python.org/3/tutorial/modules.html#the-module-search-path):

"When a module named spam is imported, the interpreter first searches for a 
built-in module with that name."


STEPS TO REPRODUCE
--

STEP 1:
On OSX, use either:

 - A Homebrew-installed version (e.g. /usr/local/bin/python2 or 3)
 - A Miniconda2 or 3 installation
 - An Anaconda3 v5.2.0 installation (not tested with other versions, nor 
Anaconda2)

NOTE: in cases 1 and 2, you need to install numpy manually.


STEP 2:
Create a directory structure as follows:

  .
  └── foo
  ├── a.py
  └── math.py

  1 directory, 2 files

where a.py contains "import numpy", and math.py contains "x++" (intentionally 
invalid).
For example, the following bash code sets this up in a temporary folder:

  D=$(mktemp -d)
  mkdir "$D/foo"
  echo "import numpy" >| "$D/foo/a.py"
  echo "x++" >| "$D/foo/math.py"


STEP 3:
Go to that directory (the one containing folder "foo"), and run:

   foo/a.py

The previous command causes the following error, for example using 
/usr/local/bin/python3 (Homebrew):

Traceback (most recent call last):
  File "foo/a.py", line 1, in 
import numpy
  File "/usr/local/lib/python3.6/site-packages/numpy/__init__.py", line 142, in 

from . import add_newdocs
  File "/usr/local/lib/python3.6/site-packages/numpy/add_newdocs.py", line 13, 
in 
from numpy.lib import add_newdoc
  File "/usr/local/lib/python3.6/site-packages/numpy/lib/__init__.py", line 3, 
in 
import math
  File 
"/private/var/folders/j7/kd8mc69j25j0yw50q_08wmlmgt/T/tmp.FfJzdVuG/foo/math.py",
 line 1
x++
  ^
SyntaxError: invalid syntax


PROBLEM:
The statement "import math" in numpy/lib/__init__.py should not resolve to 
foo/math.py, but rather, it should find the standard module "math".


ADDITIONAL INFO
---

Although I do not know what this list should look like, I would expect the list 
of builtin modules to be larger than this:

> python3 -c "import sys; print(sys.builtin_module_names)"
('_ast', '_codecs', '_collections', '_functools', '_imp', '_io', '_locale', 
'_operator', '_signal', '_sre', '_stat', '_string', '_symtable', '_thread', 
'_tracemalloc', '_warnings', '_weakref', 'atexit', 'builtins', 'errno', 
'faulthandler', 'gc', 'itertools', 'marshal', 'posix', 'pwd', 'sys', 'time', 
'xxsubtype', 'zipimport')

> python2 -c "import sys; print sys.builtin_module_names"
('__builtin__', '__main__', '_ast', '_codecs', '_sre', '_symtable', 
'_warnings', '_weakref', 'errno', 'exceptions', 'gc', 'imp', 'marshal', 
'posix', 'pwd', 'signal', 'sys', 'thread', 'xxsubtype', 'zipimport')

--
components: macOS
messages: 323609
nosy: jhadida, ned.deily, ronaldoussoren
priority: normal
severity: normal
status: open
title: Absolute imports conflict with local files
type: behavior
versions: Python 2.7, Python 3.4, Python 3.5, Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34391] test_ftplib is failing with TLS 1.3

[Christian Heimes]

Christian Heimes  added the comment:

The "OSError: [Errno 0] Error" failures are caused by session ticket handling 
in two way shutdown. I reported the issue a while ago 
https://github.com/openssl/openssl/issues/6262 and it's fixed in OpenSSL git 
master (to be released as 1.1.1-pre9).

The error in test_data_connection is actually "[SSL] shutdown while in init". 
The dummy server code is failing, because the client starts a two way shutdown 
before the full handshake has been established. A simple recv() call is good 
enough to finalize the handshake.

--
assignee:  -> christian.heimes
components: +SSL
stage:  -> patch review
type:  -> behavior
versions: +Python 2.7, Python 3.6, Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33216] [3.5] Wrong order of stack for CALL_FUNCTION_VAR and CALL_FUNCTION_VAR_KW

[Eric Snow]

Eric Snow  added the comment:

FYI, I opened bpo-34413 to address how the bytecode change is not mentioned in 
the porting section of the 3.5 "What's New" doc.  In retrospect (I didn't 
notice that this issue was still open), I suppose that doesn't need a separate 
issue so feel free to close #34413 in favor of this one. :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33216] [3.5] Wrong order of stack for CALL_FUNCTION_VAR and CALL_FUNCTION_VAR_KW

[Eric Snow]

Eric Snow  added the comment:

FTR, the change was introduced by the PEP 448 implementation. [1]

[1] https://github.com/python/cpython/pull/8338#issuecomment-406256152

--
nosy: +eric.snow

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34413] Porting section of 3.5 "What's New" doc does not mention bytecode changes.

[Eric Snow]

New submission from Eric Snow :

In the 3.5 "What's New" doc, the porting section [1] does not mention the 
bytecode changes resulting from the PEP 448 implementation. [2][3]  It should.

I've marked this issue for the versions past 3.5 because each branch has its 
own "What's New" doc for 3.5.

[1] https://docs.python.org/3/whatsnew/3.5.html#porting-to-python-3-5
[2] bpo-33216
[3] https://github.com/python/cpython/pull/8783#discussion_r210642202

--
assignee: docs@python
components: Documentation
messages: 323605
nosy: docs@python, eric.snow, serhiy.storchaka
priority: low
severity: normal
stage: needs patch
status: open
title: Porting section of 3.5 "What's New" doc does not mention bytecode 
changes.
type: enhancement
versions: Python 3.5, Python 3.6, Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17180] shutil copy* unsafe on POSIX - they preserve setuid/setgit bits

[Ronald Oussoren]

Ronald Oussoren  added the comment:

I don't understand this clarification:

> Clarification: while Mac/OS falls under "posix" in python terms - maybe
> "breakage" will need to be accepted,
> or, for "back-ports" Mac/OS will be "as if root or super-user" and use
> an additional (optional) argument in 3.8 and beyond
> to keep backwards compatibility.

AFAIK macOS should behave just like other posix-y platforms here.  In 
particular, I've verified that cp(1) behaves the same as on other platforms: 
the SUID bit is stripped when copying a setuid file.

Do you have a reason to assume that macOS is special here?


P.S. macOS is spelled macOS, not Mac/OS

--
nosy: +ronaldoussoren

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34217] windows: cross compilation fails due to headers with uppercase

[Steve Dower]

Steve Dower  added the comment:

> whatever might be the 'right' solution, it's always an advantage to
> have a consistent codebase.

Not when you are consistently wrong, it's not.

Let's figure out the right answer here before making many changes to the 
codebase. First, can you tell me where you got your lowercased named files 
from? Because it wasn't the official SDK...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34217] windows: cross compilation fails due to headers with uppercase

[Steve Dower]

Steve Dower  added the comment:

> Stop relying to case insensitive filesystem makes sense to me.

If this is the case, then we need to normalise the casing to match the actual 
files (which are mixed-case, not lowercase). You just broke this with your PR :)

So which is it? Do we match the casing of the real files? Or do we change to 
match someone's unofficial versions of the files?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34412] strsignal(3) does not exist on HP-UX

[Michael Osipov]

Change by Michael Osipov <1983-01...@gmx.net>:


--
keywords: +patch
pull_requests: +8259
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34412] strsignal(3) does not exist on HP-UX

[Michael Osipov]

Michael Osipov <1983-01...@gmx.net> added the comment:

References:
* https://github.com/google/cmockery/issues/11
* https://www.spinics.net/lists/dash/msg00547.html

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34412] strsignal(3) does not exist on HP-UX

[Michael Osipov]

New submission from Michael Osipov <1983-01...@gmx.net>:

HP-UX does not provide any mappings from signals to strings. The proper 
approach is to map just like for Windows. Alternatively, one could simply 
return the singal as an int.

--
components: Library (Lib)
messages: 323600
nosy: michael-o
priority: normal
severity: normal
status: open
title: strsignal(3) does not exist on HP-UX
type: compile error
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34401] [SOLUTION] Make test_gdb work on HP-UX

[Michael Osipov]

Change by Michael Osipov <1983-01...@gmx.net>:


Removed file: https://bugs.python.org/file47749/test_gdb.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34395] memory leaks in error handling in csv and pickle modules

[Sergey Fedoseev]

Change by Sergey Fedoseev :


--
pull_requests: +8258

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34246] Gentoo Refleaks 3.7: test_smtplib has dangling threads

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
components: +email
nosy: +barry

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33216] [3.5] Wrong order of stack for CALL_FUNCTION_VAR and CALL_FUNCTION_VAR_KW

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
pull_requests: +8257

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33216] [3.5] Wrong order of stack for CALL_FUNCTION_VAR and CALL_FUNCTION_VAR_KW

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
pull_requests: +8256

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27435] ctypes library loading and AIX - also for 2.7.X (and later)

[Michael Felt]

Michael Felt  added the comment:

as this is not (likely) to be backported to Python2 (by python, fyi: I do 
include full ctypes-load_library() support in my "independent" packaging)

and it is "resolved" for Python3-3.7 issue26439

this issue may, imho, be closed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17180] shutil copy* unsafe on POSIX - they preserve setuid/setgit bits

[Michael Felt]

Michael Felt  added the comment:

I want to believe this can be resolved - without breakage on POSIX.

Clarification: while Mac/OS falls under "posix" in python terms - maybe
"breakage" will need to be accepted,
or, for "back-ports" Mac/OS will be "as if root or super-user" and use
an additional (optional) argument in 3.8 and beyond
to keep backwards compatibility.

Short text: to proceed I think we should start with getting some
additional tests into test_shutil.py asap so that we can see how systems
respond without any changes.

My experience is that cp -r[pP] behaves the same as shutil.copy*() when
the EUID==0, aka superuser,
but strips special bits from files and cannot copy the UID/GID owner
bits of the inode.

I would appreciate someone helping me writing more extensive testing.

We need to test:
* root
* not-root, but owner
* not-root, not owner, but in group
* not-root, "other", other "read" access exists

* if the test does not already exist - also check behavior when directories
  have/do not have "search" (x-bit) enabled.

I am working on a patch to address these different conditions.

Ideally, the "not-owner" and "not-group" tests can be run
by creating the "copy.me" area as root, setting perms, etc.
and then using su -c to run the shutil.copy*() call
and back as root make the verification.

±±± Perspective ±±±

If this is too much discussion, please reply with suggestions - privately -
on what I could do better to not waste your time.

The issue seems unchanged since original posting.

The original report states:
hen copying the mode of a file with copy, copy2, copymode, copystat or
copytree, all permission bits are copied (including setuid and setgit),
but the owner of the file is not. This can be used for privilege escalation.

...snip...

The behaviour of copymode/copystat in this case is the same as `chmod
--reference', and there can be some expectation of unsafety, but
copy/copy2/copytree's behaviour differs from that of `cp -p', and this
is a non-obvious difference.

For clarity: GNU chmod states:

--reference=RFILE
    use RFILE's mode instead of MODE values

Additionally, the chmod man page reminds us the "special bit" masking
behavior is different for files and directries.
Specifically, SUID, SGID and SVTX should not be cleared unless
specifically requested by a chmod "u-s,g-s" specification.

"... a directory's unmentioned set user and group ID bits are not affected"

Additional comments discuss:
short window of opportunity (files are copied first, then mode bits copied)
breakage with the past (copytree used as "backup", regardless of version)

And the comment/opinion that shutil.copy() should emulate cp (implies
emulate "cp -r", so neither -p nor -P)

it seems shutil.copy2() is adding the -p (or -P if follow_symlinks=false)

There was a modification to test_shutil.py suggested as part of a patch.
I added that to verify the issue is still current.

±±±
diff --git a/Lib/test/test_shutil.py b/Lib/test/test_shutil.py
index 7e0a3292e0..7ceefd1ebc 100644
--- a/Lib/test/test_shutil.py
+++ b/Lib/test/test_shutil.py
@@ -1471,6 +1471,24 @@ class TestShutil(unittest.TestCase):
 rv = shutil.copytree(src_dir, dst_dir)
 self.assertEqual(['foo'], os.listdir(rv))

+    @unittest.skipUnless((os.name == "posix" and os.geteuid() != 0),
"Requires POSIX compatible OS and non-root userid")
+    def test_copy_remove_setuid(self):
+    src_dir = self.mkdtemp()
+    src_file = os.path.join(src_dir, 'foo')
+    write_file(src_file, 'foo')
+    dst_file = os.path.join(src_dir, 'bar')
+    harmful_mode = stat.S_IRUSR | stat.S_IXUSR | stat.S_ISUID
+    harmless_mode = stat.S_IRUSR | stat.S_IXUSR
+
+    # set mode and verify
+    os.chmod(src_file, harmful_mode)
+    mode = stat.S_IMODE(os.stat(src_file).st_mode)
+    self.assertTrue(oct(mode), oct(harmful_mode))
+
+    # check that copy does not preserve harmful bits
+    shutil.copy(src_file, dst_file)
+    mode = stat.S_IMODE(os.stat(dst_file).st_mode)
+    self.assertEqual(oct(mode), oct(harmless_mode))

 class TestWhich(unittest.TestCase):
±±±
The result is:
root@x066:[/data/prj/python/python3-3.8]./python -m test -v test_shutil
== CPython 3.8.0a0 (heads/master:cca4eec3c0, Aug 13 2018, 04:53:15) [C]
== AIX-1-00C291F54C00-powerpc-32bit big-endian
== cwd: /data/prj/python/python3-3.8/build/test_python_10944516
== CPU count: 8
== encodings: locale=ISO8859-1, FS=iso8859-1
Run tests sequentially
...
test_copy_remove_setuid (test.test_shutil.TestShutil) ... FAIL
...
==
FAIL: test_copy_remove_setuid (test.test_shutil.TestShutil)
--
Traceback (most recent call last):
  File "/data/prj/python/git/python3-3.8/Lib/test/test_shutil.py", line
1491, in test_copy_remove_setuid
    self.assertEqual(oct(mode), oct(harmless_mode))
AssertionError: '0o4500' != '0o500'
- 

[issue34217] windows: cross compilation fails due to headers with uppercase

[miss-islington]

miss-islington  added the comment:


New changeset bf8e9d18dd75f58ce3b9761763ae10c0800b43d8 by Miss Islington (bot) 
in branch '3.7':
bpo-34217: Use lowercase for windows headers (GH-8472)
https://github.com/python/cpython/commit/bf8e9d18dd75f58ce3b9761763ae10c0800b43d8


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34217] windows: cross compilation fails due to headers with uppercase

[INADA Naoki]

INADA Naoki  added the comment:

New windows will support case sensitive filesystem too.
Stop relying to case insensitive filesystem makes sense to me.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34217] windows: cross compilation fails due to headers with uppercase

[Erik Janssens]

Erik Janssens  added the comment:

@inada.naoki : thank you !

@steve.dower : I would not consider those bugs per se, but they are
inconsistencies, for example in the master branch :

number of "Windows.h" includes : 2
number of "windows.h" includes : 55

whatever might be the 'right' solution, it's always an advantage to
have a consistent codebase.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34217] windows: cross compilation fails due to headers with uppercase

[miss-islington]

Change by miss-islington :


--
pull_requests: +8255
stage: resolved -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34217] windows: cross compilation fails due to headers with uppercase

[INADA Naoki]

INADA Naoki  added the comment:


New changeset e6a4755e6793942b950c1595e0c34bd66a0ee13e by INADA Naoki (Erik 
Janssens) in branch 'master':
bpo-34217: Use lowercase for windows headers (GH-8472)
https://github.com/python/cpython/commit/e6a4755e6793942b950c1595e0c34bd66a0ee13e


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1529353] Squeezer - squeeze large output in IDLE's shell

[Tal Einat]

Change by Tal Einat :


--
title: Squeezer - squeeze large output in IDLE -> Squeezer - squeeze large 
output in IDLE's shell

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com