[issue29034] Fix memory leak and use-after-free in path_converter

[Xiang Zhang]

Xiang Zhang added the comment:

path_converter-new-2.patch addresses Serhiy's comments. :-)

--
Added file: http://bugs.python.org/file46170/path_converter-new-2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28180] sys.getfilesystemencoding() should default to utf-8

[Nick Coghlan]

Nick Coghlan added the comment:

And by PEP 528, I actually mean PEP 538 :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28180] sys.getfilesystemencoding() should default to utf-8

[Nick Coghlan]

Nick Coghlan added the comment:

Docker containers don't have a locale set by default - the approach proposed in 
PEP 528 actually comes from the way I configure Docker images (which in turn 
comes from Armin Ronacher's recommendations in click for Python 3 locale 
handling).

In the Dockerfile for Fedora based containers I add:

ENV LC_ALL=C.UTF-8
ENV LANG=C.UTF-8

while in CentOS 7 based containers I add:

ENV LC_ALL=en_US.UTF-8
ENV LANG=en_US.UTF-8

And with those settings, Python 3 based containers just work (my laptop is 
running en_AU.UTF-8 locally)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29170] Curses Module should check for is_keypad and not window flags

[Berker Peksag]

Changes by Berker Peksag :


--
resolution:  -> duplicate
stage:  -> resolved
superseder:  -> Fix curses module compilation with ncurses6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29172] blake2: Use lowest-common denominator signature of #pragma pack

[Martin Panter]

Martin Panter added the comment:

FWIW Issue 28290 was also opened about this pragma directive being not 
recognized by a compiler called xlC on AIX.

--
nosy: +martin.panter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28231] zipfile does not support pathlib

[Berker Peksag]

Berker Peksag added the comment:

I think the next steps are:

1. Add tests for the cases Serhiy has mentioned in msg277109:

   > Shouldn't the ZipFile.filename attribute be converted to str?

   and

   > The file name of ZipFile is only a part of the issue. There are other uses 
of file paths: paths for added files, path for extracted directory.

2. Update the docs to reflect the pathlib support.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28231] zipfile does not support pathlib

[Jeremy Freeman]

Jeremy Freeman added the comment:

OK, I understand.  How can I help get this issue fixed?

1) review the patch?
2) update the docs to reflect the patch?
3) find the other uses of pathlib in the zipfile module?
4) something else ...

I am a longtime user of python but a first time contributor but happy to help.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28848] Add CopyingMock to mock.py

[Berker Peksag]

Berker Peksag added the comment:

Thanks for releasing it on PyPI! It's probably better to keep it as an example 
in the documentation for now. We can reopen this if there's a demand for 
CopyingMock in the future.

--
nosy: +berker.peksag
resolution:  -> rejected
stage:  -> resolved
status: open -> closed
type:  -> enhancement

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29142] urllib: no_proxy variable values with leading dot not properly handled

[Berker Peksag]

Changes by Berker Peksag :


--
type:  -> behavior
versions: +Python 3.5, Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29173] Python 3.6 on Windows wastes a lot of CPU cycles in a while loop

[Eryk Sun]

Changes by Eryk Sun :


--
stage:  -> resolved

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29173] Python 3.6 on Windows wastes a lot of CPU cycles in a while loop

[Prahlad Yeri]

Prahlad Yeri added the comment:

Hi STINNER Victor,

Thanks a lot, after adding the sleep function, it has stopped wasting the CPU 
cycles! A long while ago, I remember coding a similar loop in linux and not 
faced such issue, so I thought maybe it was a bug in 3.6.

Anyway, I'm closing this.

--
resolution:  -> not a bug
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29173] Python 3.6 on Windows wastes a lot of CPU cycles in a while loop

[STINNER Victor]

STINNER Victor added the comment:

The code doesn't seem like a bug in Python, but more a classic inefficient busy 
loop pattern. Your loop doesn't sleep and so eats all the CPU.

I suggest to close the issue and ask Python questions on a place to ask 
questions, not on the Python *bug tracker*.

--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29173] Python 3.6 on Windows wastes a lot of CPU cycles in a while loop

[Prahlad Yeri]

New submission from Prahlad Yeri:

I'm running Python 3.6 on Windows 7. It wastes a lot of cpu cycles when inside 
a loop. In attached screenshot, you'll find that ~25% cpu is used by Python, 
while all I'm doing is running a pomodoro script that waits for a specific 
timeslot to complete. Here is the code for pomodoro.py that runs the while loop 
inside start_tracking() function:

https://github.com/prahladyeri/PyPomodoro/blob/master/pomodoro.py



def start_tracking(task):
global last_beep
print("Working on %s:%s (%d minutes)." % (task['category'], 
task['name'], task['duration']))
print("Started tracking at %s." % 
(datetime.datetime.now().strftime("%H:%M")))
print("Beep interval is set to %d minutes." % config.slot_interval)
session_start_time = datetime.datetime.now()
task_end_time = session_start_time + 
datetime.timedelta(minutes=task['duration'])
last_beep = datetime.datetime.now()
notified = False
while(True):
now = datetime.datetime.now()
diff = (now - last_beep).total_seconds() / 60.0 #in minutes
minutes_worked = round(diff)
reminder_text = "%s:%s" % (task['category'], task['name'])
#diff = diff.total_seconds() / (60.0) 
if (diff >= config.slot_interval): #30
#notify

--
components: Windows
files: Python_high_CPU_Windows.png
messages: 284787
nosy: paul.moore, prahladyeri, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: Python 3.6 on Windows wastes a lot of CPU cycles in a while loop
type: resource usage
versions: Python 3.6
Added file: http://bugs.python.org/file46169/Python_high_CPU_Windows.png

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29142] urllib: no_proxy variable values with leading dot not properly handled

[Milan Oberkirch]

Changes by Milan Oberkirch :


--
type: behavior -> 
versions:  -Python 3.5, Python 3.6, Python 3.7
Added file: http://bugs.python.org/file46168/ignore_dots-v2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2771] Test issue

[Ezio Melotti]

Ezio Melotti added the comment:

another test

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2771] Test issue

[Ezio Melotti]

Ezio Melotti added the comment:

testing github integration

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29172] blake2: Use lowest-common denominator signature of #pragma pack

[Eric N. Vander Weele]

New submission from Eric N. Vander Weele:

Solaris Studio emits the following during compilation:

"/tmp/Python-3.6.0/Modules/_blake2/impl/blake2.h", line 89: warning: 
ignoring malformed #pragma pack(n)
"/tmp/Python-3.6.0/Modules/_blake2/impl/blake2.h", line 119: warning: 
ignoring malformed #pragma pack(n)

To make the usage of '#pragma pack' more portable, change to the
optional, single argument form.

--
components: Build
files: blake2-pragma-pack-1.patch
keywords: patch
messages: 284784
nosy: christian.heimes, ericvw
priority: normal
severity: normal
status: open
title: blake2: Use lowest-common denominator signature of #pragma pack
type: compile error
versions: Python 3.6
Added file: http://bugs.python.org/file46167/blake2-pragma-pack-1.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29021] Custom functions in sqlite receive None on invalid UTF-8

[Aviv Palivoda]

Aviv Palivoda added the comment:

Actually had a small mistake in the patch I uploaded. Uploading a fixed one.

--
Added file: http://bugs.python.org/file46166/29021-fixed.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29171] blake2: Remove unused function to avoid undefined references

[Eric N. Vander Weele]

Changes by Eric N. Vander Weele :


--
title: Remove unused blake2 function to avoid undefined references -> blake2: 
Remove unused function to avoid undefined references

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28180] sys.getfilesystemencoding() should default to utf-8

[STINNER Victor]

STINNER Victor added the comment:

> Working with Docker I often end up with an environment where the locale isn't 
> correctly set.

The locale encoding is controlled by 3 environment variables: LC_ALL, LC_CTYPE 
and LANG.
https://www.python.org/dev/peps/pep-0540/#the-posix-locale-and-its-encoding

Can you please tell me if these variables are set and if yes, give me their 
value?

I would like to know if it would be possible to change the behaviour of Python 
when the (LC_CTYPE) locale is POSIX (aka the famous "C" locale).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29171] Remove unused blake2 function to avoid undefined references

[Eric N. Vander Weele]

New submission from Eric N. Vander Weele:

Compilers are not required to elide static functions which are unused.

Some compilers, such as Solaris Studio, always emits the function, even
if the function does not get called within the translation unit.  This
becomes problematic when a static inline function calls a non-existent
function; thus, resulting in (dynamic or static) link time errors.

Given that 'blake2' is never referenced nor called, remove the
definition of this function to increase portability for non-Linux
toolchains.

https://blogs.oracle.com/d/entry/inline_functions_in_c also indicates that this 
is case for Solaris Studio as well.

--
components: Build
files: blake2-remove-unused-function-1.patch
keywords: patch
messages: 284781
nosy: christian.heimes, ericvw
priority: normal
severity: normal
status: open
title: Remove unused blake2 function to avoid undefined references
type: behavior
versions: Python 3.6
Added file: 
http://bugs.python.org/file46165/blake2-remove-unused-function-1.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29169] update zlib to 1.2.10

[Raymond Hettinger]

Raymond Hettinger added the comment:

> I'm inclined to not cherry-pick this, which means it'd 
> ship in 3.5.4 and 3.4.7, probably in six months.

I concur.  Looking at the CVEs, these all seem minor and not exploitable 
through the Python interface.

--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29170] Curses Module should check for is_keypad and not window flags

[Roy Marples]

Roy Marples added the comment:

It is, sorry for noise.

--
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29034] Fix memory leak and use-after-free in path_converter

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

path->length and path->object are set in each way to success_exit (except 
error_exit). I think these assignment can be moved after success_exit.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29170] Curses Module should check for is_keypad and not window flags

[Masayuki Yamamoto]

Masayuki Yamamoto added the comment:

#25720 is a similar issue that is originated by opaque type WINDOW.

--
nosy: +masamoto

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29021] Custom functions in sqlite receive None on invalid UTF-8

[Aviv Palivoda]

Aviv Palivoda added the comment:

I actually was wrong and there is no problem with the type that is returned 
from the text_factory as it passes directly as argument for the create_function 
function.

Attached is a patch that change create_function and create_aggregate to use the 
text_factory when a TEXT data type is received. I am now using 
sqlite3_create_function_v2 but this is fine because we use 
sqlite3_stmt_readonly and it is from a newer version.

As for Ingo example code it still don't work with this current fix but this is 
due to a different problem. Now the _pysqlite_set_result function fail when we 
do PyUnicode_AsUTF8 on the result from py_identity. As this is a different 
problem I will fix this in a different patch.

--
keywords: +patch
Added file: http://bugs.python.org/file46164/29021.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29170] Curses Module should check for is_keypad and not window flags

[Roy Marples]

Roy Marples added the comment:

Remove WINDOW test
Add is_keypad test
Change #define in _cursesmodule.c

--
keywords: +patch
Added file: http://bugs.python.org/file46163/cursesmodule.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29159] Regression in bytes constructor

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

The code restored by 29159-index-fallback.patch is slightly different from the 
code before issue27704, but the patch LGTM except that arguments of assertEqual 
are in reversed order.

> Can you explain what your patch does?  I don't understand why you single out 
> the OverflowError.  When is __index__ expected to raise it?

PyNumber_AsSsize_t(arg, PyExc_OverflowError) raises an OverflowError when the 
result of __index__ don't fit in Py_ssize_t.

> I wonder if we could check for buffer protocol support before detecting an 
> integer argument?

This would fix an issue with NumPy arrays, but it is much harder to implement.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27961] remove support for platforms without "long long"

[STINNER Victor]

STINNER Victor added the comment:

> The backwards compatibility is not as strong as it could be: previously, 
> HAVE_LONG_LONG was defined to 1; now it's defined but empty.

Oh, right. This issue should now be fixed as well.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27961] remove support for platforms without "long long"

[Roundup Robot]

Roundup Robot added the comment:

New changeset fad67c66885f by Victor Stinner in branch '3.6':
Issue #27961: Define HAVE_LONG_LONG as 1.
https://hg.python.org/cpython/rev/fad67c66885f

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29170] Curses Module should check for is_keypad and not window flags

[Roy Marples]

New submission from Roy Marples:

checking whether WINDOW has _flags... no

The ncurses library can be built with an opaque window structure.
As such, it's not always possible to query it's flags.
Luckily there is the is_keypad function which works regardless of how ncurses 
was built.

If this test is changed, then it will help support NetBSD curses as well.

--
components: Extension Modules
messages: 284771
nosy: Roy Marples
priority: normal
severity: normal
status: open
title: Curses Module should check for is_keypad and not window flags
type: compile error
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28848] Add CopyingMock to mock.py

[wim glenn]

wim glenn added the comment:

Guess there was no interest in this.   I've released it on pypi instead (pip 
install copyingmock).  

https://github.com/wimglenn/copyingmock

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29169] update zlib to 1.2.10

[Larry Hastings]

Larry Hastings added the comment:

I cut 3.4.6rc1 and 3.5.3rc1 a couple of days ago.  Do you think the CVEs are 
bad enough to warrant cherry-picking this?  A quick google suggests they were 
all low severity:

http://www.openwall.com/lists/oss-security/2016/12/05/21

I'm inclined to not cherry-pick this, which means it'd ship in 3.5.4 and 3.4.7, 
probably in six months.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29159] Regression in bytes constructor

[Alexander Belopolsky]

Alexander Belopolsky added the comment:

On the other hand, the documentation for the bytearray constructor [1] lists 
integer argument first. 

[1]: https://docs.python.org/3/library/functions.html#bytearray

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29159] Regression in bytes constructor

[Alexander Belopolsky]

Alexander Belopolsky added the comment:

@inada.naoki

Sorry, I still don't understand the role of OverflowError.

With respect to my earlier suggestion that buffer protocol should have priority 
over __index__, note that the documentation implies the same order:

>>> help(bytes)

class bytes(object)
 |  bytes(iterable_of_ints) -> bytes
 |  bytes(string, encoding[, errors]) -> bytes
 |  bytes(bytes_or_buffer) -> immutable copy of bytes_or_buffer
 |  bytes(int) -> bytes object of size given by the parameter initialized with 
null bytes
 |  bytes() -> empty bytes object
 |
 |  Construct an immutable array of bytes from:
 |- an iterable yielding integers in range(256)
 |- a text string encoded using the specified encoding
 |- any object implementing the buffer API.
 |- an integer
..

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29057] Compiler failure on Mac OS X - sys/random.h

[Jim Nasby]

Jim Nasby added the comment:

I take it back... turns out I was building with the unsupported macports gcc.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29057] Compiler failure on Mac OS X - sys/random.h

[Larry Hastings]

Larry Hastings added the comment:

Mr. Nasby, as long as you're in a test-reproducing mood, would you mind 
downloading the source to 3.5.3rc1 and confirming that it builds correctly for 
you?  I'd appreciate it!  (Not that I don't trust Ned et al, but independent 
confirmation always helps soothe these jangled nerves.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28180] sys.getfilesystemencoding() should default to utf-8

[STINNER Victor]

STINNER Victor added the comment:

> That way each PEP can argue as strongly as it can for the respective authors 
> preferred approach to tackling the default C locale problem, even if they 
> point to a common background section in one of the PEPs (similar to the way 
> PEPs 522 and 524 shared a common problem definition, even though they 
> proposed different ways of handling it).

Ok, same players play again: as PEP 522/524 with Nick and me, I just wrote the 
PEP 540 "Add a new UTF-8 mode" and Nick wrote the PEP 538 :-D

I started a thread to discuss the PEP on python-ideas:
https://mail.python.org/pipermail/python-ideas/2017-January/044089.html

IMHO the PEP 538 should discuss the usage of the surrogateescape error handler: 
see my second mail in the thread for the details.

I proposed a change in my 3rd mail which would move my PEP closer to Nick's PEP 
538: enable "automatically" the UTF-8 mode when the locale is POSIX.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29132] shlex.shlex with punctuation_chars and posix doesn't handle punctuation next to quotes

[Vinay Sajip]

Vinay Sajip added the comment:

Evan, thanks for looking at this. Would you like to submit a completed 
contributor form?

https://www.python.org/psf/contrib/contrib-form/

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29057] Compiler failure on Mac OS X - sys/random.h

[Jim Nasby]

Jim Nasby added the comment:

This is still affecting me on 2.7, even with the new changes. 3.7 seems to be 
OK (I'm getting a different error on 3.7, but it seems unrelated to this.)

OS X 10.11.6.

--
nosy: +Jim Nasby

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29167] Race condition in enum.py:_decompose()

[Ethan Furman]

Changes by Ethan Furman :


--
assignee:  -> ethan.furman
stage:  -> test needed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29167] Race condition in enum.py:_decompose()

[Ethan Furman]

Ethan Furman added the comment:

Simon, can you post the exact line of code that causes the error?  It would be 
useful for creating a test case and the couple things I have tried to duplicate 
the error have worked fine.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29169] update zlib to 1.2.10

[Roundup Robot]

Roundup Robot added the comment:

New changeset ed172054a812 by doko in branch '2.7':
- Issue #29169: Update zlib to 1.2.10.
https://hg.python.org/cpython/rev/ed172054a812

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29034] Fix memory leak and use-after-free in path_converter

[Xiang Zhang]

Xiang Zhang added the comment:

The new patch is a try to solve the problems mentioned in this thread.

> It can be solved by making path->object referring an original argument (but 
> some code checks the type of path->object), or making path->object owning a 
> reference.

I prefer the later approach. There are codes relying on this behaviour 
(readlink).

--
title: Fix memory leak in path_converter -> Fix memory leak and use-after-free 
in path_converter
type:  -> resource usage
Added file: http://bugs.python.org/file46162/path_converter-new.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28231] zipfile does not support pathlib

[R. David Murray]

R. David Murray added the comment:

In 3.5 the stdlib is not supporting PathLib.  So this issue only affects 3.6 
and 3.7.

--
nosy: +r.david.murray

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26865] Meta-issue: support of the android platform

[Xavier de Gaye]

Xavier de Gaye added the comment:

Removing issues from the dependencies list:
issue #22747: Interpreter fails in initialize on systems where 
HAVE_LANGINFO_H is undefined
  This issue is fixed for Android.
issue #26859: unittest fails with "Start directory is not importable" when 
trying to run sourceless tests
  This issue is out of scope.

--
dependencies:  -Interpreter fails in initialize on systems where 
HAVE_LANGINFO_H is undefined, unittest fails with "Start directory is not 
importable" when trying to run sourceless tests

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29168] multiprocessing pickle error

[Davin Potts]

Davin Potts added the comment:

Simon:  You do have a lock object inside the logging handler object.  That is 
what prevents your pickling of the logging handler object.

The background information Serhiy shared is focused on why pickling a 
_thread.RLock object is problematic and unsupportable.

I do not know enough to understand how you implemented your code that worked 
under 3.5 but one of the points made by Serhiy's comments is that your code 
worked _in spite_ of the fact that you did not have a proper RLock inside your 
logging handler.  I optimistically believe you can implement what you need to 
work without needing to pickle a logging handler object.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29168] multiprocessing pickle error

[Vinay Sajip]

Vinay Sajip added the comment:

> Have a look at the sample.py program.

It fails on Python 3.4.3 on my Linux system (just renamed to mp_sample.py):

python3 mp_sample.py 
starting (without multiprocessing pool)...
worker process 0
worker process 3
worker process 4
worker process 6
worker process 5
worker process 9
worker process 8
worker process 7
worker process 1
worker process 2
starting (with multiprocessing pool)...
Traceback (most recent call last):
  File "mp_sample.py", line 82, in 
op.start()
  File "mp_sample.py", line 43, in start
result.get()
  File "/usr/lib/python3.4/multiprocessing/pool.py", line 599, in get
raise self._value
  File "/usr/lib/python3.4/multiprocessing/pool.py", line 383, in _handle_tasks
put(task)
  File "/usr/lib/python3.4/multiprocessing/connection.py", line 206, in send
self._send_bytes(ForkingPickler.dumps(obj))
  File "/usr/lib/python3.4/multiprocessing/reduction.py", line 50, in dumps
cls(buf, protocol).dump(obj)
  File "/usr/lib/python3.4/multiprocessing/queues.py", line 57, in __getstate__
context.assert_spawning(self)
  File "/usr/lib/python3.4/multiprocessing/context.py", line 347, in 
assert_spawning
' through inheritance' % type(obj).__name__
RuntimeError: Queue objects should only be shared between processes through 
inheritance

Also:

QueueHandler isn't designed to be pickleable, nor is that necessary for use 
with multiprocessing. The logging cookbook contains a working example with 
QueueHandler and multiple processes using multiprocessing, and I've just run it 
on Python 3.6 (and 3.7) with no problems. See

https://docs.python.org/3/howto/logging-cookbook.html#logging-to-a-single-file-from-multiple-processes

No idea why the sample program isn't working, and I'm sorry I haven't the time 
to look at it in detail. Please confirm if the linked sample works on your 
system, and if it does, see if you can adapt it to your needs. Note that it 
doesn't pickle any handlers.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28231] zipfile does not support pathlib

[Jeremy Freeman]

Jeremy Freeman added the comment:

This also affects python 3.5 on Windows and OSX.

Python 3.5.2 (default, Sep 21 2016, 15:07:18)
[GCC 4.2.1 Compatible Apple LLVM 7.3.0 (clang-703.0.31)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from pathlib import Path
>>> import zipfile
>>> f = Path('zipfile.zip')
>>> with zipfile.ZipFile(f) as zf:
... pass
...
Traceback (most recent call last):
  File "", line 1, in 
  File "/Users/jeremy/.pyenv/versions/3.5.2/lib/python3.5/zipfile.py", line 
1026, in __init__
self._RealGetContents()
  File "/Users/jeremy/.pyenv/versions/3.5.2/lib/python3.5/zipfile.py", line 
1089, in _RealGetContents
endrec = _EndRecData(fp)
  File "/Users/jeremy/.pyenv/versions/3.5.2/lib/python3.5/zipfile.py", line 
241, in _EndRecData
fpin.seek(0, 2)
AttributeError: 'PosixPath' object has no attribute 'seek'

--
nosy: +jtf621

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29168] multiprocessing pickle error

[Simon Schuler]

Simon Schuler added the comment:

I don't have any lock object. I just use the multiprocessing pool and a 
QueueHandler in order to be able to log from all processes.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26851] android compilation and link flags

[Xavier de Gaye]

Xavier de Gaye added the comment:

> Just tried. With my packaging scripts, CPython on ARM is compiled with 
> -mfloat-abi=softfp -mfpu=vfpv3-d16 while libffi not. test_ctypes pass as 
> usual.

This works as expected, '-mfloat-abi=softfp' and the default '-mfloat-abi=soft' 
use the same ABI [1]: "Function calls are generated to pass FP arguments 
(float, double) in integer registers (one for float, a pair of registers for 
double)".

[1] https://wiki.debian.org/ArmHardFloatPort/VfpComparison

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29168] multiprocessing pickle error

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

_thread.RLock objects were wrongly pickleable before 3.6.

>>> import pickle, pickletools, _thread
>>> lock = _thread.RLock()
>>> lock.acquire()
True
>>> lock

>>> pickle.loads(pickle.dumps(lock))

>>> pickletools.dis(pickletools.optimize(pickle.dumps(lock)))
0: \x80 PROTO  3
2: cGLOBAL '_thread RLock'
   17: )EMPTY_TUPLE
   18: \x81 NEWOBJ
   19: .STOP
highest protocol among opcodes = 2

Actually only an information about the type was saved. The state of the lock 
was lost. Unpickled lock was not a copy of the original lock, but a newly 
created object with default initial state.

There were errors in programs that pickled _thread.RLock objects. Just these 
errors were unnoticed. But programs likely didn't work as expected. Now an 
exception raised on early stage allows you to rewrite not working code. I don't 
know whether QueueHandler was designed to be pickleable. As a workaround try to 
set the lock attribute to None before pickling and set it to threading.RLock() 
(or call the createLock() method) after unpickling.

This change was made in issue22995.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29168] multiprocessing pickle error

[Simon Schuler]

Simon Schuler added the comment:

I want to handle the logging of the main and all my started processes. They 
should all log to the same Queue. 

Have a look at the sample.py program. In addition there is a inconsistency in 
using a multiprocessing pool or just the process class directly. The sample 
program illustrated this.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29169] update zlib to 1.2.10

[Matthias Klose]

New submission from Matthias Klose:

These are the changes updating zlib from 1.2.8 to 1.2.10. It is only used when 
building without a system zlib.  The new release includes fixes for security 
issues CVE-2016-9840, CVE-2016-9841, CVE-2016-9842, CVE-2016-9843.

Intending to update all active branches. Larry, is it ok to add this before the 
upcoming 3.4 and 3.5 releases, or should it wait?

Changes in 1.2.10 (2 Jan 2017)
- Avoid warnings on snprintf() return value
- Fix bug in deflate_stored() for zero-length input
- Fix bug in gzwrite.c that produced corrupt gzip files
- Remove files to be installed before copying them in Makefile.in
- Add warnings when compiling with assembler code

Changes in 1.2.9 (31 Dec 2016)
- Fix contrib/minizip to permit unzipping with desktop API [Zouzou]
- Improve contrib/blast to return unused bytes
- Assure that gzoffset() is correct when appending
- Improve compress() and uncompress() to support large lengths
- Fix bug in test/example.c where error code not saved
- Remedy Coverity warning [Randers-Pehrson]
- Improve speed of gzprintf() in transparent mode
- Fix inflateInit2() bug when windowBits is 16 or 32
- Change DEBUG macro to ZLIB_DEBUG
- Avoid uninitialized access by gzclose_w()
- Allow building zlib outside of the source directory
- Fix bug that accepted invalid zlib header when windowBits is zero
- Fix gzseek() problem on MinGW due to buggy _lseeki64 there
- Loop on write() calls in gzwrite.c in case of non-blocking I/O
- Add --warn (-w) option to ./configure for more compiler warnings
- Reject a window size of 256 bytes if not using the zlib wrapper
- Fix bug when level 0 used with Z_HUFFMAN or Z_RLE
- Add --debug (-d) option to ./configure to define ZLIB_DEBUG
- Fix bugs in creating a very large gzip header
- Add uncompress2() function, which returns the input size used
- Assure that deflateParams() will not switch functions mid-block
- Dramatically speed up deflation for level 0 (storing)
- Add gzfread(), duplicating the interface of fread()
- Add gzfwrite(), duplicating the interface of fwrite()
- Add deflateGetDictionary() function
- Use snprintf() for later versions of Microsoft C
- Fix *Init macros to use z_ prefix when requested
- Replace as400 with os400 for OS/400 support [Monnerat]
- Add crc32_z() and adler32_z() functions with size_t lengths
- Update Visual Studio project files [AraHaan]

--
assignee: doko
components: Extension Modules
files: zlib-1.2.10.diff
keywords: patch
messages: 284749
nosy: doko, larry
priority: normal
severity: normal
status: open
title: update zlib to 1.2.10
versions: Python 2.7, Python 3.4, Python 3.5, Python 3.6, Python 3.7
Added file: http://bugs.python.org/file46161/zlib-1.2.10.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29168] multiprocessing pickle error

[Vinay Sajip]

Vinay Sajip added the comment:

It may be that _thread.RLock objects were pickleable before 3.6 but aren't now, 
and if so, then this is unrelated to logging except by coincidence.

Why do you want to pickle the *handler*? It's not especially designed for 
sending over the wire, and that's not supported - so it's not a bug. If that 
code worked in Python < 3.6, that's just by luck.

Pickling of LogRecords should work.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29164] make test always fail at 218/405 ( AssertionError: ', ' not found in '1234.5' )

[Eric V. Smith]

Changes by Eric V. Smith :


--
nosy: +eric.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28180] sys.getfilesystemencoding() should default to utf-8

[Barry A. Warsaw]

Barry A. Warsaw added the comment:

On Jan 05, 2017, at 11:11 AM, STINNER Victor wrote:

>I'm sure that many Linux, UNIX and BSD systems don't have the "C.UTF-8"
>locale. For example, HP-UX has "C.utf8" which is not exactly "C.UTF-8".
>
>I'm not sure that it's ok in 2017 to always force the UTF-8 encoding if the
>user locale uses a different encoding.

It's not just any different encoding, it's specifically C (implicitly,
C.ASCII).

>I proposed an opt-in option to force UTF-8: -X utf8 command line option and
>PYTHONUTF8=1 env var. Opt-in will obviously reduce the risk of backward
>compatibility issues. With an opt-in option, users are better prepared for
>mojibake issues.

If this is true, then I would like a configuration option to default this on.
As mentioned, Debian and Ubuntu already have C.UTF-8 and most environments
(although not all, see my sbuild/schroot comment earlier) will at least be
C.UTF-8.  Perhaps it doesn't matter then, but what I really want is that for
those few odd outliers (e.g. schroot), Python would act the same inside and
out those environments.  I really don't want people to have to add that envar
or switch (or even export LC_ALL) to get proper build behavior.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29168] multiprocessing pickle error

[Davin Potts]

Changes by Davin Potts :


--
nosy: +vinay.sajip

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29168] multiprocessing pickle error

[Davin Potts]

Davin Potts added the comment:

In your first example (not sample.py), if I make the following change:

#queue = multiprocessing.Manager().Queue(-1)
queue = 42


I am still able to reproduce the exception you observe.  This suggests the 
issue has nothing to do with the use of multiprocessing but does have to do 
with the logging handlers.

I can also reproduce that this is a change in behavior between 3.5 and 3.6.

--
nosy: +davin

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26851] android compilation and link flags

[Chi Hsuan Yen]

Chi Hsuan Yen added the comment:

> Note that the system libffi must also be compiled with these same flags

Just tried. With my packaging scripts, CPython on ARM is compiled with 
-mfloat-abi=softfp -mfpu=vfpv3-d16 while libffi not. test_ctypes pass as usual. 
Maybe ctypes test suite is not complete?

And by the way, if all packages on a system should be compiled with the same 
set of flags, shouldn't those flags specified in build/package scripts rather 
than Makefile of individual packages?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29168] multiprocessing pickle error

[Simon Schuler]

Simon Schuler added the comment:

Attached is a sample program to illustrate the problem. When I use a 
multiprocessing pool the exception is raised.

--
Added file: http://bugs.python.org/file46160/sample.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26865] Meta-issue: support of the android platform

[Xavier de Gaye]

Xavier de Gaye added the comment:

issue #28833: cross compilation of third-party extension modules

--
dependencies: +byte-compile fails for cross-builds, cross compilation of 
third-party extension modules

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4431] Distutils MSVC doesn't create manifest file

[Mark Lawrence]

Changes by Mark Lawrence :


--
nosy:  -BreamoreBoy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28180] sys.getfilesystemencoding() should default to utf-8

[Nick Coghlan]

Nick Coghlan added the comment:

The trade-offs here are incredibly complex (and are mainly a matter of deciding 
whose code and configurations we want to break in 3.7+), so I think competing 
PEPs are going to be better than attempting to create a combined PEP that tries 
to cover all the options.

That way each PEP can argue as strongly as it can for the respective authors 
preferred approach to tackling the default C locale problem, even if they point 
to a common background section in one of the PEPs (similar to the way PEPs 522 
and 524 shared a common problem definition, even though they proposed different 
ways of handling it).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4431] Distutils MSVC doesn't create manifest file

[Charles Brossollet]

Charles Brossollet added the comment:

Here with python 2.7.13 and Visual Studio 2013, I get hit by the problem and it 
looks like link.exe needs the /MANIFEST keyword. Voting to get it fixed!

--
nosy: +Charles Brossollet

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29158] Possible glitch in the interaction of a thread and a multiprocessing manager

[luke_16]

luke_16 added the comment:

Relating to the idea that it is not recommended to spawn a process whenever 
there are already spawned threads running, which would be the case of the 
server side of my example, I have to disagree. If a process is supposed to be 
completely independent of the current one (no shared memory), than this should 
not be a problem. Anyway, I changed the server.py module just to clear this 
possible error source.

Trying to simplify the example, I moved everything to the client.py module and 
eliminated the "thread spawns process" part in order to see what happens. 
Actually, I don't think that that was the case because I could not see any 
process being spawned when trying to connect to the server. I think that this 
happens in the current MainThread.

In the new client, I'm doing exactly what I was trying to avoid, which is to 
wait for the connect() method to return. Surprisingly, the problem is still 
there, in a way. Let's say you start the server and then the client. The first 
attempt to send something fails and the client tries to connect to the server. 
All goes well with the connection, and the client starts sending stuff to the 
server. If you stop the server, wait a while (or not), and restart it again, 
the client can no longer send anything, and tries to reconnect again. The 
connection is then successful, but immediately after that, it cannot send 
anything because of a BrokenPipeError. Only after the exception is raised and 
the next loop of the "while" begins, the client can send something to the 
server again. If you insert a time.sleep(x) of any number of seconds inside the 
scope of the first "else" (between line 26 and 38) and right after a 
"remote_buffer.put('anything')", it will raise a BrokenPipeError. Only when a 
new l
 oop in "while" begins, then it is possible again to send something to the 
server.

This makes no sense to me. There must be something wrong with it.

--
Added file: http://bugs.python.org/file46159/case2.zip

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26851] android compilation and link flags

[Xavier de Gaye]

Xavier de Gaye added the comment:

In the code review Victor asked the following question:

> I'm a little bit surprised that you need to pass so much options which are 
> specific to the platform. GCC doesn't have a generic option "hello, please 
> compile for my architecture?"

ARM defines two incompatible floating point ABIS that specify the functions 
calling conventions, whether to use integer registers (soft floating point ABI) 
or the floating point registers (hard floating point ABI).  The compiler must 
be told which one to use and whether the hardware FPU may be used if present 
when the soft floating point ABI has been chosen. This last case is set with 
the command line option '-mfloat-abi=softfp'. The following two documents 
describe this feature in details:
  https://wiki.debian.org/ArmHardFloatPort/VfpComparison
  https://wiki.debian.org/ArmHardFloatPort

The following command prints the command line options, including implicitly 
predefined ones of the clang compiler in android-ndk-r13b and shows that the 
default is '-mfloat-abi=soft' for armv7:

$ ./clang -x c < /dev/null -dM -E - -target armv7-none-linux-androideabi -###
Android clang version 3.8.256229  (based on LLVM 3.8.256229)
Target: armv7-none-linux-android
Thread model: posix
InstalledDir: /opt/android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/.
 
"/home/opt.symlink/android-ndk-r13b/toolchains/llvm/prebuilt/linux-x86_64/bin/clang"
 "-cc1" "-triple" "armv7-none-linux-android" "-E" "-disable-free" 
"-disable-llvm-verifier" "-main-file-name" "-" "-mrelocation-model" "pic" 
"-pic-level" "1" "-mthread-model" "posix" "-mdisable-fp-elim" "-fmath-errno" 
"-masm-verbose" "-mconstructor-aliases" "-fuse-init-array" "-target-cpu" 
"cortex-a8" "-target-feature" "+soft-float-abi" "-target-abi" "aapcs-linux" 
"-mfloat-abi" "soft" "-target-linker-version" "2.24" "-dwarf-column-info" 
"-debugger-tuning=gdb" "-resource-dir" 
"/home/opt.symlink/android-ndk-r13b/toolchains/llvm/prebuilt/linux-x86_64/bin/../lib64/clang/3.8.256229"
 "-internal-isystem" "/usr/local/include" "-internal-isystem" 
"/home/opt.symlink/android-ndk-r13b/toolchains/llvm/prebuilt/linux-x86_64/bin/../lib64/clang/3.8.256229/include"
 "-internal-externc-isystem" "/include" "-internal-externc-isystem" 
"/usr/include" "-fdebug-compilation-dir" 
"/opt/android-ndk/toolchains/llvm/prebuilt/linu
 x-x86_64/bin" "-ferror-limit" "19" "-fmessage-length" "100" "-femulated-tls" 
"-fallow-half-arguments-and-returns" "-fno-signed-char" "-fobjc-runtime=gcc" 
"-fdiagnostics-show-option" "-fcolor-diagnostics" "-dM" "-o" "-" "-x" "c" "-"

Note that the system libffi must also be compiled with these same flags. The 
bundled libffi is deprecated in 3.6 by issue 27976 and has been removed in 3.7 
by issue 27979.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29163] shlex error in posix mode and empty quotes

[Berker Peksag]

Changes by Berker Peksag :


--
resolution:  -> out of date
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29168] multiprocessing pickle error

[Simon Schuler]

New submission from Simon Schuler:

Hello,

the following code doesn't work any longer in the new Python version 3.6.

import sys
import os
import subprocess
from multiprocessing import Pool, Value, Queue
import multiprocessing
import logging
import logging.handlers
import pickle

queue = multiprocessing.Manager().Queue(-1)
qh = logging.handlers.QueueHandler(queue)
pickle.dumps(qh)

It raises the following exception.
>>> TypeError: can't pickle _thread.RLock objects


Furthermore, also for customized logging handler classes it doesn't work 
anymore.

class CustomHandler(logging.Handler):
def __init__(self, queue):
logging.Handler.__init__(self)
self.queue = queue

def emit(self, record):
try:
ei = record.exc_info
if ei:
dummy = self.format(record)
record.exc_info = None  
except (KeyboardInterrupt, SystemExit):
raise
except:
self.handleError(record)

For a centralized logging facility in a multiprocess environment this is a big 
problem. How can I handle this in the 3.6 version?

--
messages: 284738
nosy: cxss
priority: normal
severity: normal
status: open
title: multiprocessing pickle error
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29160] pow with three int arguments works like it had two arguments

[Mark Dickinson]

Mark Dickinson added the comment:

> shall we make a report for 'numpy guys' directly?

Yes, please! I'll close here, since this isn't a core Python issue: what 3rd 
party libraries choose to do in their `__pow__` methods is out of the control 
of the core language.

--
nosy: +mark.dickinson
resolution:  -> not a bug
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28180] sys.getfilesystemencoding() should default to utf-8

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

While going for the full locale setting may be a good option,
perhaps just focusing on the FS encoding for now is a better
way forward (and also more in line with the ticket title).

So essentially go for the PEP 529 approach on Unix as well
(except that we use 'ascii' as fallback in legacy mode):

https://www.python.org/dev/peps/pep-0529/

The PEP also includes a section on affected modules, which we
could double check (even though the term "FS encoding" implies
that only file system relevant APIs are touched by such a change,
the encoding is used in several other places as well):

https://www.python.org/dev/peps/pep-0529/#id14

For Windows, a couple of modules such as pwd and nis are not
used, so those may need some extra attention.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29145] failing overflow checks in replace_*

[Xiang Zhang]

Xiang Zhang added the comment:

> It seems to me that unicodeobject.c should be changed in 3.x too.

I don't understand. Would you mind tell me which part? The only suspicious part 
to me is `PY_SSIZE_T_MAX >> (rkind - 1)`. It seems should be `PY_SSIZE_T_MAX / 
rkind`.

And I missed jan's patch so wrote mine. :-( But they are almost the same. :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29145] failing overflow checks in replace_*

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

It seems to me that unicodeobject.c should be changed in 3.x too.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29157] random.c: Prefer getrandom() over getentropy(), handle ENOSYS in py_getentropy()

[Christian Heimes]

Christian Heimes added the comment:

I'm doing a review now.

By the way I did not copy random.c for cryptography. I took bits and pieces out 
of it. 
https://github.com/pyca/cryptography/blob/master/src/_cffi_src/openssl/src/osrandom_engine.c
 and 
https://github.com/pyca/cryptography/blob/master/src/_cffi_src/openssl/src/osrandom_engine.h

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29163] shlex error in posix mode and empty quotes

[Evan]

Evan added the comment:

My mistake, it was addressed directly in issue21999. This is already fixed in 
2.7.13 and will be fixed in 3.5.3.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29145] failing overflow checks in replace_*

[Xiang Zhang]

Xiang Zhang added the comment:

> some instances are present in unicodeobject.c too

Thanks for your notification.

v3 adds the some changes needed I could find in unicodeobject.c. Nothing in v2 
is changed.

--
Added file: http://bugs.python.org/file46158/replace-overflow-check-v3.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29157] random.c: Prefer getrandom() over getentropy(), handle ENOSYS in py_getentropy()

[STINNER Victor]

STINNER Victor added the comment:

Nick: Thanks for the review!

Since random.c is critical for security, I would prefer to have at least a 
review from another core developer. I added Serhiy and Xiang in the nosy list. 
I'm also looking at you, Christian! ;-) (Christian reused random.c code in the 
pycrytography project.)

--
nosy: +serhiy.storchaka, xiang.zhang

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28180] sys.getfilesystemencoding() should default to utf-8

[STINNER Victor]

STINNER Victor added the comment:

Sorry, I still didn't have enough time to read carefully the PEP 538. But since 
the discussion already started on this issue, I will add my comments:

* I'm sure that many Linux, UNIX and BSD systems don't have the "C.UTF-8" 
locale. For example, HP-UX has "C.utf8" which is not exactly "C.UTF-8".

* Setting the locale has an impact on all libraries running in the Python 
process. At this point, I'm not sure that it is what we want.

* I'm not sure that it's ok in 2017 to always force the UTF-8 encoding if the 
user locale uses a different encoding. I had the same concern with the PEP 528 
(Change Windows console encoding to UTF-8) and PEP 529 (Change Windows 
filesystem encoding to UTF-8) on Windows, but these PEPs were approved and 
merged into Python 3.6. My fear is obviously mojibake with the other 
applications using the other encoding, the locale encoding. Other applications 
are not impacted by setlocale() in the Python process.

* I proposed an opt-in option to force UTF-8: -X utf8 command line option and 
PYTHONUTF8=1 env var. Opt-in will obviously reduce the risk of backward 
compatibility issues. With an opt-in option, users are better prepared for 
mojibake issues.

* I dislike "Backporting to earlier Python 3 releases". In my experience, 
changes on how Python handles text (encodings, codecs, etc.) always have subtle 
issues, and users dislike getting backward incompatible changes in minor 
releases. *Maybe* if the option is an opt-in, the risk is lower and acceptable?

* I dislike that Fedora has such downstream change. I would prefer to decide 
upstream how to convert UTF-8 slowly as a first-class citizen in Python. 
Otherwise, Fedora would behave differently than other Linux distributions and 
it can be painful to write applications having the same behaviour on all Linux 
distributions. But I also understand that Fedora has sometimes to move faster 
than the slow CPython project :-) Fedora can also seen as a toy to experiment 
changes quickly which helps to provide a wide feedback upstream to take better 
decision.

* Using strict or surrogateescape error handler is a very important choice 
which has a wide impact. If we use utf8 by default (PEP 538), people will 
problably complain less if Python magically pass undecoded bytes thanks to the 
surrogateescape. If the option is an opt-in, strict may make sense. But 
surrogateescape is maybe still more "convenient". I don't know at this point.

Nick: it seems like you have a well defined plan. But I dislike on multiple 
points. I don't know if it's better to try to convince you to change your PEP, 
or write a different PEP.

I planned to write such "UTF-8" PEP since 2015, but I never started because the 
scope is so large that I fear all tiny but annoying corner cases...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29145] failing overflow checks in replace_*

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

replace-overflow-check-v2.patch LGTM.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29157] random.c: Prefer getrandom() over getentropy(), handle ENOSYS in py_getentropy()

[Nick Coghlan]

Nick Coghlan added the comment:

New patch looks good to me, and +1 on applying the refactoring to all supported 
branches.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29167] Race condition in enum.py:_decompose()

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
components: +IO
nosy: +ethan.furman
type: crash -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29163] shlex error in posix mode and empty quotes

[Evan]

Evan added the comment:

This is also broken in 3.5.2, but not in 3.6.0:

>>> list(shlex.shlex('echo b="",echo bar', posix=True))
['echo', 'b', '=', '', ',', 'echo', 'bar']

I'm guessing it was fixed incidentally by the changes in issue1521950.

--
versions: +Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28180] sys.getfilesystemencoding() should default to utf-8

[Nick Coghlan]

Nick Coghlan added the comment:

The PEP already explains how other runtimes achieve UTF-8 and UTF-18-LE 
everywhere: by ignoring the C/C++ locale entirely. While this breaks 
integration with other C/C++ components, the developers of those languages and 
runtimes simply don't care, as they never supported integrating with those 
components in the first place.

CPython doesn't have that luxury, since it is used extensively in locale aware 
desktop applications.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29167] Race condition in enum.py:_decompose()

[Simon Percivall]

New submission from Simon Percivall:

When called by `_create_pseudo_member_()`, the dictionary iteration of 
`_value2member_map` in `_decompose()` in enum.py may lead to a "RuntimeError: 
dictionary changed size during iteration". For me, it happened in `re.compile`.

```
Traceback (most recent call last):
  ...
  File 
"/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/re.py",
 line 233, in compile
return _compile(pattern, flags)
  File 
"/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/re.py",
 line 301, in _compile
p = sre_compile.compile(pattern, flags)
  File 
"/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/sre_compile.py",
 line 562, in compile
p = sre_parse.parse(p, flags)
  File 
"/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/sre_parse.py",
 line 866, in parse
p.pattern.flags = fix_flags(str, p.pattern.flags)
  File 
"/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/sre_parse.py",
 line 835, in fix_flags
flags |= SRE_FLAG_UNICODE
  File 
"/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/enum.py",
 line 794, in __or__
result = self.__class__(self._value_ | self.__class__(other)._value_)
  File 
"/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/enum.py",
 line 291, in __call__
return cls.__new__(cls, value)
  File 
"/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/enum.py",
 line 533, in __new__
return cls._missing_(value)
  File 
"/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/enum.py",
 line 760, in _missing_
new_member = cls._create_pseudo_member_(value)
  File 
"/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/enum.py",
 line 769, in _create_pseudo_member_
_, extra_flags = _decompose(cls, value)
  File 
"/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/enum.py",
 line 849, in _decompose
for v, m in flag._value2member_map_.items()
  File 
"/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/enum.py",
 line 848, in 
(m, v)
RuntimeError: dictionary changed size during iteration
```

--
messages: 284724
nosy: simon.percivall
priority: normal
severity: normal
status: open
title: Race condition in enum.py:_decompose()
type: crash
versions: Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29157] random.c: Prefer getrandom() over getentropy(), handle ENOSYS in py_getentropy()

[STINNER Victor]

STINNER Victor added the comment:

New patch (version 2), much larger: it refactors the code, not only fix this 
specific issue (prefer getrandom() over getentropy()). Changes since 
getentropy.patch:

* Add a lot of comments to explain in depth how each function is implemented, 
which errors are handled, etc. It should help to audit the code: this code is 
very critical for security and so should be, IMHO, well documented.

* handle also EPERM and EINTR errors in getentropy(): retry on EINTR, fallback 
on /dev/urandom on EPERM -- sadly, I don't have access to a system with 
getentropy() to test this part of the code.

* call py_getrandom() and py_getentropy() in pyurandom() to make dev_urandom() 
simpler and so easy to review: dev_urandom() looses its blocking parameter

* Document the cached file descriptor, and cached st_dev+st_ino in 
dev_urandom().

* Document explicitly that functions are retried on EINTR error. Document that 
only getrandom() supports non-blocking mode. Document why we prefer an entropy 
source over others.

I'm not sure that getentropy() can fail with EPERM or EINTR in practice, but it 
shouldn't harm to handle correctly these errors :-) At least, getentropy() can 
fail with these errors on Linux since the glibc implements the getentropy() 
function using the getrandom() syscall (and it's known that getrandom() can 
fail with these errors). But on Linux, the code now prefers getrandom() over 
getentropy().

Should we use the new shiny code on all Python versions? Or only fix the 
reported issue on all Python issues, and use the refactored code in Python 
default?

Note: Python 2.7 still supports VMS. VMS is unsupported in Python 3.3 and the 
VMS code was removed in Python 3.4 (issue 16136): see the PEP 11.

I suggest to use the same code on all maintained Python versions to ease 
maintenance.

--
Added file: http://bugs.python.org/file46157/random-2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28180] sys.getfilesystemencoding() should default to utf-8

[Nick Coghlan]

Nick Coghlan added the comment:

No, requesting a locale that doesn't exist doesn't error out, because we don't 
check the return code - it just keeps working the same way it does now (i.e. 
falling back to the legacy C locale).

However, it would be entirely reasonable to put together a competing PEP 
proposing to eliminate the reliance on the problematic libc APIs, and instead 
use locale independent replacements. I'm simply not offering to implement or 
champion such a PEP myself, as I think ignoring the locale settings rather than 
coercing them to something more sensible will break integration with C/C++ GUI 
toolkits like Tcl/Tk, Gtk, and Qt, and it's reasonable for us to expect OS 
providers to offer at least one of C.UTF-8 or en_US.UTF-8 (see 
https://github.com/python/peps/issues/171 for more on that).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29163] shlex error in posix mode and empty quotes

[Evan]

Changes by Evan :


--
nosy: +evan_

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29145] failing overflow checks in replace_*

[jan matejek]

jan matejek added the comment:

some instances are present in unicodeobject.c too

--
Added file: http://bugs.python.org/file46156/unicode-overflow.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28180] sys.getfilesystemencoding() should default to utf-8

[INADA Naoki]

INADA Naoki added the comment:

Why I want to add configure option to ignore locale is:


1. C.UTF-8 is not supported by RHEL7 
(https://bugzilla.redhat.com/show_bug.cgi?id=1361965)

RHEL7 will be used for a long time.
And many people uses new Python instead of distro's Python, via pyenv or 
pythonz.
I feel deprecating C locale from Python 3.7 is bit aggressive.


2. Many admins like C locale.

locale setting will cause unintended side effects. So many admins dislike 
xx_XX.UTF-8 locale.
For example (from 
https://fumiyas.github.io/2016/12/25/dislike.sh-advent-calendar.html ):

$ mkdir tmp
$ cd tmp
$ touch a b c x y z A B C X Y Z
$ LC_ALL=C /bin/bash --noprofile --norc -c 'echo [A-Z]'
A B C X Y Z
$ LC_ALL=en_US.UTF-8 /bin/bash --noprofile --norc -c 'echo [A-Z]'
A b B c C x X y Y z Z


3. Many other languages can use UTF-8 even when C locale

node.js, Ruby, Rust, Go can use UTF-8 on Linux
People don't want to learn how to configure locale properly only for Python.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28180] sys.getfilesystemencoding() should default to utf-8

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 05.01.2017 10:26, Nick Coghlan wrote:
> 
> Anything purely on the Python side of things doesn't work in a traditional C 
> environment - CPython relies on the C lib to do conversions during startup, 
> so we need the C locale to be set correctly. We can do things differently on 
> Mac OS X and iOS because Apple ensure that *C* behaves differently on Mac OS 
> X and iOS (and apparently Google do something similar for Android, so I'll 
> update the PEP to mention that as well).

I believe IANADA-san (hope that's the right way to address him)
raised a good point though: what if a system doesn't come with
the C.UTF-8 local setup ?

The C lib would then error out when trying to use setlocale()
on such an environment.

Now, Python's main() function doesn't look at any such errors
(and neither do the other places which use it such as frozenmain.c
and readline.c), so it wouldn't even notice.

The setlocal() man-page doesn't mention how such a failure would
affect the current locale settings. My guess is that the locale
remains set to what it was before, which in case of a fresh C
application start is the "C" locale.

So in the implementation of the PEP, there should be a test
to see whether "C.UTF-8" does result in a successful
call to setlocale(). If it doesn't, there would have to be
some work-around to still make Python's FS encoding happy
while leaving the C lib locale set at "C".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28180] sys.getfilesystemencoding() should default to utf-8

[INADA Naoki]

INADA Naoki added the comment:

> Anything purely on the Python side of things doesn't work in a traditional C 
> environment - CPython relies on the C lib to do conversions during startup, 
> so we need the C locale to be set correctly. 

What I propose is non't use mbstowcs, like __ANDROID__

wchar_t*
Py_DecodeLocale(const char* arg, size_t *size)
{
#if defined(__APPLE__) || defined(__ANDROID__)
wchar_t *wstr;
wstr = _Py_DecodeUTF8_surrogateescape(arg, strlen(arg));


On Linux, command line arguments and filepath is just a byte sequence.
So using UTF-8:surrogateescape from during startup should works fine.

Am I wrong?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29142] urllib: no_proxy variable values with leading dot not properly handled

[Berker Peksag]

Berker Peksag added the comment:

Yes, thanks!

--
nosy: +martin.panter, xiang.zhang
stage:  -> patch review
type:  -> behavior
versions: +Python 3.5, Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28180] sys.getfilesystemencoding() should default to utf-8

[Nick Coghlan]

Nick Coghlan added the comment:

Anything purely on the Python side of things doesn't work in a traditional C 
environment - CPython relies on the C lib to do conversions during startup, so 
we need the C locale to be set correctly. We can do things differently on Mac 
OS X and iOS because Apple ensure that *C* behaves differently on Mac OS X and 
iOS (and apparently Google do something similar for Android, so I'll update the 
PEP to mention that as well).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29023] Results of random.seed() call with integer argument should be claimed deterministic.

[Raymond Hettinger]

Raymond Hettinger added the comment:

Attached patch simplifies, improves, and syncs the main docs with the 
docstrings.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29023] Results of random.seed() call with integer argument should be claimed deterministic.

[Raymond Hettinger]

Changes by Raymond Hettinger :


--
keywords: +patch
Added file: http://bugs.python.org/file46155/random_docs.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29023] Results of random.seed() call with integer argument should be claimed deterministic.

[Raymond Hettinger]

Changes by Raymond Hettinger :


--
Removed message: http://bugs.python.org/msg284677

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29142] urllib: no_proxy variable values with leading dot not properly handled

[Milan Oberkirch]

Milan Oberkirch added the comment:

Too bad, I was all exited when I saw that I can paste a PR link here :)

I downloaded the patch from the commit using github, I hope that works?

--
keywords: +patch
Added file: http://bugs.python.org/file46154/ignore_dots-v1.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com