[issue34172] multiprocessing.Pool and ThreadPool leak resources after being deleted

[tzickel]

tzickel  added the comment:

here is something quick I did to check if it works (it works) but I'm not 
fluent in multiprocessing code, so If i'm missing something or doing something 
wrong feel free to tell me:

https://github.com/tzickel/cpython/commit/ec63a43706f3bf615ab7ed30fb095607f6101e26

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34172] multiprocessing.Pool and ThreadPool leak resources after being deleted

[tzickel]

tzickel  added the comment:

A. It would be nice to add a test that tests this.
B. Now that Pool is cleaning up properly, any of it's functions which return 
another object (like imap's IMapIterator) need to hold a reference to the Pool, 
so it won't get cleanedup before computing.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35228] Index search in CHM help crashes viewer

[Ma Lin]

Ma Lin  added the comment:

I suffered this problem more than one years.
Here is a solution, before compiling the chm, modify like this:

--- D:\Python-3.7.1\Doc\build\htmlhelp\python371.hhpSun Dec 02 13:12:37 2018
+++ D:\fix_crash\python371.hhp  Sun Dec 02 13:05:57 2018
@@ -1,6 +1,6 @@
 [OPTIONS]
 Binary TOC=No
-Binary Index=No
+Binary Index=Yes
 Compiled file=python371.chm
 Contents file=python371.hhc
 Default Window=python371
 
The chm will generate a binary file python371.chw in the same folder when first 
opened.
No longer crash, but duplicated entries don't show Title anymore, see the 
attached picture.

--
nosy: +Ma Lin
Added file: https://bugs.python.org/file47967/binary_index.png

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34172] multiprocessing.Pool and ThreadPool leak resources after being deleted

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

This also happens on Linux, hanging in different moments:

❯ ./python test.py
Begin
4
[hangs]

❯ ./python test.py
Begin
4
3
[hangs]

❯ ./python test.py
[hangs]

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34172] multiprocessing.Pool and ThreadPool leak resources after being deleted

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

Bisecting between 3.7.0(good) and 3.7.1(bad) with the code in my previous 
commit points to:

97f998a4dfd6db6d867f446daa62445d0782bf39 is the first bad commit
commit 97f998a4dfd6db6d867f446daa62445d0782bf39
Author: Miss Islington (bot) <31488909+miss-isling...@users.noreply.github.com>
Date:   Tue Oct 2 14:17:04 2018 -0700

bpo-34172: multiprocessing.Pool leaks resources after being deleted 
(GH-8450) (GH-9676)

Fix a reference issue inside multiprocessing.Pool that caused the pool to 
remain alive if it was deleted without being closed or terminated explicitly.
(cherry picked from commit 97bfe8d3ebb0a54c8798f57555cb4152f9b2e1d0)

Co-authored-by: tzickel 

:04 04 90e0af29e82d0fcc1c1d7e19b3659f9602596e3e 
d997cfb00c1c44bbd87ce15edfd391203362a1d7 M  Lib
:04 04 6aa4273821c2c563ea69370a59284ed48576416f 
b6d46f14b6bb36cc5ae62b3ca74025c24d683bb5 M  Misc
bisect run success

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34172] multiprocessing.Pool and ThreadPool leak resources after being deleted

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

multiprocessing.Pool.imap hangs in MacOs after applying this commit:

import multiprocessing

def the_test():
print("Begin")
for x in multiprocessing.Pool().imap(int,
["4", "3"]):
print(x)
print("End")

the_test()

This also happens in the backported branches.

--
nosy: +pablogsal
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: multiple JSON documents in one file, change proposal

[Akkana Peck]

Grant Edwards  writes:
> This is what "archive" file formats are for.  Just use tar, zip, ar,
> cpio, or some other file format designed to store multiple "files" of
> arbitrary data -- there are plenty of "standard" formats to choose
> from and plenty of libraries to deal with them.

Then the data files wouldn't be human readable, making debugging
a lot more hassle.

Cameron Simpson writes:
> There's a common format called Newline Delimited JSON (NDJSON) for just this
> need.
> 
> Just format the outbound records as JSON with no newlines (i.e. make the
> separator a space or the empty string), put a newline at the end of each.
> 
> On ingest, read lines of text, and JSON parse each line separately.

I'll second that. It's very easy to deal with. A shorter name for it
is JSONL -- I use .jsonl for filenames.
https://en.wikipedia.org/wiki/JSON_streaming#Line-delimited_JSON
discusses that as well as several other solutions.

...Akkana
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: ValueError vs IndexError, unpacking arguments with string.split

[Chris Angelico]

On Sun, Dec 2, 2018 at 11:55 AM Morten W. Petersen  wrote:
>
> On Sat, Dec 1, 2018 at 1:11 AM Chris Angelico  wrote:
>>
>> On Sat, Dec 1, 2018 at 10:55 AM Morten W. Petersen  wrote:
>> > But this raises the question of how to write Python code,
>> > short and sweet, that could handle infinite iterators in
>> > such an unpack with multiple variables to assign to.
>> >
>> > Which I guess is mostly theoretical, as there are other
>> > ways of designing code to avoid needing to unpack
>> > an infinite iterator using the * prefix.
>>
>> It could only be done with the cooperation of the iterable in
>> question. For instance, a range object could implement a "remove
>> first" operation that does this, and several itertools types wouldn't
>> need to change at all. But it can't be done generically other than the
>> way it now is (pump the iterator the rest of the way).
>
> I wasn't able to follow this, could you elaborate?
>

The way *x works in unpacking is that the entire iterable gets
unpacked, and everything gets put into a list that is then assigned to
x. This is generic, works on any iterable, but doesn't take advantage
of anything. Consider:

>>> x = range(3, 20, 5)
>>> first, *rest = x
>>> first
3
>>> rest
[8, 13, 18]

If a range object were asked to yield its first-and-rest in this way,
it could instead return range(8, 20, 5) - add the step onto the start,
job done. Same if you ask for some off the beginning and some off the
end And with a number of the itertools iterables/iterators, the "rest"
wouldn't actually need to change anything, since the iterator will get
consumed. This would need explicit support from the iterable, though,
as Python can't know how to do this generically; so there would need
to be a protocol for "unpack".

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: ValueError vs IndexError, unpacking arguments with string.split

[Morten W. Petersen]

On Sat, Dec 1, 2018 at 1:11 AM Chris Angelico  wrote:

> On Sat, Dec 1, 2018 at 10:55 AM Morten W. Petersen 
> wrote:
> > But this raises the question of how to write Python code,
> > short and sweet, that could handle infinite iterators in
> > such an unpack with multiple variables to assign to.
> >
> > Which I guess is mostly theoretical, as there are other
> > ways of designing code to avoid needing to unpack
> > an infinite iterator using the * prefix.
>
> It could only be done with the cooperation of the iterable in
> question. For instance, a range object could implement a "remove
> first" operation that does this, and several itertools types wouldn't
> need to change at all. But it can't be done generically other than the
> way it now is (pump the iterator the rest of the way).
>


I wasn't able to follow this, could you elaborate?

Regards,

Morten


-- 
Videos at https://www.youtube.com/user/TheBlogologue
Twittering at http://twitter.com/blogologue
Blogging at http://blogologue.com
Playing music at https://soundcloud.com/morten-w-petersen
Also playing music and podcasting here:
http://www.mixcloud.com/morten-w-petersen/
On Google+ here https://plus.google.com/107781930037068750156
On Instagram at https://instagram.com/morphexx/
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue16516] argparse types (and actions) must be hashable

[Luna Chen]

Luna Chen  added the comment:

Hi bradengroom,
I have reviewed your PR, it's just a small thing! If you could update your PR, 
it would be amazing!

Thank you!

--
nosy: +BNMetrics

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35369] List sorting makes duplicate comparisons

[David Wyde]

David Wyde  added the comment:

Okay. Thanks!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35341] Add generic version of OrderedDict to typing module

[Raymond Hettinger]

Raymond Hettinger  added the comment:

Now that regular dicts are ordered, my expectation is that OrderedDict() is 
going to mostly fall into disuse (much like UserDict, UserList, UserString).

That said, it would be nice if all of the collections classes had generic 
counterparts in the typing module.

--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35369] List sorting makes duplicate comparisons

[Raymond Hettinger]

Raymond Hettinger  added the comment:

> I've heard of using human choices for comparisons, when fewer decisions could 
> provide a notable speedup.

Memoization is the usual solution when expensive computations are being 
repeated.  That technique preserves the work that was done and it avoids adding 
unnecessary complexity to the consumer code.

> Do you think it's worth posting to python-ideas to see 
> what people's use cases are?

No, that would be a waste of time and it wouldn't change the validity of Tim's 
insights.

--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: multiple JSON documents in one file, change proposal

[Grant Edwards]

On 2018-11-30, Marko Rauhamaa  wrote:
> Paul Rubin :
>> Maybe someone can convince me I'm misusing JSON but I often want to
>> write out a file containing multiple records, and it's convenient to
>> use JSON to represent the record data.
>>
>> The obvious way to read a JSON doc from a file is with "json.load(f)"
>> where f is a file handle. Unfortunately, this throws an exception
>
> I have this "multi-JSON" need quite often. In particular, I exchange
> JSON-encoded messages over byte stream connections. There are many ways
> of doing it. Having rejected different options ( https://en.wikipedia.org/wiki/JSON_streaming>), I settled with
> terminating each JSON value with an ASCII NUL character, which is
> illegal in JSON proper.

This is what "archive" file formats are for.  Just use tar, zip, ar,
cpio, or some other file format designed to store multiple "files" of
arbitrary data -- there are plenty of "standard" formats to choose
from and plenty of libraries to deal with them.

-- 
Grant

-- 
https://mail.python.org/mailman/listinfo/python-list

[issue27715] call-matcher breaks if a method is mocked with spec=True

[Karthikeyan Singaravelan]

Change by Karthikeyan Singaravelan :


--
nosy: +xtreak
versions: +Python 3.7, Python 3.8 -Python 3.4, Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: multiple JSON documents in one file, change proposal

[Ben Bacarisse]

Marko Rauhamaa  writes:

> Chris Angelico :
>
>> On Sat, Dec 1, 2018 at 10:16 PM Marko Rauhamaa  wrote:
>>> and the framing format is HTTP. I will need to type something like this:
>>>
>>>POST / HTTP/1.1^M
>>>Host: localhost^M
>>>Content-type: application/json^M
>>>Content-length: 54^M
>>>^M
>>>{
>>>"msgtype": "echo-req",
>>>"opid": 3487547843
>>>}
>>>
>>> That's almost impossible to type without a syntax error.
>>
>> 1) Set your Enter key to send CR-LF, at least for this operation.
>> That's half your problem solved.
>
> That can be much harder than typing ctrl-SPC. It *is* supported by
> netcat, for example, but then you have to carefully recompute the
> content-length field.
>
>> 2) Send the request like this:
>>
>> POST / HTTP/1.0
>> Content-type: application/json
>>
>> {"msgtype": "echo-req", "opid": 3487547843}
>>
>> Then shut down your end of the connection, probably with Ctrl-D. I'm
>> fairly sure I can type that without bugs, and any compliant HTTP
>> server should be fine with it.
>
> If I terminated the input, I wouldn't need any framing. The point is to
> carry a number of JSON messages/documents over a single bytestream or in
> a single file. That means the HTTP content-length would be mandatory.

I haven't been following the details of the thread, but I wonder if a
multi-part form-data POST would be useful.  Way more complex than a
simple separator, of course, but more "standard" (for some rather vague
meaning of the term).

-- 
Ben.
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue35335] msgfmt should be able to merge more than one po file

[s-ball]

s-ball  added the comment:

I have followed SilentGhost's advice and begun by thoroughly testing the 
current behaviour. I then realized that I was wrong, and that (probably by 
chance) msgfmt.py partially followed my requirements, but pybabel did not and 
GNU gettext msgfmt did not really. I wrote 2 tiny po files (joined) and played 
with them, meaning I tried to combine them with pybabel, msgfmt.py and GNU 
gettext msg. Current behaviour (Windows shell syntax):

> pybabel compile -o .\file12-fr.mo -l fr -i file1-fr.po -i file2-fr.po

only uses second file (file2-fr.po)

> msgfmt -o file12-fr.mo --no-hash file1-fr.po file2-fr.po

chokes on a repeated key on file2 (the header has "" for key...). It works fine 
anyway after commenting out the header in any of the files

> python "path\to\Tools\i18n\msgfmt.py" -o file12py-fr.mo file1-fr.po 
> file2-fr.po

unexpectedly produces the expected result and successfully combines both po 
files into one single mo file

BUT:

> python "path\to\Tools\i18n\msgfmt.py" file1-fr.po file2-fr.po

Produces file1-fr.mo which is the compiled version of file1-fr.po and 
file2-fr.mo which combines both input files. Definitely not an expected result!

This is caused by the problem identified in issue 9741 
(https://bugs.python.org/issue9741)

My initial goal was to be able to use the make function from msgfmt.py in an 
external script. I then realize that combining multiple po files is not a good 
idea because the resulting mo file can only contain one single header and the 
best behaviour is GNU gettext msgfmt one.

I now wonder whether this issue should not be closed because the requirement is 
not relevant, and it would probably better to propose a fix (including tests 
and code improvement) for issue 9741.

--
Added file: https://bugs.python.org/file47966/files.zip

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35371] Fix undefined behavior in os.utime() on Windows

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35373] PyInit_timezone() must return a value

[Serhiy Storchaka]

New submission from Serhiy Storchaka :

PyInit_timezone() is declared as returning int, but it contains return 
statements without value.

>From compiler output on Windows:

  timemodule.c
..\Modules\timemodule.c(1584): warning C4033: 'PyInit_timezone' must return a 
value [C:\py\cpython3.8\PCbuild\pythoncore.vcxproj]
..\Modules\timemodule.c(1589): warning C4033: 'PyInit_timezone' must return a 
value [C:\py\cpython3.8\PCbuild\pythoncore.vcxproj]
..\Modules\timemodule.c(1593): warning C4033: 'PyInit_timezone' must return a 
value [C:\py\cpython3.8\PCbuild\pythoncore.vcxproj]
c:\py\cpython3.8\modules\timemodule.c(1647): warning C4715: 'PyInit_timezone': 
not all control paths return a value 
[C:\py\cpython3.8\PCbuild\pythoncore.vcxproj]

--
components: Extension Modules
messages: 330858
nosy: serhiy.storchaka, vstinner
priority: normal
severity: normal
status: open
title: PyInit_timezone() must return a value
type: compile error

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31177] unittest mock's reset_mock throws an error when an attribute has been deleted

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

I am closing this as fixed since all the PRs were merged. Feel free to reopen 
this if needed. Thanks @mariocj89 and @vstinner for the review.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35365] Use wchar_t* buffer instead of Unicode object in code page decoder

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
dependencies: +Code page decoder incorrectly handles input >2GiB

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35372] Code page decoder incorrectly handles input >2GiB

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
keywords: +patch
pull_requests: +10083
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35332] shutil.rmtree(..., ignore_errors=True) doesn't ignore errors from os.close()

[Ronal Abraham]

Ronal Abraham  added the comment:

I forgot to mention: the exception raised is an OSError and the errno is 52 
(ESTALE on AIX).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35372] Code page decoder incorrectly handles input >2GiB

[Serhiy Storchaka]

New submission from Serhiy Storchaka :

>>> b = b'a'*(2**31-2)+b'\xff'*2
>>> x, y = codecs.code_page_decode(932, b, 'replace', True)
>>> len(x)
2
>>> x, y
('aa', 2147483648)

--
assignee: serhiy.storchaka
components: Interpreter Core
messages: 330855
nosy: serhiy.storchaka, vstinner
priority: normal
severity: normal
status: open
title: Code page decoder incorrectly handles input >2GiB
type: behavior
versions: Python 3.6, Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: multiple JSON documents in one file, change proposal

[Marko Rauhamaa]

Chris Angelico :

> On Sat, Dec 1, 2018 at 10:16 PM Marko Rauhamaa  wrote:
>> and the framing format is HTTP. I will need to type something like this:
>>
>>POST / HTTP/1.1^M
>>Host: localhost^M
>>Content-type: application/json^M
>>Content-length: 54^M
>>^M
>>{
>>"msgtype": "echo-req",
>>"opid": 3487547843
>>}
>>
>> That's almost impossible to type without a syntax error.
>
> 1) Set your Enter key to send CR-LF, at least for this operation.
> That's half your problem solved.

That can be much harder than typing ctrl-SPC. It *is* supported by
netcat, for example, but then you have to carefully recompute the
content-length field.

> 2) Send the request like this:
>
> POST / HTTP/1.0
> Content-type: application/json
>
> {"msgtype": "echo-req", "opid": 3487547843}
>
> Then shut down your end of the connection, probably with Ctrl-D. I'm
> fairly sure I can type that without bugs, and any compliant HTTP
> server should be fine with it.

If I terminated the input, I wouldn't need any framing. The point is to
carry a number of JSON messages/documents over a single bytestream or in
a single file. That means the HTTP content-length would be mandatory.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue14074] argparse allows nargs>1 for positional arguments but doesn't allow metavar to be a tuple

[Cyker Way]

Cyker Way  added the comment:

Can confirm this bug still exists on master branch, python3.7, python3.6, and 
very likely other versions since it's reported.

It seems only `_format_action_invocation` and `_get_action_name` need to be 
fixed. So we can do it more lightweight (<10 lines).

--
nosy: +cykerway
Added file: https://bugs.python.org/file47965/14074.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14074] argparse allows nargs>1 for positional arguments but doesn't allow metavar to be a tuple

[Roundup Robot]

Change by Roundup Robot :


--
pull_requests: +10082
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22496] urllib2 fails against IIS (urllib2 can't parse 401 reply www-authenticate headers)

[Mathieu Dupuy]

Change by Mathieu Dupuy :


--
resolution:  -> wont fix
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35371] Fix undefined behavior in os.utime() on Windows

[miss-islington]

miss-islington  added the comment:


New changeset 013832ff964a0b3b59e04a07a33bae65c1c3ae84 by Miss Islington (bot) 
in branch '3.6':
bpo-35371: Fix possible crash in os.utime() on Windows. (GH-10844)
https://github.com/python/cpython/commit/013832ff964a0b3b59e04a07a33bae65c1c3ae84


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35371] Fix undefined behavior in os.utime() on Windows

[miss-islington]

miss-islington  added the comment:


New changeset 265b41996aa3f604624a8046d1c314a1aee4b590 by Miss Islington (bot) 
in branch '3.7':
bpo-35371: Fix possible crash in os.utime() on Windows. (GH-10844)
https://github.com/python/cpython/commit/265b41996aa3f604624a8046d1c314a1aee4b590


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35371] Fix undefined behavior in os.utime() on Windows

[miss-islington]

Change by miss-islington :


--
pull_requests: +10081

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35371] Fix undefined behavior in os.utime() on Windows

[miss-islington]

Change by miss-islington :


--
pull_requests: +10080

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35371] Fix undefined behavior in os.utime() on Windows

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset 32bc11c33cf5ccea165b5f4ac3799f02fdf9c76a by Serhiy Storchaka in 
branch 'master':
bpo-35371: Fix possible crash in os.utime() on Windows. (GH-10844)
https://github.com/python/cpython/commit/32bc11c33cf5ccea165b5f4ac3799f02fdf9c76a


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35371] Fix undefined behavior in os.utime() on Windows

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
keywords: +patch
pull_requests: +10079
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35371] Fix undefined behavior in os.utime() on Windows

[Serhiy Storchaka]

New submission from Serhiy Storchaka :

The hFile variable is used uninitialized in os.utime() on Windows when an error 
is raised in arguments parsing. This is an undefined behavior, and can cause a 
crash.

--
assignee: serhiy.storchaka
components: Extension Modules
messages: 330850
nosy: serhiy.storchaka
priority: normal
severity: normal
status: open
title: Fix undefined behavior in os.utime() on Windows
type: crash
versions: Python 3.6, Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: multiple JSON documents in one file, change proposal

[Chris Angelico]

On Sat, Dec 1, 2018 at 10:16 PM Marko Rauhamaa  wrote:
>
> Chris Angelico :
> > On Sat, Dec 1, 2018 at 9:16 PM Marko Rauhamaa  wrote:
> >> The need for the format to be "typable" (and editable) is essential
> >> for ad-hoc manual testing of components. That precludes all framing
> >> formats that would necessitate a length prefix. HTTP would be
> >> horrible to have to type even without the content-length problem, but
> >> BEEP (RFC 3080) would suffer from the content-length (and CRLF!)
> >> issue as well.
> >
> > I dunno, I type HTTP manually often enough that it can't be all *that*
> > horrible.
>
> Say I want to send this piece of JSON:
>
>{
>"msgtype": "echo-req",
>"opid": 3487547843
>}
>
> and the framing format is HTTP. I will need to type something like this:
>
>POST / HTTP/1.1^M
>Host: localhost^M
>Content-type: application/json^M
>Content-length: 54^M
>^M
>{
>"msgtype": "echo-req",
>"opid": 3487547843
>}
>
> That's almost impossible to type without a syntax error.

1) Set your Enter key to send CR-LF, at least for this operation.
That's half your problem solved.
2) Send the request like this:

POST / HTTP/1.0
Content-type: application/json

{"msgtype": "echo-req", "opid": 3487547843}

Then shut down your end of the connection, probably with Ctrl-D. I'm
fairly sure I can type that without bugs, and any compliant HTTP
server should be fine with it.

> >> Finally, couldn't any whitespace character work as a terminator? Yes,
> >> it could, but it would force you to use a special JSON parser that is
> >> prepared to handle the self-delineation. A NUL gives you many more
> >> degrees of freedom in choosing your JSON tools.
> >
> > Either non-delimited or newline-delimited JSON is supported in a lot
> > of tools. I'm quite at a loss here as to how an unprintable character
> > gives you more freedom.
>
> As stated by Paul in another context, newline-delimited is a no-go
> because it forces you to restrict JSON to a subset that doesn't contain
> newlines (see the JSON example above).
>
> Of course, you could say that the terminating newline is only
> interpreted as a terminator after a complete JSON value, but that's not
> the format "supported in a lot of tools".

The subset in question is simply "JSON without any newlines between
tokens", which has the exact meaning as it would have *with* those
newlines. So what you lose is the human-readability of being able to
break an object over multiple lines. Is that a problem? Use
non-delimited instead.

> If you use any legal JSON character as a terminator, you have to make it
> contextual or add an escaping syntax.

Or just use non-delimited, strip all whitespace between objects, and
then special-case the one otherwise-ambiguous situation of two Numbers
back to back. Anything that sends newline-delimited JSON will work
with that.

> > I get it: you have a bizarre set of tools and the normal solutions
> > don't work for you. But you can't complain about the tools not
> > supporting your use-cases. Just code up your own styles of doing
> > things that are unique to you.
>
> There are numerous tools that parse complete JSON documents fine.
> Framing JSON values with NUL-termination is trivial to add in any
> programming environment. For example:
>
>def json_docs(path):
>with open(path) as f:
>for doc in f.read().split("\0")[:-1].
>yield json.loads(doc)

Yes, but many text-processing tools don't let you manually insert
NULs. Of *course* you can put anything you like in there when you
control both ends and everything in between; that's kinda the point of
coding. But I'm going to use newlines, and parse as non-delimited,
since that can be done just as easily (see my example code earlier -
it could be converted into the same style of generator as you have
here and would be about as many lines), since that will behave as text
in most applications.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: multiple JSON documents in one file, change proposal

[Marko Rauhamaa]

Chris Angelico :
> On Sat, Dec 1, 2018 at 9:16 PM Marko Rauhamaa  wrote:
>> The need for the format to be "typable" (and editable) is essential
>> for ad-hoc manual testing of components. That precludes all framing
>> formats that would necessitate a length prefix. HTTP would be
>> horrible to have to type even without the content-length problem, but
>> BEEP (RFC 3080) would suffer from the content-length (and CRLF!)
>> issue as well.
>
> I dunno, I type HTTP manually often enough that it can't be all *that*
> horrible.

Say I want to send this piece of JSON:

   {
   "msgtype": "echo-req",
   "opid": 3487547843
   }

and the framing format is HTTP. I will need to type something like this:

   POST / HTTP/1.1^M
   Host: localhost^M
   Content-type: application/json^M
   Content-length: 54^M
   ^M
   {
   "msgtype": "echo-req",
   "opid": 3487547843
   }

That's almost impossible to type without a syntax error.

>> Finally, couldn't any whitespace character work as a terminator? Yes,
>> it could, but it would force you to use a special JSON parser that is
>> prepared to handle the self-delineation. A NUL gives you many more
>> degrees of freedom in choosing your JSON tools.
>
> Either non-delimited or newline-delimited JSON is supported in a lot
> of tools. I'm quite at a loss here as to how an unprintable character
> gives you more freedom.

As stated by Paul in another context, newline-delimited is a no-go
because it forces you to restrict JSON to a subset that doesn't contain
newlines (see the JSON example above).

Of course, you could say that the terminating newline is only
interpreted as a terminator after a complete JSON value, but that's not
the format "supported in a lot of tools".

If you use any legal JSON character as a terminator, you have to make it
contextual or add an escaping syntax.

> I get it: you have a bizarre set of tools and the normal solutions
> don't work for you. But you can't complain about the tools not
> supporting your use-cases. Just code up your own styles of doing
> things that are unique to you.

There are numerous tools that parse complete JSON documents fine.
Framing JSON values with NUL-termination is trivial to add in any
programming environment. For example:

   def json_docs(path):
   with open(path) as f:
   for doc in f.read().split("\0")[:-1].
   yield json.loads(doc)


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue35370] Provide API to set the tracing function to be used for running threads.

[Fabio Zadrozny]

New submission from Fabio Zadrozny :

Right now it's hard for debuggers to set the tracing function to be used for 
running threads.

This would be really handy for debuggers when attaching to a running program to 
debug all threads.

-- Note: currently there is a way to achieve that by pausing all the threads 
then selectively switching to a thread to make it current and setting the 
tracing function using the C-API (see: 
https://github.com/fabioz/PyDev.Debugger/blob/master/pydevd_attach_to_process/dll/attach.cpp#L1224),
 but I believe this is very hacky and not portable to other Python 
implementations.

--
components: Interpreter Core
messages: 330849
nosy: fabioz
priority: normal
severity: normal
status: open
title: Provide API to set the tracing function to be used for running threads.
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35370] Provide API to set the tracing function to be used for running threads.

[Fabio Zadrozny]

Change by Fabio Zadrozny :


--
type:  -> enhancement

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: multiple JSON documents in one file, change proposal

[Chris Angelico]

On Sat, Dec 1, 2018 at 9:16 PM Marko Rauhamaa  wrote:
>
> Paul Rubin :
>
> > Marko Rauhamaa  writes:
> >> Having rejected different options ( >> https://en.wikipedia.org/wiki/JSON_streaming>), I settled with
> >> terminating each JSON value with an ASCII NUL character, which is
> >> illegal in JSON proper.
> >
> > Thanks, that Wikipedia article is helpful.  I'd prefer to not use stuff
> > like NUL or RS because I like keeping the file human readable.  I might
> > use netstring format (http://cr.yp.to/proto/netstrings.txt) but I'm even
> > more convinced now that adding a streaming feature to the existing json
> > module is the right way to do it.
>
> We all have our preferences.
>
> In my case, I need an explicit terminator marker to know when a JSON
> value is complete. For example, if I should read from a socket:
>
>123
>
> I can't yet parse it because there might be another digit coming. On the
> other hand, the peer might not see any reason to send any further bytes
> because "123" is all they wanted to send at the moment.

This is actually the only special case. Every other JSON value has a
clear end. So the only thing you need to say is that, if the sender
wishes to transmit a bare number, it must append a space. Seriously,
how often do you ACTUALLY send a bare number? I've sometimes sent a
string on its own, but even that is incredibly rare. Having to send a
simple space after a bare number is unlikely to cause much trouble.

> As for NUL, a control character that is illegal in all JSON contexts is
> practical so the JSON chunks don't need to be escaped. An ASCII-esque
> solution would be to pick ETX (= end of text). Unfortunately, a human
> operator typing ETX (= ctrl-C) to terminate a JSON value will cause a
> KeyboardInterrupt in many modern command-line interfaces.
>
> It happens NUL (= ctrl-SPC = ctrl-@) is pretty easy to generate and
> manipulate in editors and the command line.

I have no idea which editors YOU use, but if you poll across platforms
and systems, I'm pretty sure you'll find that not everyone can type
it. Furthermore, many tools use the presence of an 0x00 byte as
evidence that a file is binary, not text. (For instance, git does
this.) That might be a good choice for your personal use-case, but not
the general case, whereas the much simpler options listed on the
Wikipedia page are far more general, and actually wouldn't be THAT
hard for you to use.

> The need for the format to be "typable" (and editable) is essential for
> ad-hoc manual testing of components. That precludes all framing formats
> that would necessitate a length prefix. HTTP would be horrible to have
> to type even without the content-length problem, but BEEP (RFC 3080)
> would suffer from the content-length (and CRLF!) issue as well.

I dunno, I type HTTP manually often enough that it can't be all *that* horrible.

> Finally, couldn't any whitespace character work as a terminator? Yes, it
> could, but it would force you to use a special JSON parser that is
> prepared to handle the self-delineation. A NUL gives you many more
> degrees of freedom in choosing your JSON tools.

Either non-delimited or newline-delimited JSON is supported in a lot
of tools. I'm quite at a loss here as to how an unprintable character
gives you more freedom.

I get it: you have a bizarre set of tools and the normal solutions
don't work for you. But you can't complain about the tools not
supporting your use-cases. Just code up your own styles of doing
things that are unique to you.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue31177] unittest mock's reset_mock throws an error when an attribute has been deleted

[miss-islington]

miss-islington  added the comment:


New changeset 422c1658b7d34fdc73c5fc895b135862103d1983 by Miss Islington (bot) 
in branch '3.7':
bpo-31177: Skip deleted attributes while calling reset_mock (GH-9302)
https://github.com/python/cpython/commit/422c1658b7d34fdc73c5fc895b135862103d1983


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31177] unittest mock's reset_mock throws an error when an attribute has been deleted

[miss-islington]

miss-islington  added the comment:


New changeset c0566e0ff6c2dd1a8b814ecd65649605c090527b by Miss Islington (bot) 
in branch '3.6':
bpo-31177: Skip deleted attributes while calling reset_mock (GH-9302)
https://github.com/python/cpython/commit/c0566e0ff6c2dd1a8b814ecd65649605c090527b


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: multiple JSON documents in one file, change proposal

[Marko Rauhamaa]

Paul Rubin :

> Marko Rauhamaa  writes:
>> Having rejected different options (> https://en.wikipedia.org/wiki/JSON_streaming>), I settled with
>> terminating each JSON value with an ASCII NUL character, which is
>> illegal in JSON proper.
>
> Thanks, that Wikipedia article is helpful.  I'd prefer to not use stuff
> like NUL or RS because I like keeping the file human readable.  I might
> use netstring format (http://cr.yp.to/proto/netstrings.txt) but I'm even
> more convinced now that adding a streaming feature to the existing json
> module is the right way to do it.

We all have our preferences.

In my case, I need an explicit terminator marker to know when a JSON
value is complete. For example, if I should read from a socket:

   123

I can't yet parse it because there might be another digit coming. On the
other hand, the peer might not see any reason to send any further bytes
because "123" is all they wanted to send at the moment.

As for NUL, a control character that is illegal in all JSON contexts is
practical so the JSON chunks don't need to be escaped. An ASCII-esque
solution would be to pick ETX (= end of text). Unfortunately, a human
operator typing ETX (= ctrl-C) to terminate a JSON value will cause a
KeyboardInterrupt in many modern command-line interfaces.

It happens NUL (= ctrl-SPC = ctrl-@) is pretty easy to generate and
manipulate in editors and the command line.

The need for the format to be "typable" (and editable) is essential for
ad-hoc manual testing of components. That precludes all framing formats
that would necessitate a length prefix. HTTP would be horrible to have
to type even without the content-length problem, but BEEP (RFC 3080)
would suffer from the content-length (and CRLF!) issue as well.

Finally, couldn't any whitespace character work as a terminator? Yes, it
could, but it would force you to use a special JSON parser that is
prepared to handle the self-delineation. A NUL gives you many more
degrees of freedom in choosing your JSON tools.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue31177] unittest mock's reset_mock throws an error when an attribute has been deleted

[miss-islington]

Change by miss-islington :


--
pull_requests: +10078

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31177] unittest mock's reset_mock throws an error when an attribute has been deleted

[miss-islington]

Change by miss-islington :


--
pull_requests: +10077

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31177] unittest mock's reset_mock throws an error when an attribute has been deleted

[STINNER Victor]

STINNER Victor  added the comment:


New changeset edeca92c84a3b08902ecdfe987cde00c7e617887 by Victor Stinner 
(Xtreak) in branch 'master':
bpo-31177: Skip deleted attributes while calling reset_mock (GH-9302)
https://github.com/python/cpython/commit/edeca92c84a3b08902ecdfe987cde00c7e617887


--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34850] Emit a syntax warning for "is" with a literal

[Gregory P. Smith]

Gregory P. Smith  added the comment:

Oh neat, I didn't realize that Nathaniel.  That makes such warnings much more 
useful.

I'm fine if you go ahead with this change.  In the unlikely event it turns out 
to cause user annoyance problems in betas due to third party code that can't be 
updated, we can reconsider.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35369] List sorting makes duplicate comparisons

[David Wyde]

David Wyde  added the comment:

Thanks for the speedy and helpful response.

Keeping complexity down is fair. The wasted if-checks on subsequent iterations 
are certainly a negative trade-off. I saw that binarysort() is only called in 
one place, but I understand wanting to keep it generic.

I think that slow comparison functions, especially when repeatedly sorting 
short lists, are the main use case.

I don't know if that's common in performance-critical code. I've heard of using 
human choices for comparisons, when fewer decisions could provide a notable 
speedup. The patched code seems a bit slower in some situations, but is faster 
in others.

Do you think it's worth posting to python-ideas to see what people's use cases 
are?

--
Added file: https://bugs.python.org/file47964/sort-fix-2.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com