[issue27263] IDLE sets the HOME environment variable breaking scripts

[Eryk Sun]

Eryk Sun added the comment:

This affects all Tkinter applications on Windows. For example, in Python 2.7:

import Tkinter
import ctypes
libc = ctypes.CDLL('msvcr90')
libc.getenv.restype = ctypes.c_char_p

>>> libc.getenv(b"HOME")
>>> root = Tkinter.Tk()
>>> libc.getenv(b"HOME")
'C:\\Users\\me'

TCL sets this environment variable in TclpSetVariables [1] and has since 1995 
[2]. I don't think IDLE or any other Tkinter application should necessarily 
favor %USERPROFILE% over %HOMEDRIVE%%HOMEPATH%, and the default shouldn't be 
changed at this point. Probably the behavior should be documented for Tkinter 
and IDLE.

[1]: https://github.com/tcltk/tcl/blob/core_8_6_4/win/tclWinInit.c#L502
[2]: https://github.com/tcltk/tcl/blob/core_8_6_4/changes#L1378

--
components: +Tkinter, Windows
nosy: +eryksun, paul.moore, steve.dower, tim.golden, zach.ware
versions: +Python 2.7, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24136] document PEP 448: unpacking generalization

[Jelle Zijlstra]

Jelle Zijlstra added the comment:

This updates reference_calls_syntax_update.diff. The previous patch's grammard 
had a mistake; it was missing commas between arguments.

I believe all other patches in this diff are now obsolete.

--
Added file: http://bugs.python.org/file43302/issue24136-expressions.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20699] Document that binary IO classes work with bytes-likes objects

[Martin Panter]

Martin Panter added the comment:

New patch that I plan to commit:

* Use bytes instead of str
* bytes, bytearray or memoryview for readinto() methods
* bytes or memoryview for write() methods
* Added single “Changed in version 2.7” notice under the IOBase class (rather 
than every version of readinto and write)
* Since we only require 1-D byte array memoryviews, I resurrected the len(b) 
text
* Removed outdated XXX comment about supporting buffer API, since bytearray and 
memoryview of bytes is now all that is required

--
stage: patch review -> commit review
Added file: http://bugs.python.org/file43301/bytes-like-param.py2.v9.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24136] document PEP 448: unpacking generalization

[Jelle Zijlstra]

Jelle Zijlstra added the comment:

Here's what I found reviewing what needs to be done here:

- Neil's What's New patch apparently made it into the 3.5 release notes.
- moigagoo's two patches haven't been committed. The text of both looks ok to 
me, but the patches don't apply cleanly.

Other areas that still need updates:
- https://docs.python.org/3/reference/expressions.html#list-displays and the 
sections for set, dict, and tuple displays need to mention PEP 448-style syntax.

--
nosy: +Jelle Zijlstra

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27127] Never have GET_ITER not followed by FOR_ITER

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

As Dino noted in msg266453, this leaves the iterable on the stack while the 
loop is running. I think opcode reworking shouldn't change behavior. You should 
call GET_ITER or FOR_BEGIN before calling generator code.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15476] Index "code object" and link to code object definition

[Martin Panter]

Martin Panter added the comment:

Yes it seems you even fixed the problem with the definition list, which I 
thought was not fixable :)

--
stage: needs patch -> commit review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27127] Never have GET_ITER not followed by FOR_ITER

[Demur Rumed]

Demur Rumed added the comment:

I should've kept gitfit & forbegin more separated as issues. Attached is 
gitfit2, which only folds the GET_ITER into the comprehension if it isn't a 
generator so to pass test_genexps

--
Added file: http://bugs.python.org/file43300/gitfit2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27264] python 3.4 vs. 3.5 strftime same locale different output on Windows

[David Perra]

New submission from David Perra:

The execution of these commands in python 3.4.x (Windows 10 Home)

import locale
from datetime import datetime
locale.setlocale(locale.LC_ALL, 'Spanish')
datetime.strftime(datetime.now(), '%a %d %b %Y')

renders the output

'Spanish_Spain.1252'
'mar 07 jun 2016'

but with Python 3.5.x the output is

'Spanish_Spain.1252'
'ma. 07 jun. 2016'

--
components: Library (Lib), Windows
messages: 267786
nosy: David Perra, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: python 3.4 vs. 3.5 strftime same locale different output on Windows
type: behavior
versions: Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17797] Visual C++ 11.0 reports fileno(stdin) == 0 for non-console program

[Mark Lawrence]

Changes by Mark Lawrence :


--
nosy:  -BreamoreBoy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24567] random.choice IndexError due to double-rounding

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Ping.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25455] Some repr implementations don't check for self-referential structures

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Ping.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27260] Missing equality check for super objects

[Jelle Zijlstra]

Changes by Jelle Zijlstra :


--
priority: normal -> low
type:  -> enhancement

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25628] Make namedtuple "verbose" and "rename" parameters into keyword only arguments

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Ping.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18844] allow weights in random.choice

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Raymond, do you have a time for this issue?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27260] Missing equality check for super objects

[Jelle Zijlstra]

Jelle Zijlstra added the comment:

This came up as part of a static analysis script that compares sets of method 
calls, including calls to methods on super(). The check was giving incorrect 
results because identical super() objects were comparing as different.

super() is documented (https://docs.python.org/3/library/functions.html#super) 
as only delegating lookups for explicit attribute lookups, not for implicit 
lookups like that done by the == operator, so I think it would be safe to 
override tp_richcompare in C on super objects. In writing my patch, I'll be 
sure to test that the code you gave still works correctly though.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13451] sched.py: speedup cancel() method

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Ping.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27238] Bare except: usages in turtle.py

[Jelle Zijlstra]

Jelle Zijlstra added the comment:

Thanks for the review and comments. This patch addresses your comments and 
changes the __file__-related one to catch Exception.

--
Added file: http://bugs.python.org/file43299/issue27238-cr.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17900] Recursive OrderedDict pickling

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

I think it is too late to revert this change. Is it worth to add a workaround 
for PyYAML if it still hasn't fixed serialization of OrderedDicts?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17394] Add slicing support to collections.deque

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Raymond, do you have a time to make a review?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27242] Make the docs for NotImplemented & NotImplementedError unambigous

[Martin Panter]

Martin Panter added the comment:

Emanuel, have you seen 
? 
From my experience, install Sphinx, and then “make -C Doc/ html”.

About the patch, I think the “.. note” boxes should be intended under the 
NotImplementedError heading. The current box looks wrong sandwiched between 
NotImplemnted and Ellipsis.

(Personally I think “Note” boxes are an annoying distraction anyway, and should 
normally be written into the surrounding text. In this case “Note that” inside 
a “Note” box seems doubly redundant. But other people seem to like them, so I 
try to resist complaining :)

--
nosy: +martin.panter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27262] IDLE: move Aqua context menu code to maxosx

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

"bindtag" is an arbitrary string. Usually this is a name of Tk widget class of 
or "all". bindtags() allows to retrieve or set a list of bindtags associated 
with a widget. By default they are: full path of a widget, name of Tk class of 
widgets, full path of a toplevel window, "all".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8491] Need readline command and keybinding information

[Martin Panter]

Martin Panter added the comment:

Thanks for the patch. I left a couple review comments.

Also, this new paragraph should be outside the “Note” box, which is 
specifically about libedit aka Editline, not Gnu Readline.

--
stage: needs patch -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15476] Index "code object" and link to code object definition

[Tommy Beadle]

Tommy Beadle added the comment:

Martin, thanks for the suggestion that worked.  I believe this patch should 
have all of the issues addressed.

--
Added file: 
http://bugs.python.org/file43298/0001-Issue-15476-Make-code-object-its-own-entry-in-the-do.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26446] Mention in the devguide that core dev stuff falls under the PSF CoC

[Evelyn Mitchell]

Evelyn Mitchell added the comment:

Thank you for your help Berker.

--
nosy: +Evelyn Mitchell
Added file: http://bugs.python.org/file43297/patchCoreDevCoC

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27187] Relax __all__ location requirement in PEP 8

[Ian Lee]

Changes by Ian Lee :


Added file: http://bugs.python.org/file43296/issue-27187-patch3.txt

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27187] Relax __all__ location requirement in PEP 8

[Ian Lee]

Ian Lee added the comment:

Good catch.

I'm uploading a new patch that addresses ``from __future__`` imports issue 
explicitly.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27263] IDLE sets the HOME environment variable breaking scripts

[Jarrod Petz]

Jarrod Petz added the comment:

Worked around this by setting HOME to be USERPROFILE before IDLE starts

Rather then change the system/user environment permanently. I edited the Idle 
script which the windows shortcut seems to run below.

C:\Python35\Lib\idlelib\idle.pyw

At the top of the script I added

import os
os.environ['HOME'] = os.environ['USERPROFILE']


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27127] Never have GET_ITER not followed by FOR_ITER

[Demur Rumed]

Demur Rumed added the comment:

Didn't see Raymond's response before posting, forbegin3 at least exists as a 
completion of the experiment to a passes-tests state. The tracing hacks to 
support an instruction corresponding to two separate lines support rejecting 
this idea

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27127] Never have GET_ITER not followed by FOR_ITER

[Demur Rumed]

Demur Rumed added the comment:

Attaching forbegin3.patch. It reintroduces GET_ITER for the sole purpose of 
eagerly throwing. I decided to reuse GET_ITER over something like TEST_ITER as 
this way we can have GET_ITER flow into FOR_BEGIN & rely on the fast path of 
iter(iter(x))

GET_ITER/JUMP_FORWARD idea doesn't work because FOR_ITER is carefully setup 
currently to trace as existing on 2 separate lines. If we JUMP_FORWARD into 
FOR_ITER then that tracing triggers & our trace will say we executed the last 
line of the loop immediately before executing the iteration logic

--
Added file: http://bugs.python.org/file43295/forbegin3.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26446] Mention in the devguide that core dev stuff falls under the PSF CoC

[Berker Peksag]

Berker Peksag added the comment:

Thanks for the patch Evelyn. The patch looks good to me. I have a couple of 
suggestions:

1. It seems like https://docs.python.org/devguide/coredev.html#responsibilities 
is a better place to put this

2. Core committers -> Core developers

3. PSF CoC -> PSF Code of Conduct

4. Could you also link "PSF Code of Conduct" to 
https://www.python.org/psf/codeofconduct/ ? I always need to look at the Sphinx 
docs but we need to make it look like

 `Spam eggs `_

--
nosy: +berker.peksag
stage: needs patch -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26446] Mention in the devguide that core dev stuff falls under the PSF CoC

[Evelyn Mitchell]

Changes by Evelyn Mitchell :


Added file: http://bugs.python.org/file43294/patchCoreCoC

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8491] Need readline command and keybinding information

[Evelyn Mitchell]

Changes by Evelyn Mitchell :


Added file: http://bugs.python.org/file43293/patch8491

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25738] http.server doesn't handle RESET CONTENT status correctly

[Susumu Koshiba]

Susumu Koshiba added the comment:

A patch for 3.5 attached.

--
Added file: 
http://bugs.python.org/file43292/issue25738_http_reset_content_3.5_02.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27261] io.BytesIO.truncate does not work as advertised

[Xiang Zhang]

Changes by Xiang Zhang :


--
nosy: +alexandre.vassalotti, serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27261] io.BytesIO.truncate does not work as advertised

[Xiang Zhang]

Xiang Zhang added the comment:

Is this a bug or a documentation matter? Inspecting BytesIO.truncate's code, it 
does resizing similar to list, this seems to be a designed feature, which 
conflicts with the documented behaviour.

--
nosy: +xiang.zhang

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25738] http.server doesn't handle RESET CONTENT status correctly

[Susumu Koshiba]

Susumu Koshiba added the comment:

Great, thanks for checking. Attaching patch for 2.7. 3.5 will follow.

--
Added file: 
http://bugs.python.org/file43291/issue25738_http_reset_content_2.7_02.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25738] http.server doesn't handle RESET CONTENT status correctly

[Martin Panter]

Martin Panter added the comment:

Patch 07 looks fine. I presume you still want to do the porting to 3.5 and 2.7.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27263] IDLE sets the HOME environment variable breaking scripts

[Jarrod Petz]

Jarrod Petz added the comment:

It also makes things none deterministic. As when I am off the domain or my 
network drive H:\ is unavailable it works and uses USERPROFILE.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27263] IDLE sets the HOME environment variable breaking scripts

[Jarrod Petz]

New submission from Jarrod Petz:

It seems IDEL is setting the environment variable 'HOME' on windows. 
Specifically I am on windows 8.1

This is extremly annoying and bad as according the code and doco, 
os.path.expanduser will preference this variable above others such as 
USERPROFILE.
https://docs.python.org/3/library/os.path.html#os.path.expanduser

This is causing scripts/libraries and in my case the AWS SDK boto3/botocore 
modules to break. As they are looking for config files in the users home 
location using os.path resolution order which should be %USERPROFILE% in my 
case as I don't have HOME set. But because idle is setting HOME to what looks 
to be my HOMEPATH and HOMEDRIVE this is making the AWS SDK unable to pickup the 
credential files required.

path = "~/.aws/credentials"
print(os.path.expanduser(path))

Should be
C:\Users\myUserName/.aws/credentials

But is instead
H:\/.aws/credentials

Please stop IDLE from setting this environment variable. Running the same 
scripts with python.exe or pythonw.exe works fine.

--
components: IDLE
messages: 267761
nosy: Jarrod Petz
priority: normal
severity: normal
status: open
title: IDLE sets the HOME environment variable breaking scripts
type: behavior
versions: Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27262] IDLE: move Aqua context menu code to maxosx

[Terry J. Reedy]

New submission from Terry J. Reedy:

Put the AquaTk code added to pyshell.main in #24801 where it should have gone 
originally. (My fault, ultimately.)  See aqua_context.diff.  This is a step in 
factoring main (currently about 170 lines) into a manageable number of function 
calls.

I will try to add a test that 1. creates an editor window with a 'fixed' root 
and mocked .right_click_event method, generates a right click, and checks the 
mock event handler; 2. creates a context menu, etc.

Serhiy, the docstring for .class_bind says "Bind to widgets with bindtag 
CLASSNAME".  I have not and do not see 'bindtag' defined.  Am I correct that it 
must refer to a tk widget class, rather than a python tkinter class, or 
subclass thereof?

--
assignee: terry.reedy
files: aqua_context.diff
keywords: patch
messages: 267760
nosy: serhiy.storchaka, terry.reedy
priority: normal
severity: normal
stage: test needed
status: open
title: IDLE: move Aqua context menu code to maxosx
type: enhancement
versions: Python 3.6
Added file: http://bugs.python.org/file43290/aqua_context.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27180] Doc/pathlib: Please describe the behaviour of Path().rename() is depends on the platform (same as os.rename())

[Evelyn Mitchell]

Changes by Evelyn Mitchell :


Added file: http://bugs.python.org/file43289/patch27180

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27251] TypeError in logging.HTTPHandler.emit; possible python 2 to 3 conversion issue

[Berker Peksag]

Changes by Berker Peksag :


--
stage: test needed -> resolved

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27180] Doc/pathlib: Please describe the behaviour of Path().rename() is depends on the platform (same as os.rename())

[Berker Peksag]

Berker Peksag added the comment:

Could you please send your patch in unified diff format? See 
https://docs.python.org/devguide/patch.html for details.

--
nosy: +berker.peksag
stage: needs patch -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27261] io.BytesIO.truncate does not work as advertised

[Martin Panter]

Martin Panter added the comment:

BytesIO is supposed to implement IOBase. I would treat this as a bug in 
existing versions too.

--
nosy: +martin.panter
stage:  -> needs patch
versions: +Python 2.7, Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27187] Relax __all__ location requirement in PEP 8

[John Mark Vandenberg]

John Mark Vandenberg added the comment:

The revised text says __dunder__ should appear before any imports, however 
__future__ imports must appear first for obvious reasons.  Does this need to be 
mentioned in the pep?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27252] Make dict views copyable

[Raymond Hettinger]

Raymond Hettinger added the comment:

See also:  http://bugs.python.org/issue27253

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27253] More efficient deepcopying of Mapping

[Raymond Hettinger]

Raymond Hettinger added the comment:

Is this ever needed?  I'm not sure whether all these patches are being driven 
by reasonable use cases or whether it is just something that could be done.

I thought the whole point of the views was to avoid copying.  If so, there is a 
design principle that says roughly, "take things you want users to do and make 
them easy and fast, while leaving undesirable patterns in a less comfortable 
state".  We don't have to break the copying and deepcopying, but we don't need 
to go out of our way to make them more efficient unless that is what we want 
users to be doing.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27255] More opcode predictions

[Raymond Hettinger]

Raymond Hettinger added the comment:

Serhiy, please slow down and stop rewriting every single thing you see.  Your 
rate of patches is prolific and hard to digest.  Please give some consideration 
that the people who came before you (lots of them) put a lot of thought into 
what was done and were deliberately conservative.

Dynamic statistics can vary widely depending on code being profiled (much like 
PGO optimizations for C compilers).   

For a prediction to have value, it needs to be a commonly used opcode; it must 
occur in a highly predictive pair with an intrinsic relationship rather than a 
coincidental relationship.

FWIW, these tests aren't free.  Global prediction tables have a limited size 
and are subject to aliasing.  Mispredictions are expensive.  Also, the ceval 
loop doesn't need more clutter.

The new opcodes GET_YIELD_FROM_ITER, GET_AITER, GET_ANEXT, and GET_AWAITABLE 
haven't been considered before.  The code in Python/compile.c shows that 
LOAD_CONST is the only possible next opcode, so these are reasonable candidates 
(i.e. the pair effectively acts as a single opcode).  Of these, I think only 
GET_ANEXT and GET_AWAITABLE are likely to occur in a loop.  So, these two may 
be worthwhile.  All the rest should probably be skipped.

--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27255] More opcode predictions

[Raymond Hettinger]

Changes by Raymond Hettinger :


--
assignee:  -> rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27260] Missing equality check for super objects

[Raymond Hettinger]

Raymond Hettinger added the comment:

Why would you need this?   

Also, would it interfere with super's ability to use the __eq__ method for a 
parent class?

class A:
def __eq__(self, other):
return True

class B(A):
def __eq__(self, other):
return super(B, self).__eq__(other)

print(B() == B())

--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Christian Heimes]

Christian Heimes added the comment:

Thanks Theodore!

Your analysis was exactly what I was aiming for when I requested a thorough 
security analysis in form of a PEP. The correct choice of CPRNG is important 
for the overall security. I'm mostly concerned with the behavior of 
os.urandom(), which is Python's preferred CPRNG for any critical data from 
session keys to key material. Several people including me have been pushing 
developers towards os.urandom() for many years. For that reason I put up a veto 
against any hasty change. Larry could my bluff.

The hash randomization for hashing of strings and bytes is explained in PEP 
456, https://www.python.org/dev/peps/pep-0456 . I wrote the PEP and added DJB's 
and JP Aumasson's SipHash24 as PRF. The 24 byte Py_HashSecret struct contains 
two keys for SipHash24 and another 8 byte key for randomization of expat XML 
library, https://hg.python.org/cpython/file/tip/Include/pyhash.h#l34 .

For short-running scripts early in the boot phase, hash randomization is not 
required at all. It is only relevant for applications that reads untrusted data 
from potentially malicious peers. GET dict of HTTP requests is a famous 
example. Hash randomization can already be disabled or set to a fixed value by 
using an env var. I argued to add an option that falls back to a different 
CPRNG and sets other options at the same time (#16499) but Larry (release 
manager of 3.5) is against any option. He wants Python to always start in a 
timely fashion without any extra arguments.

Your suggestion should fix the issue on Linux (GRND_NONBLOCK, fall back to 
srandom()), although I would rather use gettimeofday() with t.tv_sec + 
t.tv_usec. I'm still concerned how we should address the issue on BSD. As far 
as I am familiar with BSD, all reads from the Kernel's CPRNG are blocking until 
the CPRNG is seeded.

I can bring up the issue with Lennart, if it is really necessary (one advantage 
of working for Red Hat ;) ). I'm going to ping JP Aumasson to get his feedback.

--
nosy: +christian.heimes

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Theodore Tso]

Theodore Tso added the comment:

I ran the experiment Colm asked me to run --- and yes, if you boot a system 
with Python 3.5.1 with the boot options "init=/usr/bin/python3", you're going 
to have a bad time.   The problem is that in a KVM environment where things are 
very quiet, especially if you are using a tickless kernel, if python calls 
getrandom(2), it will block since the entropy pool hasn't been initialized yet. 
  But since we aren't doing anything, the system becomes completely quiescent 
and so no entropy can be collected.  If systemd tries to run a python script 
very early in the boot process, and does this in a way where no further boot 
time activity takes place until the python script exits, you can indeed 
deadlock the system.

The solution is I think what Donald suggested in msg267746, which is to use 
GRND_NONBLOCK for initializing the hash which gets used for the dict, or 
whatever it's used for.   My understanding is that this is not a long-term 
cryptographic secret, and indeed it will be thrown away as soon as the python 
interpreter exits.  Since this is before networking has been brought up, the 
denial service attack or whatever requires that you use a strong SipHash for 
your Python dictionaries shouldn't be a problem.   (Which I gather has 
something to do with this?   
https://events.ccc.de/congress/2011/Fahrplan/attachments/2007_28C3_Effective_DoS_on_web_application_platforms.pdf)

Now, I can see people being concerned that if Python *always* initializes its 
hash dictionaries using getrandom with GRND_NONBLOCK, it might be opening up a 
DOS attack.   Well, in practice, once the boot sequence continues and the 
system is actually doing some real work, within a few seconds the random number 
generator will be initialized so in practice it won't be an issue once the 
system has booted.

If you want to be really paranoid, I suppose you could give some kind of 
command-line flag which tells Python to use GRND_NONBLOCK for the purposes of 
initializing its hash table for its dictionary, and only use it in the boot 
path.   In practice, I suspect very early in the systemd boot path, before it 
actually starts running the boot scripts in parallel, is the only place where 
you are likely going to run into this problem, so making it be a flag that only 
systemd scripts have to call is probably the right thing to do.   But I'll let 
someone else have the joys of negotiating with Lennart, and I won't blame the 
Python devs if using GRND_NONBLOCK unconditionally is less painful than having 
to work with the systemd folks.  :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: [issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[M.-A. Lemburg]

On 07.06.2016 22:27, Theodore Tso wrote:
> 
> Secondly, when I decided to add this behavior to getrandom(2), it was because 
> people were really worried that people would be using /dev/urandom for 
> security-critical things (e.g., initializing ssh host session keys, when 
> they'd _really_ rather not the NSA have be able to trivally pwn the server) 
> before it had been completely initialized.   (And if it is not completely 
> initialized, it would be trivially and embarassingly easy.  See 
> https://factorable.net/weakkeys12.extended.pdf for an example of where this 
> was rather disastrous.)

Thanks, Theodore, for this paper reference. It provides convincing
arguments that going back to the Python 3.4 behavior is indeed not
a good idea - even though I'm still not convinced that the main
use case for os.urandom() is cryptography. Most people will
simply use it to seed their Mersenne Twisters, like the random
module does too.

Now, raising an exception instead of blocking would likely cause
even more breakage, so I'm with Colm in keeping Victor's patch
and applying the fix to not block in dev_urandom_noraise().

We still need to fix the random module issue, though.

For 3.6, I wish we could have the getrandom() API exposed as
os.getrandom(), with all options available to applications.
That way, the application can decide what is best for them.

-- 
Marc-Andre Lemburg
eGenix.com

___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Colm Buckley]

Colm Buckley added the comment:

Ted -

I'd suggest the following to test.

Boot an arbitrary Linux system with init=/usr/bin/python3 (assuming filesystems 
mounted etc). Python 3.5.1 (on Linux) will call getrandom() in blocking mode 
very early in its startup; if this happens before the pool is initialized, 
Python will fail to start. Given that ~nothing else will be happening, I'm 
interested to see what happens to the entropy pool, and whether getrandom() 
returns.

Haven't tried this myself, but it should work.

Colm

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27261] io.BytesIO.truncate does not work as advertised

[Justus Winter]

New submission from Justus Winter:

% python3.6 -c "import io; b=io.BytesIO(); assert b.truncate(42) == 42; assert 
len(b.getbuffer()) == 42, 'expected length 42, got 
{}'.format(len(b.getbuffer()))"
Traceback (most recent call last):
  File "", line 1, in 
AssertionError: expected length 42, got 0

The documentation [0] says that truncate can extend objects, and there is no 
indication that BytesIO does not support that.  As demonstrated, truncate 
returns the new size, but the buffer obtained from that BytesIO is of size zero 
(likewise, b.getvalue() returns b''.

0: https://docs.python.org/3.6/library/io.html#io.IOBase.truncate

--
components: IO, Library (Lib)
messages: 267748
nosy: justus.winter
priority: normal
severity: normal
status: open
title: io.BytesIO.truncate does not work as advertised
type: behavior
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27260] Missing equality check for super objects

[Jelle Zijlstra]

New submission from Jelle Zijlstra:

>>> class Foo: pass
... 
>>> super(Foo) == super(Foo)
False

Will submit a patch later

--
assignee: Jelle Zijlstra
components: Library (Lib)
messages: 267747
nosy: Jelle Zijlstra
priority: normal
severity: normal
status: open
title: Missing equality check for super objects
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Donald Stufft]

Donald Stufft added the comment:

Thanks for weighing in Theodore, I think that matches what Colm's last 
suggestion was, and what I was personally OK with. To seed our SipHash function 
using GRND_NONBLOCK since it's likely that will be fine, and worst case we're 
just using it for some hash tables.

Then for our os.urandom binding, we should use getrandom() without 
GRND_NONBLOCK since we don't know why someone is calling os.urandom, but we 
know in practice people are using it for cryptographic keys and the like.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Theodore Tso]

Theodore Tso added the comment:

Hi.   Colm alerted me to this bug, so I thought I would chime in as the author 
of Linux's getrandom(2) function.

First of all, if you are OK with reading from /dev/urandom, then you might as 
well use getrandom's GRND_NONBLOCK flag.  They are logically equivalent.

Secondly, when I decided to add this behavior to getrandom(2), it was because 
people were really worried that people would be using /dev/urandom for 
security-critical things (e.g., initializing ssh host session keys, when they'd 
_really_ rather not the NSA have be able to trivally pwn the server) before it 
had been completely initialized.   (And if it is not completely initialized, it 
would be trivially and embarassingly easy.  See 
https://factorable.net/weakkeys12.extended.pdf for an example of where this was 
rather disastrous.)

Why didn't I make /dev/urandom blocking?  Because a lot of people would whine 
and complain.   But getrandom(2) was a new interface, and so this was something 
I could do.   Now, before I decided to do this, I did do some benchmarks, and 
pre-systemd in practice on real hardware (e.g., x86 servers and laptops), I 
observed that you would actually see a message indicating that we had gathered 
128 bits of entropy long before the root file system had been mounted.With 
systemd, I observed that udevd was trying to read from /dev/urandom when we had 
only gathered an estimated 7 bits of entropy --- but I devoutly hoped that 
udevd wasn't doing anything super security critical, and trying to get the 
systemd people to change what they are doing is mostly like trying to teach a 
pig to sing, so I let it be.However, in practice within a single digit 
number of seconds, the kernel printk indicating that random driver had 
considered itself initialized came quickly enough that I figured it would 
 be safe to do.

If people are claiming that they are seeing cases where it takes over 90 
seconds for the random number generator to initialize itself, please contact me 
directly; I'd love to know more, because that's input I would very much like to 
have.

However, at the end of the day, on certain hardware, if you don't have a source 
of initial entropy because the system doesn't have enough real hardware with 
real sources of entropy --- or if you don't trust your friendly cloud provider 
to provide you with some entropy from the hypervisor's entropy pool via 
virtio-random --- you can either (a) decide to pretend you are secure, when you 
really aren't, (b) wait, or (c) decide that you don't *really* need a secure 
source of randomness because you're really just initializing a hash for some 
associative array, and in fact srandom(time(0)) would have been fine, and you 
were using getrandom(2) or /dev/urandom just because you wanted to feel like 
one of the cool kids.

That being said, I do know of one potential issue which is if you happening to 
be using Microsoft Azure, the way the virtualized interrupt works, we weren't 
actually getting any entropy, and this was something I didn't discover until 
someone sent me a patch.  I have a patch[1] queued up in the random.git tree 
for the next kernel merge window to address that issue for Microsoft Azure 
servers. 

[1] 
http://git.kernel.org/cgit/linux/kernel/git/tytso/random.git/commit/?h=dev=8748971b4f5e322236154981827bf43dec4dc470

On a Google Compute Engine (GCE) system, I just did a quick test, and the 
"random: non-blocking pool initialized" message appears 5.64 seconds after the 
system is booted.  The changes I have queued up in random.git should reduce 
that to under a second.

All of this is neither here nor there, though.  The big question is *what* does 
Python expect to do with the randomness.  If you are just using it for 
computational simulation, you can do whatever you want.   If you are using it 
to create long-lived secrets that are intended to be secure against the 
depredations of a Nation-State's intelligence service, and you are on a system 
which really has almost no entropy available to be collected, then falling back 
to reading from /dev/urandom or using GRND_NONBLOCOK is going to be the 
equivalent of saying La-La-La-La-La-Nobody-Knows-How-Secure-I-Am while keeping 
your ears plugged.(Now, if you are on an Intel system with RDRAND, and you 
trust Intel not to have given a back door to the NSA, you probably are safe, 
because we do actually mix in RDRAND.  On the other hand, if you are using some 
crappy ARM SOC for some Internet of Things device, and are firing up Python 
right after the system boots for the first time, and creating long-lived RS
 A private keys within milliseconds after the system is first booted --- please 
tell me so, I can avoid your product like the Plague.  :-)

--
nosy: +Theodore Tso

___
Python tracker 

___

[issue27187] Relax __all__ location requirement in PEP 8

[Barry A. Warsaw]

Barry A. Warsaw added the comment:

Thanks for the patch!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27187] Relax __all__ location requirement in PEP 8

[Roundup Robot]

Roundup Robot added the comment:

New changeset cf8e888b9555 by Barry Warsaw in branch 'default':
Relax __all__ location.
https://hg.python.org/peps/rev/cf8e888b9555

--
nosy: +python-dev
resolution:  -> fixed
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27251] TypeError in logging.HTTPHandler.emit; possible python 2 to 3 conversion issue

[Vinay Sajip]

Changes by Vinay Sajip :


--
resolution:  -> fixed
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27251] TypeError in logging.HTTPHandler.emit; possible python 2 to 3 conversion issue

[Roundup Robot]

Roundup Robot added the comment:

New changeset 11ebd14076b4 by Vinay Sajip in branch '3.5':
Fixed #27251: corrected string/bytes handling in credentials.
https://hg.python.org/cpython/rev/11ebd14076b4

New changeset 12d939477b4f by Vinay Sajip in branch 'default':
Fixed #27251: merged fix from 3.5.
https://hg.python.org/cpython/rev/12d939477b4f

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Colm Buckley]

Colm Buckley added the comment:

I've spoken with Ted Ts'o (one advantage of working for Google) and taken a 
look in the Linux kernel source, and things are actually better than we'd 
feared.

Firstly, calling getrandom() with GRND_NONBLOCK and a buffer size of less than 
or equal to 32 bytes will always succeed (so, for the hash seed initialization 
at least, the EAGAIN logic is superfluous - it's still possibly needed for the 
general case and other operating systems, though).

Secondly, the quality of the getrandom data *before* the kernel PRNG is 
initialized is still pretty good - it's seeded from a combination of RDRAND, 
interrupt timing, several kernel parameters like uname -a, and RTC. Ted is 
confident that at least 24 bytes of real entropy will be present by a few 
seconds into boot time (due to interrupts etc), and that the predictability of 
the data will be very low.

Finally - note that any network-facing applications are *extremely* unlikely to 
encounter this issue, as they will be started well after networking and other 
good entropy sources have started. In particular, getrandom() will no longer 
block once fastinit has completed (on my system, this was less than one second 
after kernel load).

In other words, I think we are very safe to proceed with changeset 9de508dc4837 
+ the nonblocking_urandom_noraise.patch

Note that this solves the problem for *Linux* - if other operating systems do 
indeed have blocking /dev/urandom reads, this still needs to be addressed. I am 
not aware of any reports from non-Linux systems, though.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Donald Stufft]

Donald Stufft added the comment:

> Once this has been done, it will never block again, and happily send you poor 
> random data if the entropy pool has been completely wiped of any entropy data 
> - without telling you.

This doesn't actually happen in real life, once urandom has been initialized 
you will never be able to get "poor random" out of it. You will get 
cryptographically secure random out of it always. *ACTUAL* Cryptographers 
pretty much universally agree on this statement. You can even use them for 
cryptographic keys, no matter how long it's been since your system booted as 
long as the urandom pool has had a chance to initialize.

> Or put differently: Where is the attack vector that blocking behavior of 
os.urandom() would help remedy ?

Someone attempting to use cryptographic random before the urandom pool has been 
sufficiently initialized to provide said random.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Christian Heimes]

Changes by Christian Heimes :


--
nosy:  -christian.heimes

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27258] Exception in BytesGenerator.flatten

[Hans-Peter Jansen]

New submission from Hans-Peter Jansen:

Attached mail, parsed with email.message_from_binary_file results in:

Traceback (most recent call last):
  File "./mail_filter.py", line 616, in 
ret = main.run()
  File "./mail_filter.py", line 605, in run
self.process(fp)
  File "./mail_filter.py", line 589, in process
self.save_message(msg, self._fname + '.out')
  File "./mail_filter.py", line 103, in save_message
ofd.write(msg.as_bytes())
  File "/usr/lib64/python3.4/email/message.py", line 179, in as_bytes
g.flatten(self, unixfrom=unixfrom)
  File "/usr/lib64/python3.4/email/generator.py", line 115, in flatten
self._write(msg)
  File "/usr/lib64/python3.4/email/generator.py", line 195, in _write
self._write_headers(msg)
  File "/usr/lib64/python3.4/email/generator.py", line 422, in _write_headers
self._fp.write(self.policy.fold_binary(h, v))
  File "/usr/lib64/python3.4/email/policy.py", line 190, in fold_binary
folded = self._fold(name, value, refold_binary=self.cte_type=='7bit')
  File "/usr/lib64/python3.4/email/policy.py", line 204, in _fold
return self.header_factory(name, ''.join(lines)).fold(policy=self)
  File "/usr/lib64/python3.4/email/headerregistry.py", line 255, in fold
return header.fold(policy=policy)
  File "/usr/lib64/python3.4/email/_header_value_parser.py", line 300, in fold
self._fold(folded)
  File "/usr/lib64/python3.4/email/_header_value_parser.py", line 1228, in _fold
rest._fold(folded)
  File "/usr/lib64/python3.4/email/_header_value_parser.py", line 338, in _fold
if folded.append_if_fits(part, tstr):
  File "/usr/lib64/python3.4/email/_header_value_parser.py", line 149, in 
append_if_fits
token._fold(self)
  File "/usr/lib64/python3.4/email/_header_value_parser.py", line 324, in _fold
for part in self.parts:
  File "/usr/lib64/python3.4/email/_header_value_parser.py", line 254, in parts
if token.startswith_fws():
  File "/usr/lib64/python3.4/email/_header_value_parser.py", line 267, in 
startswith_fws
return self[0].startswith_fws()
  File "/usr/lib64/python3.4/email/_header_value_parser.py", line 267, in 
startswith_fws
return self[0].startswith_fws()
  File "/usr/lib64/python3.4/email/_header_value_parser.py", line 267, in 
startswith_fws
return self[0].startswith_fws()
  File "/usr/lib64/python3.4/email/_header_value_parser.py", line 267, in 
startswith_fws
return self[0].startswith_fws()
  File "/usr/lib64/python3.4/email/_header_value_parser.py", line 267, in 
startswith_fws
return self[0].startswith_fws()
IndexError: list index out of range

when flattened with BytesGenerator.

--
components: email
files: flatten-exception.mail
messages: 267736
nosy: barry, frispete, r.david.murray
priority: normal
severity: normal
status: open
title: Exception in BytesGenerator.flatten
versions: Python 3.4, Python 3.5, Python 3.6
Added file: http://bugs.python.org/file43288/flatten-exception.mail

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 07.06.2016 21:12, Cory Benfield wrote:
> 
>> Python hash randomization only happens once.  So it's not a matter of how 
>> early we try the attack, it's a matter of how early we seed Python hash 
>> randomization.
> 
> Sorry Larry, I was insufficiently clear (relying on context from earlier). I 
> totally agree that Python startup should not block. I'm saying that having 
> getrandom() called in "blocking mode" for os.urandom, random.SystemRandom, 
> and secrets is not a DoS vector.

I'm not sure I follow. A delay of 90 seconds on startup of a VM
or container can easily lead the supervisor style management tool
to think that something is wrong and issue a retry. Depending on
the configuration it'll then try this a couple of times and
give up.

Overall, I have a hard time following the arguments.

To be clear: getrandom() on Linux is just a wrapper with some
additional control around /dev/random and /dev/urandom.

http://lxr.free-electrons.com/source/drivers/char/random.c#L1601

Unlike /dev/urandom, getrandom() without flag GRND_NONBLOCK will
block, but only in the case where the entropy pool has not been
initialized yet. Once this has been done, it will never block
again, and happily send you poor random data if the entropy pool
has been completely wiped of any entropy data - without telling
you.

So now, you're all arguing: oh my, it's so insecure to use
data from /dev/urandom when the entropy pool is not initialized.
But you're not worried about os.urandom() happily sending you data
which is no longer based on any external entropy half an hour later:

http://lxr.free-electrons.com/source/drivers/char/random.c#L1458

This doesn't make sense. Either you're worried all the time,
or you're not :-)

The whole discussion is centering around whether to block
on an uninitialized entropy pool or not. This can only happen
during startup. By falling back to reading /dev/urandom
in case of an uninitialized pool, you are reading data from
a not fully initialized pool, but you still get random data.
That's really all that's needed for basic operations like
hash seeding or seeding the PRNG in the random module.
And it's limited to Python processes which are run very
early in the VM/container startup phase.

Note that "uninitialized" only means that the kernel entropy
pool has not yet reached an "entropy level" of 128 (whatever
that means):

http://lxr.free-electrons.com/source/drivers/char/random.c#L676

It does not mean that you're just reading a list of zeros.

So given all this information, why is it that you get so
tangled up in wanting os.urandom() to block during system
startup ?

Or put differently: Where is the attack vector that blocking
behavior of os.urandom() would help remedy ?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27259] Possible missing deprecation warnings?

[Mark Summerfield]

New submission from Mark Summerfield:

In the docs for the array module:
https://docs.python.org/dev/library/array.html#module-array
Note 1 in the table of type codes says that the 'u' type is deprecated and will 
go in Python 4.0.

Since the array.fromunicode() and array.tounicode() methods depend on type code 
'u' shouldn't they also be marked as deprecated?

If people use the 'u' type code maybe it would be helpful to provide an example 
of how to store unicode chars in an array.array? (Presumably they'd use type 
code 'L' and use ord() and chr()?)

--
assignee: docs@python
components: Documentation
messages: 267738
nosy: docs@python, mark
priority: normal
severity: normal
status: open
title: Possible missing deprecation warnings?
type: enhancement
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Larry Hastings]

Larry Hastings added the comment:

I fear I may be changing my mind a little bit.  However, I skipped 
breakfast--and now it's looking like a late lunch--so I simply have to step 
away for a while.  Expect me to post in about two hours when I get some 
calories down and finally make up my tiny mind.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Donald Stufft]

Donald Stufft added the comment:

Colm,

Great, then I think there's general agreement, we just need someone to review 
the nonblocking_urandom_noraise.patch (which my C is not strong enough to feel 
comfortable doing). That still leaves the `import random` issue, but I think we 
can reopen #25420 and figure out over there what the right answer is for that.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26632] @public - an __all__ decorator

[Barry A. Warsaw]

Barry A. Warsaw added the comment:

On Jun 04, 2016, at 07:47 PM, Zachary Ware wrote:

>So, +1 anyway.  I think this would be rather worthwhile, especially in the
>stdlib.

Thanks!

I still like it and plan on continuing to use it in my code.  I would
recommend you playing with the third party module:

https://pypi.python.org/pypi/atpublic

and seeing how you like it.  If you do like it maybe help get some momentum
behind it.  Then we can approach python-dev and try to get it into builtins.
I'd be willing to bring it up there (it didn't get such an overwhelming
reception at the Pycon 2016 language summit ;).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27257] get_addresses results in traceback with a valid? header

[Hans-Peter Jansen]

New submission from Hans-Peter Jansen:

In the course of replacing an old Python 2.7 email filter tool with a rewritten 
Python3 version, I stumbled across a ugly case, where such an header:

To: unlisted-recipients: ;,
""@pop.kundenserver.de (no To-header on input)

results in a Traceback (most recent call last):
  File "./mail_filter.py", line 606, in 
ret = main.run()
  File "./mail_filter.py", line 595, in run
self.process(fp)
  File "./mail_filter.py", line 520, in process
config.recipients = self.get_addresses('to', msg)
  File "./mail_filter.py", line 103, in get_addresses
vals = msg.get_all(field, [])
  File "/usr/lib64/python3.4/email/message.py", line 511, in get_all
values.append(self.policy.header_fetch_parse(k, v))
  File "/usr/lib64/python3.4/email/policy.py", line 145, in header_fetch_parse
return self.header_factory(name, ''.join(value.splitlines()))
  File "/usr/lib64/python3.4/email/headerregistry.py", line 584, in __call__
return self[name](name, value)
  File "/usr/lib64/python3.4/email/headerregistry.py", line 195, in __new__
cls.parse(value, kwds)
  File "/usr/lib64/python3.4/email/headerregistry.py", line 342, in parse
for mb in addr.all_mailboxes]))
  File "/usr/lib64/python3.4/email/headerregistry.py", line 342, in 
for mb in addr.all_mailboxes]))
  File "/usr/lib64/python3.4/email/_header_value_parser.py", line 837, in 
local_part
return self[0].local_part
  File "/usr/lib64/python3.4/email/_header_value_parser.py", line 889, in 
local_part
return self[0].local_part
  File "/usr/lib64/python3.4/email/_header_value_parser.py", line 984, in 
local_part
tok[0].token_type == 'cfws'):
IndexError: list index out of range

I'm not completely sure, if the Top header, as added from my email provider, is 
perfectly valid, but none of the other parts of my mail infrastructure neither 
complained, nor behave strange with such headers.

This happens with 3.4.4, but also with the email module from current hg for 
testing.

--
components: email
files: lkml-exception.mail
messages: 267733
nosy: barry, frispete, r.david.murray
priority: normal
severity: normal
status: open
title: get_addresses results in traceback with a valid? header
versions: Python 3.4, Python 3.5, Python 3.6
Added file: http://bugs.python.org/file43287/lkml-exception.mail

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27081] Cannot capture sys.stderr output from an uncaught exception in a multiprocessing Process using a multiprocessing Queue

[ppperry]

Changes by ppperry :


--
title: Multiprocessing is not robust against sys.stderr changes involving 
queues -> Cannot capture sys.stderr output from an uncaught exception in a 
multiprocessing Process using a multiprocessing Queue

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27256] header indentation destroyed

[Hans-Peter Jansen]

Changes by Hans-Peter Jansen :


Added file: http://bugs.python.org/file43286/mf.9__mi0bf.out

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27256] header indentation destroyed

[Hans-Peter Jansen]

New submission from Hans-Peter Jansen:

In the course of replacing an old Python 2.7 email filter tool with a rewritten 
Python3 version, I stumbled across a ugly case, where such an header:

X-Microsoft-Exchange-Diagnostics: 
=?utf-8?B?MTtCTDJQUjAyTUI1MTQ7MjM6bEtRRlNaUHQvVTk5WCttdktlOUVrUGQvVFBH?=
 =?utf-8?B?cDFJemVUeXFzOGNzYnZOYWlwMDZpR0YzbXZyY09WaTBKM2pkeUl4S1VDMkxw?=
 =?utf-8?B?eVRkNWthRW9waUhJTzczTWd5WDZOQ3hMNU1haGFvQTVzVTdRZmxJUnZlblpW?=
 ...

is regenerated as:

X-Microsoft-Exchange-Diagnostics:
 1;BL2PR02MB514;23:lKQFSZPt/U99X+mvKe9EkPd/TPG
p1IzeTyqs8csbvNaip06iGF3mvrcOVi0J3jdyIxKUC2Lp
yTd5kaEopiHIO73MgyX6NCxL5MahaoA5sU7QflIRvenZV

which is plain wrong of course.

I'm using email.message_from_binary_file for parsing and BytesGenerator.flatten 
for regeneration. Since those are LKML public mails, I'm attaching both 
versions.

I'm using 3.4.4, but also the email module from current hg for testing.

--
components: email
files: utf8-header-failure.mail
messages: 267732
nosy: barry, frispete, r.david.murray
priority: normal
severity: normal
status: open
title: header indentation destroyed
versions: Python 3.4, Python 3.5, Python 3.6
Added file: http://bugs.python.org/file43285/utf8-header-failure.mail

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Cory Benfield]

Cory Benfield added the comment:

> Python hash randomization only happens once.  So it's not a matter of how 
> early we try the attack, it's a matter of how early we seed Python hash 
> randomization.

Sorry Larry, I was insufficiently clear (relying on context from earlier). I 
totally agree that Python startup should not block. I'm saying that having 
getrandom() called in "blocking mode" for os.urandom, random.SystemRandom, and 
secrets is not a DoS vector.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Larry Hastings]

Larry Hastings added the comment:

> This is only a DoS vector if you can hit the server so early in the boot 
> process that it doesn't have enough entropy.

Python hash randomization only happens once.  So it's not a matter of how early 
we try the attack, it's a matter of how early we seed Python hash randomization.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Cory Benfield]

Cory Benfield added the comment:

> So you are intentionally accepting a new vector for DoS attacks, and calling
this non-reduced security?

This is only a DoS vector if you can hit the server so early in the boot 
process that it doesn't have enough entropy. The *second* enough entropy has 
been gathered getrandom() will never block again.

In essence, then, the situation where it becomes possible to DoS a server is 
entirely outside an attackers control and extremely unlikely to ever actually 
occur in real life: you can only DoS the server if you can demand entropy 
before the system has gathered enough, and if the server has managed to *boot* 
by then then the alternative is that it is incapable of generating secure 
random numbers and shouldn't be running exposed against the web anyway.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Colm Buckley]

Colm Buckley added the comment:

Donald -

Oh, that's interesting; thank you. I guess that system-crontab-generator is 
being called before that in the boot process.

The most common trigger case, I guess, will then be cloud containers and VMs 
which are spun up for single applications. I think Larry's comments are still 
very valid.

Colm

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27254] heap overflow in Tkinter module

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Minimal reproducer:

from tkinter import Tk
Tk(className='A'*0xf)

This looks as Tcl/Tk problem.

--
nosy: +serhiy.storchaka
type: security -> crash
versions: +Python 3.5, Python 3.6 -Python 3.3, Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25958] Implicit ABCs have no means of "anti-registration"

[Ethan Furman]

Changes by Ethan Furman :


--
nosy: +ethan.furman

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Donald Stufft]

Donald Stufft added the comment:

> specifically, early in the boot process, although not I think just on the 
> first boot of a system; Debian at least by default does not attempt to 
> preserve its entropy pool across a reboot.)

Look at /etc/init.d/urandom in the initscripts package in Jessie 
(https://anonscm.debian.org/cgit/collab-maint/sysvinit.git/tree/debian/src/initscripts/etc/init.d/urandom).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Colm Buckley]

Colm Buckley added the comment:

To clarify what the various patches do:

3.5.1 as released: os.urandom and hash secret initialization both attempt 
getrandom() in preference to reading /dev/urandom. Under certain circumstances, 
this will block, possibly indefinitely.

Changeset 9de508dc4837: both os.urandom and hash secret initialization call 
getrandom() in nonblocking mode, falling back to (possibly low-entropy) 
/dev/urandom should getrandom() block due to lack of entropy.

Changeset 9de508dc4837 + nonblocking_urandom_noraise.patch: hash secret 
initialization calls getrandom() in nonblocking mode (ie: will always succeed, 
although with a silent fallback to low-entropy data if called when the system 
has no entropy). os.urandom will always block until there's enough entropy.

I think this final case implements what you need for the 3.5.2 RC.

The issue of "import random" still needs to be resolved; maybe we should 
de-merge #25420 and pursue Donald's approach there.

Thanks,

Colm

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27242] Make the docs for NotImplemented & NotImplementedError unambigous

[Emanuel Barry]

Emanuel Barry added the comment:

More tweaks after Ethan's comments. This is starting to get way outside of the 
realm of NotImplemented, but it still makes for a good reference to point 
confused users at.

--
Added file: 
http://bugs.python.org/file43284/NotImplemented_exceptions_wording_4.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Matthias Klose]

Matthias Klose added the comment:

On 07.06.2016 16:47, Christian Heimes wrote:
> 
> Christian Heimes added the comment:
> 
> PSRT VETO!
> 
> In the light of the upcoming release 3.5.2 I'm now putting on my PSRT hat 
> (Python Security Response Team) and proclaim a veto against any and all 
> changes to os.urandom(). The security properties of os.urandom() must not be 
> modified or reduced compared to 3.5.1. Please restore the behavior of 
> os.urandom().

So you are intentionally accepting a new vector for DoS attacks, and calling
this non-reduced security?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27127] Never have GET_ITER not followed by FOR_ITER

[Raymond Hettinger]

Raymond Hettinger added the comment:

I don't think this should go forward.  The current FOR_ITER and JUMP_ABSOLUTE 
combination is very efficient and shouldn't be changed lightly.  It is the 
inside of the loop that matters -- the GET_ITER step is outside of the loop and 
isn't expensive.

Also, I really like that GET_ITER and FOR_ITER correspond exactly to our high 
level understanding of how for-loops operate:

   for x in s:
   f(x)

corresponds to:

   it = iter(s)
   while True:
   try:
   x = next(it)
   except StopIteration:
   break
   f(x)

I really don't think this should be pursued further.  It messes with my 
understanding of for-loops, it combines features that should remain decoupled, 
it focuses its efforts on the exterior of the loop, and it alters code that is 
very mature (having been examined and tweaked by hundreds of your predecessors).

--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Larry Hastings]

Larry Hastings added the comment:

That reminds me.  I want to be clear: I think it's preferable that os.urandom() 
blocks when insufficient entropy is available.  If Victor's patch changed that, 
it should be backed out.

(Since non-blocking urandom is useful, perhaps in 3.6 os.urandom() should take 
a new "block=True" parameter.  But it's too late to add it for 3.5.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Colm Buckley]

Colm Buckley added the comment:

Larry -

To the first point:

The combination of Victor's changeset 9de508dc4837 (based on my patch) and my 
most recent nonblocking_urandom_noraise patch (which is on top of 9de508dc4837) 
will do what you suggest for the hash secret initialization - ie: it is allowed 
to fall back to predictable sources when there is insufficient entropy to 
securely seed it.

I suspect that it is simply impossible to reconcile "os.urandom will never 
block" with "os.urandom is always cryptographically reasonable". If the system 
has no entropy, it has no entropy. The only escape I see is to add an exception 
condition, instead of the silent fallback which some platforms currently have. 
There is a judgement call to be made here; whether silent fallback is 
acceptable or not.

As Donald points out, this will fail only in very unusual circumstances 
(specifically, early in the boot process, although not I think just on the 
first boot of a system; Debian at least by default does not attempt to preserve 
its entropy pool across a reboot.)

This should not affect things like web servers etc. as they start much later in 
the boot process; in particular after networking has started, which I believe 
is the principal source of entropy for /dev/urandom.

Colm

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27255] More opcode predictions

[Serhiy Storchaka]

New submission from Serhiy Storchaka:

Currently the PREDICT() macros is used for predicticting following pair of 
opcodes in the ceval loop:

LIST_APPEND JUMP_ABSOLUTE
SET_ADD JUMP_ABSOLUTE
MAP_ADD JUMP_ABSOLUTE
COMPARE_OP POP_JUMP_IF_FALSE
COMPARE_OP POP_JUMP_IF_TRUE
GET_ITER FOR_ITER
FOR_ITER STORE_FAST (if continue iterating)
FOR_ITER UNPACK_SEQUENCE (if continue iterating)
WITH_CLEANUP_START WITH_CLEANUP_FINISH
WITH_CLEANUP_FINISH END_FINALLY

I collected static and dynamic opcode statistics. Static statistics is a 
statistics about what opcodes are used in code objects after running the 
compileall script on the Lib directory. Dynamic statistics is is a statistics 
about what opcodes are executed when run all Python tests. Resulting numbers 
are close, but there are some reasonable differences (opcodes used in loops 
have larger numbers in dynamic statistics, and opcodes used for creating 
classes and functions have larger numbers in static statistics). In following 
lines the fist number is from dynamic statistics and the second number is from 
static statistics.

LIST_APPEND, SET_ADD and MAP_ADD are always followed by JUMP_ABSOLUTE.
COMPARE_OP is followed by POP_JUMP_IF_FALSE in 83% (78%) and by 
POP_JUMP_IF_TRUE in 13% (5%).
GET_ITER is followed by FOR_ITER in 72% (80%) and CALL_FUNCTION in 28% (28%).
FOR_ITER is followed by STORE_FAST in 50% (80%), UNPACK_SEQUENCE in 21% (18%) 
and POP_BLOCK in 20% (0%).
WITH_CLEANUP_START is always followed by WITH_CLEANUP_FINISH.
WITH_CLEANUP_FINISH is always followed by END_FINALLY

According to this statistic existing predictions are correct.

But the statistics suggests other predictions.

JUMP_ABSOLUTE is followed by FOR_ITER in 77% (0%).
UNPACK_SEQUENCE is followed by STORE_FAST in 97% (94%).
SETUP_LOOP is followed by LOAD_FAST in 81% (52%) and LOAD_GLOBAL in 18% (31%).
BUILD_SLICE is followed by BINARY_SUBSCR in 99% (87%).
ROT_TWO is followed by STORE_FAST in 85% (40%).
LOAD_CLOSURE is followed by BUILD_TUPLE in 94% (68%).
SETUP_WITH is followed by POP_TOP in 94% (54%) and STORE_FAST in 6% (44%).
GET_YIELD_FROM_ITER, GET_AITER, GET_ANEXT and GET_AWAITABLE are always followed 
by LOAD_CONST.
DUP_TOP_TWO is always followed by BINARY_SUBSCR.
BEFORE_ASYNC_WITH is always followed by GET_AWAITABLE.
All INPLACE_ instructions almost always are followed by STORE_FAST.

Proposed patch adds predictions of following pairs:

DUP_TOP_TWO BINARY_SUBSCR
INPLACE_POWER STORE_FAST
INPLACE_MULTIPLY STORE_FAST
INPLACE_MATRIX_MULTIPLY STORE_FAST
INPLACE_TRUE_DIVIDE STORE_FAST
INPLACE_FLOOR_DIVIDE STORE_FAST
INPLACE_MODULO STORE_FAST
INPLACE_ADD STORE_FAST
INPLACE_SUBTRACT STORE_FAST
INPLACE_LSHIFT STORE_FAST
INPLACE_RSHIFT STORE_FAST
INPLACE_AND STORE_FAST
INPLACE_XOR STORE_FAST
INPLACE_OR STORE_FAST
GET_AITER LOAD_CONST
GET_ANEXT LOAD_CONST
GET_AWAITABLE LOAD_CONST
UNPACK_SEQUENCE STORE_FAST
LOAD_CLOSURE BUILD_TUPLE
GET_ITER CALL_FUNCTION
GET_YIELD_FROM_ITER LOAD_CONST
FOR_ITER POP_BLOCK
BEFORE_ASYNC_WITH GET_AWAITABLE
SETUP_WITH POP_TOP
SETUP_WITH STORE_FAST
BUILD_SLICE BINARY_SUBSCR

Note that the effect of this change is not very significant since the PREDICT() 
macros is no-op if computed GOTOs are used.

--
components: Interpreter Core
files: more_opcode_predicts.patch
keywords: patch
messages: 267719
nosy: serhiy.storchaka
priority: normal
severity: normal
stage: patch review
status: open
title: More opcode predictions
type: enhancement
versions: Python 3.6
Added file: http://bugs.python.org/file43283/more_opcode_predicts.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Donald Stufft]

Donald Stufft added the comment:

> As a final note, let me steer you towards this comment in Python/random.c:
>
> /* Issue #25003: Don' use getentropy() on Solaris (available since
>  * Solaris 11.3), it is blocking whereas os.urandom() should not block. */
>
> Yes: we already had this discussion for Solaris, nine months ago, on issue 
> #25003.  Both Guido and Tim Peters were involved in the discussion.  The 
> decision there: use lower-quality random bits to seed the MT when importing 
> the random module.  Keeping the slowdown was so obviously wrong it wasn't 
> even debated.

I will point out, that was a somewhat different situation as ``getentropy`` on 
Solaris is more like /dev/random in that it tries to decide how much 
random-ness is in the pool and will randomly block throughout the execution of 
the program. The ``getrandom()`` call on Linux (and Solaris) will, by default, 
only block on the first boot at the very beginning before the kernel has 
collected enough entropy.

I don't think this changes anything, I just want to be clear because there are 
two kinds of "blocking" in this discussion, one that only occurs in very 
specific scenarios and one that occurs regularly in the operation of the 
program.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27255] More opcode predictions

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
priority: normal -> low

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27252] Make dict views copyable

[Raymond Hettinger]

Raymond Hettinger added the comment:

I think we should stop adding features like this without actual user requests 
and real use cases to show necessity and to inform the design.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Larry Hastings]

Larry Hastings added the comment:

Everybody: let's drop discussing "hashlib" unless someone says it actually is a 
problem.  I think it was always, as we say in English, a "red herring".


> The secret for SipHash is composed of two 64bit integers. The entire 
> _Py_HashSecret_t struct is 24 bytes. The remaining 8 bytes are used for XML 
> hash randomization of libexpat. Only the manual seed with PYTHONHASHSEED is a 
> 32bit integer which is stretched to 24 bytes with a LCG.

Okay, I have misunderstood the code.  Have I misunderstood the strength of 
SipHash?  Is it regarded as "cryptographically secure"?

Predictability of the hash function on web servers was the original use case of 
the "hash seed"; I remember a demonstration of an attack where the attacker 
produced pathologically bad hash behavior on a Python-based web server with 
very little data.  So it seems like web servers running on cloud instances is 
exactly the sort of use case where we'd want less-predictable hashing.

--

Nevertheless, a 90 second startup time is simply unacceptable.  I am officially 
making a pronouncement as Release Manager: Python 3.5 *must not* take 90 
seconds to start up under *any* circumstances.  I view this as a performance 
regression, and it is and will remain a release blocker for 3.5.2.

Python *must not* require special command-line flags to avoid a 90 second 
startup time.  Python *must not* require a special environment-variable to 
avoid a 90 second startup time.  This is no longer open to debate, and I will 
only be overruled by Guido.

--

If I understand the technical issues correctly, here's how I expect it to work. 
 For seeding the hash randomization, and seeding the _inst in the random 
module, we will use getrandom() in a non-blocking way (GRND_NONBLOCK?).  If it 
succeeds, we use those bits.  If it fails because it would have blocked 
(EAGAIN?), we fall back to a less-random source of random bits.  Under no 
circumstances will Python block when seeding the hash randomization function or 
seeding the MT for the random module.

This means cloud instances may inadvertently use lower-quality hash 
randomization seeds.  I judge this as obviously better than cloud instances 
taking 90 seconds to start up.  Also, as Christian points out, the people 
running these cloud instances should be managing their entropy pools anyway.  
Additionally, there are many uses of cloud instances that aren't exposed to 
tainted data that permit these predictable-hash abuses.

--

As a final note, let me steer you towards this comment in Python/random.c:

/* Issue #25003: Don' use getentropy() on Solaris (available since
 * Solaris 11.3), it is blocking whereas os.urandom() should not block. */

Yes: we already had this discussion for Solaris, nine months ago, on issue 
#25003.  Both Guido and Tim Peters were involved in the discussion.  The 
decision there: use lower-quality random bits to seed the MT when importing the 
random module.  Keeping the slowdown was so obviously wrong it wasn't even 
debated.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Donald Stufft]

Donald Stufft added the comment:

I've attached a minimal patch for making it so ``import random`` does not 
block, it does this by changing what the default instance of Random() is seeded 
with, from os.urandom() to the time based fallback it currently employs. It 
does not change the behavior of any documented behavior that I can see (it's 
documented that calling seed(None) or seed() will use urandom if available).

This could be improved by mixing in id(self) and using SipHash or LCG on the 
value, but this represents a minimal patch that is already possible in cases 
where os.urandom doesn't exist.

--
Added file: http://bugs.python.org/file43282/no-urandom-by-default.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27254] heap overflow in Tkinter module

[Emin Ghuliev]

New submission from Emin Ghuliev:

/* This is used to get the application class for Tk 4.1 and up */
argv0 = (char*)attemptckalloc(strlen(className) + 1); //<=== classname 
allocated
if (!argv0) {
PyErr_NoMemory();
Py_DECREF(v);
return NULL;
}

strcpy(argv0, className); < //classname copy to argv0
if (Py_ISUPPER(Py_CHARMASK(argv0[0])))
argv0[0] = Py_TOLOWER(Py_CHARMASK(argv0[0]));
Tcl_SetVar(v->interp, "argv0", argv0, TCL_GLOBAL_ONLY); // argv0 passed to 
v->interp and freed;
ckfree(argv0);

then v->interp passed to the Tcl_AppInit function
if (Tcl_AppInit(v->interp) != TCL_OK)

in Tcl_AppInit call to (and passed the v->interp) the Tcl_DStringAppend. 
allocates the specified byte Tcl_DStringAppend function then heap memory passed 
to memcpy.

Realloc arguments
presentation in the native tcl allocator; 
char * 
Tcl_Realloc(ptr, size) 

disassemble:
gdb>  print /x $rdi
$4 = 0x703c8810
0x703c8814: 0x41414141 ...
gdb>  print /x $rsi
$2 = 0x3e
   0x73a07dfe <+46>:call   0x73935040 
after return to the caller function. Performed memory copy operation.

   0x73a07e0a <+58>:leardi,[rax+rdx*1] < === destination buffer

   $rax = 0x7fffeffc5810 - $rdx = 0x10
   $rax+$rdx = 0x700c5810



   0x73a07e0e <+62>:movrsi,r12 < === source buffer
   0x73a07e11 <+65>:movsxd rdx,ebp <=== 0xf
   0x73a07e14 <+68>:call   0x739155c0 

copy to $rdi bytes to $rsi buffer with 0xf byte;

ASAN report.

=
==27988==ERROR: AddressSanitizer: heap-use-after-free on address 0x7f4e6ba64810 
at pc 0x4665ea bp 0x7fff89a4ab80 sp 0x7fff89a4a340
READ of size 1048575 at 0x7f4e6ba64810 thread T0
==27988==WARNING: Trying to symbolize code, but external symbolizer is not 
initialized!
#0 0x4665e9 (/home/eminus/Downloads/Python-2.7.11/python+0x4665e9)
#1 0x7f4e6f0a3e18 (/usr/lib/x86_64-linux-gnu/libtcl8.6.so+0x116e18)
#2 0x7f4e6f38744e (/usr/lib/x86_64-linux-gnu/libtk8.6.so+0x6244e)
#3 0x7f4e6f6b6e4c 
(/home/eminus/Downloads/Python-2.7.11/build/lib.linux-x86_64-2.7/_tkinter.so+0x19e4c)
#4 0x7f4e6f6a7fc5 
(/home/eminus/Downloads/Python-2.7.11/build/lib.linux-x86_64-2.7/_tkinter.so+0xafc5)
#5 0x5e1813 (/home/eminus/Downloads/Python-2.7.11/python+0x5e1813)
#6 0x5d319c (/home/eminus/Downloads/Python-2.7.11/python+0x5d319c)
#7 0x721353 (/home/eminus/Downloads/Python-2.7.11/python+0x721353)
#8 0x4acb2a (/home/eminus/Downloads/Python-2.7.11/python+0x4acb2a)
#9 0x4b6c62 (/home/eminus/Downloads/Python-2.7.11/python+0x4b6c62)
#10 0x4acb2a (/home/eminus/Downloads/Python-2.7.11/python+0x4acb2a)
#11 0x5f0823 (/home/eminus/Downloads/Python-2.7.11/python+0x5f0823)
#12 0x4b0a08 (/home/eminus/Downloads/Python-2.7.11/python+0x4b0a08)
#13 0x4acb2a (/home/eminus/Downloads/Python-2.7.11/python+0x4acb2a)
#14 0x5e2d19 (/home/eminus/Downloads/Python-2.7.11/python+0x5e2d19)
#15 0x5d319c (/home/eminus/Downloads/Python-2.7.11/python+0x5d319c)
#16 0x5d2041 (/home/eminus/Downloads/Python-2.7.11/python+0x5d2041)
#17 0x660980 (/home/eminus/Downloads/Python-2.7.11/python+0x660980)
#18 0x65fc8a (/home/eminus/Downloads/Python-2.7.11/python+0x65fc8a)
#19 0x48e46c (/home/eminus/Downloads/Python-2.7.11/python+0x48e46c)
#20 0x7f4e72389ec4 (/lib/x86_64-linux-gnu/libc.so.6+0x21ec4)
#21 0x48c5bc (/home/eminus/Downloads/Python-2.7.11/python+0x48c5bc)

0x7f4e6ba64810 is located 16 bytes inside of 2097166-byte region 
[0x7f4e6ba64800,0x7f4e6bc6480e)
freed by thread T0 here:
#0 0x4766d3 (/home/eminus/Downloads/Python-2.7.11/python+0x4766d3)
#1 0x7f4e6f09b52d (/usr/lib/x86_64-linux-gnu/libtcl8.6.so+0x10e52d)

previously allocated by thread T0 here:
#0 0x4764d9 (/home/eminus/Downloads/Python-2.7.11/python+0x4764d9)
#1 0x7f4e6f09b0cc (/usr/lib/x86_64-linux-gnu/libtcl8.6.so+0x10e0cc)

SUMMARY: AddressSanitizer: heap-use-after-free ??:0 ??
Shadow bytes around the buggy address:
  0x0fea4d7448b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0fea4d7448c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0fea4d7448d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0fea4d7448e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0fea4d7448f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0fea4d744900: fd fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0fea4d744910: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0fea4d744920: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0fea4d744930: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0fea4d744940: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0fea4d744950: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:   00
  Partially addressable: 01 02 03 04 05 06 07 
  

[issue27252] Make dict views copyable

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

I didn't encounter this.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Christian Heimes]

Christian Heimes added the comment:

On 2016-06-07 19:36, Donald Stufft wrote:
> 
> Donald Stufft added the comment:
> 
>> Possible resolutions:
>>  * accept possible low-entropy initialization of the hash secret; using the 
>> patches supplied here by myself and Victor.
>>  * add a command-line flag to disable "strong" initialization of the hash 
>> secret (or revive the old -R flag).
>>  * simply require user-space workarounds like setting PYTHONHASHSEED
> 
> I think either the first or second here are good solutions, the third is kind 
> of crummy on it's own because it's not always possible to pass in an 
> environment variable. Pairing the third with a CLI flag option might work out 
> nice though, perhaps a -XPYTHONHASHSEED=(random/int()) or something. Then 
> folks who are in early boot can easily just hardcode a hash seed, removing 
> the need to hit the entropy pools while still maintaining strong random for 
> everyone else.
> 
> So I guess I would lean towards adding a CLI flag, but just allowing SipHash 
> to fall back to possibly bad randomness for it's initialization is OK.

I don't like the fact that applications can fall back to insecure RNG
without user involvement or warning.

Therefore I'm in favor of a command line argument that allows pyhash.c
to fall back to a less secure RNG. System scripts must use the -I option
(isolated mode without user-site dir and PY* env vars) anyway. The new
option would enable less secure RNG as fallback and -I.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Christian Heimes]

Christian Heimes added the comment:

On 2016-06-07 19:46, Larry Hastings wrote:
> 
> Larry Hastings added the comment:
> 
> Thank you for summarizing the debate. It made it a lot easier to 
> 
>> * blocking initialization of the hash secret. This occurs regardless of 
>> script contents; at present Python simply can't be used at all in 
>> low-entropy situations. I feel that this issue is a release blocker.
>>
>> Possible resolutions:
>>   * accept possible low-entropy initialization of the hash secret; using the 
>> patches supplied here by myself and Victor.
>>   * add a command-line flag to disable "strong" initialization of the hash 
>> secret (or revive the old -R flag).
>>   * simply require user-space workarounds like setting PYTHONHASHSEED
> 
> The latter two approaches are unacceptable IMO.  They result in a poor user 
> experience.  Python should do the "right" thing by default; the "right" thing 
> includes not taking 90 seconds to start up.
> 
> By process of elimination, this leaves only the first approach as viable.  
> Ergo, let's do that.
> 
> The hash secret is a 32-bit integer, even on 64-bit builds of Python.  It is 
> not and cannot be cryptographically secure.  It's frankly ridiculous to fret 
> about "strong" initialization of it at the cost of a 90 second startup time.
> 
> (For posterity: when people mention "SipHash", they're talking about the 
> hashing algorithm used for str/dict/etc.  The seed for SipHash is the "hash 
> secret" we're talking about here.)

The secret for SipHash is composed of two 64bit integers. The entire
_Py_HashSecret_t struct is 24 bytes. The remaining 8 bytes are used for
XML hash randomization of libexpat. Only the manual seed with
PYTHONHASHSEED is a 32bit integer which is stretched to 24 bytes with a LCG.

Christian

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Donald Stufft]

Donald Stufft added the comment:

> I don't follow whose patch does what.  But here's what I find acceptable, 
> from a high level.
> 
> * The semantics as presented by the documentation must be preserved.  
> os.urandom() and other operations that declare they're safe for cryptographic 
> use must remain safe for cryptographic use.
> * "import random" must not block.
> * "import hashlib" must not block.
>
> Is there a patch set that accomplishes that?

I *think* nonblocking_urandom_noraise.patch will solve the 90+ second start up 
without affecting os.urandom which should solve the first one (once the already 
applied patch gets reverted), but I'm afraid I don't know C well enough to 
meaningfully review that for accuracy.

None of the current patches solve the second without invalidating the first, 
but it would be, I believe, an additional patch ontop of 
nonblocking_urandom_noraise.patch.

The third is already the case.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23496] Steps for Android Native Build of Python 3.4.2

[Chris Rebert]

Changes by Chris Rebert :


--
nosy: +cvrebert

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com