[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Larry Hastings]

Larry Hastings added the comment:

AFAICT, the history is, Linux added /dev/urandom, then all the other UNIXes 
followed suit.  I've read a lot of man pages for /dev/urandom (or equivalent) 
on a lot of different platforms and most of them say "this was added to Linux, 
so we added our own version".

/dev/random and getrandom() seem to be playing out the same way.  Linux added 
getrandom() first, then Solaris added it.  My bet is most of the other UNIXes 
will add it over the next couple years.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[STINNER Victor]

STINNER Victor added the comment:

IHMO we only need two functions. os.urandom() is now documented as the
reference for security. So we need a new function. I proposed a new
urandom() parameter, but a new function without parameter would also be ok.

If os.getrandom() is directly a binding of Linux/Solaris getrandom(), I
will not use it since it would not be portable.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27276] FileFinder.find_spec() incompatible with finder specification

[Paul Marinescu]

Changes by Paul Marinescu :


--
resolution:  -> not a bug
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27276] FileFinder.find_spec() incompatible with finder specification

[Paul Marinescu]

New submission from Paul Marinescu:

importlib.machinery.FileFinder.find_spec is incompatible with 
importlib.abc.MetaPathFinder.find_spec (different number of arguments).

The following leads to a runtime error:

loader = (importlib.machinery.SourceFileLoader, 
importlib.machinery.SOURCE_SUFFIXES)
sys.meta_path.append(importlib.machinery.FileFinder('/', loader))

--
messages: 267961
nosy: paulmar
priority: normal
severity: normal
status: open
title: FileFinder.find_spec() incompatible with finder specification
versions: Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27275] KeyError thrown by optimised collections.OrderedDict.popitem()

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Could you please provide short example?

--
nosy: +serhiy.storchaka
stage:  -> test needed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16132] ctypes incorrectly encodes .format attribute of memory views

[Martin Panter]

Martin Panter added the comment:

In Python 3.5 (since Issue 15944), you can now cast normal C-contiguous 
memoryviews (including ones from ctypes) to bytes:

>>> a = (ctypes.c_double * 3)(1,2,3)
>>> m = memoryview(a)
>>> m.format
'>> byteview = m.cast("B")
>>> byteview.format
'B'
>>> byteview[0]
0

Also, the format has changed at some point. See '

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27272] random.Random should not read 2500 bytes from urandom

[Tim Peters]

Tim Peters added the comment:

Donald, it does matter.  The code you found must be using some older version of 
Python, because the Python 3 version of randint() uses _randbelow(), which is 
an accept/reject method that consumes an _unpredictable_ number of 32-bit 
Twister outputs.  That utterly destroys the theoretical framework the code you 
found relies on.

I know all about this, because I did extensive research during the `secrets` 
module discussion - Python 3 isn't systematically vulnerable to any of the 
attacks in the paper.  Deduction of the Twister's internal state is blocked in 
"almost all" cases because of the accept/reject logic in _randbelow() (BTW, 
.choice() is of far more practical concern here than .randint(), but in Python 
3 they both build on _randbelow()).

Code attempting to deduce the state can't know how many Twister outputs were 
consumed, so can't know which bit variables in its equations are involved.  It 
can make a relatively high-probability _guess_, but that's not good enough 
either:  it has to get many consecutive high-probability guesses right to 
deduce all the bits of the Twister's very large state.

The usual outcome:  at least one (usually more than one) guess is wrong, and if 
the equation solver is careful it notices that its equations have become 
mutually inconsistent.  Dead end.  More rarely, the equations remain 
consistent, but the "deduced" state is pure fantasy due to a wrong guess along 
the way.

There's nothing about this that's a mystery to me.  I wrote my own solver more 
capable than the one you found:  it can deduce full MT states quickly from 
partial outputs of any kind whatsoever (e.g., it only sees the bits of the form 
2**i for prime i in every 7th Twister output).  However, it too needs to know 
exactly _which_ MT partial outputs it's seeing.  If it believes it's seeing 
every 7th, but actually sees the 8th in one case, all bets are off:  the 
equations may become inconsistent, or they remain consistent but deliver a 
nonsense state.

So I remain strongly -1 on any attempt to make Python's seeding stupid again.  
Stupid seeding makes Python vulnerable to attacks by script kiddies; no 
relatively sophisticated equation solvers are needed if the seeding is lame.

Yes, the Twister remains unsuitable for crypto purposes.  That's why the 
`secrets` module is being introduced.  But that's no excuse for deliberately 
making existing naive code laughably easy to attack.

And, also yes, we have made some decisions based on worst-case scenarios about 
naivety.  That's why random switched to using urandom() to begin with.  Get 
over it ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27275] KeyError thrown by optimised collections.OrderedDict.popitem()

[William Pitcock]

New submission from William Pitcock:

The C-based optimised version of collections.OrderedDict occasionally throws 
KeyErrors when deleting items.

See https://github.com/mailgun/expiringdict/issues/16 for an example of this 
regression.

Backporting 3.6's patches to 3.5.1 does not resolve the issue. :(

--
components: Library (Lib)
messages: 267957
nosy: kaniini
priority: normal
severity: normal
status: open
title: KeyError thrown by optimised collections.OrderedDict.popitem()
type: behavior
versions: Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26955] Implement equivalent to `pip.locations.distutils_scheme` in distutils

[Sylvain Corlay]

Sylvain Corlay added the comment:

We use it in the pybind11 project. I am not sure that homebrew does anything 
wrong.

It is just that it is the only case I am aware of where the `install_headers` 
command does not install headers in a subdirectory of 
`sysconfig.get_path('include')`. On the other hand, 
pip.locations.distutil_scheme provides the location used by install_headers.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27272] random.Random should not read 2500 bytes from urandom

[Donald Stufft]

Donald Stufft added the comment:

Tim,

If MT is used in any of the security sensitive contexts that paper mentions, 
then it doesn't matter if you seed it with a static zero or a billion bytes 
read from the purest of purestrain randomness, your application is broken. In 
other words, it doesn't matter what we seed it with, random.py (outside of 
SystemRandom) is vulnerable to everything said in that paper.

It took me 5 minutes of googling to find https://github.com/fx5/not_random, 
which took 22 minutes on my iMac to generate my own copy of it's `magic_data` 
file, and then 15 seconds to clone the state of the MT using nothing but the 
output of it. This was on CPython 2.7.11 where MT is seeded with 2500 bytes off 
urandom.

Surely we're not making engineering trade offs based off whether or not someone 
who doesn't understand the difference between a CSPRNG and a PRNG would make 
fun of us for not using a CSPRNG where it's not needed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27274] [ctypes] Allow from_pointer creation

[Memeplex]

Memeplex added the comment:

I would like to add some information about my use case. Many c structs have 
pointers to arrays of data plus some field indicating the length of those 
arrays. Sometimes I need to pickle that kind of structs and a bytes object has 
to somehow be created from each pointer, given the length (the alternative 
ptr[:len] is too expensive for large arrays). So I need to cast the pointer to 
a ctypes array first and then convert the array to bytes (sadly, there is no 
way to pickle a memoryview, so a copy is unavoidable).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26565] [ctypes] Add value attribute to non basic pointers.

[Memeplex]

Memeplex added the comment:

Related: http://bugs.python.org/issue27274.

Maybe ptr.toaddress? As opposed to addressof(ptr). I think ptr.address might be 
confusing as it's intutive meaning is closer to addressof(ptr).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27272] random.Random should not read 2500 bytes from urandom

[Tim Peters]

Tim Peters added the comment:

Didn't anyone here follow the discussion about the `secrets` module?  PHP was 
crucified by security wonks for its horridly naive ways of initializing its 
PRNGs:

https://media.blackhat.com/bh-us-12/Briefings/Argyros/BH_US_12_Argyros_PRNG_WP.pdf

Please don't even think about making Python a target of similar ridicule ;-)

The only sane approach is to use an _excellent_ source of randomness for 
initialization, and `urandom()` is the only obvious such source.  While the 
more the merrier, I agree 2500 utterly unpredictable bytes isn't necessary.

If this has to change, use the most possible without creating other problems on 
a major platform, but certainly no less than 128 crypto-strength bytes.

-1 on any poke-and-hope gibberish trying to brew our own out of time.time(), 
PID, id(), etc.  That stuff is easy for a malicious program to attack.  That's 
why Python switched to using `urandom()` to begin with, before security wonks 
noticed how poorly most libraries handle this.

It's not about supplying "enough randomness" for applications, it's about 
making it computationally intractable for well-funded expert attackers to 
out-guess.  That's why urandom() is used.

--
nosy: +tim.peters

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27274] [ctypes] Allow from_pointer creation

[Memeplex]

New submission from Memeplex:

This real life example is pretty terrible:

(ct.c_float * self._nfeats).from_address(
   ct.addressof(self._vals.contents))

The alternative of casting the pointer to pointer-to-array, then pick 
ptr.contents is not really better.

What about a from_pointer(ptr) method? Or overloading from_address to take a 
pointer? Or a simple shortcut to get the address pointed to by a pointer 
(related: https://bugs.python.org/issue26565).

I think this part of ctypes api needs to get more concise and pythonic.

--
components: ctypes
messages: 267951
nosy: memeplex
priority: normal
severity: normal
status: open
title: [ctypes] Allow from_pointer creation
type: enhancement

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27272] random.Random should not read 2500 bytes from urandom

[Martin Panter]

Martin Panter added the comment:

If the seed is to be based on time.time(), why not use something based on 
hash(time.time()) to avoid the 1/256 s resolution?

If there is a practical way to get pseudo-random data from the platform, it 
would be better to use that, rather than cooking up our own entropy from 
time.time(), PID, etc. But I guess that depends on the future of os.urandom() 
and friends.

In the meantime, limiting to os.urandom(256) seems reasonable, though I’m not a 
random number or cryptography expert.

--
nosy: +martin.panter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27271] asyncio lost udp packets

[valdemar pavesi]

valdemar pavesi added the comment:

thanks Guido

I will keep working with it. 

I am not able to debug between socket and asyncio-read-udp.

There is no bottleneck on cpu/memory or network card.

I cannot write debug with this heavy udp load.

if I decrease the UDP per second then this problem go away. 

I know when it will start dropping and i never got any drop inside the network. 
( tcpdump to validate it ).

I will send it to python-tulip

regards!
Valdemar

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Larry Hastings]

Larry Hastings added the comment:

Victor: just to confirm, would "os.getrandom(n, block=True)" be okay with you?  
Do you strongly prefer adding the "block" parameter to os.urandom(), or do you 
not care much?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27066] SystemError if custom opener returns -1

[Berker Peksag]

Changes by Berker Peksag :


--
stage: patch review -> resolved

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27179] subprocess uses wrong encoding on Windows

[Dāvis]

Changes by Dāvis :


Added file: 
http://bugs.python.org/file43316/subprocess_fix_encoding_v4fixed.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27179] subprocess uses wrong encoding on Windows

[Dāvis]

Changes by Dāvis :


Removed file: http://bugs.python.org/file43315/subprocess_fix_encoding_v4.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[STINNER Victor]

STINNER Victor added the comment:

About the "add block=False parameter to os.urandom()" part of the issue title: 
I don't really care of the exact API. It can be a new function. The parameter 
can be named differently. The idea is to give access to users to Linux 
non-blocking /dev/urandom, and do something helpful for users on platforms 
without non-blocing urandom.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27179] subprocess uses wrong encoding on Windows

[Dāvis]

Dāvis added the comment:

> Note that patch 3 requires setting `encoding` for even python.exe as a child 
> process, because sys.std* default to ANSI when isatty(fd) isn't true.

I've updated my patch so that Python outputs in consoles encoding for pipes too.

So now in PowerShell

>[Console]::OutputEncoding = [System.Text.Encoding]::UTF8
>python -c "print('ā')" | Out-String
ā
> python -c "import subprocess; print(subprocess.getoutput('python -c 
> ""print(\'ā\')""'))"
ā
>[Console]::OutputEncoding = [System.Text.Encoding]::GetEncoding(775)
>python -c "print('ā')" | Out-String
ā
> python -c "import subprocess; print(subprocess.getoutput('python -c 
> ""print(\'ā\')""'))"
ā


> What I wish is for Python to default to using UTF-8 for its own pipe and disk 
> file I/O. The old behavior could be selected by setting some hypothetical 
> environment variable, such as PYTHONIOUSELOCALE.

I actually don't really see need for this, specifying PYTHONIOENCODING="UTF-8" 
it will be used for pipes.


> If subprocess defaults to the console's current codepage (when available), it 
> would be nice to have a way to conveniently select the OEM or ANSI codepage. 
> The codecs module could define string constants based on GetOEMCP() and 
> GetACP(), such as codecs.CP_OEMCP (e.g. 'cp437') and codecs.CP_ACP (e.g. 
> 'cp1252'). subprocess could import these constants on Windows.

also updated in my patch and implemented something like this but IMO easier, 
basically "ansi" and "oem" is a valid encoding on Windows and can be used 
anywhere where encoding can be specified as a parameter. Look at patch to see 
how it's implemented.


Ok, so now does my patch look acceptable? What would be issues with it? IMO it 
greatly improves current situation (fixes #27048 and solves #6135) and I don't 
see any issues with it.

Things that are changed:
* "ansi" and "oem" are valid encodings on Windows
* console's code page is used for console and pipe (if there's no console then 
ANSI is used like now)
* subprocess uses "ansi" for DETACHED_PROCESS and "oem" for CREATE_NEW_CONSOLE, 
CREATE_NO_WINDOW
* encoding and errors parameters can be specified for Popen
* custom parameters (including encoding and errors) can be specified for 
subprocess.getstatusoutput and getoutput

Also if it's needed I see how easily can add support for separate encodings and 
errors for stdin/out/err
for example with

if (type(encoding) is str):
encoding_stdin = encoding_stdout = encoding_stderr = encoding
elif (type(encoding) is tuple):
encoding_stdin, encoding_stdout, encoding_stderr = encoding
else:
encoding_stdin = encoding_stdout = encoding_stderr = None

then could use 
subprocess.check_output('', encoding='oem')
and
subprocess.check_output('', encoding=('oem', 'ansi', 'ansi'))



Known issues (present in both cases with and without my patch):
* when using cmd.exe and python is outputting to pipe then for some unknown 
reason error happens

with cmd.exe
> python -c "print('\n')" | echo
ECHO is on.
Exception ignored in: <_io.TextIOWrapper name='' mode='w' 
encoding='cp775'>
OSError: [Errno 22] Invalid argument

It doesn't matter which code page for console is set and what is being 
outputted.
It happens for both released 3.5.1 and repo default branch but it doesn't 
happen when PowerShell is used.

I looked into it but didn't found why it happens, only that
n = write(fd, buf, (int)count);
in _Py_write_impl (fileutils.c) returns -1 and errno is EINVAL
I verified that all parameters are correct fd, buf (it isn't NULL) and count 
(parameters are same as when running without pipe)
so I've no idea what causes it.


* Python corrupts characters when reading from stdin

with PowerShell
> Out-String -InputObject "ā" | python -c "import sys; 
> print(sys.stdin.encoding,sys.stdin.read())"
cp1257 ?

It happens for both released 3.5.1 and repo default branch.
With my patch used encoding will be based on console's code page, but it 
doesn't matter because seems it gets corrupted even before it gets used. I 
tested it when using console encodings: oem, ansi and utf-8 and also these same 
with PYTHONIOENCODING too and in all cases it was corrupted, replaced with "?".

I didn't looked further into this.

--
Added file: http://bugs.python.org/file43315/subprocess_fix_encoding_v4.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Donald Stufft]

Donald Stufft added the comment:

> Again, I cite #25003: he implicitly ruled there that it's not permissible for 
> os.urandom() to block on Solaris, so I think he agrees with me.

To be clear, that's an entirely different kind of blocking. That made 
os.urandom on Solaris behave similarly to /dev/random, where it would randomly 
block for all users of that API regularly. I would have made the same ruling, 
this blocking is not that blocking.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Larry Hastings]

Larry Hastings added the comment:

> Larry, I'm sorry but I think you're just flat wrong here and I don't know 
> what else to say about it.

Yeah, I think maybe asking Guido to make a ruling would be the right thing 
here.  Again, I cite #25003: he implicitly ruled there that it's not 
permissible for os.urandom() to block on Solaris, so I think he agrees with me. 
 But if he made a ruling naturally I'd acquiesce to it regardless of what he 
decided.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Larry Hastings]

Larry Hastings added the comment:

> As a process comment: I agree with what Victor wrote in 
> http://haypo-notes.readthedocs.io/pep_random.html#status-of-python-3-5-2, 
> when he suggests that we leave 3.5.2 as is for now [...]

I agree in principle.  Certainly we all agree it'd be inappropriate to add the 
block parameter to 3.5.2.

I only allowed the possibility of adding os.getrandom() in 3.5.2 to try and 
placate the authors of cryptography libraries.  They could use os.getrandom() 
if it was available, and fall back to os.urandom() if not.  Since they want the 
blocking behavior, that means they'd block in 3.5.2+ (they'd call getrandom), 
and they'd also block in 3.5.[0-1] (they'd call os.urandom() which blocks on 
Linux in those versions).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Larry Hastings]

Larry Hastings added the comment:

> I'd also *STRONGLY* request that we avoid adding any new APIs in relation to 
> this that mean "Use os.urandom" is no longer the preferred option to obtain 
> cryptographically strong random numbers in Python.

According to the documentation, os.urandom() does not actually *guarantee* 
cryptographically strong random numbers in Python.  Indeed, it has never 
guaranteed this; you can check the 2.6 documentation, it says the same thing.

There are debates among cryptographers about how strong the numbers are.  The 
paper cited by Mr. Ts'o shows that the numbers can be pretty awful under Linux 
when the entropy pool is empty.

The only way of fixing this this is by using a different API, which can block 
in an unbounded fashion.  It's simply unreasonable to change to this API by 
default, because on Linux /dev/urandom is guaranteed to never block.

I don't see it as an improvement to add a block= parameter to os.urandom(); the 
parameter behaves differently on different platforms (currently ignored on most 
platforms) and if implemented would make the underlying implementation and 
behavior far more difficult to predict and reason about.

Mr. Ts'o had exactly the same problem on Linux.  /dev/urandom was the preferred 
way of obtaining cryptographically strong random numbers.  But it didn't 
produce them in all cases, and in order to produce them he really had to block. 
 Instead of changing /dev/urandom so it would sometimes block, he added a new 
kernel API (getrandom()) which is permitted to block, and he basically left 
/dev/urandom alone.  I see Python as facing the same problem, and I think it 
should solve the problem in the same way: leave os.urandom() alone and add a 
new function getrandom() where possible.

I would support a new function in 3.6, something like random.get_cprng_bytes(n) 
(probably a bad name), which always returns cryptographically strong PRNG bits 
and is permitted to block in an unbounded fashion.  The complication here is 
I'm not sure we can provide it on every platform; the quality of bits available 
from /dev/random on OS X is apparently highly debatable.  (OTOH, what aspect of 
cryptography is not highly, endlessly, debatable? *sigh*)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Nick Coghlan]

Nick Coghlan added the comment:

As a process comment: I agree with what Victor wrote in 
http://haypo-notes.readthedocs.io/pep_random.html#status-of-python-3-5-2, when 
he suggests that we leave 3.5.2 as is for now (i.e. reverted to be consistent 
with 2.7 and 3.4, which favours a risk of subtle security failures in some 
situations over a certainty of Python failing to start in those same 
situations), and only reconsider what to do for 3.5.x and 2.7.x after we have 
agreed on an acceptable solution for 3.6.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Donald Stufft]

Donald Stufft added the comment:

Python should strive do the right thing, and the right thing for something as 
security sensitive as accessing a CSPRNG is to use the interface that most 
strongly promises to in fact, give cryptographically secure random numbers. 
Obsessing over the purity of matching /dev/urandom because the function happens 
to be called os.urandom is only making Python more dangerous to use.

This is exactly the kind of change like making urllib validate HTTPS by 
default, it doesn't matter what you document something as behaving, what 
matters is how people use it and what the expectations are of the average user. 
The nice thing about this change, is the downside is massively smaller than 
that urllib change, because it's basically not going to negatively affect the 
vast vast bulk of people. 

I think this is doubly so since to get the behavior you desire on Linux is 
trivial to do regardless of what os.urandom does, which is using the code 
snippet that Nick pasted.

Larry, I'm sorry but I think you're just flat wrong here and I don't know what 
else to say about it. Since you're the RM for 3.5 and you've made it clear 
you're against the behavior I'm advocating for, I'm going to respect your 
decision on that and I'm not going to pursue getting it into 3.5, however I am 
going to pursue getting it into 3.6.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Larry Hastings]

Larry Hastings added the comment:

> Regardless of the behavior of os.urandom (and 'import random'), is it agreed 
> that the current state of _PyRandom_Init is acceptable for 3.5.2?

I'll get back to you with a specific yes or no.  What I want is that it the 
behavior removed where "import random" can block unboundedly on Linux because 
it's waiting for the entropy pool to fill.  If the code behaves like that, then 
yes, but I'm not giving it my official blessing until I read it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Nick Coghlan]

Nick Coghlan added the comment:

Transferring a comment I previously made on #27250:

One request I'd make in these discussions is that we avoid using the term 
"block" - it makes people think of the /dev/random behaviour (i.e. blocking 
intermittently and unhelpfully), rather than the usually-desired "wait for 
sufficient entropy on system startup" behaviour. (The only documented case 
where that behaviour clearly *isn't* desired is for people writing startup 
scripts on Linux that may run before the system entropy pool is initialised, 
since doing that has been shown to deadlock the system until the systemd script 
watchdog times out)

I'd also request that we keep in mind that any Linux user always remains free 
to write the 3-line utility function:

def read_urandom(num_bytes):
with open('/dev/urandom', 'rb') as urandom:
return urandom.read(num_bytes)

If they want to get precisely the Linux /dev/urandom semantics, and not a 
Python level abstraction that provides the same kinds of assurances offered by 
other *nix platforms and by the Windows crypto APIs. While "os" originated as a 
relatively thin wrapper around POSIX APIs, that's far from universally true 
these days, especially given the introduction of things like implicit Unicode 
handling, implicit EINTR handling, os.scandir, dir_fd handling, and more.

I'd also *STRONGLY* request that we avoid adding any new APIs in relation to 
this that mean "Use os.urandom" is no longer the preferred option to obtain 
cryptographically strong random numbers in Python. Any such APIs can't be used 
in single source Python 2/3 code, they invalidate existing third party 
documentation like https://cryptography.io/en/latest/random-numbers/ and they 
invalidate answers on Q sites like 
http://stackoverflow.com/questions/20936993/how-can-i-create-a-random-number-that-is-cryptographically-secure-in-python

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Larry Hastings]

Larry Hastings added the comment:

> I don't think most people calling os.urandom have any idea how /dev/urandom 
> behaves on their machine nor do I think most people have any idea how 
> /dev/urandom behaves on other people's machines.

Here I invoke the "consenting adults" rule.  In Python development, we never 
say "The user doesn't know what they're doing here, so we need to do the right 
thing for them."  You must treat Python programmers as adults and assume they 
know what they're doing.

If the user is calling os.urandom(), which is documented as behaving like 
/dev/urandom, then it must behave like /dev/urandom.  We can optionally make it 
behave better than /dev/urandom, but not at the cost of unpredictable 
complexity, and not at the cost of degraded performance (unbounded blocking).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27273] subprocess.run(cmd, input='text') should pass universal_newlines=True to Popen

[Akira Li]

New submission from Akira Li:

At the moment, subprocess.run(cmd, input='text') raises TypeError.
It would be nice if universal_newlines=isinstance(input, str) if *input* is set.

I've attached a corresponding patch with the necessary changes to the docs, 
tests and the subprocess.run() code.

--
components: Library (Lib)
files: text_input.diff
keywords: patch
messages: 267936
nosy: akira
priority: normal
severity: normal
status: open
title: subprocess.run(cmd, input='text') should pass universal_newlines=True to 
Popen
type: enhancement
versions: Python 3.6
Added file: http://bugs.python.org/file43314/text_input.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Larry Hastings]

Larry Hastings added the comment:

Wait a minute.  If I read the code correctly, currently os.urandom() is 
implemented strictly using getrandom() *on all platforms*.  And if block=false, 
it... returns an empty buffer?  Am I reading that right?

And what does it do on platforms that don't have a getrandom() call?  Does 
os.urandom() literally always raise an exception on OS X or FreeBSD?

This is so crazy.  Surely I'm misunderstanding the code.  What am I missing?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Donald Stufft]

Donald Stufft added the comment:

> The block= parameter confuses the matter greatly, which is why I oppose it.

I don't like the block= parameter.

> I assert that people are calling os.urandom() because they expect it to 
> behave like /dev/urandom on their machine.

I don't think most people calling os.urandom have any idea how /dev/urandom 
behaves on their machine nor do I think most people have any idea how 
/dev/urandom behaves on other people's machines. In fact, in the original 
thread there were long time developers stating factually wrong information 
about the properties of os.urandom. That's not a mark against them, the 
behavior of random is an esoteric and nuanced thing and there is a lot of 
misinformation out there about the properties and behavior of it on various 
systems.


> If you read the documentation for os.urandom(), it specifically does *not* 
> guarantee the result is suitable for cryptography.

Sure, but the cases where os.urandom doesn't provide results suitable for 
cryptography are:

* Systems where you have a random number generator that is completely broken, 
in which case there's basically no help for you and you're screwed 10 times 
over before you even execute Python.
* Linux, in the window prior to the pool being initialized.

That statement is an obvious hedge against the first case, not an allowance for 
the second case in situations where we don't have to continue to use a worse 
interface for the same underlying functionality.

If chdir on Linux would, in 0.01% of cases, cause a segfault, but 99.9% of 
cases work as you'd expect and a new, chdir2 function was added that still 
changed the directory, but did so in 100% of cases, would you be opposed to 
changing os.chdir to use chdir2, even though it's no longer a thin wrapper 
around chdir or would you demand that all developers switch to os.chdir2? 
Especially if this behavior only really occurred on Linux.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27250] Add os.urandom_block()

[Nick Coghlan]

Nick Coghlan added the comment:

Victor, given the assumption that internal hashing and the random module will 
change to no longer depend on os.urandom, are you happy to close this one in 
favour of #27250?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27250] Add os.urandom_block()

[Nick Coghlan]

Nick Coghlan added the comment:

One request I'd make in these discussions is that we avoid using the term 
"block" - it makes people think of the /dev/random behaviour (i.e. blocking 
intermittently and unhelpfully), rather than the usually-desired "wait for 
sufficient entropy on system startup" behaviour.

I'd also request that we keep in mind that any Linux user always remains free 
to write the 3-line utility function:

def read_urandom(num_bytes):
with open('/dev/urandom', 'rb') as urandom:
return urandom.read(num_bytes)

If they want to get precisely the Linux /dev/urandom semantics, and not a 
Python level abstraction that provides the same kinds of assurances offered by 
other *nix platforms.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Larry Hastings]

Larry Hastings added the comment:

Let me make one more thing clear.  I'm willing for os.urandom() to try to use 
getrandom(GRND_NOBLOCK) (or whatever the flag is called).  This will not block 
unless the entropy pool is low--which almost never happens.  So 99.% of the 
time, os.urandom() should return lovely high-quality, cryptographically-safe 
random numbers.

Really this entire debacle is an argument about this funny edge cases, like 
"you create a new VM and run python3 as PID 0 and your bad sysadmins don't 
manage the entropy pool and the entropy pool never fills".

What should the code do in that situation?  Is it acceptable to use low-quality 
bits from /dev/urandom?  Or do they need cryptographically-strong random 
numbers?  I quote the Zen: "In the face of ambiguity, refuse the temptation to 
guess."  Thus, we shouldn't block on Linux, and we should behave predictably 
like /dev/urandom does on the local machine.

This leads me to one reason why I oppose block=.  It's hiding the complexity of 
the local system with even more complexity, and makes it hard to reason about 
what the code is doing.  The os module should behave in a predictable manner, 
as if it was a thin shell around the local system call.  Given that it's 
impossible to block on Linux and get higher-quality random bits, and it's 
impossible to *not* block on FreeBSD and get lower-quality random bits, adding 
block= to os.urandom() means its behavior becomes inobvious and hard to predict.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Larry Hastings]

Larry Hastings added the comment:

> There are a number of functions in os.py that add additional logic ontop of 
> the system calls, like:

All the functions you name don't have POSIX equivalents, except popen and 
scandir.

popen provides a lot of functionality around popen (which internally does 
fork()/exec()).  But fundamentally it behaves like popen, and its behavior is 
predictable across platforms.

scandir() says right in the documentation: it's implemented using opendir() and 
readdir().  This makes the behavior predictable across platforms.

You might have also cited os.utime(), which hides byzantine complexity 
internally and may be implemented with one of about a half-dozen local calls.  
But is behavior is predictable across platforms.

/dev/urandom behaves very differently on different platforms, to an extent that 
CPython can't reasonably be expected to hide the cross-platform behavior and 
behave in an outwardly predictable manner.  On Linux, by default it should 
never block, and we could theoretically provide additional functionality to let 
it block and guarantee high-quality random bits.  On OS X, we don't have any 
option: it will never block, and it will never provide high-quality random 
bits.  On FreeBSD, we don't have any option: it could locally block, and if the 
user would prefer to not block and get lower-quality random numbers as a 
result, sorry they're out of luck.

It's not reasonable for os.urandom() to hide this complexity.  It is better 
that os.urandom() is a thin layer around /dev/urandom, and behaves predictably 
like /dev/urandom on the local computer.  The block= parameter confuses the 
matter greatly, which is why I oppose it.


> This statement doesn't make any sense to me... you're asserting that because 
> a lot of people are using os.urandom assuming it's going to give them 
> cryptographically secure random numbers... we shouldn't change the 
> implementation of this function to assure that they are going to get 
> cryptographically secure random numbers?

You misunderstand me.  I assert that people are calling os.urandom() because 
they expect it to behave like /dev/urandom on their machine.  I make no 
assertion what they expect to happen as a result.  What is returned by 
os.urandom() is platform-dependent.

If you read the documentation for os.urandom(), it specifically does *not* 
guarantee the result is suitable for cryptography.  The one-line summary seems 
to suggest that it does, but then in the very next paragraph it clarifies the 
situation:

"This function returns random bytes from an OS-specific randomness source. The 
returned data should be unpredictable enough for cryptographic applications, 
though its exact quality depends on the OS implementation."

I *am* asserting that /dev/urandom is guaranteed to not block on Linux, 
therefore os.urandom() should never block on Linux.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27250] Add os.urandom_block()

[Nick Coghlan]

Changes by Nick Coghlan :


--
nosy: +ncoghlan

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Nick Coghlan]

Changes by Nick Coghlan :


--
nosy: +ncoghlan

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Nick Coghlan]

Changes by Nick Coghlan :


--
nosy: +ncoghlan

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27271] asyncio lost udp packets

[Guido van Rossum]

Guido van Rossum added the comment:

You have not convinced anyone that this is asyncio's fault. I'm closing the 
issue. If you need help debugging this please use the python-tulip Google 
Group; if in the end it is found to be an asyncio issue we'll help you file a 
useful bug report on the asyncio GitHub project.

--
resolution:  -> not a bug
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27271] asyncio lost udp packets

[valdemar pavesi]

valdemar pavesi added the comment:

hi,

I did made a change, removing the queue and calling corotine. and now lost udp 
is bigger.

 def datagram_received(self, data, addr):
   asyncio.ensure_future(process_report(data))


@asyncio.coroutine
def process_report(data):


tcmpdump got  38122
and asyncio got just 20711 

2016-06-08 18:19:23,209 [INFO] 546  Total udp from fns: 20711 , queue size: 0
2016-06-08 18:19:23,209 [INFO] 417  Got 20711 json report from traffica server



all udp received will be send by tcp to another server. (Got 20711 json report )

--
Added file: 
http://bugs.python.org/file43313/calling_corotine_wihout_queue_lost_bigger.png

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Donald Stufft]

Donald Stufft added the comment:

> > there is a lot of code out there using os.urandom for it's security 
> > properties
>
> This is exactly why we should not change the behavior of os.urandom().  
> os.urandom() must not block on Linux.  So defaulting to block=True on Linux 
> is a non-starter.

This statement doesn't make any sense to me... you're asserting that because a 
lot of people are using os.urandom assuming it's going to give them 
cryptographically secure random numbers... we shouldn't change the 
implementation of this function to assure that they are going to get 
cryptographically secure random numbers?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Larry Hastings]

Larry Hastings added the comment:

os.getrandom() would be a thin shell around what the local OS provides, and has 
the advantage of behaving in a predictable manner.  If you provide block=, its 
implementation and semantics will vary wildly between platforms.

* On Linux, block=False should be the default.  block=True means it will use 
getrandom(), and block=False means it will use /dev/urandom except possibly 
getrandom().

* On FreeBSD, block=True will be the default, and block=False will either be 
ignored, or return an empty string (or throw an exception, depending on what 
would be a better error-handling mechanism, IDK).

* On OS X, block=False would be the default, and block=True would be ignored, 
because the interface doesn't permit blocking to wait for additional entropy 
and generating higher-quality random bits.

So, yes, the os module should *expose* the behavior of the underlying OS, 
rather than trying to paper over it.

> there is a lot of code out there using os.urandom for it's security properties

This is exactly why we should not change the behavior of os.urandom().  
os.urandom() must not block on Linux.  So defaulting to block=True on Linux is 
a non-starter.

The argument "if we add a block parameter, then users can use it and benefit", 
is equivalent to saying "if we add a new function os.getrandom(), then users 
can use it and benefit".  The user needs to be educated, and we can do that 
with either approach.

However, if users want to write backwards-compatible code, it's a lot easier to 
detect the presence or lack of a new function (hasattr(os, "getrandom")) than 
to detect the presence or lack of a new parameter to an existing function 
(which, if we're lucky, we could do with inspect.signature, and if we're 
unlucky involves a try/except block).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Donald Stufft]

Donald Stufft added the comment:

I also contest the idea that os functions must only be thin shells around 
system provided calls (even though I don't think that using getrandom instead 
of reading from /dev/urandom violates that assertion).

There are a number of functions in os.py that add additional logic ontop of the 
system calls, like:

* os.makedirs
* os.removedirs
* os.renames
* os.walk
* os.popen (which actually imports and uses the entire subprocess module)
* os.scandir

That's just from a quick scan of the pure Python os.py file.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Donald Stufft]

Donald Stufft added the comment:

I think a new function is a bad idea TBH, os.urandom provides 99% of the right 
interface on Linux, and 100% of the right interface on other OSes and there is 
a lot of code out there using os.urandom for it's security properties, and a 
lot of people educating users to use os.urandom for security sensitive tasks.

All of a sudden going, "wait! You actually want Y" is not going to make 
anything more clear, especially since the difference in behavior only matters 
in an edge case.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Donald Stufft]

Donald Stufft added the comment:

It would still be a thin shell around what the OS provides, it'd just be a thin 
wrapper around getrandom() instead of around /dev/urandom specifically-- but 
accessing the same content.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Larry Hastings]

Larry Hastings added the comment:

This is why I'm unwilling to accept this change in 3.5.2.  This sort of "we 
think it's right" behavior is only appropriate in a point release like 3.6.

I think odds are better than even that I'm going to get an os.getrandom(n, 
block=True) call in 3.5.2.  It solves problems, and its behavior is predictable 
on a platform-by-platform basis.  It also won't change the behavior of any 
reasonable existing code.  (Yes, someone could say "if hasattr(os, 
'getrandom'): detonate_the_device()" but that's unreasonable.)

If os.getrandom(n, block=True) goes in 3.5 and 3.6, this makes adding a block 
parameter to os.urandom() less interesting.

Also I disagree with adding the block parameter to os.urandom() in the first 
place.  os.urandom() should behave like the /dev/urandom device on the local 
OS.  Whether or not /dev/urandom blocks is system-dependent behavior. On Linux, 
it's guaranteed to never block; on FreeBSD it is permitted to block under 
certain conditions.  At the point that you say "but on Linux we're implementing 
os.urandom() using the getrandom() system call, which has the ability to block 
if you want", you are straying quite far from the behavior of os.urandom().

Functions in the os module are intended to provide a thin shell around the 
equivalent local OS function, and should behave predictably like that local 
function.  os.stat() should behave like the local stat(); os.utime() should 
behave like the local utime(); and os.urandom() should behave like the local 
/dev/urandom.  It's okay to add functionality for free, but it's not okay to 
degrade behavior (like "it might block forever").

Since the behavior of /dev/urandom already varies so widely between different 
platforms, I think it adds unhelpful complexity and confusion to add the block= 
parameter.  I think we're much better off with a new function providing the 
local system call: os.getrandom(), if available, and perhaps os.getentropy(), 
if available.  Then we preserve the "behaves predictably like the local system 
call" approach of the os module.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27136] sock_connect fails for bluetooth (and probably others)

[A. Jesse Jiryu Davis]

A. Jesse Jiryu Davis added the comment:

The asyncio fix was proposed and reviewed here:

https://github.com/python/asyncio/pull/357

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27250] Add os.urandom_block()

[STINNER Victor]

STINNER Victor added the comment:

IMHO "Should os.urandom() block before the kernel collected enough entropy?" is 
the last major question in the large discussion around random:
https://haypo-notes.readthedocs.io/pep_random.html#os-urandom

It became clear that Python at startup should use a weak entropy if 
high-quality entropy is not available (read would block):
https://haypo-notes.readthedocs.io/pep_random.html#python-startup

So Python startup should no more be impacted if os.urandom() blocks or not. In 
this case, I'm in favor of making os.urandom() the most secure as possible: 
block until the kernel collected enough entropy. I'm in favor in the issue 
#27266 instead of this one.

If it is not possible to agree on a solution, I fear that a PEP will be 
required. I hope that once people understood that the Python startup issue is 
(no more) unrelated to the behaviour of os.urandom() (block or not), most 
people will be in favor of making os.urandom() as secure as possible.

See also https://haypo-notes.readthedocs.io/pep_random.html : my summary of the 
issue #26839.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27250] Add os.urandom_block()

[STINNER Victor]

STINNER Victor added the comment:

> Should this ticket be named "Add os.random()" ? After all, blocking in case 
> of missing entropy is what /dev/random is all about.

In short, /dev/random must not be used :-) See:
https://haypo-notes.readthedocs.io/pep_random.html#rng-denial-of-service

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27249] Add os.urandom_info

[STINNER Victor]

STINNER Victor added the comment:

The current trend in the random discussion (see 
https://haypo-notes.readthedocs.io/pep_random.html and the issue #26839) is 
more to use weak entropy to start Python (hash seed and random constructor) and 
os.urandom() blocks until the kernel collected enough entropy. In this case, 
such information is no more needed. So I close this issue, superseded by the 
issue #27266.

--
resolution:  -> postponed
status: open -> closed
superseder:  -> Always use getrandom() in os.random() on Linux and add 
block=False parameter to os.urandom()

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[STINNER Victor]

STINNER Victor added the comment:

> Second, unless you can guarantee you support blocking / non-blocking behavior 
> on all platforms, this is a bad move.

Well, there are two options:

* Detect when os.urandom() is going to block, and falls back to weaker entropy 
(Linux: /dev/urandom in non-blocking mode, others: getpid+time?).
* Never use os.urandom() to seed random.Random when the randim module is 
imported: I suggest to discuss this option in the issue #27272

I would prefer to collect enough technical information before taking a decision 
(ensure that it's not possible to get a portable function to check if 
/dev/urandom is going to block.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27256] header indentation destroyed

[R. David Murray]

R. David Murray added the comment:

Hmm.  There appear to be at least two bugs here, using the SMTP policy.  I 
thought I had test cases like this, but clearly I don't :(

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27272] random.Random should not read 2500 bytes from urandom

[Donald Stufft]

Donald Stufft added the comment:

One of the key take aways though is that MT doesn't really *need* to be 
initialized with a CSPRNG, all it needs it some moderately random data. So we 
can eliminate the case all together by just not using it. Though a sticking 
point is that it's documented to seed itself from os.urandom if seed(None) is 
called.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[STINNER Victor]

STINNER Victor added the comment:

> The current behavior (as of 9de508dc4837) is that it will never block on 
> Linux, but could still block on other OS if called before /dev/urandom is 
> initialized.

In practice, only Linux is impacted. See the rationale:
https://haypo-notes.readthedocs.io/pep_random.html#scope-of-the-python-blocks-at-startup-issue


> We have not determined a satisfactory solution for other operating systems.

Stp. This issue is specific to Linux. If you want to fix the issue on other 
operating systems, please open a new issue.

Oh, you know what? I already opened such issue :-) The issue #27266 wants to 
fix the issue on all platforms, not only Linux. Open a second issue if you 
prefer.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27239] Make idlelib.macosx self-contained.

[Terry J. Reedy]

Terry J. Reedy added the comment:

I ran into this issue again with the test I am writing for another issue, so I 
pushed it as is, after testing.

Ned, neither of the working OSX buildbots run gui tests.  I am presuming that 
you run the test suite occasionally and will notice if test_idle fails.

Serhiy, almost ditto, except for Zach Ware's Gentoo bot.  What do you think of 
initializing, using a temporary Tk(), during import?  Or best to leave as is?

--
resolution:  -> fixed
stage: patch review -> needs patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[STINNER Victor]

STINNER Victor added the comment:

I opened the issue #27272: "random.Random should not read 2500 bytes from 
urandom".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27272] random.Random should not read 2500 bytes from urandom

[Donald Stufft]

Donald Stufft added the comment:

To be clear, that was a minimal patch using a method that already existing. It 
could be made a lot better by mixing in other sources of entropy like PID#, 
id(self), etc.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27272] random.Random should not read 2500 bytes from urandom

[STINNER Victor]

STINNER Victor added the comment:

Donald Stufft proposed the patch no-urandom-by-default.diff in the issue #26839 
which replaces os.urandom() with int(time.time()*256) in random.Random 
constructor.

I dislike this change because it becomes more likely that two Python processes 
produce the same random number sequence:
https://bugs.python.org/issue26839#msg267819

--
nosy: +dstufft

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27272] random.Random should not read 2500 bytes from urandom

[STINNER Victor]

STINNER Victor added the comment:

PHP uses GENERATE_SEED() to initialize mt_rand():
https://github.com/php/php-src/blob/1c295d4a9ac78fcc2f77d6695987598bb7abcb83/ext/standard/php_rand.h#L50

This macro uses: time() (resolution of 1 second), process identifier, 
php_combined_lcg()

php_combined_lcg() is seeded using: gettimeofday(), and getpid() or thread id 
in ZTS mode, gettimeofday() (called again after getpid()):
https://github.com/php/php-src/blob/1c295d4a9ac78fcc2f77d6695987598bb7abcb83/ext/standard/lcg.c#L55

mt_rand():
https://github.com/php/php-src/blob/1c295d4a9ac78fcc2f77d6695987598bb7abcb83/ext/standard/rand.c#L308

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27239] Make idlelib.macosx self-contained.

[Roundup Robot]

Roundup Robot added the comment:

New changeset cc7f63b6847e by Terry Jan Reedy in branch 'default':
Issue #27239: idlelib.macosx.isXyzTk functions initialize as needed.
https://hg.python.org/cpython/rev/cc7f63b6847e

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27257] get_addresses results in traceback with an addrspec with an empty local part.

[R. David Murray]

R. David Murray added the comment:

Yeah, it never occurred to me that the local part could be empty, so I never 
made a test case for that.  The correct behavior should indeed to be to 
register a defect and set the local part to blank.  I will not be surprised if 
there are other bits of the code (on the output side) that expect local part to 
be non-blank, so there may be some additional test cases and fixes needed.

--
title: get_addresses results in traceback with a valid? header -> get_addresses 
results in traceback with an addrspec with an empty local part.

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27272] random.Random should not read 2500 bytes from urandom

[STINNER Victor]

STINNER Victor added the comment:

Related issue #26839. Don't read this very long issue! You can read my summary 
:-D https://haypo-notes.readthedocs.io/pep_random.html#bug-reports

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27066] SystemError if custom opener returns -1

[Barry A. Warsaw]

Changes by Barry A. Warsaw :


--
resolution:  -> fixed
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27272] random.Random should not read 2500 bytes from urandom

[STINNER Victor]

New submission from STINNER Victor:

Reading more than 256 bytes from os.urandom() is different than reading 256 
bytes or less. For example, see getrandom() and getentropy() manual pages for 
more details.

random.Random doesn't need a very high quality entropy. The glib library only 
reads 128 bits from /dev/urandom to initialize the Mersenne Twister RNG for 
example.

--
messages: 267905
nosy: christian.heimes, haypo
priority: normal
severity: normal
status: open
title: random.Random should not read 2500 bytes from urandom
versions: Python 2.7, Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27066] SystemError if custom opener returns -1

[Roundup Robot]

Roundup Robot added the comment:

New changeset 4af64ab34eef by Barry Warsaw in branch '3.5':
Issue #27066: Fixed SystemError if a custom opener (for open()) returns
https://hg.python.org/cpython/rev/4af64ab34eef

New changeset 84c91d7d4667 by Barry Warsaw in branch 'default':
Issue #27066: Fixed SystemError if a custom opener (for open()) returns a
https://hg.python.org/cpython/rev/84c91d7d4667

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

aixutil.py renamed as _aixutil.py and other changes in response to Martin's 
comments of 4 June

delta of changes to __init__.py, util.py, and test/test_loading.py

--
Added file: http://bugs.python.org/file43312/python.Lib.ctypes.160608.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

aixutil.py renamed as _aixutil.py and other changes in response to Martin's 
comments of 4 June

--
Added file: http://bugs.python.org/file43311/_aixutil.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27270] 'parentheses-equality' warnings when building with clang and ccache

[Raymond Hettinger]

Raymond Hettinger added the comment:

I like this suggestion.

--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[aixtools]

aixtools added the comment:

Answering this again - now that the new patch is ready.

On 04-Jun-16 16:24, Martin Panter wrote:
> Martin Panter added the comment:
>
> Okay here are some more thoughts about your latest patch:
>
> ## Automatic RTLD_MEMBER ##
>
> I was still uneasy about the automatic setting of RTLD_MEMBER. But I looked 
> for how others handle this, and I found Libtool’s LTDL library, and Apache 
> Portable Runtime (APR). Both have a similar (but stricter) automatic addition 
> based on detecting the archive(member) notation:
>
> http://git.savannah.gnu.org/cgit/libtool.git/commit/libltdl/loaders/dlopen.c?id=8fa719e
> https://github.com/apache/apr/commit/c27f8d2
>
> So I guess I can accept this way, if you think it is better than my other 
> ideas:
a) I think it is more in line with what I perceive as normal practice
  libFOO = CDLL(find_library("FOO"))
As CDLL() has always had a 'simple' string as input and not a "tuple".
I have also added some lines to test/test_loading.py to test direct 
calling of CDLL() with fixed strings and a test of os.maxsize
and in util.py - but using e.g., CDLL(find_library("c") as behavior is 
dependent on 32 or 64-bit mode

depending on mode - different output:
  note: find_library("libssl64") is expected to return None - as it 
would be "abnormal" to have an archive libssl64.a or a file libssl64.so

cd Lib/ctypes
../../python util.py
# 32-bit mode:
None
libc.a(shr.o)
libbz2.a(libbz2.so)
find_library("c")returns: libc.a(shr.o)
find_library("libc") returns: libc.a(shr.o)
find_library("libssl")   returns: libssl.a(libssl.so)
find_library("libssl64") returns: None
find_library("ssl")  returns: libssl.a(libssl.so)
find_library("libiconv") returns: libiconv.a(libiconv.so.2)
find_library("intl") returns: libintl.a(libintl.so.8)
libcrypt.a(shr.o)



# 64-bit mode:
None
libc.a(shr_64.o)
libbz2.a(libbz2.so.1)
find_library("c")returns: libc.a(shr_64.o)
find_library("libc") returns: libc.a(shr_64.o)
find_library("libssl")   returns: libssl.a(libssl64.so.1.0.0)
find_library("libssl64") returns: None
find_library("ssl")  returns: libssl.a(libssl64.so.1.0.0)
find_library("libiconv") returns: libiconv.a(shr4_64.o)
find_library("intl") returns: libintl.a(libintl.so.1)
libcrypt.a(shr_64.o)



>
> # Always uses RTLD_MEMBER, never loads a plain file outside an archive
> name = "libcrypto.a(libcrypto.so.1.0.0)"
>
> # Other options, that could be returned by find_library() and would not 
> conflict with a plain file name
> name = ("libcrypto.a", "libcrypto.so.1.0.0")  # (archive, member)
> name = ("libcrypto.a(libcrypto.so.1.0.0)", RTLD_MEMBER)  # (name, extra-flags)
>
> libcrypto = CDLL(name)
>
> ## find_library() modes ##
>
> In your find_library() function, you still have three parts. Can you confirm 
> that each behaviour is intended:
I was being "Q" here, not changing the aixutils.py (now _aixutils.py). 
My intent is to be comparable with other "posix" behaviors.
So, if you believe I am still not compatible with their behavior, 
forgive me - but also shoot me!
>
> A) If I have a file called "crypto" in the current directory, 
> find_library("crypto") returns "crypto". This does not seem right. On Linux, 
> “gcc [. . .] -lcrypto” does not look for a file exactly called “crypto”.
>
> B) You are still stripping bits off the library name if it contains “lib” or 
> a dot (.), so find_library("glib-2.0") is almost equivalent to 
> find_library("b-2"). Isn’t this a bug?
>
> C) find_library("crypto") will return "/usr/lib/crypto" if such a file 
> exists. Just like in A), this does not seem right to me.
All should be seen as bugs, and I hope I coded it correctly to not do 
this anymore.
> ## Other things ##
>
> * You don’t need to prefix most names with underscores, unless they could be 
> confused with a public API. If you follow my earlier suggestion of renaming 
> the new file to _aixutil.py (so it is obvious it is not a public module), 
> then you can freely write “import re, os, sys”, etc.
I had missed, certainly not understood the context, before. aixutil.py 
is now _aixutil.py. Originally I had done this to make the diff in 
util.py much simplier, but also because I incorrectly thought CDLL() was 
frequently called with "foo" or "libfoo". In short, trying to prevent a 
non-existent problem.
__init__.py delta is also much much simpler to grasp.
> * No need to add the internal variable names to the function signatures. Just 
> write find_library(name), and if you need to initialize a variable, do that 
> in the body.
Oops - not removed those yet. That was done to be sure there was no 
global scope interference. If you feel it is vital they be removed - 
will be done.
>
> * I suggest to go over all the regular expressions, and either change them to 
> plain string searching, or make sure special characters and external 
> variables are escaped as necessary. A comment explaining what the RE is 
> trying to do might help too.
Ugh. I actually 

[issue26826] Expose new copy_file_range() syscall in os module.

[Marcos Dione]

Marcos Dione added the comment:

I added a couple of unit tests, which lead me to fix a couple of bugs (yay!).

I discarded the idea of removing any reference to flags.

--
Added file: http://bugs.python.org/file43310/copy_file_range.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Colm Buckley]

Colm Buckley added the comment:

Larry -

Regardless of the behavior of os.urandom (and 'import random'), is it agreed 
that the current state of _PyRandom_Init is acceptable for 3.5.2?

The current behavior (as of 9de508dc4837) is that it will never block on Linux, 
but could still block on other OS if called before /dev/urandom is initialized. 
We have not determined a satisfactory solution for other operating systems. 
Note that no other OS have reported a problem 'in the wild', probably because 
of their extreme rarity in VM/container environments and the lack of Python in 
their early init sequence.

Colm

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[ppperry]

Changes by ppperry :


--
title: Python 3.5 running on Linux kernel 3.17+ can block at startup or on 
importing /arguinthe random module on getrandom() -> Python 3.5 running on 
Linux kernel 3.17+ can block at startup or on importing the random module on 
getrandom()

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing /arguinthe random module on getrandom()

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 08.06.2016 22:49, Larry Hastings wrote:
> 
> Third, because the os module is in general a thin wrapper over what the OS 
> provides, I disapprove of "cryptorandom()" and "pseudorandom()" going into 
> the os module.  There are no functions with these names on any OS of which 
> I'm aware.  This is why I proposed "os.getrandom(n, block=True)".  From its 
> signature, the function it calls on your OS will be obvious, and its 
> semantics on your OS will be documented by your OS.
> 
> Thus I am completely unwilling to add os.cryptorandom() and os.pseudorandom() 
> in 3.5.2.

That was a sketch for 3.6 to resolve the ambiguity between the
different use cases.

You're right, it's better to move such things to the random
module.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27271] asyncio lost udp packets

[Guido van Rossum]

Guido van Rossum added the comment:

How do you know the OS kernel isn't dropping them?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27271] asyncio lost udp packets

[valdemar pavesi]

valdemar pavesi added the comment:

I do understand the possibility to lose on udp.

and I am monitoring all machines involved.
I am not losing on network, I am losing between our network-card-dumpcap and 
read udp from socket. 

is it possible that read will be blocked and not able to read ? 
during read and write to queue?  
I do have udp packets that will come in same microsecond.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27271] asyncio lost udp packets

[Yury Selivanov]

Yury Selivanov added the comment:

Even if you increase buffers you will lose data over UDP.  The reason can be 
the network, or the kernel may simply skip sending the data.

To work with buffer sizes use transport.set_write_buffer_limits

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing /arguinthe random module on getrandom()

[Larry Hastings]

Larry Hastings added the comment:

I am increasingly convinced that I'm right.

--

First, consider that the functions in the os module, as a rule, are a thin 
shell over the equivalent function provided by the operating system.  If Python 
exposes a function called os.XYZ(), and it calls the OS, then with few 
exceptions it does so by calling a function called XYZ().**

This has several ramifications, and these are effectively guarantees for the 
Python programmer:

* You can read your local man pages (or equivalent) to see how the function 
behaves oh your system.  Python occasionally improves on the functionality 
provided; os.utime() provides a lot more functionality than POSIX utime.  But 
it never *degrades* the functionality provided.

* It's implied, and strongly preferred, that the function is atomic: it will 
make exactly one system call.  I once proposed simulating behavior for an os 
module function using a series of system calls, and this approach was rejected 
because it wasn't atomic.  So if you see a function os.XYZ(), you may predict 
that Python will call XYZ() exactly once, and with only a few exceptions you'll 
be right.

Now read this snippet of the documentation for os.urandom():

"The returned data should be unpredictable enough for cryptographic 
applications, though its exact quality depends on the OS implementation. On a 
Unix-like system this will query /dev/urandom, and on Windows it will use 
CryptGenRandom()."

That text has been in the documentation for os.urandom() since at least Python 
2.6.  (That's as old as we have on the web site; I didn't go hunting for older 
documentation.)

Thus the documentation for os.urandom():

* explicitly says it uses /dev/urandom, and

* explicitly *does not* guarantee cryptographic strength random numbers on all 
platforms at all times.

Thus, while it's laudable to try and give the user higher-quality random bits 
when they call os.urandom(), you cannot degrade the behavior of the system's 
/dev/urandom when doing so.  On Linux /dev/urandom is *guaranteed* to never 
block.  This guarantee is so strong, Mr. Ts'o had to add a separate facility to 
Linux (/dev/random) to permit blocking.  os.urandom() *must* replicate this 
behavior.

What I'm proposing is that os.urandom() may use getrandom(RND_NOBLOCK) to 
attempt to get higher-quality random bits, but it *must not block*.  If it 
fails, it will use /dev/urandom, *exactly as it is documented to do*.

(Naturally this flunks the "atomic operation" test.  But in the case of 
procuring random bits, the atomicity of its operation is obviously irrelevant.)


** The exception to this, naturally, is Windows.  Internally the os module is 
called "posixmodule"--and this is no coincidence.  AFAIK every platform 
supported by CPython is POSIX-based except Windows.  The choice was made long 
ago to simulate POSIX behavior on Windows so as to present a consistent API to 
the programmer.  If you're curious about this, and have the time, read the 
implementation of os.stat for Windows.  What a rush!

--

Second, I invoke the "consenting adults" rule.  Python provides well-documented 
behavior for os.urandom().  You cannot make assumptions about the use case of 
the caller and decide for them that they would prefer the function block in an 
unbounded fashion rather than provide low-quality random bits.

And yes, unbounded.  As covered earlier in the thread, it only blocked for 90 
seconds before systemd killed it.  We don't know how long it would actually 
have blocked.  This is completely unacceptable--for startup, for "import 
random", and for "os.urandom()" on Linux.

--

Third, because the os module is in general a thin wrapper over what the OS 
provides, I disapprove of "cryptorandom()" and "pseudorandom()" going into the 
os module.  There are no functions with these names on any OS of which I'm 
aware.  This is why I proposed "os.getrandom(n, block=True)".  From its 
signature, the function it calls on your OS will be obvious, and its semantics 
on your OS will be documented by your OS.

Thus I am completely unwilling to add os.cryptorandom() and os.pseudorandom() 
in 3.5.2.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27271] asyncio lost udp packets

[Yury Selivanov]

Yury Selivanov added the comment:

> I think we could lose inside the network, but we cannot lose inside of our 
> application.

UDP is fundamentally unreliable. It *will* lose data even over UNIX sockets.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27271] asyncio lost udp packets

[valdemar pavesi]

valdemar pavesi added the comment:

I am not getting  any pause, or any message about buffer full.

def pause_reading(self):
print('pause_reading')

def resume_reading(self):
print('resume_reading')


and I cannot find a way to increase the receive buffer by asyncio.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing /arguinthe random module on getrandom()

[Colm Buckley]

Colm Buckley added the comment:

@pitti -

We already discussed this; there are cases where it's not practical to set an 
environment variable. The discussion eventually converged on "it is not 
desirable that Python should block on startup, regardless of system RNG status".

Re: the triggering bug; it was actually 
/lib/systemd/system-generators/systemd-crontab-generator (in systemd-cron) 
which caused the behavior to be noticed in Debian. It wasn't a change in 
systemd behavior, per se (that has been a Python script for some time), it was 
the fact that it was being called before the system PRNG had been initialized. 
With the change from /dev/urandom to getrandom() in 3.5.1, this caused a 
deadlock at boot.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27271] asyncio lost udp packets

[valdemar pavesi]

valdemar pavesi added the comment:

thanks Yury,

I think we could lose inside the network, but we cannot lose inside of our 
application.


regards!
Valdemar

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26955] Implement equivalent to `pip.locations.distutils_scheme` in distutils

[Tim Smith]

Tim Smith added the comment:

As a Homebrew maintainer I'm happy to consider improving Homebrew's 
configuration if someone can point me to an extant package that uses this 
mechanism.

--
nosy: +tdsmith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing /arguinthe random module on getrandom()

[Martin Pitt]

Martin Pitt added the comment:

> you could give some kind of command-line flag

That already exists -- set PYTHONHASHSEED=0.

> But I'll let someone else have the joys of negotiating with Lennart, and I 
> won't blame the Python devs if using GRND_NONBLOCK unconditionally is less 
> painful than having to work with the systemd folks. 

In case it's of any relief: This has nothing to do with having to change 
anything in systemd itself -- none of the services that systemd ships use 
Python. The practical case where this bug appeared was cloud-init (which is 
written in Python), and that wants to run early in the boot sequence even 
before the network is up (so that tools like "pollinate" which gather entropy 
from the cloud host don't kick in yet). So if there's any change needed at all, 
it would be in cloud-init and similar services which run Python during early 
boot.

--
nosy: +pitti
title: Python 3.5 running on Linux kernel 3.17+ can block at startup or on 
importing the random module on getrandom() -> Python 3.5 running on Linux 
kernel 3.17+ can block at startup or on importing /arguinthe random module on 
getrandom()

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27257] get_addresses results in traceback with a valid? header

[Hans-Peter Jansen]

Hans-Peter Jansen added the comment:

Dear Stephen,

thanks for your care. I'm glad, that you're able to reproduce it.

This header is added from the email provider (the biggest here in Germany), so 
yes, it deserves an entry in the defects list, but must not traceback, of 
course. It is not expected to provide a sensible way of interoperability 
otherwise. The unlisted-recipients part is a bit more useful in this respect.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27271] asyncio lost udp packets

[Yury Selivanov]

Yury Selivanov added the comment:

Isn't it normal for UDP to lose some packets?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27271] asyncio lost udp packets

[valdemar pavesi]

New submission from valdemar pavesi:

hello,

I am using asyncio to handle 165 udp-packet per second

everything received by (datagram_received) will be put into a queue.

+++
124605 UDP packets send from client.

and received by server network:

dumpcap  ( filter "port 5  and len > 100" ) 
Packets: 124605

correct number send and received.  124605

++

received by application:

def datagram_received(self, data, addr):

[2016-06-08 14:59:49] total udp = 124255,queue size =0
[2016-06-08 14:59:49] Got 124255 json report from server.

only 124255 received by application.

124605 - 124255 = 350 udp , received by network card , but application never 
got it.

+++


code:
#
class UDPProtocolServerTraffica:

def __init__(self):
self.transport = None
# heart beat timer
self.HEARTBEAT_TIMEOUT = 10.0
self.REPORTSHOWTOTAL_TIMEOUT=60.0
self.MSG_UDP_HEARTBEAT = 
b'\x01\x00\x00\x00\x11\x00\x00\x00\x01\x00\x00\x00\x00\x05\x00\x00\x00'
self.UDPCount = 0

def connection_made(self, transport):
self.transport = transport
# traffica startup message
self.transport.sendto(self.MSG_UDP_HEARTBEAT, (fns_remote_host, 
fns_remote_port))
# start 10 seconds timeout timer
self.h_timeout = 
asyncio.get_event_loop().call_later(self.HEARTBEAT_TIMEOUT, 
self.timeout_heartbeat)
# show total report
self.h_timeout = 
asyncio.get_event_loop().call_later(self.REPORTSHOWTOTAL_TIMEOUT, 
self.timeout_report_showtotal)

 
def datagram_received(self, data, addr):
#fns_mmdu_ipaddr = addr [0]
#fns_mmdu_port = addr [1]
Report_Id = (int.from_bytes(data[0:2], byteorder='big'))
if Report_Id != 327:
self.UDPCount += 1
# send message to queue
asyncio_queue.put_nowait(data)
  
def pause_reading(self):
print('pause_reading')

def resume_reading(self):
print('resume_reading')

def error_received(self, exc):
print('Error received:', exc)

def connection_lost(self, exc):
print('stop', exc)

def timeout_heartbeat(self):
self.transport.sendto(self.MSG_UDP_HEARTBEAT, (fns_remote_host, 
fns_remote_port))
self.h_timeout = 
asyncio.get_event_loop().call_later(self.HEARTBEAT_TIMEOUT, 
self.timeout_heartbeat)
#print('queue size ',asyncio_queue.qsize())
 
def timeout_report_showtotal(self):
self.h_timeout = 
asyncio.get_event_loop().call_later(self.REPORTSHOWTOTAL_TIMEOUT, 
self.timeout_report_showtotal)
self.displayReportTotalCount()
elasticsearch_get_all_reports()

def displayReportTotalCount(self):
logging.info('Total udp from fns: ' + str(self.UDPCount) + ' , queue 
size: ' + str(asyncio_queue.qsize()) )



regards!
Valdemar

--
components: asyncio
messages: 267884
nosy: gvanrossum, haypo, valdemar.pavesi, yselivanov
priority: normal
severity: normal
status: open
title: asyncio lost udp packets
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7063] Memory errors in array.array

[Mark Lawrence]

Changes by Mark Lawrence :


--
nosy:  -BreamoreBoy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Larry Hastings]

Larry Hastings added the comment:

> getentropy() is non-blocking. By the way, os.urandom() is now implemented 
> with getentropy() on OpenBSD.

I quote issue #25003:

> You can not substitute getentropy() for getrandom(), if you need randomness 
> then entropy does not suffice.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[STINNER Victor]

STINNER Victor added the comment:

Larry Hastings added the comment:
>
> First, DO NOT merge this change into 3.5.2 without my explicit permission.
>

I explicitly removed Python 3.5 from this issue to follow your request. See
my previous comment :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27270] 'parentheses-equality' warnings when building with clang and ccache

[Xavier de Gaye]

New submission from Xavier de Gaye:

Building Python with clang and ccache produces about 200 'parentheses-equality' 
warnings with the message:
warning: equality comparison with extraneous parentheses 
[-Wparentheses-equality]

It seems that clang does not recognize that these extraneous parentheses are 
valid [1] after a macro expansion because the preprocessing is done as a 
separate stage by ccache.

[1] For example:
With the macros:
#define Py_TYPE(ob)   (((PyObject*)(ob))->ob_type)
#define PyLong_CheckExact(op) (Py_TYPE(op) == _Type)
The statement:
if (PyLong_CheckExact(v))
is expanded to:
if (PyObject*)(v))->ob_type) == _Type))
and produces the warning with ccache.

--
components: Devguide
files: clang_ccache.patch
keywords: patch
messages: 267881
nosy: eric.araujo, ezio.melotti, ncoghlan, willingc, xdegaye
priority: normal
severity: normal
stage: patch review
status: open
title: 'parentheses-equality' warnings when building with clang and ccache
type: enhancement
Added file: http://bugs.python.org/file43309/clang_ccache.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27129] Wordcode, part 2

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Here is a patch that just refactors the code. It renames OPCODE and OPCODE to 
_Py_OPCODE and _Py_OPCODE and moves them to code.h for reusing in other files. 
Introduces _Py_CODEUNIT as an alias to unsigned short (if it is 16-bit, 
otherwise a compile error is raised). Makes compiler and peepholer to operate 
with _Py_CODEUNIT units instead of bytes. Replaces or scale magic numbers with 
sizeof(_Py_CODEUNIT). Adds fill_nops() for filling specified region with NOPs. 
Decreases memory consumption for peepholer (doesn't allocate memory for unused 
odd addresses).

--
Added file: http://bugs.python.org/file43308/wordcode-refactor.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Theodore Tso]

Theodore Tso added the comment:

Larry, at least on FreeBSD, it sounds like the implementation could just the 
kern.random.sys.seeded sysctl, and return .  (Note: what is the 
proposed behaviour if the PRNG is not seeded?  Return Null?)

As far as OpenBSD is concerned, it's true that it's getentropy(2) never blocks. 
 But that's because OpenBSD assumes that the bootloader can read a seeded 
entropy file from the previous boot session, and that the CD-ROM installer will 
be able to gather enough entropy to save a entropy file from the beginning of 
the installation.So if you don't have worry about booting your OS on an ARC 
Internet of Things device, you can afford to make certain simplifying 
assumptions.

Could Linux on x86 do something similar (read the entropy file in the 
bootloader)?  Sure, the design isn't difficult.  If someone can fund the 
headcount to do the work, I'd be happy to help supervise the GSOC intern or 
work with some engineer at some other company who is interested in getting a 
change like this upstream.  But there will still be cases where getrandom(2) 
could block, simply because I can't control all of the hardware platforms where 
Linux might be ported to.   The real problem is that since on real hardware 
platforms it's only a problem in very early boot, it's hard to make a business 
case to invest in solving this problem.

--
nosy: +Theodore Tso

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Larry Hastings]

Larry Hastings added the comment:

First, DO NOT merge this change into 3.5.2 without my explicit permission.

Second, unless you can guarantee you support blocking / non-blocking behavior 
on all platforms, this is a bad move.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27262] IDLE: move Aqua context menu code to maxosx

[Roundup Robot]

Roundup Robot added the comment:

New changeset 09ec7f7322b5 by Terry Jan Reedy in branch 'default':
Issue #27262: move Aqua unbinding code, which enable context menus, to maxosx.
https://hg.python.org/cpython/rev/09ec7f7322b5

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27262] IDLE: move Aqua context menu code to maxosx

[Terry J. Reedy]

Terry J. Reedy added the comment:

[The light dawns, as the pieces connect together] The pseudoclass 'MyButton' 
would then consist of all widgets with 'MyButton' inserted into its bindtags 
list.  Similarly for 'Editor'. I presume that a) tk has the equivalent of a 
master bind dict mapping bindtags to dicts mapping event sequences strings to 
event handlers, and b) .bind_class(cname, evseq, func) does the equivalent of 
"tagdict.get(cname, {})[evseq] = func".

I also notice that the introspective versions of .bind... call will let us 
discover the pre-defined bindings on difference systems and check our 
alterations.  Anyway, thanks.  I will experiment when ready to refactor 
bindings.  In the meanwhile, I will apply the patch and close this.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27268] Incorrect error message on float('')

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
nosy: +mark.dickinson, serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27260] Missing equality check for super objects

[Raymond Hettinger]

Raymond Hettinger added the comment:

I don't think this should be done.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com