Changes by Adam Bielański abg...@gmail.com:
--
nosy: +Adam.Bielański
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18756
___
___
Python-bugs-list
Roundup Robot added the comment:
New changeset acc7439b1406 by Antoine Pitrou in branch 'default':
Issue #18756: os.urandom() now uses a lazily-opened persistent file descriptor,
so as to avoid using many file descriptors when run in parallel from multiple
threads.
Antoine Pitrou added the comment:
Ok, I've committed the patch for the lazy opening approach.
--
resolution: - fixed
stage: patch review - committed/rejected
status: open - closed
___
Python tracker rep...@bugs.python.org
Charles-François Natali added the comment:
New changeset fe949918616c by Antoine Pitrou in branch 'default':
Issue #18756: Improve error reporting in os.urandom() when the failure is due
to something else than /dev/urandom not existing.
http://hg.python.org/cpython/rev/fe949918616c
Charles-François Natali added the comment:
Or more precisely, just run the test in a subprocess. That should fix
the OS X failure if we don't restore the RLIMIT_NOFILE limits, and
will make the test more robust (but you can't reuse the new test,
since it won't work with lazy-opening).
Roundup Robot added the comment:
New changeset b9e62929460e by Antoine Pitrou in branch '3.3':
Issue #18756: make test_urandom_failure more robust by executing its code in a
subprocess
http://hg.python.org/cpython/rev/b9e62929460e
New changeset 68ff013b194c by Antoine Pitrou in branch
Antoine Pitrou added the comment:
Ok, the tiger should feel better now :-)
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18756
___
___
Antoine Pitrou added the comment:
So, to come back to the original topic, is everyone sold on the idea of caching
the urandom fd lazily?
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18756
Donald Stufft added the comment:
Lazily opening urandom and holding it open sounds like a sane thing to do to me
+1
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18756
___
Antoine Pitrou added the comment:
Ok, you're gonna laugh, the simplified patch has a complication (not
theoretical, it would trip test_cmd_line). Attaching patch.
--
Added file: http://bugs.python.org/file31449/persistent_urandom_fd4.patch
___
STINNER Victor added the comment:
Tarek: try to use ssl.RAND_bytes(), it is secure, fast and don't use a file
descriptor.
IMO if something can be improved, it is in the random.SystemRandom() class:
it can keep the FD open. Does the class have a method to generate random
bytes?
--
Donald Stufft added the comment:
haypo: It's been suggested by a number of security professionals that using the
OpenSSL random (or really any random) instead of urandom is likely to be a
smarter idea. The likelyhood that urandom is broken is far less than any other
source of random. This can
New submission from Christian Heimes:
I have seen complains from e.g. Tarek that os.urandom() fails under high load:
https://twitter.com/tarek_ziade/status/362281268215418880
The problem is caused by file descriptor limits. os.urandom() opens
/dev/urandom for every call. How about
STINNER Victor added the comment:
I have seen complains from e.g. Tarek that os.urandom() fails under high
load: https://twitter.com/tarek_ziade/status/362281268215418880
dev_urandom_python() should handle ENFILE and ENOENT differently to raise a
different exception. Or it should always
Antoine Pitrou added the comment:
I don't think that's bug in os.urandom(). If os.urandom() doesn't fail,
something else will fail soon after.
OTOH, the error is clearly misleading. The NotImplementedError should only be
raised for certain errnos (such as ENOENT, ENODEV, ENXIO and EACCES), not
Jesús Cea Avión added the comment:
I agree with Antoine. Exhausting the FDs is not the problem, the problem is the
misleading error.
--
nosy: +jcea
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18756
Tarek Ziadé added the comment:
If os.urandom() doesn't fail, something else will fail soon after.
the random pool can be exhausted, but this is not soon after I think. In
Linux and Mac OS X, ulimit -n defaults to 512 and 256.
It's very easy to reach that limit if you write a web app that
Tarek Ziadé added the comment:
Can tarek tell us more about its usecases: is he directly calling
os.urandom() or does he use the random module? How many threads?
I was using ws4py inside greenlets. ws4py uses os.urandom() to generate some
keys. So one single thread, many greenlets.
Charles-François Natali added the comment:
If os.urandom() doesn't fail, something else will fail soon after.
the random pool can be exhausted, but this is not soon after I think. In
Linux and Mac OS X, ulimit -n defaults to 512 and 256.
I don't think he's referring to the entropy pool, but
Christian Heimes added the comment:
Tarek Ziadé added the comment:
If os.urandom() doesn't fail, something else will fail soon after.
the random pool can be exhausted, but this is not soon after I think. In
Linux and Mac OS X, ulimit -n defaults to 512 and 256.
It's highly unlikely
Tarek Ziadé added the comment:
What does high load mean?
a web app with a few hundreds concurrent requests.
If you mean many concurrent threads, then you should probably go for
the random module, no?
I use greenlets. But, I don't know - are you suggesting os.urandom() should be
marked in
Charles-François Natali added the comment:
2013/8/16, Tarek Ziadé rep...@bugs.python.org:
I use greenlets. But, I don't know - are you suggesting os.urandom() should
be marked in the documentation as DOES NOT SCALE and I should use another
API ? Which one ?
Well, even with greenlets, I
Tarek Ziadé added the comment:
Well, even with greenlets, I assume you're using at least one FD
(socket) per client, no?
So you can get EMFILE on socket() just as on os.urandom().
I do many calls on urandom() so that's the FD bottleneck.
So os.urandom() isn't your biggest problem here.
Of
Christian Heimes added the comment:
Am 16.08.2013 18:24, schrieb Charles-François Natali:
Well, first we'll have to make the code thread-safe, if we want to
keep a persistent FD open. Which means we'll have to add a lock, which
is likely to reduce concurrency, and overall throughput.
Why
Antoine Pitrou added the comment:
Well, even with greenlets, I assume you're using at least one FD
(socket) per client, no?
So you can get EMFILE on socket() just as on os.urandom().
I do many calls on urandom() so that's the FD bottleneck.
Unless you're doing many calls *in parallel*
Charles-François Natali added the comment:
I do many calls on urandom() so that's the FD bottleneck.
So os.urandom() isn't your biggest problem here.
Of course it is. But it looks like you know better without having looked at
the code. :)
So please explain me :-)
os.urandom() can only be
Antoine Pitrou added the comment:
Am 16.08.2013 18:24, schrieb Charles-François Natali:
Well, first we'll have to make the code thread-safe, if we want to
keep a persistent FD open. Which means we'll have to add a lock, which
is likely to reduce concurrency, and overall throughput.
Why
Christian Heimes added the comment:
Am 16.08.2013 18:47, schrieb Charles-François Natali:
I don't think it can be fixed. I think Christian's working on a PEP
for random number generators, which would probably make it easier,
although I din't have a look at it.
In the light of the recent
Tarek Ziadé added the comment:
Unless you're doing many calls *in parallel* it's unlikely to be a
bottleneck.
That's what we're saying since message 1. Antoine, allo quoi! :)
os.urandom() is a convenience function, it doesn't have to be extremely
optimized
I suggest that you tell it the
Charles-François Natali added the comment:
Why locking? /dev/urandom is a pseudo char device. You can have multiple
readers on the same fd without any locking.
You must put a lock around the open() call, though, to avoid calling it
several times and losing an fd.
Exactly (unless the FD is
Charles-François Natali added the comment:
In the light of the recent Android issue with PRNGs [1] I don't think
that Python should roll out its own CPRNG. I'd rather use the operation
system's CPRNG or OpenSSL's CPRNG. After all we aren't crypto experts.
I'd rather point my finger to
Antoine Pitrou added the comment:
Attaching a patch to make error reporting better.
--
keywords: +patch
Added file: http://bugs.python.org/file31317/urandom_error.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18756
Tarek Ziadé added the comment:
So please explain me :-).
it sounded like you did not really want any explanation
os.urandom() can only be called by one thread/greenlet at a time.
do you mean that we cannot have two parallel calls of that function ?
e.g. two opened FD at the same time ?
Donald Stufft added the comment:
Just to be explicit, ``open(/dev/urandom)`` only works on POSIX platforms
while ``os.usrandom`` should work on any supported platform that has an OS
level source of randomness. So advocating *for* simply using ``open()`` is
probably a bad idea unless the
Antoine Pitrou added the comment:
Good point, Donald. os.urandom() is the only (simple) way to access the Windows
randomness pool.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18756
___
Changes by Alex Gaynor alex.gay...@gmail.com:
--
nosy: +alex
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18756
___
___
Python-bugs-list mailing
Charles-François Natali added the comment:
Attaching a patch to make error reporting better.
Why didn't you include ENODEV?
Apparently it can be reported in some corner cases, e.g. in this patch:
http://lfs-matrix.net/patches/downloads/linux/linux-2.6.14.2-pseudo_random-1.patch
Otherwise,
Antoine Pitrou added the comment:
Why didn't you include ENODEV?
Apparently it can be reported in some corner cases, e.g. in this patch:
http://lfs-matrix.net/patches/downloads/linux/linux-2.6.14.2-pseudo_random-1.patch
That isn't mentioned in the POSIX open() spec:
Donald Stufft added the comment:
Looking at random.SystemRandom it appears it would suffer from the same FD
exhaustion problem.
So as of right now afaik none of the sources of cryptographically secure random
in the python stdlib offer a way to open a persistent FD. The primary question
on my
Antoine Pitrou added the comment:
So as of right now afaik none of the sources of cryptographically
secure random in the python stdlib offer a way to open a persistent
FD. The primary question on my mind is if os.urandom can't be modified
to maintain a persistent FD can Python offer a
Antoine Pitrou added the comment:
Attached patch to make os.urandom's fd persistent.
--
type: behavior - resource usage
Added file: http://bugs.python.org/file31318/persistent_urandom_fd.patch
___
Python tracker rep...@bugs.python.org
Antoine Pitrou added the comment:
Updated error handling patch testing for ENODEV.
--
Added file: http://bugs.python.org/file31319/urandom_error2.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18756
Charles-François Natali added the comment:
Updated error handling patch testing for ENODEV.
LGTM, you can apply to 2.7 and 3.x (I just hope all those errnos are
available on every POSIX platform ;-).
--
___
Python tracker rep...@bugs.python.org
Roundup Robot added the comment:
New changeset 193bcc12575d by Antoine Pitrou in branch '3.3':
Issue #18756: Improve error reporting in os.urandom() when the failure is due
to something else than /dev/urandom not existing.
http://hg.python.org/cpython/rev/193bcc12575d
New changeset
Roundup Robot added the comment:
New changeset ec296a36156b by Antoine Pitrou in branch '2.7':
Issue #18756: Improve error reporting in os.urandom() when the failure is due
to something else than /dev/urandom not existing.
http://hg.python.org/cpython/rev/ec296a36156b
--
Antoine Pitrou added the comment:
Ok, committed. We're left with the persistent fd patch for 3.4.
--
stage: needs patch - patch review
versions: -Python 2.7, Python 3.3
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18756
Antoine Pitrou added the comment:
Updated patch for persistent fd.
--
Added file: http://bugs.python.org/file31323/persistent_urandom_fd2.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18756
Antoine Pitrou added the comment:
Updated patch after Christian's comments.
--
Added file: http://bugs.python.org/file31327/persistent_urandom_fd3.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18756
Christian Heimes added the comment:
LGTM
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18756
___
___
Python-bugs-list mailing list
Unsubscribe:
49 matches
Mail list logo
