Re: [Python-Dev] cpython: Implement PEP 380 - 'yield from' (closes #11682)

2012-01-13 Thread Georg Brandl
On 01/14/2012 07:53 AM, Nick Coghlan wrote: >>> +PyAPI_FUNC(int) PyGen_FetchStopIterationValue(PyObject **); >> >> Does this API need to be public? If yes, it needs to be documented. > > Hmm, good point - that one needs a bit of thought, so I've put it on > the tracker: http://bugs.python.org/iss

Re: [Python-Dev] Status of the fix for the hash collision vulnerability

2012-01-13 Thread Nick Coghlan
On Sat, Jan 14, 2012 at 4:24 PM, Jack Diederich wrote: >> This is depending on how the counting is done (I didn't look at MAL's >> patch), and assuming that increasing the hash table size will generally >> reduce collisions if items collide but their hashes are different. > > The patch counts conf

Re: [Python-Dev] cpython: Implement PEP 380 - 'yield from' (closes #11682)

2012-01-13 Thread Nick Coghlan
On Sat, Jan 14, 2012 at 1:17 AM, Georg Brandl wrote: > On 01/13/2012 12:43 PM, nick.coghlan wrote: >> diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst > > There should probably be a "versionadded" somewhere on this page. Good catch, I added versionchanged notes to this p

Re: [Python-Dev] Status of the fix for the hash collision vulnerability

2012-01-13 Thread Jack Diederich
On Thu, Jan 12, 2012 at 9:57 PM, Guido van Rossum wrote: > Hm... I started out as a big fan of the randomized hash, but thinking more > about it, I actually believe that the chances of some legitimate app having >>1000 collisions are way smaller than the chances that somebody's code will > break d

Re: [Python-Dev] Status of the fix for the hash collision vulnerability

2012-01-13 Thread Terry Reedy
On 1/13/2012 8:58 PM, Gregory P. Smith wrote: It is perfectly okay to break existing users who had anything depending on ordering of internal hash tables. Their code was already broken. Given that the doc says "Return the hash value of the object", I do not think we should be so hard-nosed. T

Re: [Python-Dev] [Python-checkins] cpython: add test, which was missing from d64ac9ab4cd0

2012-01-13 Thread Nick Coghlan
On Sat, Jan 14, 2012 at 5:39 AM, benjamin.peterson wrote: > http://hg.python.org/cpython/rev/be85914b611c > changeset:   74363:be85914b611c > parent:      74361:609482c6710e > user:        Benjamin Peterson > date:        Fri Jan 13 14:39:38 2012 -0500 > summary: >  add test, which was missing fr

Re: [Python-Dev] Status of the fix for the hash collision vulnerability

2012-01-13 Thread Guido van Rossum
On Fri, Jan 13, 2012 at 5:58 PM, Gregory P. Smith wrote: > It is perfectly okay to break existing users who had anything depending on > ordering of internal hash tables. Their code was already broken. We > *will*provide a flag and/or environment variable that can be set to turn the > feature off

Re: [Python-Dev] Status of the fix for the hash collision vulnerability

2012-01-13 Thread martin
What an implementation looks like: http://pastebin.com/9ydETTag some stuff to be filled in, but this is all that is really required. I think this statement (and the patch) is wrong. You also need to change the byte string hashing, at least for 2.x. This I consider the biggest flaw in that app

Re: [Python-Dev] Sphinx version for Python 2.x docs

2012-01-13 Thread Éric Araujo
Hi Sandro, Thanks for getting the ball rolling on this. One style for markup, one Sphinx version to code our extensions against and one location for the documenting guidelines will make our work a bit easier. During the build process, there are some warnings that I can understand: I assume yo

Re: [Python-Dev] Status of the fix for the hash collision vulnerability

2012-01-13 Thread Barry Warsaw
On Jan 13, 2012, at 05:38 PM, Guido van Rossum wrote: >On Fri, Jan 13, 2012 at 5:17 PM, Antoine Pitrou wrote: > >> Breaking due to variable hashing is deterministic: you notice it as >> soon as you upgrade (and then you use PYTHONHASHSEED to disable >> variable hashing). That seems better than un

Re: [Python-Dev] Status of the fix for the hash collision vulnerability

2012-01-13 Thread Gregory P. Smith
On Fri, Jan 13, 2012 at 5:58 PM, Gregory P. Smith wrote: > > On Fri, Jan 13, 2012 at 5:38 PM, Guido van Rossum wrote: > >> On Fri, Jan 13, 2012 at 5:17 PM, Antoine Pitrou wrote: >> >>> On Thu, 12 Jan 2012 18:57:42 -0800 >>> Guido van Rossum wrote: >>> > Hm... I started out as a big fan of the ra

Re: [Python-Dev] Status of the fix for the hash collision vulnerability

2012-01-13 Thread Steven D'Aprano
On 14/01/12 12:58, Gregory P. Smith wrote: I do like *randomly seeding the hash*. *+1*. This is easy. It can easily be back ported to any Python version. It is perfectly okay to break existing users who had anything depending on ordering of internal hash tables. Their code was already broken.

Re: [Python-Dev] Status of the fix for the hash collision vulnerability

2012-01-13 Thread Gregory P. Smith
btw, Tim's commit message on this one is amusingly relevant. :) http://hg.python.org/cpython/diff/8d2f2cb9/Objects/dictobject.c On Fri, Jan 13, 2012 at 6:25 PM, Gregory P. Smith wrote: > >> Clearly these ideas are more complex than adding randomization, but >> adding randomization doesn't

Re: [Python-Dev] Status of the fix for the hash collision vulnerability

2012-01-13 Thread Gregory P. Smith
> > > Clearly these ideas are more complex than adding randomization, but adding > randomization doesn't seem to be produce immunity from attack, when data > about the randomness is leaked. > Which will not normally happen. I'm firmly in the camp that believes the random seed can be probed and de

Re: [Python-Dev] Status of the fix for the hash collision vulnerability

2012-01-13 Thread Glenn Linderman
On 1/13/2012 5:35 PM, Victor Stinner wrote: - Glenn Linderman proposes to fix the vulnerability by adding a new "safe" dict type (only accepting string keys). His proof-of-concept (SafeDict.py) uses a secret of 64 random bits and uses it to compute the hash of a key. We could mix Marc's collisio

Re: [Python-Dev] Status of the fix for the hash collision vulnerability

2012-01-13 Thread Gregory P. Smith
On Fri, Jan 13, 2012 at 5:38 PM, Guido van Rossum wrote: > On Fri, Jan 13, 2012 at 5:17 PM, Antoine Pitrou wrote: > >> On Thu, 12 Jan 2012 18:57:42 -0800 >> Guido van Rossum wrote: >> > Hm... I started out as a big fan of the randomized hash, but thinking >> more >> > about it, I actually believ

Re: [Python-Dev] Status of the fix for the hash collision vulnerability

2012-01-13 Thread Guido van Rossum
On Fri, Jan 13, 2012 at 5:17 PM, Antoine Pitrou wrote: > On Thu, 12 Jan 2012 18:57:42 -0800 > Guido van Rossum wrote: > > Hm... I started out as a big fan of the randomized hash, but thinking > more > > about it, I actually believe that the chances of some legitimate app > having > > >1000 colli

Re: [Python-Dev] Status of the fix for the hash collision vulnerability

2012-01-13 Thread Victor Stinner
> - Glenn Linderman proposes to fix the vulnerability by adding a new > "safe" dict type (only accepting string keys). His proof-of-concept > (SafeDict.py) uses a secret of 64 random bits and uses it to compute > the hash of a key. We could mix Marc's collision counter with SafeDict idea (being ab

Re: [Python-Dev] Status of the fix for the hash collision vulnerability

2012-01-13 Thread Antoine Pitrou
On Thu, 12 Jan 2012 18:57:42 -0800 Guido van Rossum wrote: > Hm... I started out as a big fan of the randomized hash, but thinking more > about it, I actually believe that the chances of some legitimate app having > >1000 collisions are way smaller than the chances that somebody's code will > brea

Re: [Python-Dev] Status of the fix for the hash collision vulnerability

2012-01-13 Thread Benjamin Peterson
2012/1/13 Guido van Rossum : > Really? Even though you came up with specifically to prove me wrong? Coming up with a counterexample now invalidates it? -- Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mail

Re: [Python-Dev] Python as a Metro-style App

2012-01-13 Thread Dino Viehland
Dino wrote: > Martin wrote: > > See the start of the thread: I tried to create a "WinRT Component > > DLL", and that failed, as VS would refuse to compile any C file in > > such a project. Not sure whether this is triggered by defining > > WINAPI_FAMILY=2, or any other compiler setting. > > > > I'd

Re: [Python-Dev] PEP 380 ("yield from") is now Final

2012-01-13 Thread Terry Reedy
On 1/13/2012 7:14 AM, Nick Coghlan wrote: print("\n".join(list((lambda:(yield from ("Cheers,", "Nick")))( I pulled, rebuilt, and it indeed works (on Win 7). I just remembered that Tim Peters somewhere (generator.c?) left a large comment with examples of recursive generators, such as knigh

Re: [Python-Dev] Status of the fix for the hash collision vulnerability

2012-01-13 Thread Guido van Rossum
On Fri, Jan 13, 2012 at 10:13 AM, Mark Dickinson wrote: > On Fri, Jan 13, 2012 at 5:43 PM, Guido van Rossum > wrote: > >> How pathological do you consider the set > >> > >> {1 << n for n in range(2000)} > >> > >> to be? What about the set: > >> > >> ieee754_powers_of_two = {2.0**n for n in

Re: [Python-Dev] Status of the fix for the hash collision vulnerability

2012-01-13 Thread Mark Dickinson
On Fri, Jan 13, 2012 at 5:43 PM, Guido van Rossum wrote: >> How pathological do you consider the set >> >>   {1 << n for n in range(2000)} >> >> to be?  What about the set: >> >>   ieee754_powers_of_two = {2.0**n for n in range(-1074, 1024)} >> >> ?  The > 2000 elements of the latter set have only

Re: [Python-Dev] PEP 380 ("yield from") is now Final

2012-01-13 Thread Antoine Pitrou
On Fri, 13 Jan 2012 22:14:43 +1000 Nick Coghlan wrote: > I marked PEP 380 as Final this evening, after pushing the tested and > documented implementation to hg.python.org: > http://hg.python.org/cpython/rev/d64ac9ab4cd0 I don't know if this is supposed to work, but the exception looks wrong: >>>

Re: [Python-Dev] Status of the fix for the hash collision vulnerability

2012-01-13 Thread Guido van Rossum
On Fri, Jan 13, 2012 at 9:08 AM, Mark Dickinson wrote: > On Fri, Jan 13, 2012 at 2:57 AM, Guido van Rossum > wrote: > > How > > pathological the data needs to be before the collision counter triggers? > I'd > > expect *very* pathological. > > How pathological do you consider the set > > {1 <<

Re: [Python-Dev] Status of the fix for the hash collision vulnerability

2012-01-13 Thread Mark Dickinson
On Fri, Jan 13, 2012 at 2:57 AM, Guido van Rossum wrote: > How > pathological the data needs to be before the collision counter triggers? I'd > expect *very* pathological. How pathological do you consider the set {1 << n for n in range(2000)} to be? What about the set: ieee754_powers_of

[Python-Dev] Summary of Python tracker Issues

2012-01-13 Thread Python tracker
ACTIVITY SUMMARY (2012-01-06 - 2012-01-13) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open3210 (+30) closed 22352 (+30) total 25562 (+60) Open issues wit

Re: [Python-Dev] Backwards incompatible sys.stdout.write() behavior in Python 3 (Was: [Python-ideas] Pythonic buffering in Py3 print())

2012-01-13 Thread Xavier Morel
On 2012-01-13, at 17:19 , Antoine Pitrou wrote: > > "-u" forces line-buffering mode for stdout/stderr, which is already the > default if they are wired to an interactive device (isattr() returning > True). Oh, I had not noticed the documentation had changed in Python 3 (in Python 2 it stated that

Re: [Python-Dev] Backwards incompatible sys.stdout.write() behavior in Python 3 (Was: [Python-ideas] Pythonic buffering in Py3 print())

2012-01-13 Thread Antoine Pitrou
On Fri, 13 Jan 2012 17:00:57 +0100 Xavier Morel wrote: > FWIW this is not restricted to Linux (the same behavior change can > be observed in OSX), and the script is overly complex you can expose > the change with 3 lines > > import sys > sys.stdout.write('promt>') > sys.stdin.read(1)

Re: [Python-Dev] Backwards incompatible sys.stdout.write() behavior in Python 3 (Was: [Python-ideas] Pythonic buffering in Py3 print())

2012-01-13 Thread Xavier Morel
On 2012-01-13, at 16:34 , anatoly techtonik wrote: > Posting to python-dev as it is no more relates to the idea of improving > print(). > > > sys.stdout.write() in Python 3 causes backwards incompatible behavior that > breaks recipe for unbuffered character reading from stdin on Linux - > http://

Re: [Python-Dev] PEP 380 ("yield from") is now Final

2012-01-13 Thread Guido van Rossum
AWESOME!!! On Fri, Jan 13, 2012 at 4:14 AM, Nick Coghlan wrote: > I marked PEP 380 as Final this evening, after pushing the tested and > documented implementation to hg.python.org: > http://hg.python.org/cpython/rev/d64ac9ab4cd0 > > As the list of names in the NEWS and What's New entries suggest

Re: [Python-Dev] Backwards incompatible sys.stdout.write() behavior in Python 3 (Was: [Python-ideas] Pythonic buffering in Py3 print())

2012-01-13 Thread Guido van Rossum
I think this may be because in Python 2, there is a coupling between stdin and stderr (in the C stdlib code) that flushes stdout when you read stdin. This doesn't seem to be required by the C std, but most implementations seem to do it. http://stackoverflow.com/questions/2123528/does-reading-from-s

[Python-Dev] Backwards incompatible sys.stdout.write() behavior in Python 3 (Was: [Python-ideas] Pythonic buffering in Py3 print())

2012-01-13 Thread anatoly techtonik
Posting to python-dev as it is no more relates to the idea of improving print(). sys.stdout.write() in Python 3 causes backwards incompatible behavior that breaks recipe for unbuffered character reading from stdin on Linux - http://code.activestate.com/recipes/134892/ At first I though that the

Re: [Python-Dev] cpython: Implement PEP 380 - 'yield from' (closes #11682)

2012-01-13 Thread Georg Brandl
Caution, long review ahead. On 01/13/2012 12:43 PM, nick.coghlan wrote: > http://hg.python.org/cpython/rev/d64ac9ab4cd0 > changeset: 74356:d64ac9ab4cd0 > user:Nick Coghlan > date:Fri Jan 13 21:43:40 2012 +1000 > summary: > Implement PEP 380 - 'yield from' (closes #11682) > dif

Re: [Python-Dev] Status of the fix for the hash collision vulnerability

2012-01-13 Thread And Clover
On 2012-01-13 11:20, Lennart Regebro wrote: The vulnerability is basically only in the dictionary you keep the form data you get from a request. I'd have to disagree with this statement. The vulnerability is anywhere that creates a dictionary (or set) from attacker-provided keys. That would i

Re: [Python-Dev] PEP 380 ("yield from") is now Final

2012-01-13 Thread Matt Joiner
Great work Nick, I've been looking forward to this one. Thanks all for putting the effort in. On Fri, Jan 13, 2012 at 11:14 PM, Nick Coghlan wrote: > I marked PEP 380 as Final this evening, after pushing the tested and > documented implementation to hg.python.org: > http://hg.python.org/cpython/r

Re: [Python-Dev] Status of the fix for the hash collision vulnerability

2012-01-13 Thread Frank Sievertsen
Unfortunately it requires only a few seconds to compute enough 32bit collisions on one core with no precomputed data. Are you running the hash function "backward" to generate strings with the same value, or you are more trying something like brute forcing? If you try it brute force to hit a s

[Python-Dev] PEP 380 ("yield from") is now Final

2012-01-13 Thread Nick Coghlan
I marked PEP 380 as Final this evening, after pushing the tested and documented implementation to hg.python.org: http://hg.python.org/cpython/rev/d64ac9ab4cd0 As the list of names in the NEWS and What's New entries suggests, it was quite a collaborative effort to get this one over the line, and th

Re: [Python-Dev] Status of the fix for the hash collision vulnerability

2012-01-13 Thread Lennart Regebro
On Fri, Jan 13, 2012 at 02:24, Victor Stinner wrote: > - Glenn Linderman proposes to fix the vulnerability by adding a new > "safe" dict type (only accepting string keys). His proof-of-concept > (SafeDict.py) uses a secret of 64 random bits and uses it to compute > the hash of a key. This is my p

Re: [Python-Dev] Status of the fix for the hash collision vulnerability

2012-01-13 Thread Victor Stinner
> Unfortunately it requires only a few seconds to compute enough 32bit > collisions on one core with no precomputed data. Are you running the hash function "backward" to generate strings with the same value, or you are more trying something like brute forcing? And how do you get the hash secret?

Re: [Python-Dev] Status of the fix for the hash collision vulnerability

2012-01-13 Thread Frank Sievertsen
Am 13.01.2012 02:24, schrieb Victor Stinner: My patch doesn't fix the DoS, it just make the attack more complex. The attacker cannot pregenerate data for an attack: (s)he has first to compute the hash secret, and then compute hash collisions using the secret. The hash secret is a least 64 bits lo