Re: [Python-Dev] Should urlencode() sort the query parameters (if they come from a dict)?
Joao S. O. Bueno writes: > I don't think this behavior is only desirable to unit tests: having > URL's been formed in predictable way a good thing in any way one > thinks about it. Especially if you're a hacker. One more thing you may be able to use against careless sites that don't expect the unexpected to occur in URLs. I'm not saying this is a bad thing, but we should remember that the whole point of PYTHONHASHSEED is that regularities can be exploited for devious and malicious purposes, and reducing regularity makes many attacks more difficult. "*Any* way one thinks about it" is far too strong a claim. Steve ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Should urlencode() sort the query parameters (if they come from a dict)?
On 17 August 2012 18:52, Chris Angelico wrote: > On Sat, Aug 18, 2012 at 5:27 AM, Guido van Rossum wrote: >> I just fixed a unittest for some code used at Google that was >> comparing a url generated by urllib.encode() to a fixed string. The >> problem was caused by turning on PYTHONHASHSEED=1. Because of this, >> the code under test would generate a textually different URL each time >> the test was run, but the intention of the test was just to check that >> all the query parameters were present and equal to the expected >> values. >> >> >> query = sorted(query.items()) > > Hmm. ISTM this is putting priority on the unit test above the > functionality of actual usage. Although on the other hand, sorting > parameters on a URL is nothing compared to the cost of network > traffic, so it's unlikely to be significant. > I don't think this behavior is only desirable to unit tests: having URL's been formed in predictable way a good thing in any way one thinks about it. js -><- > Chris Angelico > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Should urlencode() sort the query parameters (if they come from a dict)?
On Sat, Aug 18, 2012 at 5:27 AM, Guido van Rossum wrote: > I just fixed a unittest for some code used at Google that was > comparing a url generated by urllib.encode() to a fixed string. The > problem was caused by turning on PYTHONHASHSEED=1. Because of this, > the code under test would generate a textually different URL each time > the test was run, but the intention of the test was just to check that > all the query parameters were present and equal to the expected > values. > > > query = sorted(query.items()) Hmm. ISTM this is putting priority on the unit test above the functionality of actual usage. Although on the other hand, sorting parameters on a URL is nothing compared to the cost of network traffic, so it's unlikely to be significant. Chris Angelico ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Should urlencode() sort the query parameters (if they come from a dict)?
Thanks, I filed http://bugs.python.org/issue15719 to track this. On Fri, Aug 17, 2012 at 12:50 PM, "Martin v. Löwis" wrote: > On 17.08.2012 21:27, Guido van Rossum wrote: >> query = sorted(query.items()) >> >> This would not prevent breakage of unit tests, but it would make a >> much simpler fix possible: simply sort the parameters in the URL. >> >> Thoughts? > > Sounds good. For best backwards compatibility, I'd restrict the sorting > to the exact dict type, since people may be using non-dict mappings > which already have a different stable order. > >> for all versions of Python that support PYTHONHASHSEED? > > I think this cannot be done, in particular not for 2.6 and 3.1 - it's > not a security fix (*). > > Strictly speaking, it isn't even a bug fix, since it doesn't restore > the original behavior that some people (like your test case) relied > on. In particular, if somebody has fixed PYTHONHASHSEED to get a stable > order, this change would break such installations. By that policy, it > could only go into 3.4. > > OTOH, if it also checked whether there is randomized hashing, and sort > only in that case, I think it should be backwards compatible in all > interesting cases. > > Regards, > Martin > > (*) I guess some may claim that the current implementation leaks > some bits of the hash seed, since you can learn the seed from > the parameter order, so sorting would make it more secure. However, > I would disagree that this constitutes a feasible threat. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Should urlencode() sort the query parameters (if they come from a dict)?
On 17.08.2012 21:27, Guido van Rossum wrote: > query = sorted(query.items()) > > This would not prevent breakage of unit tests, but it would make a > much simpler fix possible: simply sort the parameters in the URL. > > Thoughts? Sounds good. For best backwards compatibility, I'd restrict the sorting to the exact dict type, since people may be using non-dict mappings which already have a different stable order. > for all versions of Python that support PYTHONHASHSEED? I think this cannot be done, in particular not for 2.6 and 3.1 - it's not a security fix (*). Strictly speaking, it isn't even a bug fix, since it doesn't restore the original behavior that some people (like your test case) relied on. In particular, if somebody has fixed PYTHONHASHSEED to get a stable order, this change would break such installations. By that policy, it could only go into 3.4. OTOH, if it also checked whether there is randomized hashing, and sort only in that case, I think it should be backwards compatible in all interesting cases. Regards, Martin (*) I guess some may claim that the current implementation leaks some bits of the hash seed, since you can learn the seed from the parameter order, so sorting would make it more secure. However, I would disagree that this constitutes a feasible threat. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Should urlencode() sort the query parameters (if they come from a dict)?
Only if it is not an OrderedDict > query = sorted(query.items()) > > This would not prevent breakage of unit tests, but it would make a > much simpler fix possible: simply sort the parameters in the URL. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Should urlencode() sort the query parameters (if they come from a dict)?
I just fixed a unittest for some code used at Google that was comparing a url generated by urllib.encode() to a fixed string. The problem was caused by turning on PYTHONHASHSEED=1. Because of this, the code under test would generate a textually different URL each time the test was run, but the intention of the test was just to check that all the query parameters were present and equal to the expected values. The solution was somewhat painful, I had to parse the url, split the query parameters, and compare them to a known dict. I wonder if it wouldn't make sense to change urlencode() to generate URLs that don't depend on the hash order, for all versions of Python that support PYTHONHASHSEED? It seems a one-line fix: query = query.items() with this: query = sorted(query.items()) This would not prevent breakage of unit tests, but it would make a much simpler fix possible: simply sort the parameters in the URL. Thoughts? -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mountain Lion drops sign of zero, breaks test_cmath...
On Fri, Aug 17, 2012 at 06:50:09AM -0700, Stefan Krah wrote: > R. David Murray wrote: > > > --- a/Lib/test/test_cmath.pyThu Aug 16 22:14:43 2012 +0200 > > > +++ b/Lib/test/test_cmath.pyFri Aug 17 07:54:05 2012 + > > > > Open an issue on the tracker and make mark.dickinson (and maybe skrah) > > nosy. > > I think this issue covers the problem: > > http://bugs.python.org/issue15477 Ah! I'll update that with my notes. (FWIW, I've changed my mind; I think the correct action is to remove the erroneous entries from the test file, rather than patch rAssertAlmostEqual.) Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2012-08-10 - 2012-08-17) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open3640 (+52) closed 23856 (+48) total 27496 (+100) Open issues with patches: 1584 Issues opened (80) == #15623: Init time relative imports no longer work from __init__.so mod http://bugs.python.org/issue15623 opened by scoder #15625: Support u and w codes in memoryview http://bugs.python.org/issue15625 opened by loewis #15626: unittest.main negates -bb option and programmatic warning conf http://bugs.python.org/issue15626 opened by Ben.Darnell #15627: Add a method to importlib.abc.SourceLoader for converting sour http://bugs.python.org/issue15627 opened by brett.cannon #15629: Run doctests in Doc/*.rst as part of regrtest http://bugs.python.org/issue15629 opened by cjerdonek #15631: Python 3.3 beta 1 installation issue lib/lib64 folders http://bugs.python.org/issue15631 opened by ita1024 #15632: regrtest.py: spurious leaks with -R option http://bugs.python.org/issue15632 opened by skrah #15633: httplib.response is not closed after all data has been read http://bugs.python.org/issue15633 opened by Nikratio #15634: synchronized decorator for the threading module http://bugs.python.org/issue15634 opened by jjdominguezm #15636: base64.decodebytes is only available in Python3.1+ http://bugs.python.org/issue15636 opened by lurchman #15637: Segfault reading null VMA (works fine in python 2.x) http://bugs.python.org/issue15637 opened by albertomilone #15639: csv.Error description is incorrectly broad http://bugs.python.org/issue15639 opened by xmorel #15640: Document importlib.abc.Finder as deprecated http://bugs.python.org/issue15640 opened by brett.cannon #15641: Clean up importlib for Python 3.4 http://bugs.python.org/issue15641 opened by brett.cannon #15642: Integrate pickle protocol version 4 GSoC work by Stefan Mihail http://bugs.python.org/issue15642 opened by alexandre.vassalotti #15643: Support OpenCSW in setup.py http://bugs.python.org/issue15643 opened by flub #15645: 2to3 Grammar pickles not created when upgrading to 3.3.0b2 http://bugs.python.org/issue15645 opened by stefanholek #15648: stderr "refs" output does not respect PYTHONIOENCODING http://bugs.python.org/issue15648 opened by cjerdonek #15649: subprocess.Popen.communicate: accept str for input parameter i http://bugs.python.org/issue15649 opened by asvetlov #15650: PEP 3121, 384 refactoring applied to dbm module http://bugs.python.org/issue15650 opened by Robin.Schreiber #15651: PEP 3121, 384 refactoring applied to elementtree module http://bugs.python.org/issue15651 opened by Robin.Schreiber #15652: PEP 3121, 384 refactoring applied to gdbm module http://bugs.python.org/issue15652 opened by Robin.Schreiber #15653: PEP 3121, 384 refactoring applied to hashopenssl module http://bugs.python.org/issue15653 opened by Robin.Schreiber #15654: PEP 384 Refactoring applied to bz2 module http://bugs.python.org/issue15654 opened by Robin.Schreiber #15655: PEP 384 Refactoring applied to json module http://bugs.python.org/issue15655 opened by Robin.Schreiber #15657: Error in Python 3 docs for PyMethodDef http://bugs.python.org/issue15657 opened by sandro.tosi #15660: In str.format there is a misleading error message about alignm http://bugs.python.org/issue15660 opened by py.user #15661: OS X installer packages should be signed for OS X 10.8 Gatekee http://bugs.python.org/issue15661 opened by ned.deily #15662: PEP 3121 refactoring applied to locale module http://bugs.python.org/issue15662 opened by Robin.Schreiber #15663: Investigate providing Tcl/Tk 8.5 with OS X installers http://bugs.python.org/issue15663 opened by ned.deily #15665: PEP 3121, 384 refactoring applied to lsprof module http://bugs.python.org/issue15665 opened by Robin.Schreiber #15666: PEP 3121, 384 refactoring applied to lzma module http://bugs.python.org/issue15666 opened by Robin.Schreiber #15667: PEP 3121, 384 refactoring applied to pickle module http://bugs.python.org/issue15667 opened by Robin.Schreiber #15668: PEP 3121, 384 Refactoring applied to random module http://bugs.python.org/issue15668 opened by Robin.Schreiber #15669: PEP 3121, 384 Refactoring applied to sre module http://bugs.python.org/issue15669 opened by Robin.Schreiber #15670: PEP 3121, 384 Refactoring applied to ssl module http://bugs.python.org/issue15670 opened by Robin.Schreiber #15671: PEP 3121, 384 Refactoring applied to struct module http://bugs.python.org/issue15671 opened by Robin.Schreiber #15672: PEP 3121, 384 Refactoring applied to testbuffer module http://bugs.python.org/issue15672 opened by Robin.Schreiber #15673: PEP 3121, 384 Refactoring applied to testcapi module http://bugs.python.org/issue15673 opened by Robin.Schreiber #15674: PEP 3121, 384 Refactoring applied to thread mod
Re: [Python-Dev] Mountain Lion drops sign of zero, breaks test_cmath...
R. David Murray wrote: > > --- a/Lib/test/test_cmath.pyThu Aug 16 22:14:43 2012 +0200 > > +++ b/Lib/test/test_cmath.pyFri Aug 17 07:54:05 2012 + > > Open an issue on the tracker and make mark.dickinson (and maybe skrah) > nosy. I think this issue covers the problem: http://bugs.python.org/issue15477 Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mountain Lion drops sign of zero, breaks test_cmath...
On Fri, 17 Aug 2012 04:00:50 -0400, Trent Nelson wrote:
> This is the patch I came up with against test_cmath.py:
>
> xenon% hg diff Lib/test/test_cmath.py
> diff -r ce49599b9fdf Lib/test/test_cmath.py
> --- a/Lib/test/test_cmath.pyThu Aug 16 22:14:43 2012 +0200
> +++ b/Lib/test/test_cmath.pyFri Aug 17 07:54:05 2012 +
> @@ -121,8 +121,10 @@
> # if both a and b are zero, check whether they have the same sign
> # (in theory there are examples where it would be legitimate for a
> # and b to have opposite signs; in practice these hardly ever
> -# occur).
> -if not a and not b:
> +# occur) -- the exception to this is if we're on a system that drops
> +# the sign on zeros.
> +drops_zero_sign = sysconfig.get_config_var('LOG1P_DROPS_ZERO_SIGN')
> +if not drops_zero_sign and not a and not b:
> if math.copysign(1., a) != math.copysign(1., b):
> self.fail(msg or 'zero has wrong sign: expected {!r}, '
>'got {!r}'.format(a, b))
>
> With that applied, all the test_cmath tests pass again (without any
> changes to the test file).
>
> Thoughts?
Open an issue on the tracker and make mark.dickinson (and maybe skrah)
nosy.
--David
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Mountain Lion drops sign of zero, breaks test_cmath...
The Mountain Lion build slave I set up earlier this evening fails on
test_cmath:
==
FAIL: test_specific_values (test.test_cmath.CMathTests)
--
Traceback (most recent call last):
File
"/Volumes/bay2/buildslave/cpython/2.7.snakebite-mountainlion-amd64/build/Lib/test/test_cmath.py",
line 352, in test_specific_values
msg=error_message)
File
"/Volumes/bay2/buildslave/cpython/2.7.snakebite-mountainlion-amd64/build/Lib/test/test_cmath.py",
line 94, in rAssertAlmostEqual
'got {!r}'.format(a, b))
AssertionError: atan: atan(complex(0.0, 0.0))
Expected: complex(0.0, 0.0)
Received: complex(0.0, -0.0)
Received value insufficiently close to expected value.
Mountain Lion's atan/log1p appear to drop the negative sign when
passed in -0.0, whereas previous versions of OS X didn't:
Mountain Lion:
% ~/log1p-viper
log1p_drops_zero_sign_test:
atan2(log1p(-0.), -1.) != atan2(-0., -1.)
3.14159 vs -3.14159
atan_drops_zero_sign_test:
atan2(-0., 0.): -0.0
atan2( 0., -0.): 3.14159
atan2(-0., -0.): -3.14159
atan2( 0., 0.): 0.0
log1p(-0.): 0.0
log1p( 0.): 0.0
Lion:
% ./log1p
log1p_drops_zero_sign_test:
atan2(log1p(-0.), -1.) == atan2(-0., -1.)
-3.14159 vs -3.14159
atan_drops_zero_sign_test:
atan2(-0., 0.): -0.0
atan2( 0., -0.): 3.14159
atan2(-0., -0.): -3.14159
atan2( 0., 0.): 0.0
log1p(-0.): -0.0
log1p( 0.): 0.0
(The C code for that is below.)
configure.ac already has a test for this (it makes mention of AIX
having similar behaviour), and the corresponding sysconfig entry
named 'LOG1P_DROPS_ZERO_SIGN' is already being used on a few tests,
i.e.:
# The algorithm used for atan and atanh makes use of the system
# log1p function; If that system function doesn't respect the sign
# of zero, then atan and atanh will also have difficulties with
# the sign of complex zeros.
@requires_IEEE_754
@unittest.skipIf(sysconfig.get_config_var('LOG1P_DROPS_ZERO_SIGN'),
"system log1p() function doesn't preserve the sign")
def testAtanSign(self):
for z in complex_zeros:
self.assertComplexIdentical(cmath.atan(z), z)
@requires_IEEE_754
@unittest.skipIf(sysconfig.get_config_var('LOG1P_DROPS_ZERO_SIGN'),
"system log1p() function doesn't preserve the sign")
def testAtanhSign(self):
for z in complex_zeros:
self.assertComplexIdentical(cmath.atanh(z), z)
Taking a look at cmath_testcases.txt, and we can see this:
-- These are tested in testAtanSign in test_cmath.py
-- atan atan 0.0 0.0 -> 0.0 0.0
-- atan0001 atan 0.0 -0.0 -> 0.0 -0.0
-- atan0002 atan -0.0 0.0 -> -0.0 0.0
-- atan0003 atan -0.0 -0.0 -> -0.0 -0.0
However, a few lines down, those tests crop up again:
-- special values
atan1000 atan -0.0 0.0 -> -0.0 0.0
atan1014 atan 0.0 0.0 -> 0.0 0.0
which is what causes the current test failures. I hacked
test_cmath.py a bit to spit out all the errors it finds after
it's finished parsing the test file (instead of bombing out on
the first one), and it yielded this:
FAIL: test_specific_values (test.test_cmath.CMathTests)
--
Traceback (most recent call last):
File
"/Volumes/bay2/buildslave/cpython/3.2.snakebite-mountainlion-amd64/build/Lib/test/test_cmath.py",
line 446, in test_specific_values
self.fail("\n".join(failures))
AssertionError: atan1000: atan(complex(-0.0, 0.0))
Expected: complex(-0.0, 0.0)
Received: complex(-0.0, -0.0)
Received value insufficiently close to expected value.
atan1014: atan(complex(0.0, 0.0))
Expected: complex(0.0, 0.0)
Received: complex(0.0, -0.0)
Received value insufficiently close to expected value.
atanh0225: atanh(complex(-0.0, 5.6067e-320))
Expected: complex(-0.0, 5.6067e-320)
Received: complex(0.0, 5.6067e-320)
Received value insufficiently close to expected value.
atanh0227: atanh(complex(-0.0, -3.0861101e-316))
Expected: complex(-0.0, -3.0861101e-316)
Received: complex(0.0, -3.0861101e-316)
Received value ins
