[issue13858] readline fails on nonblocking, unbuffered io.FileIO objects
Charles-François Natali neolo...@free.fr added the comment: Can this be handled some other way? Yeah, that's an hairy issue. See #13322 for the details. -- nosy: +neologix resolution: - duplicate stage: - committed/rejected status: open - closed superseder: - buffered read() and write() does not raise BlockingIOError ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13858 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13841] multiprocessing should use sys.exit() where possible
Charles-François Natali neolo...@free.fr added the comment: * atexit callbacks are NOT run (although multiprocessing.util._exit_function IS run), It may be a good thing after a fork() (e.g. you don't want to remove the same file twice), but it most definitely looks wrong for a new interpreter process (à la Windows / fork() + exec()). * the main thread does NOT wait for non-daemonic background threads. Same thing here. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13841 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13322] buffered read() and write() does not raise BlockingIOError
Matt Joiner anacro...@gmail.com added the comment: The patches only fix write? What about read? http://bugs.python.org/issue13858 -- nosy: +anacrolix ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13322 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13852] Doc fixes with patch
Changes by Boštjan Mejak bostjan.me...@gmail.com: -- nosy: +terry.reedy ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13852 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9285] Add a profile decorator to profile and cProfile
Changes by Yuval Greenfield ubershme...@gmail.com: -- nosy: +ubershmekel ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9285 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13857] Add textwrap.indent() as counterpart to textwrap.dedent()
Nick Coghlan ncogh...@gmail.com added the comment:
David Miller pointed out a shorter spelling:
s.replace('\n', '\n' + (4 * ' '))
Still not particularly obvious to the reader (or writer), though.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13857
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13703] Hash collision security issue
Dave Malcolm dmalc...@redhat.com added the comment: I'm attaching a patch which implements a hybrid approach: hybrid-approach-dmalcolm-2012-01-25-001.patch This is a blend of various approaches from the discussion, taking aspects of both hash randomization *and* collision-counting. It incorporates code from amortized-probe-counting-dmalcolm-2012-01-21-003.patch backport-of-hash-randomization-to-2.7-dmalcolm-2012-01-23-001.patch random-8.patch along with ideas from: http://mail.python.org/pipermail/python-dev/2012-January/115812.html The patch is against the default branch (although my primary goal here is eventual backporting). As per haypo's random-8.patch, a randomization seed is read at startup. By default, the existing hash() values are preserved, and no randomization is performed until a dict comes under attack. This preserves existing behaviors (such as dict ordering) under non-attack conditions. For large dictionaries, it reuses the ma_smalltable area to track the amortized cost of all modifications to this dictionary. When the cost exceeds a set threshold, we convert the dictionary's ma_lookup function from lookdict/lookdict_unicode to a paranoid variant. These variants ignore the hash passed in, and instead uses a new function: PyObject_RandomizedHash(obj) to give a second hash value, which is fixed value for a given object within the process, but not predictable to an attacker for the most high-risk types (PyUnicodeObject and PyBytesObject). This patch is intended as a base for backporting, and takes it as given that we can't expand PyTypeObject or hide something in one of the Py*Methods tables; iirc we've run out of tp_flags in 2.*, hence we're forced to implement PyObject_RandomizedHash via direct ob_type comparison, for the most high-risk types. As noted in http://bugs.python.org/issue13703#msg151870: I would NOT worry about hash repeatability for integers and complex data structures. It is not at the core of the common problem (maybe a couple application specific problems but not a general all python web apps severity problem). Doing it for base byte string and unicode string like objects is sufficient. [We can of course implement hash randomization by default in 3.3, but I care more about getting a fix into the released branches ASAP] Upon transition of a dict to paranoid mode, the hash values become unpredictable to an attacker, and all PyDictEntries are rebuilt based on the new hash values. Handling the awkward case within custom ma_lookup functions allows us to move most of the patch from out of the fast path, and lookdict/lookdict_unicode only need minimal changes (stat gathering for the above cost analysis tracking). Once a dict has transitioned to paranoid mode, it isn't using PyObject_Hash anymore, and thus isn't using cached object values, performing a more expensive calculation, but I believe this calculation is essentially constant-time. This preserves hash() and dict order for the cases where you're not under attack, and gracefully handles the attack without having to raise an exception: it doesn't introduce any new exception types. It preserves ABI, assuming no-one else is reusing ma_smalltable. It is suitable for backporting to 3.2, 2.7, and earlier (I'm investigating fixing this going all the way back to Python 2.2) Under the old implementation, there were 4 types of PyDictObject, given these two booleans: * small vs large i.e ma_table == ma_smalltable vs ma_table != ma_smalltable * all keys are str vs arbitary keys i.e ma_lookdict == lookdict_unicode vs lookdict Under this implementation, this doubles to 8 kinds, adding the boolean: * normal hash vs randomized hash (normal vs paranoid). This is expressed via the ma_lookdict callback, adding two new variants, lookdict_unicode_paranoid and lookdict_paranoid Note that if a paranoid dict goes small again (ma_table == ma_smalltable), it stays paranoid. This is for simplicity: it avoids having to rebuild all of the non-randomized me_hash values again (which could fail). Naturally the patch adds selftests. I had to add some diagnostic methods to support them; dict gains _stats() and _make_paranoid() methods, and sys gains a _getrandomizedhash() method. These could be hidden more thoroughly if need be (see DICT_PROTECTION_TRACKING in dictobject.c). Amongst other things, the selftests measure wallclock time taken for various dict operations (and so might introduce failures on a heavily-loaded machine, I guess). Hopefully this approach is a viable way forward. Caveats and TODO items: TODO: I haven't yet tuned the safety threshold. According to http://bugs.python.org/issue13703#msg151850: slot collisions are much more frequent than hash collisions, which only account for less than 0.01% of all collisions. It also shows that slot collisions in the low 1-10 range are most frequent, with very few instances of a dict lookup reaching 20
[issue13857] Add textwrap.indent() as counterpart to textwrap.dedent()
Amaury Forgeot d'Arc amaur...@gmail.com added the comment: If such a function is added, I'd like the option to not indent empty lines: trailing spaces are often not a good idea. -- nosy: +amaury.forgeotdarc ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13857 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13703] Hash collision security issue
Dave Malcolm dmalc...@redhat.com added the comment: I've found a bug in my patch; insertdict writes the old non-randomized hash value into me_hash at: ep-me_hash = hash; rather than using the randomized hash, leading to issues when tested against a real attack. I'm looking into fixing it. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13703 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13703] Hash collision security issue
Alex Gaynor alex.gay...@gmail.com added the comment: On Wed, Jan 25, 2012 at 7:45 AM, Dave Malcolm rep...@bugs.python.orgwrote: Dave Malcolm dmalc...@redhat.com added the comment: I've found a bug in my patch; insertdict writes the old non-randomized hash value into me_hash at: ep-me_hash = hash; rather than using the randomized hash, leading to issues when tested against a real attack. I'm looking into fixing it. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13703 ___ What happens if I have a dict with str keys that goes into paranoid mode, and I then do: class A(object): def __init__(self, s): self.s = s def __eq__(self, other): return self.s == other def __hash__(self): return hash(self.s) d[A(some str that's a key in d)] Is it still able to find the value? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13703 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11457] os.stat(): add new fields to get timestamps as Decimal objects with nanosecond resolution
STINNER Victor victor.stin...@haypocalc.com added the comment:
Attached patch adds an optional format argument to time.time():
time.time(float) is the same than time.time(), but
time.time(decimal) returns a Decimal object. The Decimal object
stores the resolution of the clock and doesn't loose lower bits for
big numbers. I configured the Decimal context to be able to store
10,000 years in seconds with a resolution of 1 nanosecond and
ROUND_HALF_EVEN rounding method.
Example: time.time(decimal) returns Decimal('1327495951.346024').
It is really cool for have directly the resolution in the result,
because the resolution may change at each call: time.time() has 3
different implementations (with fallbacks), each has a different
resolution. time.clock() has also 2 implementations (one is used as a
fallback) with different resolution.
The internal time_to_format() takes integer arguments: the floating
part is written as (floatpart, divisor).
If you like the idea, I will also write a patch for time.clock(),
time.wallclock() and time.clock_gettime(), and also maybe for
time.clock_getres().
We may use a registry to allow to add user defined formats, but I
prefer to keep the patch simple (only allow float and decimal
right now).
--
Added file: http://bugs.python.org/file24321/time_decimal.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11457
___changeset: 74495:6fcdb1d46abb
tag: tip
user:Victor Stinner vstin...@wyplay.com
date:Tue Jan 24 14:27:20 2012 +0100
files: Doc/library/time.rst Modules/timemodule.c
description:
Add a format optional argument to time.time() to return the current time
as a decimal.Decimal object, instead of a float.
diff --git a/Doc/library/time.rst b/Doc/library/time.rst
--- a/Doc/library/time.rst
+++ b/Doc/library/time.rst
@@ -444,9 +444,10 @@ The module defines the following functio
:exc:`TypeError` is raised.
-.. function:: time()
+.. function:: time(format=float)
- Return the time as a floating point number expressed in seconds since the
epoch,
+ Return the time as a floating point number, or a :class:`decimal.Decimal` if
+ format is ``decimal``, expressed in seconds since the epoch,
in UTC. Note that even though the time is always returned as a floating
point
number, not all systems provide time with a better precision than 1 second.
While this function normally returns non-decreasing values, it can return a
diff --git a/Modules/timemodule.c b/Modules/timemodule.c
--- a/Modules/timemodule.c
+++ b/Modules/timemodule.c
@@ -44,16 +44,246 @@
static int floatsleep(double);
static double floattime(void);
+typedef unsigned long floatpart_t;
+
+static PyObject*
+PyLong_FromTimeT(time_t value)
+{
+#ifdef HAVE_LONG_LONG
+return PyLong_FromUnsignedLongLong(value);
+#else
+assert(sizeof(time_t) = sizeof(unsigned long));
+return PyLong_FromUnsignedLong(value);
+#endif
+}
+
+static PyObject*
+PyLong_FromFloatpartT(floatpart_t value)
+{
+#ifdef HAVE_LONG_LONG
+return PyLong_FromUnsignedLongLong(value);
+#else
+assert(sizeof(floatpart_t) = sizeof(unsigned long));
+return PyLong_FromUnsignedLong(value);
+#endif
+}
+
+/* Convert a timestamp to a decimal.Decimal object of the requested
+ resolution. */
+static PyObject*
+time_to_decimal(time_t value,
+floatpart_t floatpart, floatpart_t divisor,
+int resolution)
+{
+static PyObject* module = NULL;
+static PyObject* decimal = NULL;
+static PyObject* context = NULL;
+/* 10^resolution cache, dictionary of int=Decimal */
+static PyObject* exponents = NULL;
+PyObject *t = NULL;
+
+if (!module) {
+module = PyImport_ImportModuleNoBlock(decimal);
+if (module == NULL)
+return NULL;
+}
+
+if (!decimal) {
+decimal = PyObject_GetAttrString(module, Decimal);
+if (decimal == NULL)
+return NULL;
+}
+
+if (context == NULL)
+{
+/* context = decimal.Context(22, rounding=decimal.ROUND_HALF_EVEN) */
+PyObject *context_class, *rounding;
+/* Use 12 decimal digits to store 10,000 years in seconds + 9
+ decimal digits for the floating part in nanoseconds + 1 decimal
+ digit to round correctly */
+context_class = PyObject_GetAttrString(module, Context);
+if (context_class == NULL)
+return NULL;
+rounding = PyObject_GetAttrString(module, ROUND_HALF_EVEN);
+if (rounding == NULL)
+{
+Py_DECREF(context_class);
+return NULL;
+}
+context = PyObject_CallFunction(context_class, iO, 22, rounding);
+Py_DECREF(context_class);
+Py_DECREF(rounding);
+if (context == NULL)
+return NULL;
+}
+
+/* t = decimal.Decimal(value, context) */
+if (value) {
+PyObject *f =
[issue13703] Hash collision security issue
Frank Sievertsen pyt...@sievertsen.de added the comment: Is it still able to find the value? Probably not. :( That's exactly why I stopped thinking about all two-hash-functions or rehashing ideas. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13703 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13857] Add textwrap.indent() as counterpart to textwrap.dedent()
Nick Coghlan ncogh...@gmail.com added the comment: I'd actually suggest that as the default behaviour (and is a good argument in favour of a dedicated function in textwrap - both suggested alternatives will blithely add whitespace to otherwise empty lines). To handle the empty line requires either switching to an re.sub() based solution or adding a conditional expression: '\n'.join(((4 * ' ') + x if x else x) for x in s.splitlines()) I should probably also explicitly address the why not textwrap.fill()? alternative: because fill() does a lot more than simple indenting. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13857 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13857] Add textwrap.indent() as counterpart to textwrap.dedent()
Jon Brandvein jon.brandv...@gmail.com added the comment: If such a function is added, I'd like the option to not indent empty lines: trailing spaces are often not a good idea. From dedent's documentation, it wasn't immediately clear to me that it ignores blank lines when determining common whitespace. (In fact the comment in the example suggests otherwise.) Perhaps a note could be added to the documentation when this change is made? -- nosy: +brandj ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13857 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13051] Infinite recursion in curses.textpad.Textbox
Tycho Andersen ty...@tycho.ws added the comment: Hi, any movement on this? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13051 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13856] xmlrpc / httplib changes to allow for certificate verification
Changes by Senthil Kumaran sent...@uthcode.com: -- nosy: +orsenthil ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13856 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13859] Lingering StandardError in logging module
Changes by Brian Curtin br...@python.org: -- components: -2to3 (2.x to 3.x conversion tool) nosy: +vinay.sajip stage: - patch review ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13859 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13859] Lingering StandardError in logging module
New submission from Matt Joiner anacro...@gmail.com: There's a lingering StandardError referenced in the logging module. StandardError was removed in Python 3, and execution across this code path generates a NameError: File /home/matt/src/cpython/Lib/logging/__init__.py, line 291, in __init__ except StandardError: #pragma: no cover NameError: global name 'StandardError' is not defined Patch attached. -- components: 2to3 (2.x to 3.x conversion tool), Library (Lib) files: logging-uncovered-standarderror.patch keywords: patch messages: 151948 nosy: anacrolix priority: normal severity: normal status: open title: Lingering StandardError in logging module type: behavior versions: Python 3.3 Added file: http://bugs.python.org/file24322/logging-uncovered-standarderror.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13859 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10042] total_ordering
Changes by Catalin Iacob iacobcata...@gmail.com: -- nosy: +catalin.iacob ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10042 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13859] Lingering StandardError in logging module
Matt Joiner anacro...@gmail.com added the comment: Interesting this also occurs in 3.2 and 2.7, but not 2.6 or 3.1. It's probably not an error in 2.x tho. -- versions: +Python 2.7, Python 3.2 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13859 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13853] SystemExit/sys.exit() doesn't print boolean argument
Brett Cannon br...@python.org added the comment: Thanks for going to the trouble to report this, but this is working as intended since bool is a subclass of int. -- nosy: +brett.cannon resolution: - invalid status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13853 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13854] multiprocessing: SystemExit from child with non-int, non-str arg causes TypeError
Jon Brandvein jon.brandv...@gmail.com added the comment: Also, as Brett pointed out to me in #13853, bool is a subclass of int, so they should follow the same code path. I suggest replacing elif type(e.args[0]) is int: exitcode = e.args[0] with something like elif isinstance(e.args[0], int): exitcode = e.args[0] which assumes that a subtype of int is convertible to int. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13854 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13843] Python doesn't compile anymore on our Solaris buildbot: undefined libintl_* symbols
Changes by Jesús Cea Avión j...@jcea.es: -- nosy: +jcea ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13843 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13859] Lingering StandardError in logging module
Vinay Sajip vinay_sa...@yahoo.co.uk added the comment: Interesting this also occurs in 3.2 and 2.7, but not 2.6 or 3.1 Most likely it happened because a fix was backported and this got missed. I'll take care of it. -- assignee: - vinay.sajip ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13859 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13860] PyBuffer_FillInfo() return value
New submission from Stefan Krah stefan-use...@bytereef.org: This came up in #10419, which is already a little crowded. PyBuffer_FillInfo() returns success if the value for 'view' is NULL. I can't really see a reason for that. Any thoughts? -- components: Interpreter Core messages: 151953 nosy: ncoghlan, pitrou, skrah priority: normal severity: normal status: open title: PyBuffer_FillInfo() return value type: behavior versions: Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13860 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)
Changes by Stefan Krah stefan-use...@bytereef.org: Added file: http://bugs.python.org/file24323/8dd9f0ea4876.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10181 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13411] Hashable memoryviews
Stefan Krah stefan-use...@bytereef.org added the comment: Done in: http://hg.python.org/features/pep-3118/rev/508d5e3c579c -- dependencies: -Problems with Py_buffer management in memoryobject.c (and elsewhere?) ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13411 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13411] Hashable memoryviews
Changes by Stefan Krah stefan-use...@bytereef.org: -- resolution: remind - fixed status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13411 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13859] Lingering StandardError in logging module
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset e506848d6381 by Vinay Sajip in branch '3.2': Closes #13859: Replaced reference to StandardError with reference to Exception. Thanks to Matt Joiner for spotting this and submitting a patch. http://hg.python.org/cpython/rev/e506848d6381 New changeset fec45282dc28 by Vinay Sajip in branch 'default': Closes #13859: Merged fix from 3.2 - thanks to Matt Joiner for spotting this and the patch. http://hg.python.org/cpython/rev/fec45282dc28 -- nosy: +python-dev resolution: - fixed stage: patch review - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13859 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12834] PyBuffer_ToContiguous() incorrect for non-contiguous arrays
Stefan Krah stefan-use...@bytereef.org added the comment: I removed the dependency since PyBuffer_ToContiguous() still needs to be fixed in abstract.c while memoryview.tobytes() now works in the PEP-3118 repo. -- dependencies: -Problems with Py_buffer management in memoryobject.c (and elsewhere?) title: memoryview.tobytes() incorrect for non-contiguous arrays - PyBuffer_ToContiguous() incorrect for non-contiguous arrays ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12834 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13703] Hash collision security issue
Antoine Pitrou pit...@free.fr added the comment: I'm attaching a revised version of the patch that should fix the above issue: hybrid-approach-dmalcolm-2012-01-25-002.patch It looks like that approach will break any non-builtin type (in either C or Python) which can compare equal to bytes or str objects. If that's the case, then I think the likelihood of acceptance is close to zero. Also, the level of complication is far higher than in any other of the proposed approaches so far (I mean those with patches), which isn't really a good thing. So I'm rather -1 myself on this approach, and would much prefer to randomize hashes in all conditions. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13703 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13703] Hash collision security issue
Jim Jewett jimjjew...@gmail.com added the comment: On Wed, Jan 25, 2012 at 6:06 AM, Dave Malcolm dmalc...@redhat.com added the comment: hybrid-approach-dmalcolm-2012-01-25-001.patch As per haypo's random-8.patch, a randomization seed is read at startup. Why not wait until it is needed? I suspect a lot of scripts will never need it for any dict, so why add the overhead to startup? Once a dict has transitioned to paranoid mode, it isn't using PyObject_Hash anymore, and thus isn't using cached object values The alternative hashes could be stored in an id-keyed dict performing a more expensive calculation, but I believe this calculation is essentially constant-time. This preserves hash() and dict order for the cases where you're not under attack, and gracefully handles the attack without having to raise an exception: it doesn't introduce any new exception types. It preserves ABI, assuming no-one else is reusing ma_smalltable. It is suitable for backporting to 3.2, 2.7, and earlier (I'm investigating fixing this going all the way back to Python 2.2) Under the old implementation, there were 4 types of PyDictObject, given these two booleans: * small vs large i.e ma_table == ma_smalltable vs ma_table != ma_smalltable * all keys are str vs arbitary keys i.e ma_lookdict == lookdict_unicode vs lookdict Under this implementation, this doubles to 8 kinds, adding the boolean: * normal hash vs randomized hash (normal vs paranoid). This is expressed via the ma_lookdict callback, adding two new variants, lookdict_unicode_paranoid and lookdict_paranoid Note that if a paranoid dict goes small again (ma_table == ma_smalltable), it stays paranoid. This is for simplicity: it avoids having to rebuild all of the non-randomized me_hash values again (which could fail). Naturally the patch adds selftests. I had to add some diagnostic methods to support them; dict gains _stats() and _make_paranoid() methods, and sys gains a _getrandomizedhash() method. These could be hidden more thoroughly if need be (see DICT_PROTECTION_TRACKING in dictobject.c). Amongst other things, the selftests measure wallclock time taken for various dict operations (and so might introduce failures on a heavily-loaded machine, I guess). Hopefully this approach is a viable way forward. Caveats and TODO items: TODO: I haven't yet tuned the safety threshold. According to http://bugs.python.org/issue13703#msg151850: slot collisions are much more frequent than hash collisions, which only account for less than 0.01% of all collisions. It also shows that slot collisions in the low 1-10 range are most frequent, with very few instances of a dict lookup reaching 20 slot collisions (less than 0.0002% of all collisions). This suggests that the threshold of 32 slot/hash collisions per lookup may already be high enough. TODO: in a review of an earlier version of the complexity detection idea, Antoine Pitrou suggested that make the protection scale factor be a run-time configurable value, rather than a #define. This isn't done yet. TODO: run more extensive tests (e.g. Django and Twisted), monitoring the worst-case complexity that's encountered TODO: not yet benchmarked and optimized. I want to get feedback on the approach before I go in and hand-optimize things (e.g. by hand-inlining check_iter_count, and moving the calculations out of the loop etc). I believe any performance issues ought to be fixable, in that the we can get the cost of this for the we're not under attack case to be negligible, and the under attack case should transition from O(N^2) to O(N), albeit it with a larger constant factor. TODO: this doesn't cover sets, but assuming this approach works, the patch can be extended to cover it in an analogous way. TODO: should it cover PyMemoryViewObject, buffer object, etc? TODO: should it cover the hashing in Modules/expat/xmlparse.c? FWIW I rip this code out when doing my downstream builds in RHEL and Fedora, and instead dynamically link against a system copy of expat TODO: only tested on Linux so far (which is all I've got). Fedora 15 x86_64 fwiw Doc/using/cmdline.rst | 6 Include/bytesobject.h | 2 Include/object.h | 8 Include/pythonrun.h | 2 Include/unicodeobject.h | 2 Lib/os.py | 17 -- Lib/test/regrtest.py | 5 Lib/test/test_dict.py | 298 + Lib/test/test_hash.py | 53 ++ Lib/test/test_os.py | 35 +++- Makefile.pre.in | 1 Modules/posixmodule.c | 126 ++- Objects/bytesobject.c | 7 Objects/dictobject.c | 369 +- Objects/object.c | 37 Objects/unicodeobject.c | 51 ++ PCbuild/pythoncore.vcproj | 4 Python/pythonrun.c | 3 Python/sysmodule.c | 16 +
[issue13703] Hash collision security issue
Jim Jewett jimjjew...@gmail.com added the comment: Sorry; hit the wrong key... intended message below: On Wed, Jan 25, 2012 at 6:06 AM, Dave Malcolm dmalc...@redhat.com added the comment: [lots of good stuff] hybrid-approach-dmalcolm-2012-01-25-001.patch As per haypo's random-8.patch, a randomization seed is read at startup. Why not wait until it is needed? I suspect a lot of scripts will never need it for any dict, so why add the overhead to startup? Once a dict has transitioned to paranoid mode, it isn't using PyObject_Hash anymore, and thus isn't using cached object values The alternative hashes could be stored in an id-keyed WeakKeyDictionary; that would handle at least the normal case of using exactly the same string for the lookup. Note that if a paranoid dict goes small again (ma_table == ma_smalltable), it stays paranoid. As I read it, that couldn't happen, because paranoid dicts couldn't shrink at all. (Not letting them shrink beneath 2*PyDict_MINSIZE does seem like a reasonable solution.) Additional TODOs... The checks for Unicode and Dict should not be exact; it is OK to do on a subclass so long as they are using the same lookdict (and, for unicode, the same eq). Additional small strings should be excluded from the new hash, to avoid giving away the secret. At a minimum, single-char strings should be excluded, and I would prefer to exclude all strings of length = N (where N defaults to 4). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13703 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13861] test_pydoc failure
New submission from Stefan Krah stefan-use...@bytereef.org: test_pydoc fails on Ubuntu Lucid: == FAIL: test_apropos_with_bad_package (test.test_pydoc.PydocImportTest) -- Traceback (most recent call last): File /home/stefan/pydev/cpython/Lib/test/test_pydoc.py, line 409, in test_apropos_with_bad_package self.assertEqual(b'', result) AssertionError: b'' != b'Crypto.Protocol.AllOrNothing - This file implements all-or-nothing package transformations.\nCrypto.SelfTest.Protocol.test_AllOrNothing' == FAIL: test_apropos_with_unreadable_dir (test.test_pydoc.PydocImportTest) -- Traceback (most recent call last): File /home/stefan/pydev/cpython/Lib/test/test_pydoc.py, line 419, in test_apropos_with_unreadable_dir self.assertEqual(b'', result) AssertionError: b'' != b'Crypto.Protocol.AllOrNothing - This file implements all-or-nothing package transformations.\nCrypto.SelfTest.Protocol.test_AllOrNothing' -- -- components: Tests messages: 151962 nosy: skrah priority: normal severity: normal stage: needs patch status: open title: test_pydoc failure type: behavior versions: Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13861 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13862] test_zlib failure
New submission from Stefan Krah stefan-use...@bytereef.org: test_zlib currently fails on Ubuntu Lucid: == FAIL: test_library_version (test.test_zlib.VersionTestCase) -- Traceback (most recent call last): File /home/stefan/pydev/cpython/Lib/test/test_zlib.py, line 24, in test_library_version self.assertEqual(zlib.ZLIB_RUNTIME_VERSION, zlib.ZLIB_VERSION) AssertionError: '1.2.3.3' != '1.2.5' - 1.2.3.3 + 1.2.5 -- components: Tests messages: 151963 nosy: nadeem.vawda, skrah priority: normal severity: normal stage: needs patch status: open title: test_zlib failure type: behavior versions: Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13862 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13862] test_zlib failure
Nadeem Vawda nadeem.va...@gmail.com added the comment: Do you have a self-compiled version of zlib (1.2.5) installed on this system? It looks like this is due to a (benign) version mismatch between zlib.h and the actual shared lib. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13862 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13703] Hash collision security issue
Jim Jewett jimjjew...@gmail.com added the comment: On Wed, Jan 25, 2012 at 1:05 PM, Antoine Pitrou pit...@free.fr added the comment: It looks like that approach will break any non-builtin type (in either C or Python) which can compare equal to bytes or str objects. If that's the case, then I think the likelihood of acceptance is close to zero. (1) Isn't that true of *any* patch that changes hashing? (Thus the PYTHONHASHSEED=0 escape hatch.) (2) I think it would still work for the lookdict_string (or lookdict_unicode) case ... which is the normal case, and also where most vulnerabilities should appear. (3) If the alternate hash is needed for non-string keys, there is no perfect resolution, but I suppose you could get closer with if obj == str(obj): newhash=hash(str(obj)) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13703 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13703] Hash collision security issue
Antoine Pitrou pit...@free.fr added the comment: Jim Jewett jimjjew...@gmail.com added the comment: On Wed, Jan 25, 2012 at 1:05 PM, Antoine Pitrou pit...@free.fr added the comment: It looks like that approach will break any non-builtin type (in either C or Python) which can compare equal to bytes or str objects. If that's the case, then I think the likelihood of acceptance is close to zero. (1) Isn't that true of *any* patch that changes hashing? (Thus the PYTHONHASHSEED=0 escape hatch.) If a third-party type wants to compare equal to bytes or str objects, its __hash__ method will usually end up calling hash() on the equivalent bytes/str object. That's especially true for Python types (I don't think anyone wants to re-implement a slow str-alike hash in pure Python). (2) I think it would still work for the lookdict_string (or lookdict_unicode) case ... which is the normal case, and also where most vulnerabilities should appear. It would probably still work indeed. (3) If the alternate hash is needed for non-string keys, there is no perfect resolution, but I suppose you could get closer with if obj == str(obj): newhash=hash(str(obj)) That may be slowing down things quite a bit. It looks correct though. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13703 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13703] Hash collision security issue
Dave Malcolm dmalc...@redhat.com added the comment: On Wed, 2012-01-25 at 18:05 +, Antoine Pitrou wrote: Antoine Pitrou pit...@free.fr added the comment: I'm attaching a revised version of the patch that should fix the above issue: hybrid-approach-dmalcolm-2012-01-25-002.patch It looks like that approach will break any non-builtin type (in either C or Python) which can compare equal to bytes or str objects. If that's the case, then I think the likelihood of acceptance is close to zero. How? Also, the level of complication is far higher than in any other of the proposed approaches so far (I mean those with patches), which isn't really a good thing. So would I. I want something I can backport, though. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13703 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6056] socket.setdefaulttimeout affecting multiprocessing Manager
Derek Wilson jderekwil...@gmail.com added the comment: Any chance this patch will be accepted (or at least reviewed) soon? -- type: behavior - crash versions: +Python 3.3, Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6056 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13863] Using py_compile does not prevent recompilation due to 'bad mtime'.
New submission from Mark Dickinson dicki...@gmail.com:
(Marking this as 'Interpreter Core' because the issue really seems to come from
Python/import.c rather than from the py_compile module.)
On a Windows 7 VM (64-bit, NTFS) running Python 2.7 (also reproduced on a
non-VM Windows 7 installation), I'm seeing the following surprising behaviour:
C:\Python27\Libdir decimal.py*
Volume in drive C is Local
Volume Serial Number is 5083-D43D
Directory of C:\Python27\Lib
06/10/2011 08:46 PM 219,383 decimal.py
01/25/2012 07:05 PM 165,322 decimal.pyc
01/25/2012 04:54 PM 169,386 decimal.pyo
3 File(s)554,091 bytes
0 Dir(s) 966,266,880 bytes free
C:\Python27\Libpython
Enthought Python Distribution -- www.enthought.com
Version: 7.2-2 (64-bit)
Python 2.7.2 |EPD 7.2-2 (64-bit)| (default, Sep 14 2011, 11:25:00) [MSC v.1500
64 bit (AMD64)] on win32
Type packages, demo or enthought for more information.
import py_compile; py_compile.compile('decimal.py')
import os; os.stat('decimal.pyc').st_mtime
1327518430.306176
import decimal
os.stat('decimal.pyc').st_mtime
1327518443.9094632
Notice that in spite of calling py_compile.compile on decimal.py, the .pyc file
is still regenerated on import.
Relevant details: note that the decimal.py timestamp is in the summer, and that
it's currently winter time. Also, my Windows 7 timezone setting is UTC, with
'automatically adjust for DST' set to true.
This bit me today when working with a 'post-install' script for an MSI
installer for a large Python application, where the post-install script did a
compileall.compile_path to make sure that all the .pyc files were up to date,
and thereby avoid a *long* ( 90 second) application first startup time. It
turned out that on application startup some of the .pyc files got regenerated
while others didn't, and it took me far too long to notice that it was the .py
files with summer timestamps that lead to .pyc regeneration, and that those
with winter timestamps were okay.
When I set my timezone to plain UTC with no DST adjustments, the above issue
disappears. Also, when I reset my timezone to that of Saudi Arabia (no DST),
the issue also disappears.
The cause of the issue seems to be that:
(1) import.c uses an 'fstat' call to get mtime for a .py file.
(2) on my Windows installation, the result of fstat on a file varies (a) with
time of year, and (b) with computer DST and timezone settings. (No such
problems on OS X.)
(3) in contrast, py_compile uses os.stat(...).st_mtime, which appears to be
invariant under time changes.
The difference between the 'fstat' result and the os.stat result can be seen
directly:
Python 2.7.2 |EPD 7.2-2 (64-bit)| (default, Sep 14 2011, 11:25:00) [MSC v.1500
64 bit (AMD64)] on win32
Type packages, demo or enthought for more information.
import decimal # make sure .pyc file is up to date
import os; os.stat('decimal.py').st_mtime # actual mtime of .py file
1307738784.0
import struct; struct.unpack('LL', open('decimal.pyc', 'rb').read(8))[1] #
stored mtime from .pyc; uses fstat
1307735184
This presumably also means that in this part of the world, .pyc files will be
regenerated on import after any DST change, since the reported 'fstat' result
used by import.c will no longer match the stored mtime timestamp in the .pyc
file.
--
components: Interpreter Core
messages: 151969
nosy: mark.dickinson
priority: normal
severity: normal
status: open
title: Using py_compile does not prevent recompilation due to 'bad mtime'.
type: behavior
versions: Python 2.7
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13863
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13703] Hash collision security issue
Antoine Pitrou pit...@free.fr added the comment: Le mercredi 25 janvier 2012 à 19:19 +, Dave Malcolm a écrit : Dave Malcolm dmalc...@redhat.com added the comment: On Wed, 2012-01-25 at 18:05 +, Antoine Pitrou wrote: Antoine Pitrou pit...@free.fr added the comment: I'm attaching a revised version of the patch that should fix the above issue: hybrid-approach-dmalcolm-2012-01-25-002.patch It looks like that approach will break any non-builtin type (in either C or Python) which can compare equal to bytes or str objects. If that's the case, then I think the likelihood of acceptance is close to zero. How? This kind of type, for example: class C: def __hash__(self): return hash(self._real_str) def __eq__(self, other): if isinstance(other, C): other = other._real_str return self._real_str == other If I'm not mistaken, looking up C(abc) will stop matching abc when there are too many collisions in one of your dicts. Also, the level of complication is far higher than in any other of the proposed approaches so far (I mean those with patches), which isn't really a good thing. So would I. I want something I can backport, though. Well, your approach sounds like it subtly and unpredictably changes the behaviour of dicts when there are too many collisions, so I'm not sure it's a good idea to backport it, either. If we don't want to backport full hash randomization, I think I much prefer raising a BaseException when there are too many collisions, rather than this kind of (excessively) sophisticated workaround. You *are* changing a fundamental datatype in a rather complicated way. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13703 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13864] Python 2.7.2 refuses to open
New submission from stephen Andel elden.an...@gmail.com: I recently changed one of the keybindings in python to just control. Python did not like this and, when I tried to fix this by swapping back to the default settings it closed itself and now will not open. Th program will attempt to open then stop, and the process with cancel. I have reinstalled and deleted all apparent files associated with python but this was not enough and continues to fail, badly. Sorry for the informal dialogue I'm just a bit peeved right now. -- components: IDLE messages: 151971 nosy: stephen.Andel priority: normal severity: normal status: open title: Python 2.7.2 refuses to open versions: Python 2.7 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13864 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13863] Using py_compile does not prevent recompilation due to 'bad mtime'.
Mark Dickinson dicki...@gmail.com added the comment: Issue reproduced on stock Python 2.7 and Python 3.2 from www.python.org. I don't have a working 3.3 install on Windows, but looking at the source, it seems likely that it's an issue there, too. -- versions: +Python 3.2, Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13863 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13703] Hash collision security issue
Dave Malcolm dmalc...@redhat.com added the comment: I think you're right: it will stop matching it during lookup within such a dict, since the dict will be using the secondary hash for abc, but hash() for the C instance. It will still match outside of the dict, and within other dicts. So yes, this would be a subtle semantic change when under attack. Bother. Having said that, note that within the typical attack scenarios (HTTP headers, HTTP POST, XML-RPC, JSON), we have a pure-str dict (or sometimes a pure-bytes dict). Potentially I could come up with a patch that only performs this change for such a case (pure-str is easier, given that we already track this), which would avoid the semantic change you identify, whilst still providing protection against a wide range of attacks. Is it worth me working on this? Also, the level of complication is far higher than in any other of the proposed approaches so far (I mean those with patches), which isn't really a good thing. So would I. I want something I can backport, though. Well, your approach sounds like it subtly and unpredictably changes the behaviour of dicts when there are too many collisions, so I'm not sure it's a good idea to backport it, either. If we don't want to backport full hash randomization, I think I much prefer raising a BaseException when there are too many collisions, rather than this kind of (excessively) sophisticated workaround. You *are* changing a fundamental datatype in a rather complicated way. Well, each approach has pros and cons, and we've circled around between hash randomization vs collision counting vs other approaches for several weeks. I've supplied patches for 3 different approaches. Is this discussion likely to reach a conclusion soon? Would it be regarded as rude if I unilaterally ship something close to: backport-of-hash-randomization-to-2.7-dmalcolm-2012-01-23-001.patch in RHEL/Fedora, so that my users have some protection they can enable if they get attacked? (see http://bugs.python.org/msg151847). If I do this, I can post the patches here in case other distributors want to apply them. As for python.org, who is empowered to make a decision here? How can we move this forward? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13703 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13865] distutils documentation says Extension has optional argument
New submission from Miki Tebeka miki.teb...@gmail.com: The Extension documentation says: 2.3.5. Other options There are still some other options which can be used to handle special cases. The optional option is a boolean; if it is true, a build failure in the extension will not abort the build process, but instead simply not install the failing extension. - However there not such option, and distutils will complain: /usr/lib/python2.7/distutils/extension.py:133: UserWarning: Unknown Extension options: 'optional' -- messages: 151974 nosy: tebeka priority: normal severity: normal status: open title: distutils documentation says Extension has optional argument ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13865 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13865] distutils documentation says Extension has optional argument
Changes by Miki Tebeka miki.teb...@gmail.com: -- assignee: - docs@python components: +Documentation nosy: +docs@python versions: +Python 2.7 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13865 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13862] test_zlib failure
Nadeem Vawda nadeem.va...@gmail.com added the comment: Either way, the failure should be fixed now. -- resolution: - fixed stage: needs patch - committed/rejected status: open - pending ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13862 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13862] test_zlib failure
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset a5f8611ce81d by Nadeem Vawda in branch 'default': Issue #13862: Relax zlib version test to avoid spurious failures. http://hg.python.org/cpython/rev/a5f8611ce81d -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13862 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13703] Hash collision security issue
Frank Sievertsen pyt...@sievertsen.de added the comment:
For the sake of completeness:
Collision-counting (with Exception) has interesting effects, too.
d={((1(65+i))-2**(i+4)): 9 for i in range(1001)}
for i in list(d):
... del d[i]
d
{}
9 in d
False
0 in d
Traceback (most recent call last):
File stdin, line 1, in module
KeyError: 'too many slot collisions'
d[9] = 1
d
{9: 1}
d == {0: 1}
False
{0: 1} == d
Traceback (most recent call last):
File stdin, line 1, in module
KeyError: 'too many slot collisions'
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13703
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13852] Doc fixes with patch
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset e2b98a332070 by Georg Brandl in branch '2.7': #13852: some small doc fixes. http://hg.python.org/cpython/rev/e2b98a332070 New changeset 5b8800012e88 by Georg Brandl in branch '3.2': #13852: some small doc fixes. http://hg.python.org/cpython/rev/5b8800012e88 -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13852 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13852] Doc fixes with patch
Georg Brandl ge...@python.org added the comment: Done. Thanks! -- nosy: +georg.brandl resolution: - fixed status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13852 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13866] {urllib, urllib.parse}.urlencode should not use quote_plus
New submission from Stephen Day stevv...@gmail.com:
The current behavior of the urlencode function (2.7: urllib, 3.x: urllib.parse)
encodes spaces as pluses:
from urllib import urlencode
urlencode({'a': 'some param'})
'a=some+param'
However, in most instances, it would be desirable to merely encode spaces using
percent encoding:
urlencode({'a': 'some param'})
'a=some%20param'
But there is no way to get this behavior in the standard library.
It would probably best to change this so it defaults to use the regular quote
function, but allows callers who need the legacy quote_plus behavior to pass
that in as a function parameter.
An acceptable fix would be to have the quote function taken as a keyword
parameter, so legacy behavior remains:
urlencode({'a': 'some param'})
'a=some+param'
Then the behavior could be adjusted where needed:
from urllib import quote
urlencode({'a': 'some param'}, quote=quote)
'a=some%20param'
--
components: Library (Lib)
messages: 151980
nosy: Stephen.Day
priority: normal
severity: normal
status: open
title: {urllib,urllib.parse}.urlencode should not use quote_plus
versions: Python 2.7, Python 3.1, Python 3.2, Python 3.3, Python 3.4
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13866
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13862] test_zlib failure
Stefan Krah stefan-use...@bytereef.org added the comment: Thanks! There was a 32-bit 1.2.5 version in /usr/local/lib and a 64-bit 1.2.3.3 version in /lib. gcc picks up the header from /usr/local/include, while ld is smart enough to choose the 64-bit version. So it was a multilib issue. While this is not the most robust setup, it's very convenient on a dev machine, so I'm glad that this is fixed. -- status: pending - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13862 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6056] socket.setdefaulttimeout affecting multiprocessing Manager
Jim Jewett jimjjew...@gmail.com added the comment: The wording in 138415 suggested this patch was changing socket to not support timeouts -- which would be unacceptable. But the actual patch only seems to touch multiprocessing/connection.py -- a far more reasonable change. Unfortunately, the patch no longer applies to the development tip. I *think* the places you wanted to change are still there, and just moved. (1) Is it sufficiently clear that this is not-a-feature to justify a backport? (2) Are the problems already fixed by some of the other changes? (It doesn't look like it, but I'm not sure.) (3) Can you produce an updated patch? (The current tip is http://hg.python.org/cpython/file/fec45282dc28/Lib/multiprocessing/connection.py ) (4) If I understand the intent, then s.setblocking(True) would be slightly more clear than s.settimeout(None), though that change obviously isn't essential. -- nosy: +Jim.Jewett ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6056 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13867] misleading comment in weakrefobject.h
New submission from Jim Jewett jimjjew...@gmail.com: http://hg.python.org/cpython/file/fec45282dc28/Include/weakrefobject.h#l54 The comment makes sense -- but doesn't appear to be true, so perhaps it is the macro that should change. /* This macro calls PyWeakref_CheckRef() last since that can involve a function call; this makes it more likely that the function call will be avoided. */ #define PyWeakref_Check(op) \ (PyWeakref_CheckRef(op) || PyWeakref_CheckProxy(op)) -- assignee: docs@python components: Documentation, Extension Modules messages: 151983 nosy: Jim.Jewett, docs@python priority: normal severity: normal status: open title: misleading comment in weakrefobject.h ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13867 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13703] Hash collision security issue
Antoine Pitrou pit...@free.fr added the comment: I think you're right: it will stop matching it during lookup within such a dict, since the dict will be using the secondary hash for abc, but hash() for the C instance. It will still match outside of the dict, and within other dicts. So yes, this would be a subtle semantic change when under attack. Bother. Hmm, you're right, perhaps it's not as important as I thought. By the way, have you run benchmarks on some of your patches? Is this discussion likely to reach a conclusion soon? Would it be regarded as rude if I unilaterally ship something close to: backport-of-hash-randomization-to-2.7-dmalcolm-2012-01-23-001.patch in RHEL/Fedora, so that my users have some protection they can enable if they get attacked? I don't think Fedora shipping its own patches can be considered rude by anyone else than its users. And deciding what is best for your users is indeed your job as a distro maintainer, not python-dev's. As for python.org, who is empowered to make a decision here? How can we move this forward? I don't know. Guido is empowered if he wants to make a pronouncement. Otherwise, we have the following data points: - hash randomization is generally considered the cleanest solution - hash randomization cannot be enabled by default in bugfix, let alone security releases - collision counting can mitigate some of the attacks, although it can have weaknesses (see Frank's emails) and it comes with its own problems (breaking the program later on) So I'd suggest the following course of action: - ship and enable some form of collision counting on bugfix and security releases - ship and enable hash randomization in 3.3 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13703 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13868] Add hyphen doc fix
New submission from Boštjan Mejak bostjan.me...@gmail.com: When you have time, incorporate this patch. Thanks. -- assignee: docs@python components: Documentation files: smallfix.diff keywords: patch messages: 151985 nosy: Retro, docs@python priority: normal severity: normal status: open title: Add hyphen doc fix type: enhancement versions: Python 2.7, Python 3.2, Python 3.3 Added file: http://bugs.python.org/file24325/smallfix.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13868 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13869] CFLAGS=-UNDEBUG build failure
New submission from Stefan Krah stefan-use...@bytereef.org: Passing -UNDEBUG as CFLAGS currently doesn't work: ./configure CFLAGS=-UNDEBUG make [...] libpython3.3m.a(object.o): In function `PyObject_Str': /home/stefan/pydev/cpython/Objects/object.c:441: undefined reference to `_PyUnicode_CheckConsistency' libpython3.3m.a(unicodeobject.o): In function `unicode_result_ready': /home/stefan/pydev/cpython/Objects/unicodeobject.c:476: undefined reference to `_PyUnicode_CheckConsistency' /home/stefan/pydev/cpython/Objects/unicodeobject.c:468: undefined reference to `_PyUnicode_CheckConsistency' libpython3.3m.a(unicodeobject.o): In function `PyUnicode_New': /home/stefan/pydev/cpython/Objects/unicodeobject.c:1066: undefined reference to `_PyUnicode_CheckConsistency' libpython3.3m.a(unicodeobject.o): In function `get_latin1_char': /home/stefan/pydev/cpython/Objects/unicodeobject.c:1647: undefined reference to `_PyUnicode_CheckConsistency' libpython3.3m.a(unicodeobject.o):/home/stefan/pydev/cpython/Objects/unicodeobject.c:1787: more undefined references to `_PyUnicode_CheckConsistency' follow -- components: Build messages: 151986 nosy: skrah priority: normal severity: normal stage: needs patch status: open title: CFLAGS=-UNDEBUG build failure type: behavior versions: Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13869 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11457] os.stat(): add new fields to get timestamps as Decimal objects with nanosecond resolution
Larry Hastings la...@hastings.org added the comment: Victor: I think your patch merits its own tracker issue; it's only tangentially related to the proposed changes to os.stat. That said, please do add me to the nosy list if you create one. One more thing: I haven't given it a lot of thought, so there might be an even better API out there. But given your proposed API, wouldn't it be slightly better if it took the type object rather than the string? time.time(float) or time.time(Decimal) as examples. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11457 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13864] Python 2.7.2 refuses to open
Changes by Ezio Melotti ezio.melo...@gmail.com: -- nosy: +terry.reedy ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13864 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12776] argparse: type conversion function should be called only once
Arnaud Fontaine ar...@debian.org added the comment: ping? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12776 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10042] total_ordering
Jim Jewett jimjjew...@gmail.com added the comment: I like Nick Coghlan's suggestion in msg140493, but I think he was giving up too soon in the or cases, and I think the confusion could be slightly reduced by some re-spellings around return values and comments about short-circuiting. def not_op(op, other): # not a b handles a = b # not a = b handles a b # not a = b handles a b # not a b handles a = b op_result = op(other) if op_result is NotImplemented: return NotImplemented return not op_result def op_or_eq(op, self, other): # a b or a == b handles a = b # a b or a == b handles a = b op_result = op(other) if op_result is NotImplemented return self.__eq__(other) or NotImplemented if op_result: return True return self.__eq__(other) def not_op_and_not_eq(op, self, other): # not (a b or a == b) handles a b # not a b and a != b is equivalent # not (a b or a == b) handles a b # not a b and a != b is equivalent op_result = op(other) if op_result is NotImplemented: return NotImplemented if op_result: return False return self.__ne__(other) def not_op_or_eq(op, self, other): # not a = b or a == b handles a = b # not a = b or a == b handles a = b op_result = op(other) if op_result is NotImplemented: return self.__eq__(other) or NotImplemented if op_result: return self.__eq__(other) return True def op_and_not_eq(op, self, other): # a = b and not a == b handles a b # a = b and not a == b handles a b op_result = op(other) if op_result is NotImplemented: return NotImplemented if op_result: return self.__ne__(other) return False -- nosy: +Jim.Jewett ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10042 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13859] Lingering StandardError in logging module
Matt Joiner anacro...@gmail.com added the comment: Cheers, thanks for the fast turn around. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13859 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11437] IDLE crash on startup with typo in config-keys.cfg
Terry J. Reedy tjre...@udel.edu added the comment: #13864 is a duplicate, where I mentioned the remedy of deleting the bad file. This is similar to #5707, but I am not sure if exactly the same. The patch there fixed one problem but not the bad key binding problem. -- nosy: +serwy versions: -Python 3.1 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11437 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5707] IDLE will not load
Terry J. Reedy tjre...@udel.edu added the comment: The patch fixed a real issue, the difference return of filter in 3.x versus 2.x. Bad key bindings came up in #11437 and #13864 also. -- nosy: +serwy, terry.reedy versions: +Python 3.2 -Python 3.0, Python 3.1 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5707 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11457] os.stat(): add new fields to get timestamps as Decimal objects with nanosecond resolution
Raymond Hettinger raymond.hettin...@gmail.com added the comment: Have you researched how other languages plan to expose sub-millisecond times? The isn't an API that will get points for originality. Also, it needs to be an API that is time efficient (many scripts use os.stat() frequently to scan files for changes and that check needs to be fast). Please do not just check it in. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11457 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4765] IDLE fails to Delete Custom Key Set properly
Changes by Terry J. Reedy tjre...@udel.edu: -- nosy: +serwy ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4765 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11437] IDLE crash on startup with typo in config-keys.cfg
Terry J. Reedy tjre...@udel.edu added the comment: #13071 is another duplicate closed in favor of this. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11437 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13071] IDLE accepts, then crashes, on invalid key bindings.
Changes by Terry J. Reedy tjre...@udel.edu: -- nosy: +serwy resolution: - duplicate status: open - closed superseder: - IDLE crash on startup with typo in config-keys.cfg ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13071 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13864] IDLE: Python 2.7.2 refuses to open
Terry J. Reedy tjre...@udel.edu added the comment: What system are you running on? I presume by 'keybindings in Python', you mean 'keybindings in IDLE'. Correct? By 'change to control', do you mean you made a line in the IDLE Preferences dialog, Keys tab, binding box, look like keyname - Control-Key instead of keyname - Control-Key-X ? The default configuration files are stored in and loaded from pythonxp/Lib/idlelib/. The user configuration files are in userhome/.idlerc/. They are not deleted when you delete or install Python. The reason is so that users do not lose their custom configuration when they upgrade or re-install for non-idle reasons. The location of userhome depends on the system. On my Win7 system, userhome appears as C;/user/Terry. On XP, I believe it was C:/Documents and Settings/Users/Terry/, possibly with 'Application Settings/idle' (or something) added. In any case, delete the bad file, perhaps after uploading config-keys.cfg here with the name 'poisonous_config_keys'. PS. Usage questions are generally best directed to python-list, where you generally get faster answers. In this case, though, there seems to be a nasty bug. A bad key binding should give a nice error message when one first tries to store it. Also, try searching the tracker before opening a new issue. Searching for 'key binding' in #11437 Component: IDLE turns up IDLE crash on startup with typo in config-keys.cfg. So I am closing this as a duplicate. Please post any answers there. -- nosy: +serwy resolution: - duplicate stage: - test needed status: open - closed superseder: - IDLE crash on startup with typo in config-keys.cfg title: Python 2.7.2 refuses to open - IDLE: Python 2.7.2 refuses to open type: - behavior versions: +Python 3.2, Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13864 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13870] false comment in collections/__init__.py ordered dict
New submission from Jim Jewett jimjjew...@gmail.com: http://hg.python.org/cpython/file/tip/Lib/collections/__init__.py#l37 states that the prev/next links are weakref proxies; as of http://hg.python.org/cpython/diff/3977dc349ae7/Lib/collections.py this is no longer true of the next links. It could be fixed by changing # The prev/next links are weakref proxies (to prevent circular references). to # The prev links are weakref proxies (to prevent circular references). -- components: Library (Lib) files: collections_init.patch keywords: patch messages: 151996 nosy: Jim.Jewett priority: normal severity: normal status: open title: false comment in collections/__init__.py ordered dict Added file: http://bugs.python.org/file24326/collections_init.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13870 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13871] namedtuple does not normalize field names when sanitizing
New submission from Jim Jewett jimjjew...@gmail.com:
collections.namedtuple raises a ValueError if any of the field names are not
valid identifiers, or are duplicates.
It does not normalize the identifiers when checking for duplicates.
(Similar issue with the typename)
namedtuple(dup_fields, [a, a])
Traceback (most recent call last):
File pyshell#23, line 1, in module
namedtuple(dup_fields, [a, a])
File C:\python32\lib\collections.py, line 345, in namedtuple
raise ValueError('Encountered duplicate field name: %r' % name)
ValueError: Encountered duplicate field name: 'a'
namedtuple(nfk_tester, [a, ª])
Traceback (most recent call last):
File pyshell#22, line 1, in module
namedtuple(nfk_tester, [a, ª])
File C:\python32\lib\collections.py, line 365, in namedtuple
raise SyntaxError(e.msg + ':\n\n' + class_definition)
File string, line None
SyntaxError: duplicate argument 'a' in function definition:
...
and
namedtuple(justª, ª)
Traceback (most recent call last):
File pyshell#24, line 1, in module
namedtuple(justª, ª)
File C:\python32\lib\collections.py, line 366, in namedtuple
result = namespace[typename]
KeyError: 'justª'
--
messages: 151997
nosy: Jim.Jewett
priority: normal
severity: normal
status: open
title: namedtuple does not normalize field names when sanitizing
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13871
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13871] namedtuple does not normalize field names when sanitizing
Changes by Raymond Hettinger raymond.hettin...@gmail.com: -- assignee: - rhettinger nosy: +rhettinger priority: normal - low ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13871 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13871] namedtuple does not normalize field names when sanitizing
Raymond Hettinger raymond.hettin...@gmail.com added the comment: The SyntaxError is fine. The dupcheck isn't about sanitization anyway; rather, it was part of a suite of tests designed to add a more helpful error messages than the usual ones you get if you had rolled a class by hand. I would close as invalid but want to think it over for a while -- we'll see how all your other normalization bugs resolve. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13871 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13870] false comment in collections/__init__.py ordered dict
Changes by Raymond Hettinger raymond.hettin...@gmail.com: -- assignee: - rhettinger nosy: +rhettinger priority: normal - low ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13870 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13871] namedtuple does not normalize field names when checking for duplicates
Changes by Raymond Hettinger raymond.hettin...@gmail.com: -- title: namedtuple does not normalize field names when sanitizing - namedtuple does not normalize field names when checking for duplicates ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13871 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13870] Out-of-date comment in collections/__init__.py ordered dict
Raymond Hettinger raymond.hettin...@gmail.com added the comment: This patch is fine. Anyone can feel free to apply it to Py3.2 and Py3.3 if I don't get to it first. -- title: false comment in collections/__init__.py ordered dict - Out-of-date comment in collections/__init__.py ordered dict versions: +Python 3.2, Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13870 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
