[issue13703] Hash collision security issue

[Frank Sievertsen]

Frank Sievertsen  added the comment:

>> That's true. But without the suffix, I can pretty easy and efficient
>> guess the prefix by just seeing the result of a few well-chosen and
>> short repr(dict(X)). I suppose that's harder with the suffix.

> Since the hash function is known, it doesn't make things much
> harder. Without suffix you just need hash('') to find out what
> the prefix is. With suffix, two values are enough

This is obvious and absolutely correct!

But it's not what I talked about. I didn't talk about the result of
hash(X), but about the result of repr(dict([(str: val), (str:
val)])), which is more likely to happen and not so trivial
(if you want to know more than the last 8 bits)

IMHO this problem shows that we can't advice dict() or set() for
(potential dangerous) user-supplied keys at the moment.

I prefer randomization because it fixes this problem. The
collision-counting->exception prevents a software from becoming slow,
but it doesn't make it work as expected.

Sure, you can catch the exception. But when you get the exception,
probably you wanted to add the items for a reason: Because you want
them to be in the dict and that's how your software works.

Imagine an irc-server using a dict to store the connected users, using
the nicknames as keys. Even if the irc-server catches the unexpected
exception while connecting a new user (when adding his/her name to the
dict), an attacker could connect 999 special-named users to prevent a
specific user from connecting in future.

Collision-counting->exception can make it possible to inhibit a
specific future add to the dict. The outcome is highly application
dependent.

I think it fixes 95% of the attack-vectors, but not all and it adds a
few new risks. However, of course it's much better then doing nothing
to fix the problem.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13405] Add DTrace probes

[Dave Malcolm]

Dave Malcolm  added the comment:

On Fri, 2012-01-20 at 04:46 +, Benjamin Peterson wrote:
> Benjamin Peterson  added the comment:
> 
> As great as a tool it maybe, it's still only available on a minority 
> platform. So I couldn't really try it.

FWIW, the analogous systemtap patch works great on Linux, or, at least
Fedora 13+/RHEL 6 (though beware: not all Linux distributions have
systemtap working properly out of the box, alas).

See:
http://fedoraproject.org/wiki/Features/SystemtapStaticProbes#Python_2
and http://bugs.python.org/issue4111 though this stalled due to lack of
documentation (I started writing some, but got bogged down in the
differences between the tapsets and the static markers; sorry).

Dave

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13405] Add DTrace probes

[Benjamin Peterson]

Benjamin Peterson  added the comment:

As great as a tool it maybe, it's still only available on a minority platform. 
So I couldn't really try it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13405] Add DTrace probes

[John Levon]

John Levon  added the comment:

I would strongly suggest those of you who are not "getting it" to actually try 
Jesus's patch out in some real scenarios. You'll quickly see what the benefit 
is. I think some of you are missing that it's a *not* just about performance: 
it's a system comprehension tool. It's there to help answer questions, on a 
live system; sometimes those are performance questions, but more often they're 
*not*.

I can't speak as to the maintenance burden. I'm sure we can all agree that the  
ceval.c changes are not pretty - it's why I never submitted my original 
changes. But speaking for the Solaris organization, the burden of maintaining 
our patches across the various Python releases has been minimal.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13806] Audioop decompression frames size check fix

[Oleg Plakhotnyuk]

Changes by Oleg Plakhotnyuk :


Added file: http://bugs.python.org/file24285/audioop_size_check.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13806] Audioop decompression frames size check fix

[Oleg Plakhotnyuk]

Changes by Oleg Plakhotnyuk :


Removed file: http://bugs.python.org/file24284/audioop_size_check.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13806] Audioop decompression frames size check fix

[Oleg Plakhotnyuk]

Oleg Plakhotnyuk  added the comment:

Yep, you're right. Didn't noticed audioop_check_size() function at first.

The fact that audioop accepts unicode strings seems weird to me too. I've 
replaced strings with bytes in tests. However, I'm afraid to add restrictions 
to library itself because of backward compatibility.

--
Added file: http://bugs.python.org/file24284/audioop_size_check.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13405] Add DTrace probes

[Glyph Lefkowitz]

Glyph Lefkowitz  added the comment:

Charles-François:
> Also, I must admit I'm quite skeptical about the real benefit of
explicit probes for user-land, especially for CPython which isn't used
for performance-critical systems...

I beg to differ.  CPython is totally used on performance-critical systems, and 
I know I'm not the only user who thinks that.  "Performance-critical" doesn't 
necessarily mean "goes as fast as it ever possibly can", clearly PyPy is the 
place to go for that, but "can process at least X work in Y time".  Meeting 
performance goals with CPython is already challenging enough, please don't make 
it artificially hard by refusing to integrate tools which help users understand 
and improve performance.

Benjamin:
> I'm -1 on this patch for essentially the same reasons as Charles-François. It 
> introduces a lot of code (and hacks!) in critical pathways of the 
> interpreter. Someone would have to be constantly maintaining and testing it. 
> In return, what do we get?

You get support for a highly sophisticated and low-impact profiling and tracing 
technology which provides support for illuminating performance problems *as 
well as* complicated behavioral problems that only happen under load, without 
slowing down the interpreter as a whole.  Not to mention possible integration 
with a whole slew of tools that know how to deal with data from that system.

I'm not saying that this is necessarily worth the maintenance burden; your 
analysis of the tradeoff may ultimately be correct.  I can't presume to know 
that because I am not intimately familiar with all the code it touches.  But 
it's definitely not nothing.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12949] Documentation of PyCode_New() lacks kwonlyargcount argument

[Meador Inge]

Meador Inge  added the comment:

Fixed.  Thanks for the report Stefan.

--
nosy: +meador.inge
resolution:  -> fixed
stage: needs patch -> committed/rejected
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13806] Audioop decompression frames size check fix

[Oleg Plakhotnyuk]

Changes by Oleg Plakhotnyuk :


Removed file: http://bugs.python.org/file24260/audioop_size_check.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12949] Documentation of PyCode_New() lacks kwonlyargcount argument

[Roundup Robot]

Roundup Robot  added the comment:

New changeset b22a35c14a91 by Meador Inge in branch '3.2':
Issue #12949: Document the kwonlyargcount argument for the PyCode_New C API 
function.
http://hg.python.org/cpython/rev/b22a35c14a91

New changeset 218b167ff521 by Meador Inge in branch 'default':
Issue #12949: Document the kwonlyargcount argument for the PyCode_New C API 
function.
http://hg.python.org/cpython/rev/218b167ff521

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13405] Add DTrace probes

[Jesús Cea Avión]

Jesús Cea Avión  added the comment:

NOTE to myself: In OpenIndiana we could have this error:

"""
dtrace: failed to compile script ./Include/phelper.d: Preprocessor not found
"""

In that case, we must install the C preprocessor:

"""
pfexec pkg install cpp
"""

I confirm that current 3.3 patch works in the 32 bits OpenIndiana Python 
buildbots, after adding this patch (already available in my mercurial 
repository, but not in the patch published in this issue):

"""
diff --git a/Lib/test/test_dtrace.py b/Lib/test/test_dtrace.py
--- a/Lib/test/test_dtrace.py
+++ b/Lib/test/test_dtrace.py
@@ -173,7 +173,7 @@
 # Verify that we are checking:
 opcodes = set(["CALL_FUNCTION", "CALL_FUNCTION_VAR",
 "CALL_FUNCTION_KW", "CALL_FUNCTION_VAR_KW"])
-obj = compile(open(sample).read(), "sample", "exec")
+obj = compile(open(sample, encoding="utf-8").read(), "sample", "exec")
 class dump() :
 def __init__(self) :
 self.buf = []
"""

Also, of course, the user must have dtrace permissions, as documented in a 
previous message.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12949] Documentation of PyCode_New() lacks kwonlyargcount argument

[Meador Inge]

Changes by Meador Inge :


--
keywords: +easy
stage:  -> needs patch
versions:  -Python 3.1

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13405] Add DTrace probes

[Benjamin Peterson]

Benjamin Peterson  added the comment:

I'm -1 on this patch for essentially the same reasons as Charles-François. It 
introduces a lot of code (and hacks!) in critical pathways of the interpreter. 
Someone would have to be constantly maintaining and testing it. In return, what 
do we get?

--
nosy: +benjamin.peterson

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13405] Add DTrace probes

[Jesús Cea Avión]

Jesús Cea Avión  added the comment:

Code ready for commit.

Please, review.

After the first commit, I will try to make it compatible with MacOS and 
FreeBSD. Current target is Solaris and derivatives (OpenIndiana, for instance)

Stan Cox, if you want systemtap compatibility at this moment, please provide a 
patch. The first patch doesn't need "ustack" compatibility. In fact, MacOS 
dtrace, for instance, seems not to have "ustack" support at all (according to 
Google), so the feature is very convenient but optional.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13405] Add DTrace probes

[Jesús Cea Avión]

Changes by Jesús Cea Avión :


Removed file: http://bugs.python.org/file23921/43d1a819a63d.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13807] logging.Handler.handlerError() may raise AttributeError in traceback.print_exception()

[Vinay Sajip]

Changes by Vinay Sajip :


--
assignee:  -> vinay.sajip
nosy: +vinay.sajip

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13405] Add DTrace probes

[Jesús Cea Avión]

Changes by Jesús Cea Avión :


Removed file: http://bugs.python.org/file23920/f73be85b9a7e.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13405] Add DTrace probes

[Jesús Cea Avión]

Changes by Jesús Cea Avión :


Added file: http://bugs.python.org/file24283/f86bb02fd8f4.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13405] Add DTrace probes

[Jesús Cea Avión]

Changes by Jesús Cea Avión :


Added file: http://bugs.python.org/file24282/f86bb02fd8f4.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13405] Add DTrace probes

[Jesús Cea Avión]

Changes by Jesús Cea Avión :


Added file: http://bugs.python.org/file24281/5458412752d5.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10278] add time.wallclock() method

[Roundup Robot]

Roundup Robot  added the comment:

New changeset 8502a9236c2e by Victor Stinner in branch 'default':
Issue #10278: Be more explicit in tests than wallclock() is monotonic (cannot
http://hg.python.org/cpython/rev/8502a9236c2e

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13828] Further improve casefold documentation

[Benjamin Peterson]

Benjamin Peterson  added the comment:

It's a bit unfriendly to launch into discussion of "compatiblity caseless 
matching" when the new reader probably has no idea what 
"compatibility-equivalence" is.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13703] Hash collision security issue

[STINNER Victor]

STINNER Victor  added the comment:

> Since the hash function is known, it doesn't make things much
> harder. Without suffix you just need hash('') to find out what
> the prefix is. With suffix, two values are enough.

With my patch, hash('') always return zero. I don't remember who asked
me to do that, but it avoids to leak too easily the secret :-) I wrote
some info how to compute the secret:
http://bugs.python.org/issue13703#msg150706

I don't see how to compute the secret, but it doesn't mean that it is
impossible :-) I suppose that you have to brute force some bits, at
least if you only have repr(dict) which gives only (indirectly) the
lower bits of the hash.

> (things obviously get tricky once overflow kicks in)

hash() doesn't overflow: if you know the string, you can run the
algorithm backward. To divide, you can compute 1/103 mod 2^32 (or
mod 2^64): 2021759595 and 16109806864799210091. So x/103 mod 2^32
= x*2021759595 mod 2^32.

See my invert_mod() function of:
https://bitbucket.org/haypo/misc/src/tip/python/mathfunc.py

> With Victor's approach hash(0) would output the whole seed,
> but even if the seed is not known, creating an attack data
> set is trivial, since hash(x) = P ^ x ^ S.

I suppose that it would be too simple to compute the secret of a
randomized integer hash, so it is maybe better to leave them
unchanged. Using a different secret from strings and integer would not
protect Python against an attack only using integers, but integer keys
are less common than string keys (especially on web applications).

Anyway, I changed my mind about randomized hash: I now prefer counting
collisions :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13811] In str.format, if invalid fill and alignment are specified, the text of the ValueError message is misleading.

[Roundup Robot]

Roundup Robot  added the comment:

New changeset 5c33ebb50702 by Eric V. Smith in branch 'default':
Improve exception text. Closes issue 13811.
http://hg.python.org/cpython/rev/5c33ebb50702

--
nosy: +python-dev
resolution:  -> fixed
stage: needs patch -> committed/rejected
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13703] Hash collision security issue

[Marc-Andre Lemburg]

Marc-Andre Lemburg  added the comment:

Frank Sievertsen wrote:
> 
> Frank Sievertsen  added the comment:
> 
>> The suffix only introduces a constant change in all hash values
>> output, so even if you don't know the suffix, you can still
>> generate data sets with collisions by just having the prefix.
> 
> That's true. But without the suffix, I can pretty easy and efficient guess 
> the prefix by just seeing the result of a few well-chosen and short 
> repr(dict(X)). I suppose that's harder with the suffix.

Since the hash function is known, it doesn't make things much
harder. Without suffix you just need hash('') to find out what
the prefix is. With suffix, two values are enough.

Say P is your prefix and S your suffix. Let's say you can get the
hash values of A = hash('') and B = hash('\x00').

With Victor's hash function you have (IIRC):

A = hash('') = P ^ (0<<7) ^ 0 ^ S = P ^ S
B = hash('\x00') = ((P ^ (0<<7)) * 103) ^ 0 ^ 1 ^ S = (P * 103) ^ 1 ^ S

Let X = A ^ B, then

X = P ^ (P * 103) ^ 1

since S ^ S = 0 and 0 ^ Y = Y (for any Y), i.e. the suffix doesn't
make any difference.

For P < 50, you can then easily calculate P from X
using:

P = X // 102

(things obviously get tricky once overflow kicks in)

Note that for number hashes the randomization doesn't work at all,
since there's no length or feedback loop involved.

With Victor's approach hash(0) would output the whole seed,
but even if the seed is not known, creating an attack data
set is trivial, since hash(x) = P ^ x ^ S.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13763] Potentially hard to understand wording in devguide

[Terry J. Reedy]

Terry J. Reedy  added the comment:

I do not much like sentences starting with lowercase, so I combined our 
sentences. The result is good enough, I think. Closing.

--
assignee:  -> terry.reedy
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13763] Potentially hard to understand wording in devguide

[Roundup Robot]

Roundup Robot  added the comment:

New changeset 081106c142ec by Terry Jan Reedy in branch 'default':
#13763 Clarify 'hg' usage.
http://hg.python.org/devguide/rev/081106c142ec

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13832] tokenization assuming ASCII whitespace; missing multiline case

[Benjamin Peterson]

Benjamin Peterson  added the comment:

The tokenizer doesn't consider unicode spaces, either.

--
resolution:  -> fixed
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13832] tokenization assuming ASCII whitespace; missing multiline case

[Roundup Robot]

Roundup Robot  added the comment:

New changeset 00c4efbf57c3 by Benjamin Peterson in branch 'default':
check after comments, too (#13832)
http://hg.python.org/cpython/rev/00c4efbf57c3

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13833] No documentation for PyStructSequence

[Torsten Landschoff]

New submission from Torsten Landschoff :

While writing a C extension I wanted to create a namedtuple like object as 
os.statvfs and friends do.

I was unable to find a simple way to do this from C and was wondering how the 
posixmodule does it. It turned out that there is a PyStructSequence type for 
this.

It would be nice to have it documented. First draft for the documentation 
update attached.

--
assignee: docs@python
components: Documentation
files: structseq_doc.diff
keywords: patch
messages: 151657
nosy: docs@python, torsten
priority: normal
severity: normal
status: open
title: No documentation for PyStructSequence
type: enhancement
versions: Python 3.3
Added file: http://bugs.python.org/file24280/structseq_doc.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13020] structseq.c: refleak

[Torsten Landschoff]

Torsten Landschoff  added the comment:

+1 for the patch.
All the error paths above the change do

  Py_DECREF(arg);
  return NULL;

arg is initialized with PySequence_Fast, which returns a new reference.
Hard to create a test case for this...

--
nosy: +torsten

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13832] tokenization assuming ASCII whitespace; missing multiline case

[Antoine Pitrou]

Changes by Antoine Pitrou :


--
nosy: +benjamin.peterson

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13605] document argparse's nargs=REMAINDER

[Roundup Robot]

Roundup Robot  added the comment:

New changeset 1b481e76cd16 by Sandro Tosi in branch '2.7':
Issue #13605: more meaningful example + fixes
http://hg.python.org/cpython/rev/1b481e76cd16

New changeset d6e53d1f46eb by Sandro Tosi in branch '3.2':
Issue #13605: more meaningful example + fixes
http://hg.python.org/cpython/rev/d6e53d1f46eb

New changeset 4c3271527794 by Sandro Tosi in branch 'default':
Issue #13605: merge with 3.2
http://hg.python.org/cpython/rev/4c3271527794

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13605] document argparse's nargs=REMAINDER

[Sandro Tosi]

Changes by Sandro Tosi :


--
nosy: +sandro.tosi
resolution:  -> fixed
stage: needs patch -> committed/rejected
status: open -> closed
versions: +Python 2.7, Python 3.2

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13605] document argparse's nargs=REMAINDER

[Roundup Robot]

Roundup Robot  added the comment:

New changeset 35665f6f3674 by Sandro Tosi in branch '2.7':
Issue #13605: add documentation for nargs=argparse.REMAINDER
http://hg.python.org/cpython/rev/35665f6f3674

New changeset 6f3d55f5a31e by Sandro Tosi in branch '3.2':
Issue #13605: add documentation for nargs=argparse.REMAINDER
http://hg.python.org/cpython/rev/6f3d55f5a31e

New changeset 6b4cec0719a3 by Sandro Tosi in branch 'default':
Issue #13605: merge with 3.2
http://hg.python.org/cpython/rev/6b4cec0719a3

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11948] Tutorial/Modules - small fix to better clarify the modules search path

[Terry J. Reedy]

Terry J. Reedy  added the comment:

Definitely out-of-scope for the tutorial. I consider this akin to monkey 
patching imported modules.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13832] tokenization assuming ASCII whitespace; missing multiline case

[Jim Jewett]

New submission from Jim Jewett :

Parser/parsetok.c was recently changed (e.g. 
http://hg.python.org/cpython/rev/2bd7f40108b4 ) to raise an error if multiple 
statements were found in a single-statement compile call.  It sensibly ignores 
trailing whitespace and comments.  Unfortunately,

(1)  It looks only at (c == ' ' || c == '\t' || c == '\n' || c == '\014') as 
opposed to using Py_UNICODE_ISSPACE(ch)
(2)  It assumes that a "#" means the rest of the line is OK, instead of looking 
for additional linebreaks.

Not sure whether to mark this a bug or an enhancement, since it is already 
strictly better than the 3.2 behavior of never warning about extra text.

--
components: Interpreter Core
messages: 151652
nosy: Jim.Jewett
priority: normal
severity: normal
status: open
title: tokenization assuming ASCII whitespace; missing multiline case
versions: Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13831] get method of multiprocessing.pool.Async should return full traceback

[Faheem Mitha]

New submission from Faheem Mitha :

The documentation in 
http://docs.python.org/library/multiprocessing.html#module-multiprocessing.pool

says

"""class multiprocessing.pool.AsyncResult¶
The class of the result returned by Pool.apply_async() and Pool.map_async().

get([timeout])
Return the result when it arrives. If timeout is not None and the result does 
not arrive within timeout seconds then multiprocessing.TimeoutError is raised. 
If the remote call raised an exception then that exception will be reraised by 
get()."""

Consider the example code



from multiprocessing import Pool

def go():
print 1
raise Exception("foobar")
print 2

p = Pool()
x = p.apply_async(go)
x.get()
p.close()
p.join()

###

The traceback from this is

Traceback (most recent call last):
  File "", line 10, in 
  File "/usr/lib/python2.6/multiprocessing/pool.py", line 422, in get
raise self._value
Exception: foobar
1

As is clear in this example, this is not a full traceback - it only shows the 
traceback to the line where get is located and gives no further information. 
This is the case in all the other places I have used get. It seems to me that 
it *should* return the full traceback, which may contain important information 
missing in such a partial one. I don't know whether one would call this a 
feature request or a bug report. Maybe there is some technical reason why this 
is not possible, but I can't think of one.

--
components: Library (Lib)
messages: 151651
nosy: fmitha
priority: normal
severity: normal
status: open
title: get method of  multiprocessing.pool.Async should return full traceback
type: enhancement
versions: Python 2.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13830] codecs error handler is called with a UnicodeDecodeError with the same args

[Amaury Forgeot d'Arc]

New submission from Amaury Forgeot d'Arc :

The script below shows that the error handler is always called with the same 
error object.  The 'start', 'end', and 'reason' properties are correctly 
updated, but the 'args' is always the same and holds the values used for the 
first call.

It's a bit weird that error.args[2] is not equal to error.start, for example. 
All versions are affected: 2.7, 3.2, 3.3.
And by the way, I could not find where these are attributes documented.



def custom_handler(error):
print(error.args,
  (error.start, error.end, error.reason))
return b'?'.decode(), error.end

import codecs
codecs.register_error('custom', custom_handler)
b'\x80\xd0'.decode('utf-8', 'custom')

--
components: Unicode
messages: 151650
nosy: amaury.forgeotdarc, ezio.melotti
priority: normal
severity: normal
status: open
title: codecs error handler is called with a UnicodeDecodeError with the same 
args
type: behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13826] Having a shlex example in the subprocess.Popen docs is confusing

[Julian Berman]

Julian Berman  added the comment:

Sounds reasonable to me.

I'll take a look at adding one unless someone manages to beat me to it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13829] exception error

[Dan kamp]

New submission from Dan kamp :

Get this error when trying to run Moviegrabber on a mac running v2.7. Crash 
report below.

Process: Python [2444]
Path:
/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/C 
ontents/MacOS/Python
Identifier:  Python
Version: ??? (???)
Code Type:   X86-64 (Native)
Parent Process:  Python [2431]
 
 
Date/Time:   2011-12-21 20:32:41.233 -0500
OS Version:  Mac OS X 10.7.2 (11C74)
Report Version:  9
 
 
Interval Since Last Report:  357638 sec
Crashes Since Last Report:   17
Per-App Crashes Since Last Report:   405
Anonymous UUID:  3C387DB7-0AA3-4F34-A100-31D1736D2668
 
 
Crashed Thread:  0
 
 
Exception Type:  EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x0108
 
 
VM Regions Near 0x108:
-->
__TEXT 00010c5c5000-00010c5c6000 [4K] r-x/rwx 
SM=COW  
/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/C 
ontents/MacOS/Python
 
 
Application Specific Information:
objc[2429]: garbage collection is OFF
 
 
Thread 0 Crashed:
0   libdispatch.dylib   0x7fff8b14ace9 _dispatch_wakeup 
+ 108
1   libdispatch.dylib   0x7fff8b14d876 
_dispatch_resume_slow + 20
2   com.apple.CoreFoundation0x7fff8af8262c 
-[NSXPCConnection start] + 92
3   com.apple.CoreFoundation0x7fff8af84ca3 
__CFXNotificationCenterSetupConnection + 387
4   com.apple.CoreFoundation0x7fff8af84b11 
__CFXNotificationCenterCreate + 273
5   com.apple.CoreFoundation0x7fff8af849ea 
__CFNotificationCenterGetDistributedCenter_block_invoke_1 + 26
6   libdispatch.dylib   0x7fff8b14d224 dispatch_once_f 
+ 53
7   com.apple.CoreFoundation0x7fff8af73e0a 
CFNotificationCenterGetDistributedCenter + 74
8   com.apple.CoreFoundation0x7fff8afb2eb8 
CFXPreferencesGetSourceForTriplet_block_invoke_1 + 40
9   libdispatch.dylib   0x7fff8b14d224 dispatch_once_f 
+ 53
10  com.apple.CoreFoundation0x7fff8af7fa4a 
__CFXPreferencesGetSourceForTriplet + 58
11  com.apple.CoreFoundation0x7fff8af8be67 
__CFXPreferencesGetSearchListForBundleID + 215
12  com.apple.CoreFoundation0x7fff8af8bd48 
___CFXPreferencesCopyAppValue_block_invoke_1 + 24
13  com.apple.CoreFoundation0x7fff8af8bcea 
CFPreferencesCopyAppValue + 218
14  com.apple.SystemConfiguration   0x7fff928f388c 
SCDynamicStoreCopyProxies + 43
15  _scproxy.so 0x00010ceae9ba 0x10ceae000 + 
2490
16  org.python.python   0x00010c651b58 
PyEval_EvalFrameEx + 13318
17  org.python.python   0x00010c654df7 0x10c5ca000 + 
568823
18  org.python.python   0x00010c651e0a 
PyEval_EvalFrameEx + 14008
19  org.python.python   0x00010c654df7 0x10c5ca000 + 
568823
20  org.python.python   0x00010c651e0a 
PyEval_EvalFrameEx + 14008
21  org.python.python   0x00010c654cd8 
PyEval_EvalCodeEx + 1996
22  org.python.python   0x00010c5f2abf 0x10c5ca000 + 
166591
23  org.python.python   0x00010c5d1d32 PyObject_Call + 
97
24  org.python.python   0x00010c5e06e9 0x10c5ca000 + 
91881
25  org.python.python   0x00010c5d1d32 PyObject_Call + 
97
26  org.python.python   0x00010c64dc40 
PyEval_CallObjectWithKeywords + 180
27  org.python.python   0x00010c5dd489 PyInstance_New + 
273
28  org.python.python   0x00010c5d1d32 PyObject_Call + 
97
29  org.python.python   0x00010c651f63 
PyEval_EvalFrameEx + 14353
30  org.python.python   0x00010c654cd8 
PyEval_EvalCodeEx + 1996
31  org.python.python   0x00010c654e6c 0x10c5ca000 + 
568940
32  org.python.python   0x00010c651e0a 
PyEval_EvalFrameEx + 14008
33  org.python.python   0x00010c654cd8 
PyEval_EvalCodeEx + 1996
34  org.python.python   0x00010c654e6c 0x10c5ca000 + 
568940
35  org.python.python   0x00010c651e0a 
PyEval_EvalFrameEx + 14008
36  org.python.python   0x00010c654cd8 
PyEval_EvalCodeEx + 1996
37  org.python.python   0x00010c5f2abf 0x10c5ca000 + 
166591
38  org.python.python   0x00010c5d1d32 PyObject_Call + 
97
39  org.python.python   0x00010c6525ec 
PyEval_EvalFrameEx + 16026
40  org.python.python   0x00010c654df7 0x10c5ca000 + 
568823
41  org.python.python  

[issue13703] Hash collision security issue

[Frank Sievertsen]

Frank Sievertsen  added the comment:

> The suffix only introduces a constant change in all hash values
> output, so even if you don't know the suffix, you can still
> generate data sets with collisions by just having the prefix.

That's true. But without the suffix, I can pretty easy and efficient guess the 
prefix by just seeing the result of a few well-chosen and short repr(dict(X)). 
I suppose that's harder with the suffix.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13641] decoding functions in the base64 module could accept unicode strings

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Thanks for the updated patch!
Two comments:
- I see no tests for map01 and altchars being passed as an str, is this 
supported by the patch or am I reading it wrong?
- apparently b16decode is not tackled, is it deliberate?

Thanks again.

--
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13828] Further improve casefold documentation

[Jim Jewett]

Jim Jewett  added the comment:

Frankly, I do think that sample code is too long, but correctness matters ... 
perhaps a better solution would be to add either a method or a unicodedata 
function that does the work, then the extra note could just say

Note that most case-insensitive matches should also match compatibility 
equivalent characters; see unicodedata.compatibity_casefold

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13828] Further improve casefold documentation

[Jim Jewett]

New submission from Jim Jewett :

> http://hg.python.org/cpython/rev/0b5ce36a7a24
> changeset:   74515:0b5ce36a7a24


> +   Casefolding is similar to lowercasing but more aggressive because it is
> +   intended to remove all case distinctions in a string. For example, the 
> German
> +   lowercase letter ``'ß'`` is equivalent to ``"ss"``. Since it is already
> +   lowercase, :meth:`lower` would do nothing to ``'ß'``; :meth:`casefold`
> +   converts it to ``"ss"``.

Perhaps add the recommendation to canonicalize as well.

A complete, but possibly too long, try is below:


Casefolding is similar to lowercasing but more aggressive because it is 
intended to remove all case distinctions in a string. For example, the German 
lowercase letter ``'ß'`` is equivalent to ``"ss"``. Since it is already 
lowercase, :meth:`lower` would do nothing to ``'ß'``; :meth:`casefold` converts 
it to ``"ss"``.  Note that most case-insensitive matches should also match 
compatibility equivalent characters.  

The casefolding algorithm is described in section 3.13 of the Unicode Standard. 
 Per D146, a compatibility caseless match can be achieved by

from unicodedata import normalize
def caseless_compat(string):
nfd_string = normalize("NFD", string)
nfkd1_string = normalize("NFKD", nfd_string.casefold())
return normalize("NFKD", nfkd1_string.casefold())

--
assignee: docs@python
components: Documentation
messages: 151644
nosy: Jim.Jewett, benjamin.peterson, docs@python
priority: normal
severity: normal
status: open
title: Further improve casefold documentation
versions: Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11948] Tutorial/Modules - small fix to better clarify the modules search path

[Eric Snow]

Eric Snow  added the comment:

FYI: unless importlib took undue liberties (unlikely), frozen modules also 
precede path-based modules.  See the implicit additions to sys.meta_path in 
Lib/importlib/_bootstrap.py.

Whether or not to include a mention of frozen modules in the tutorial...I'll 
leave that to you.  :)

--
nosy: +eric.snow

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13825] Datetime failing while reading active directory time attribute

[scape]

scape  added the comment:

I dug a little deeper using an error trap and found some of the problematic 
accounts in AD have their attribute set to a wildly long number and not 0 (as 
are others when 'never' is specified.) i'll dig further, it also does not seem 
to be an issue with python but more of an issue with the module I am using 
(active_directory) and its datetime handling (likely not fixed as was Python)

I don't think the issue is necessarily solved, but I'll close it anyways as I 
think I have atleast my solution now

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11551] test_dummy_thread.py test coverage improvement

[Sandro Tosi]

Sandro Tosi  added the comment:

Hi Denver, do you have the time to address the review on rietveld and propose a 
new patch?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13827] Unexecuted import changes namespace

[Ezio Melotti]

Changes by Ezio Melotti :


--
components:  -None
resolution:  -> invalid
stage:  -> committed/rejected
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13827] Unexecuted import changes namespace

[Ezio Melotti]

Ezio Melotti  added the comment:

>>> OVERRIDE = False
>>> SOMETHING = "original"
>>> 
>>> def main():
... if OVERRIDE:
... SOMETHING = None
... print SOMETHING
... 
>>> main()
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 4, in main
UnboundLocalError: local variable 'SOMETHING' referenced before assignment

http://docs.python.org/faq/programming.html#why-am-i-getting-an-unboundlocalerror-when-the-variable-has-a-value

--
nosy: +ezio.melotti
resolution: invalid -> 
stage: committed/rejected -> 
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13827] Unexecuted import changes namespace

[Michael Foord]

Michael Foord  added the comment:

hippmr: the problem is that by importing SOMETHING inside that function you're 
creating a *local variable* called SOMETHING. If the override isn't executed, 
and SOMETHING isn't global, then that local variable doesn't exist - which is 
why you get that error.

So even if the import isn't executed, its existence in the function tells 
Python that name is local to the function.

--
nosy: +michael.foord
resolution:  -> invalid
stage:  -> committed/rejected
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13827] Unexecuted import changes namespace

[Michael Hipp]

Michael Hipp  added the comment:

Even an *unexecuted* import assignment statement?

--
resolution: invalid -> 
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13827] Unexecuted import changes namespace

[Benjamin Peterson]

Benjamin Peterson  added the comment:

Not a bug. Basically, import is an explicit assignment statement.

--
nosy: +benjamin.peterson
resolution:  -> invalid
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13827] Unexecuted import changes namespace

[Michael Hipp]

Michael Hipp  added the comment:

Add'l over.py file

--
Added file: http://bugs.python.org/file24279/over.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13827] Unexecuted import changes namespace

[Michael Hipp]

New submission from Michael Hipp :

A local *unexecuted* import appears to be changing the namespace. Attached 
files are ready to run.

# over.py
SOMETHING = "overridden"

# main.py
OVERRIDE = False
SOMETHING = "original"

def main():
#global SOMETHING  # uncomment and it works
if OVERRIDE:
from over import SOMETHING  # comment out and it works
pass
print SOMETHING  # UnboundLocalError: local variable 'SOMETHING' referenced 
before assignment

The SOMETHING variable has a value from the module global namespace, but it 
gets lost due to an import that is never executed.

I would think an unexecuted statement shouldn't have any effect on anything.

The second file will have to be submitted in a follow-on, it appears

--
components: None
files: main.py
messages: 151635
nosy: hippmr
priority: normal
severity: normal
status: open
title: Unexecuted import changes namespace
type: behavior
versions: Python 2.7
Added file: http://bugs.python.org/file24278/main.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13478] No documentation for timeit.default_timer

[Sandro Tosi]

Changes by Sandro Tosi :


Added file: http://bugs.python.org/file24277/issue13478-py32.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13478] No documentation for timeit.default_timer

[Sandro Tosi]

Sandro Tosi  added the comment:

I propose these 2 patches: I'm not exactly proud of 

- 'in a platform specific manner' in the first hunk
- the second hunk

I know that there should be a better way to express it, but since I can't get 
to it I'd just ask for suggestions :)

--
keywords: +patch
stage: needs patch -> patch review
Added file: http://bugs.python.org/file24276/issue13478-py27.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13703] Hash collision security issue

[Marc-Andre Lemburg]

Marc-Andre Lemburg  added the comment:

[Reposting, since roundup removed part of the Python output]

M.-A. Lemburg wrote:
> Note that the integer attack also applies to other number types
> in Python:
> 
> --> (hash(3), hash(3.0), hash(3+0j)
> (3, 3, 3)
> 
> See Tim's post I referenced earlier on for the reasons. Here's
> a quick summary ;-) ...
> 
> --> {3:1, 3.0:2, 3+0j:3}
> {3: 3}

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13703] Hash collision security issue

[Marc-Andre Lemburg]

Marc-Andre Lemburg  added the comment:

STINNER Victor wrote:
> 
> I tried the collision counting with a low number of collisions:
> ... no false positives with a limit of 50 collisions ...

Thanks for running those tests. Looks like a limit lower than 1000
would already do just fine.

Some timings showing how long it would take to hit a limit:

# 100
python2.7 -m timeit -n 100 "dict((x*(2**64 - 1), 1) for x in xrange(1, 100))"
100 loops, best of 3: 297 usec per loop

# 250
python2.7 -m timeit -n 100 "dict((x*(2**64 - 1), 1) for x in xrange(1, 250))"
100 loops, best of 3: 1.46 msec per loop

# 500
python2.7 -m timeit -n 100 "dict((x*(2**64 - 1), 1) for x in xrange(1, 500))"
100 loops, best of 3: 5.73 msec per loop

# 750
python2.7 -m timeit -n 100 "dict((x*(2**64 - 1), 1) for x in xrange(1, 750))"
100 loops, best of 3: 12.7 msec per loop

# 1000
python2.7 -m timeit -n 100 "dict((x*(2**64 - 1), 1) for x in xrange(1, 1000))"
100 loops, best of 3: 22.4 msec per loop

These timings have to matched against the size of the payload
needed to trigger those limits.

In any case, the limit needs to be configurable like the hash seed
in the randomization patch.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13752] add a str.casefold() method

[Giampaolo Rodola']

Changes by Giampaolo Rodola' :


--
nosy: +giampaolo.rodola

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13817] deadlock in subprocess while running several threads using Popen

[Charles-François Natali]

Charles-François Natali  added the comment:

Here's the patch.
It's probably possible to add a test for this, however I don't have access to 
my development machine, so I can't write it now.

--
Added file: http://bugs.python.org/file24275/reinit_tls.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13826] Having a shlex example in the subprocess.Popen docs is confusing

[Sandro Tosi]

Sandro Tosi  added the comment:

Maybe we can add a very small example before the whole note to show just how to 
use Popen in simple situation, and so the shlex part below will add more 
details for more advanced cases.

--
nosy: +sandro.tosi
versions:  -Python 2.6, Python 3.1

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13703] Hash collision security issue

[Alex Gaynor]

Alex Gaynor  added the comment:

Django's tests will *not* be run with HASHEED=0, if they're broken with hash 
randomization then they are likely broken on random.choice(["32-bit", "64-bit", 
"pypy", "jython", "ironpython"]) and we strive to run on all those platforms. 
If our tests are order dependent then they're broken, and we'll fix the tests.

Further, most of the failures I can think of would be failures in the tests 
that wouldn't actually be failures in a real application, such as the rendered 
HTML being different because a tag's attributes are in a different order.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13703] Hash collision security issue

[Marc-Andre Lemburg]

Marc-Andre Lemburg  added the comment:

Antoine Pitrou wrote:
> 
> Antoine Pitrou  added the comment:
> 
>> Please note, that you'd have to extend the randomization to
>> all other Python data types as well in order to reach the same level
>> of security as the collision counting approach.
> 
> You also have to extend the collision counting to sets, by the way.

Indeed, but that's easy, since the set implementation derives from
the dict implementation.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13826] Having a shlex example in the subprocess.Popen docs is confusing

[R. David Murray]

R. David Murray  added the comment:

It is not particularly intuitive what goes in to a Popen non-shell argument 
list, unless you are an experienced programmer.  The real purpose of the note 
is to convey a lot of information about how tokenization works in a short 
example, and it also demonstrates how to investigate other complex cases the 
user may have to deal with.  Because of the first part of that (showing 
tokenization quirks) I don't think it should be relegated to a footnote.

That said, the example could perhaps be reworded slightly to make its 
expositional purpose clearer.  Suggestions welcome.

--
nosy: +r.david.murray

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13826] Having a shlex example in the subprocess.Popen docs is confusing

[R. David Murray]

Changes by R. David Murray :


--
priority: normal -> low

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13703] Hash collision security issue

[Antoine Pitrou]

Antoine Pitrou  added the comment:

> Please note, that you'd have to extend the randomization to
> all other Python data types as well in order to reach the same level
> of security as the collision counting approach.

You also have to extend the collision counting to sets, by the way.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13703] Hash collision security issue

[Marc-Andre Lemburg]

Marc-Andre Lemburg  added the comment:

STINNER Victor wrote:
> ...
> So I expect something similar in applications: no change in the
> applications, but a lot of hacks/tricks in tests.

Tests usually check output of an application given a certain
input. If those fail with the randomization, then it's likely
real-world application uses will show the same kinds of failures
due to the application changing from deterministic to
non-deterministic via the randomization.

>> BTW: The patch still includes the unnecessary _Py_unicode_hash_secret.suffix
>> which needlessly complicates the code and doesn't any additional
>> protection against hash value collisions
> 
> How does it complicate the code? It adds an extra XOR to hash(str) and
> 4 or 8 bytes in memory, that's all. It is more difficult to compute
> the secret from hash(str) output if there is a prefix *and* a suffix.
> If there is only a prefix, knowning a single hash(str) value is just
> enough to retrieve directly the secret.

The suffix only introduces a constant change in all hash values
output, so even if you don't know the suffix, you can still
generate data sets with collisions by just having the prefix.

>> I don't think it affects more than 0.01% of applications/users :)
> 
> It would help to try a patched Python on a real world application like
> Django to realize how much code is broken (or not) by a randomized
> hash function.

That would help for both approaches, indeed.

Please note, that you'd have to extend the randomization to
all other Python data types as well in order to reach the same level
of security as the collision counting approach.

As-is the randomization patch does not solve the integer key attack and
even though parsers such as JSON and XML-RPC aren't directly affected,
it is well possible that stringified integers such as IDs are converted
back to integers later during processing, thereby triggering the
attack.

Note that the integer attack also applies to other number types
in Python:

(3, 3, 3)

See Tim's post I referenced earlier on for the reasons. Here's
a quick summary ;-) ...

{3: 3}

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13826] Having a shlex example in the subprocess.Popen docs is confusing

[Julian Berman]

New submission from Julian Berman :

The example at 
http://docs.python.org/dev/library/subprocess.html#popen-constructor seems a 
bit misplaced, as it seems to suggest that one should use the shlex module. 
Most of the other examples in the module seem to use a list to provide the 
args, so if there was a need to just point out that shlex could be used for a 
corner case perhaps it'd be better suited as a footnote or another subsection 
somewhere.

--
assignee: docs@python
components: Documentation
messages: 151624
nosy: Julian, docs@python
priority: normal
severity: normal
status: open
title: Having a shlex example in the subprocess.Popen docs is confusing
type: enhancement
versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13817] deadlock in subprocess while running several threads using Popen

[Antoine Pitrou]

Antoine Pitrou  added the comment:

I think you forgot to attach the patch :)

--
nosy: +pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13825] Datetime failing while reading active directory time attribute

[Chris Gill]

New submission from Chris Gill :

I believe I am having a similar issue to this: http://bugs.python.org/issue7150



I am in the middle of programming a quick script and now I cannot seem to get 
beyond this issue; as it is printing up the expiration times from the AD user 
listings (many of which print 1601 year) it finally fails after the same user 
account, I have compared accounts and the expiration is the same as other 
accounts in AD: which is set to (never)

any ideas of what's going on here and how I can bypass this error?


error report: 

Traceback (most recent call last):
... line 14, in 
print user.name + ": " + str(user.accountExpires)
  File "C:\Python27\lib\site-packages\active_directory.py", line 425, in 
__getattr__
self._delegate_map[name] = converter (attr)
  File "C:\Python27\lib\site-packages\active_directory.py", line 335, in 
convert_to_datetime
return ad_time_to_datetime (item)
  File "C:\Python27\lib\site-packages\active_directory.py", line 319, in 
ad_time_to_datetime
return BASE_TIME + delta
OverflowError: date value out of range



code:

import active_directory
from datetime import datetime,timedelta

##check AD for account expirations
users = active_directory.AD_object ("LDAP://ou=administration,dc=domain,dc=com")
for user in users.search (objectCategory='Person'):

dn = user.distinguishedName
dn = dn.encode("utf-8") #for the occasional apostrophe

if "Adjuncts" in str(dn):
print user.name + ": " + str(user.accountExpires)

print "done"


example output:

CN=John Hancock: 1601-01-01 00:00:00
CN=Jane Smith: 1601-01-01 00:00:00
...

--
components: IDLE, Windows
messages: 151622
nosy: Chris.Gill
priority: normal
severity: normal
status: open
title: Datetime failing while reading active directory time attribute
versions: Python 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13817] deadlock in subprocess while running several threads using Popen

[Charles-François Natali]

Charles-François Natali  added the comment:

Here's the backtrace:

"""
#0  0x003bfb20c9b1 in sem_wait () from /lib64/libpthread.so.0
#1  0x0051a7c3 in PyThread_acquire_lock (lock=0x17db0750, waitflag=1)
at Python/thread_pthread.h:321
#2  0x0051a9b4 in find_key (key=1, value=0x0) at Python/thread.c:268
#3  0x0051abdc in PyThread_get_key_value (key=1) at Python/thread.c:360
#4  0x005025b1 in PyGILState_GetThisThreadState () at 
Python/pystate.c:598
#5  0x005024f5 in _PyGILState_Reinit () at Python/pystate.c:547
#6  0x00521fc7 in PyOS_AfterFork () at ./Modules/signalmodule.c:979
#7  0x0052461d in posix_fork (self=0x0, noargs=0x0) at 
./Modules/posixmodule.c:3695
"""

It's stuck in _PyGILState_Reinit(), when calling 
PyGILState_GetThisThreadState().
That's because in 2.7, TLS is emulated (see Python/thread.c), and it uses a 
global mutex.
If this mutex is locked at the time of fork(), then the next call to TLS 
primitives (even PyGILState_GetThisThreadState()) will deadlock.

Now, this particular bug is fixed in 2.7 since #13156, which backed-out  
_PyGILState_Reinit() because it was only relevant for native TLS 
implementations.

The code is still present in 3.2 and and default, but this problem doesn't 
affect native TLS implementations.

Just to be extra safe, we PyThread_ReInitTLS() - which resets this global mutex 
on emulated implementations, and is just a no-op on pthread and windows - 
should be moved earlier in PyOS_AfterFork(), to avoid this type of deadlock (I 
mean, PyGILState_GetThisThreadState() deadlock after fork() is bad).
Patch attached.

--
keywords: +needs review, patch
stage:  -> patch review
versions: +Python 3.2, Python 3.3 -Python 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13703] Hash collision security issue

[STINNER Victor]

STINNER Victor  added the comment:

I tried the collision counting with a low number of collisions:

less than 15 collisions
---

Fail at startup.

5 collisions (32 buckets, 21 used=65.6%): hash=ceb3152f => f
10 collisions (32 buckets, 21 used=65.6%): hash=ceb3152f => f

dict((str(k), 0) for k in range(200))
-

15 collisions (32,768 buckets, 18024 used=55.0%): hash=0e4631d2 => 31d2
20 collisions (131,072 buckets, 81568 used=62.2%): hash=12660719 => 719
25 collisions (1,048,576 buckets, 643992 used=61.4%): hash=6a1f6d21 => f6d21
30 collisions (1,048,576 buckets, 643992 used=61.4%): hash=6a1f6d21 => f6d21
35 collisions => ? (more than 10,000,000 integers)

random_dict('', 5, charset, 1, 3)
--

charset = 'abcdefghijklmnopqrstuvwxyz0123456789'

15 collisions (8192 buckets, 5083 used=62.0%): hash=1526677a => 77a
20 collisions (32768 buckets, 19098 used=58.3%): hash=5d7760e6 => 60e6
25 collisions => 

random_dict('', 5, charset, 1, 3)
--

charset = 
'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789.=+_(){}%'

15 collisions (32768 buckets, 20572 used=62.8%): hash=789fe1e6 => 61e6
20 collisions (2048 buckets, 1297 used=63.3%): hash=2052533d => 33d
25 collisions => nope

random_dict('', 5, charset, 1, 10)
--

charset = 
'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789.=+_(){}%'

15 collisions (32768 buckets, 18964 used=57.9%): hash=94d7c4f5 => 44f5
20 collisions (32768 buckets, 21548 used=65.8%): hash=acb5b39e => 339e
25 collisions (8192 buckets, 5395 used=65.9%): hash=04d367ae => 7ae
30 collisions => nope

random_dict() comes from the following script:
***
import random

def random_string(charset, minlen, maxlen):
strlen = random.randint(minlen, maxlen)
return ''.join(random.choice(charset) for index in xrange(strlen))

def random_dict(prefix, count, charset, minlen, maxlen):
dico = {}
keys = set()
for index in xrange(count):
for tries in xrange(1):
key = prefix + random_string(charset, minlen, maxlen)
if key in keys:
continue
keys.add(key)
break
else:
raise ValueError("unable to generate a new key")
dico[key] = None

charset = 
'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789.=+_(){}%'
charset = 'abcdefghijklmnopqrstuvwxyz0123456789'
random_dict('', 5, charset, 1, 3)
***

I ran the Django test suite. With a limit of 20 collisions, 60 tests
fail. With a limit of 50 collisions, there is no failure. But I don't
think that the test suite uses large data sets.

I also triend the Django test suite with a randomized hash function.
There are 46 failures. Many (all?) are related to the order of dict
keys: repr(dict) or indirectly in a HTML output. I didn't analyze all
failures. I suppose that Django can simply run the test suite using
PYTHONHASHSEED=0 (disable the randomized hash function), at least in a
first time.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13763] Potentially hard to understand wording in devguide

[Éric Araujo]

Éric Araujo  added the comment:

Thanks for commenting.  Please commit your wording, or this alternate version:

“``hg`` is the name of the Mercurial command-line program, and is often used as 
an abbreviation for "Mercurial".”

(Just to avoid repetition and to add “command-line”.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9631] Python 2.7 installation issue for Linux gcc-4.1.0-3 (Fedora Core 5?)

[mike]

mike  added the comment:

Hi,

I downloaded source and did the following instructions.


We use Red Hat Enterprise Linux Server release 5.5.

./configure --prefix=/home/mike/python_rh_32
make
make install

I also changed the line in site.py

from:

s = os.path.join(os.path.dirname(sys.path.pop()), s)

to:

s = os.path.join(os.path.dirname(sys.path[-1]), s)

but I still get:

/usr/bin/install -c -m 644 ./LICENSE 
/home/mike/python_rh_32/lib/python2.7/LICENSE.txt
PYTHONPATH=/home/mike/python_rh_32/lib/python2.7   \
./python -Wi -tt 
/home/mike/python_rh_32/lib/python2.7/compileall.py \
-d /home/mike/python_rh_32/lib/python2.7 -f \
-x 'bad_coding|badsyntax|site-packages|lib2to3/tests/data' \
/home/mike/python_rh_32/lib/python2.7
Traceback (most recent call last):
  File "/home/mike/python_rh_32/lib/python2.7/compileall.py", line 16, in 

import struct
  File "/home/mike/python_rh_32/lib/python2.7/struct.py", line 1, in 
from _struct import *
ImportError: No module named _struct
make: *** [libinstall] Error 1


Did anyone slove this problem?

--
nosy: +eraonel

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13703] Hash collision security issue

[Éric Araujo]

Éric Araujo  added the comment:

> Even Lib/packaging/create.py change is related to fixing tests. The test can 
> be changed
> differently, but I like the idea of having always the same output in 
> packaging (e.g. it is
> more readable for the user if files are sorted).

See #13712 for why this is a fake fix.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13813] "sysconfig.py" and "distutils/util.py" redundancy

[Éric Araujo]

Éric Araujo  added the comment:

Background on the distutils freeze?  Right now I don’t have the time and I’m 
going to be offline until the end of the month.  You can look for Tarek Ziadé’s 
blog posts after PyCon 2010, or wait until I come back and put the links on a 
wiki page :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13817] deadlock in subprocess while running several threads using Popen

[Antoine Pitrou]

Changes by Antoine Pitrou :


--
nosy: +neologix

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13824] argparse.FileType opens a file without excepting resposibility for closing it

[David Layton]

New submission from David Layton :

argparse.FileType.__call__ opens the specified file and returns it. This is 
well documented as an anit-idiom in 
http://docs.python.org/howto/doanddont.html#exceptions. 

"...a serious problem — due to implementation details in CPython, the file 
would not be closed when an exception is raised until the exception handler 
finishes; and, worse, in other implementations (e.g., Jython) it might not be 
closed at all regardless of whether or not an exception is raised."

Disregarding the above, handling a file which may or may not have been opened 
depending the users input requires a bit of boilerplate code compared to the 
usual with-open idiom.  

Additionally, there is no way to prevent FileType from clobbering an existing 
file when used with write mode. 

Given these issues and others, it seems to me that the usefulness of FileType 
is outweighed by propensity to encourage bad coding. Perhaps, it would be best 
if FileType (or some replacement) simply checked that the file exists (when 
such a check is appropriate), it can be opened in the specified mode, and, 
curry the call to open (i.e. return lambda: open(string, self._mode, 
self._bufsize))

--
components: Library (Lib)
messages: 151615
nosy: David.Layton, Paolo.Elvati, Stefan.Pfeiffer, bethard, manveru
priority: normal
severity: normal
status: open
title: argparse.FileType opens a file without excepting resposibility for 
closing it
type: behavior
versions: Python 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11948] Tutorial/Modules - small fix to better clarify the modules search path

[Sandro Tosi]

Sandro Tosi  added the comment:

Thanks Terry for the ping, I've just committed it - thanks again to everyone 
for the help/suggestions.

--
resolution:  -> fixed
stage: commit review -> committed/rejected
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11948] Tutorial/Modules - small fix to better clarify the modules search path

[Roundup Robot]

Roundup Robot  added the comment:

New changeset d01208ba482f by Sandro Tosi in branch '2.7':
Issue #11948: clarify modules search path
http://hg.python.org/cpython/rev/d01208ba482f

New changeset 6d663db63705 by Sandro Tosi in branch '3.2':
Issue #11948: clarify modules search path
http://hg.python.org/cpython/rev/6d663db63705

New changeset 93769b8ff40b by Sandro Tosi in branch 'default':
Issue #11948: merge with 3.2
http://hg.python.org/cpython/rev/93769b8ff40b

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13823] xml.etree.ElementTree.ElementTree.write - argument checking

[patrick vrijlandt]

New submission from patrick vrijlandt :

(1) The docs say:
xml_declaration controls if an XML declaration should be added to the file. Use 
False for never, True for always, None for only if not US-ASCII or UTF-8 or 
Unicode (default is None). 

The method also accepts other values, like xml_declaration = "yes". This 
behavior should be documented, or raise a ValueError (up to effbot, I think)

(2) The docs say (in a note):
The encoding string included in XML output should conform to the appropriate 
standards. For example, “UTF-8” is valid, but “UTF8” is not. See 
http://www.w3.org/

But the method accepts both values, (“UTF-8” and “UTF8”). Since this will 
result in invalid xml, (but not invalid python) it should probably raise 
ValueError too.

(3) Open issue 9458 also refers to this method. It might be wise to raise 
ValueError if the encoding does not match the (mode of the) file target (binary 
or text).

--
assignee: docs@python
components: Documentation, XML
messages: 151612
nosy: docs@python, patrick.vrijlandt
priority: normal
severity: normal
status: open
title: xml.etree.ElementTree.ElementTree.write - argument checking
versions: Python 3.2

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13804] Python library structure creates hard to read code when using higher order functions

[Martin Häcker]

Martin Häcker  added the comment:

Jup - oh the joys of writing code in a bugtracker :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com