[issue8525] Small enhancement to help()

2010-04-25 Thread Georg Brandl

Changes by Georg Brandl ge...@python.org:


--
assignee:  - georg.brandl
nosy: +georg.brandl

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8525
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue7865] io close() swallowing exceptions

2010-04-25 Thread Pascal Chambon

Changes by Pascal Chambon chambon.pas...@gmail.com:


Removed file: http://bugs.python.org/file17046/release_io_close_exceptions.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7865
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue7865] io close() swallowing exceptions

2010-04-25 Thread Pascal Chambon

Pascal Chambon chambon.pas...@gmail.com added the comment:

SHould be better this way then B-)

--
Added file: http://bugs.python.org/file17077/no_swallow_on_close2.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7865
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8528] typo in argparse documentation

2010-04-25 Thread akira

New submission from akira 4kir4...@gmail.com:

`messges` should be replaced by `messages` on 
http://docs.python.org/dev/library/argparse.html#upgrading-optparse-code page.

--
assignee: d...@python
components: Documentation
messages: 104144
nosy: akira, d...@python
severity: normal
status: open
title: typo in argparse documentation
type: behavior

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8528
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8528] typo in argparse documentation

2010-04-25 Thread Georg Brandl

Georg Brandl ge...@python.org added the comment:

Thanks, fixed in r80460.

--
assignee: d...@python - 
nosy: +georg.brandl
resolution:  - fixed
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8528
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8522] enhacement proposal in howto/doanddont

2010-04-25 Thread Georg Brandl

Georg Brandl ge...@python.org added the comment:

Thanks, applied in r80461.

--
nosy: +georg.brandl
resolution:  - fixed
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8522
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue4147] xml.dom.minidom toprettyxml: omit whitespace for text-only elements

2010-04-25 Thread Dan Kenigsberg

Changes by Dan Kenigsberg dan...@redhat.com:


--
nosy: +danken

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4147
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue7946] Convoy effect with I/O bound threads and New GIL

2010-04-25 Thread Ray.Allen

Changes by Ray.Allen ysj@gmail.com:


--
nosy: +ysj.ray

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7946
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8514] Create fs_encode() and fs_decode() functions in os.path

2010-04-25 Thread STINNER Victor

STINNER Victor victor.stin...@haypocalc.com added the comment:

 Please follow the naming convention used in os.path. The functions
 would have to be called os.path.fsencode() and os.path.fsdecode().

Ok

 Other than that, I'm +0 on the patch: the sys.filesystemencoding
 logic doesn't really work well in practice - on Unix and BSD
 platforms, there's no such thing as a single system-wide file
 system

Today, most POSIX system uses utf8 by default for all partitions.  If you mount 
an USB key, CD-Rom or network shared directory with the wrong options, you may 
get filenames in a different encoding. But this issue is not about fixing your 
OS configuration, but helping the most common case: a system using the same 
encoding everywhere (for the whole file system).

You are still free to use directly the native OS type (unicode on Windows, 
bytes on other OS), ie. don't use fsencode()/fsdecode().

Python3 prefers unicode, eg. print expects an unicode string, not a byte 
string. I mean it's more pratical to use unicode everywhere in Python, and so 
fsencode()/fsdecode() can be really useful on POSIX systems.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8514
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8529] subclassing builtin class (str, unicode, list...) needs to override __getslice__

2010-04-25 Thread Florent Xicluna

New submission from Florent Xicluna florent.xicl...@gmail.com:

It looks like a bug, because __getslice__ is deprecated since 2.0.

If you subclass a builtin type and override the __getitem__ method, you need to 
override the (deprecated) __getslice__ method too.
And if you run your program with python -3, it 

Example script:


class Upper(unicode):

def __getitem__(self, index):
return unicode.__getitem__(self, index).upper()

#def __getslice__(self, i, j):
#return self[i:j:]


if __name__ == '__main__':
text = Upper('Lorem ipsum')

print text[:]
print text[::]

--
components: Interpreter Core
messages: 104148
nosy: flox
priority: normal
severity: normal
status: open
title: subclassing builtin class (str, unicode, list...) needs to override 
__getslice__
type: behavior
versions: Python 2.7

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8529
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8530] Stringlib fastsearch can read beyond the front of an array

2010-04-25 Thread Alex

New submission from Alex alex.gay...@gmail.com:

In Objects/stringlib/fastsearch.h the lines:

if (!STRINGLIB_BLOOM(mask, s[i-1]))

and

if (!STRINGLIB_BLOOM(mask, s[i-1]))

can read beyond the front of the array that is passed to it when the loop 
enters with i = 0.

I originally noticed this when porting the algorithm to PyPy (which has bounds 
checking :)), all tests pass if I simple add `if i-1 = 0` before the 
conditional.  This doesn't appear to actually cause the algorithm to ever 
break, but it is unsafe.

--
messages: 104149
nosy: alex
severity: normal
status: open
title: Stringlib fastsearch can read beyond the front of an array

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8530
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8530] Stringlib fastsearch can read beyond the front of an array

2010-04-25 Thread Antoine Pitrou

Changes by Antoine Pitrou pit...@free.fr:


--
components: +Interpreter Core
nosy: +flox
priority:  - normal
stage:  - needs patch
type:  - behavior
versions: +Python 2.6, Python 2.7, Python 3.1, Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8530
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8530] Stringlib fastsearch can read beyond the front of an array

2010-04-25 Thread Florent Xicluna

Florent Xicluna florent.xicl...@gmail.com added the comment:

I guess we don't have the same issue with the find() implementation?

 if (!STRINGLIB_BLOOM(mask, s[i+m]))


Because:
 * len(s) = n = (w + m)
 * the loop condition is (i = w)
  == s[w+m] is beyond the array, but it is '\0' probably

Is it correct?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8530
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8530] Stringlib fastsearch can read beyond the front of an array

2010-04-25 Thread Alex

Alex alex.gay...@gmail.com added the comment:

Yes, as the comment of the top of the file notes, reading to s[n] (where n == 
len(s)) is safe because strings are null padded.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8530
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8510] update to autoconf2.65

2010-04-25 Thread Matthias Klose

Matthias Klose d...@debian.org added the comment:

r80474: Replace AC_HELP_STRING with AS_HELP_STRING

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8510
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8530] Stringlib fastsearch can read beyond the front of an array

2010-04-25 Thread Florent Xicluna

Florent Xicluna florent.xicl...@gmail.com added the comment:

This patch should fix it.
Since there's no failure, I don't find any test to add.

--
keywords: +patch
stage: needs patch - patch review
Added file: http://bugs.python.org/file17078/issue8530_rfind.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8530
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue6085] Logging in BaseHTTPServer.BaseHTTPRequestHandler causes lag

2010-04-25 Thread Charles-Francois Natali

Charles-Francois Natali neolo...@free.fr added the comment:

The attached patch caches the result of FQDN lookup.

--
keywords: +patch
Added file: http://bugs.python.org/file17079/base_http_server_fqdn_lag.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6085
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8530] Stringlib fastsearch can read beyond the front of an array

2010-04-25 Thread Antoine Pitrou

Antoine Pitrou pit...@free.fr added the comment:

I can't manage to trigger any crash on a Linux machine, so I think we'll live 
without a test.

--
nosy: +pitrou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8530
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8530] Stringlib fastsearch can read beyond the front of an array

2010-04-25 Thread Antoine Pitrou

Antoine Pitrou pit...@free.fr added the comment:

Of course your patch might slow down the loop, so perhaps you want to run some 
benchmarks.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8530
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8530] Stringlib fastsearch can read beyond the front of an array

2010-04-25 Thread Antoine Pitrou

Changes by Antoine Pitrou pit...@free.fr:


--
versions:  -Python 2.6, Python 3.1

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8530
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8510] update to autoconf2.65

2010-04-25 Thread Matthias Klose

Matthias Klose d...@debian.org added the comment:

r80475: s/AC_AIX/AC_USE_SYSTEM_EXTENSIONS/

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8510
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8526] msilib doesn't support multiple CAB instances in same installer

2010-04-25 Thread Bill Janssen

Bill Janssen bill.jans...@gmail.com added the comment:

I'm certainly using the API provided by msilib, but perhaps I'm using it badly. 
 Which API did you have in mind?  I'm using msilib directly, not through 
bdist_msi.

This seems like an artificial limitation to put on the Python library; the 
standard MSI spec supports 65K different cabs in a single installer.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8526
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8523] shutil.rmtree and os.listdir cannot recover on error conditions

2010-04-25 Thread Brett Cannon

Changes by Brett Cannon br...@python.org:


--
assignee:  - tarek
nosy: +tarek

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8523
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue6085] Logging in BaseHTTPServer.BaseHTTPRequestHandler causes lag

2010-04-25 Thread Santoso Wijaya

Santoso Wijaya santa@me.com added the comment:

Doesn't that only cache the first remote client it encounters, though? Maybe a 
dictionary of caches?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6085
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8526] msilib doesn't support multiple CAB instances in same installer

2010-04-25 Thread Martin v . Löwis

Martin v. Löwis mar...@v.loewis.de added the comment:

I still don't see the need to create multiple CABs. Just use the Directory 
class to add files, and that will automatically record them in the singleton 
CAB.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8526
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue6085] Logging in BaseHTTPServer.BaseHTTPRequestHandler causes lag

2010-04-25 Thread Charles-Francois Natali

Changes by Charles-Francois Natali neolo...@free.fr:


Removed file: http://bugs.python.org/file17079/base_http_server_fqdn_lag.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6085
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue6483] Modules are not deallocated correctly if m_size = -1

2010-04-25 Thread Antoine Pitrou

Antoine Pitrou pit...@free.fr added the comment:

Well, in 2.x you didn't even have the choice. The dict of an extension module 
was always copied, becoming essentially immortal. In 3.x you can use an 
m_size=0 so as to disable this behaviour.

--
nosy: +pitrou
versions: +Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6483
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue5099] subprocess.POpen.__del__() AttributeError (os module == None!)

2010-04-25 Thread Brett Cannon

Brett Cannon br...@python.org added the comment:

New patch updated to at least r80476.

--
Added file: http://bugs.python.org/file17080/subprocess_shutdown.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5099
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue5099] subprocess.POpen.__del__() AttributeError (os module == None!)

2010-04-25 Thread Brett Cannon

Changes by Brett Cannon br...@python.org:


Removed file: http://bugs.python.org/file16976/subprocess__del__.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5099
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8510] update to autoconf2.65

2010-04-25 Thread Matthias Klose

Matthias Klose d...@debian.org added the comment:

r80478 converts all obsolete AC_TRY_* macros to AC_*_IFELSE, the outcome is ... 
nothing (but whitespace changes):

$ svn diff --diff-cmd diff -x -uEwB configure
Index: configure
===
--- configure   (Revision 80475)
+++ configure   (Arbeitskopie)
@@ -1,5 +1,5 @@
 #! /bin/sh
-# From configure.in Revision: 80474 .
+# From configure.in Revision: 80475 .
 # Guess values for system-dependent variables and create Makefiles.
 # Generated by GNU Autoconf 2.65 for python 3.2.
 #

running configure with the old and the new version shows no diffs (besides 
config.log).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8510
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue6085] Logging in BaseHTTPServer.BaseHTTPRequestHandler causes lag

2010-04-25 Thread Charles-Francois Natali

Charles-Francois Natali neolo...@free.fr added the comment:

 Doesn't that only cache the first remote client it encounters, though? Maybe 
 a dictionary of caches?

A BaseHTTPRequestHandler is instantiated every time a client connects, so there 
should be only one client per handler, no (the cache is an attribute of the 
handler, not the server) ?

--
Added file: http://bugs.python.org/file17081/base_http_server_fqdn_lag.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6085
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8530] Stringlib fastsearch can read beyond the front of an array

2010-04-25 Thread Benjamin Peterson

Benjamin Peterson benja...@python.org added the comment:

Why add a bounds check if it can't be caused to fail. How about just a comment?

--
nosy: +benjamin.peterson

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8530
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8530] Stringlib fastsearch can read beyond the front of an array

2010-04-25 Thread Alex

Alex alex.gay...@gmail.com added the comment:

Well, the fact that it hasn't been shown to fail doesn't mean it can't fail.  
It relies on reading undefined memory, which is usually bad ;).  However, since 
we're at i=0, regardless of what we add to the value it's going to end up 
terminating the loop, so I'm not sure if it can actually break in practice.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8530
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8510] update to autoconf2.65

2010-04-25 Thread Matthias Klose

Matthias Klose d...@debian.org added the comment:

r80481: configure.in: Avoid autoconf warning: Assume C89 semantics that
RETSIGTYPE is always void (issue #8510).

Keep the definition, although the python code itself doesn't use it anymore.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8510
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8510] update to autoconf2.65

2010-04-25 Thread Matthias Klose

Matthias Klose d...@debian.org added the comment:

r80483: Makefile.pre.in (autoconf): Call autoconf/autoheader with -Wall to help 
the configure script to stay warning free.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8510
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8530] Stringlib fastsearch can read beyond the front of an array

2010-04-25 Thread Antoine Pitrou

Antoine Pitrou pit...@free.fr added the comment:

It could read into an invalid page and segfault. It depends on specifics of the 
memory allocator.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8530
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8529] subclassing builtin class (str, unicode, list...) needs to override __getslice__

2010-04-25 Thread Benjamin Peterson

Benjamin Peterson benja...@python.org added the comment:

This is because unicode implements __getslice__.

--
nosy: +benjamin.peterson
resolution:  - invalid
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8529
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8529] subclassing builtin class (str, unicode, list...) needs to override __getslice__

2010-04-25 Thread Florent Xicluna

Florent Xicluna florent.xicl...@gmail.com added the comment:

OK, but it yields Python 3 DeprecationWarning in the subclass.
And there's no workaround to get rid of the deprecation. 

If it is the correct behaviour, maybe some words could be added about 
subclassing builtin types:
http://docs.python.org/reference/datamodel.html#additional-methods-for-emulation-of-sequence-types

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8529
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8529] subclassing builtin class (str, unicode, list...) needs to override __getslice__

2010-04-25 Thread Florent Xicluna

Florent Xicluna florent.xicl...@gmail.com added the comment:

OK, I said nothing, it is already in the doc.

:-)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8529
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue2302] Uses of SocketServer.BaseServer.shutdown have a race

2010-04-25 Thread Antoine Pitrou

Antoine Pitrou pit...@free.fr added the comment:

Fixed with a test in r80484 (trunk), r80486 (2.6), r80487 (py3k), r80491 (3.1).

--
resolution:  - fixed
stage:  - committed/rejected
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue2302
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue7319] Silence DeprecationWarning by default

2010-04-25 Thread Brett Cannon

Brett Cannon br...@python.org added the comment:

-Q now works like -3 by causing the DeprecationWarning silencing to be skipped. 
Committed in r80492.

--
resolution:  - fixed
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7319
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8391] os.execvpe() doesn't support surrogates in env

2010-04-25 Thread STINNER Victor

STINNER Victor victor.stin...@haypocalc.com added the comment:

I blocked the fix in Python 3.1 because it's non trivial and I prefer to avoid 
complex changes in Python 3.1. But then I realized that Python 3.1 has two bugs 
about environment variables.

It uses sys.getfilesystemencoding()+surrogateecape to decode variables and 
sys.getdefaultencoding()+strict to encode variables: the encoding is different!

It counts the number of *characters* to allocate the *byte* string buffer and 
so non-ASCII values are truncated.

So I decided to backport the fix: r80494.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8391
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue7288] Detect improper leading whitespace in C files for Vim

2010-04-25 Thread Brett Cannon

Changes by Brett Cannon br...@python.org:


--
resolution:  - wont fix
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7288
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8512] os.execv*e(): fix formatting of the environment variables (Python 3.1)

2010-04-25 Thread STINNER Victor

STINNER Victor victor.stin...@haypocalc.com added the comment:

I choosed to backport #8391 fix to Python 3.1 (which is better) instead of 
writing a custom (shorter but incomplete) patch for Python 3.1: see msg104175.

(os_execve_env.patch doesn't work if LANG environment variable is not set)

--
resolution:  - fixed
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8512
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue7319] Silence DeprecationWarning by default

2010-04-25 Thread Benjamin Peterson

Benjamin Peterson benja...@python.org added the comment:

This still hasn't been ported to py3.

--
resolution: fixed - 
status: closed - open

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7319
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue7319] Silence DeprecationWarning by default

2010-04-25 Thread Brett Cannon

Brett Cannon br...@python.org added the comment:

Making this a 3.2 issue now.

--
versions: +Python 3.2 -Python 2.7

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7319
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8531] libffi: selected processor does not support stfeqd/ldfd (ARMv7Thumb)

2010-04-25 Thread STINNER Victor

New submission from STINNER Victor victor.stin...@haypocalc.com:

Errors on ARMv7Thumb Ubuntu 3.1 buildbot.

http://www.python.org/dev/buildbot/builders/ARMv7Thumb Ubuntu 
3.1/builds/37/steps/compile/logs/stdio

creating build/temp.linux-armv7l-3.1-pydebug/libffi
checking build system type... armv7l-unknown-linux-gnu
checking host system type... armv7l-unknown-linux-gnu
checking target system type... armv7l-unknown-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
...
config.status: executing src commands
building '_ctypes' extension
creating 
build/temp.linux-armv7l-3.1-pydebug/home/doko/buildarea/3.1.klose-linux-arm/build/Modules/_ctypes/libffi
creating 
build/temp.linux-armv7l-3.1-pydebug/home/doko/buildarea/3.1.klose-linux-arm/build/Modules/_ctypes/libffi/src
creating 
build/temp.linux-armv7l-3.1-pydebug/home/doko/buildarea/3.1.klose-linux-arm/build/Modules/_ctypes/libffi/src/arm
gcc -pthread -fPIC -fno-strict-aliasing -g -O2 -g -Wall -Wstrict-prototypes 
-Ibuild/temp.linux-armv7l-3.1-pydebug/libffi/include 
-Ibuild/temp.linux-armv7l-3.1-pydebug/libffi 
-I/home/doko/buildarea/3.1.klose-linux-arm/build/Modules/_ctypes/libffi/src -I. 
-I./Include -I/usr/local/include -IInclude 
-I/home/doko/buildarea/3.1.klose-linux-arm/build -c 
/home/doko/buildarea/3.1.klose-linux-arm/build/Modules/_ctypes/_ctypes.c -o 
build/temp.linux-armv7l-3.1-pydebug/home/doko/buildarea/3.1.klose-linux-arm/build/Modules/_ctypes/_ctypes.o
gcc -pthread -fPIC -fno-strict-aliasing -g -O2 -g -Wall -Wstrict-prototypes 
-Ibuild/temp.linux-armv7l-3.1-pydebug/libffi/include 
-Ibuild/temp.linux-armv7l-3.1-pydebug/libffi 
-I/home/doko/buildarea/3.1.klose-linux-arm/build/Modules/_ctypes/libffi/src -I. 
-I./Include -I/usr/local/include -IInclude 
-I/home/doko/buildarea/3.1.klose-linux-arm/build -c 
/home/doko/buildarea/3.1.klose-linux-arm/build/Modules/_ctypes/callbacks.c -o 
build/temp.linux-armv7l-3.1-pydebug/home/doko/buildarea/3.1.klose-linux-arm/build/Modules/_ctypes/callbacks.o
gcc -pthread -fPIC -fno-strict-aliasing -g -O2 -g -Wall -Wstrict-prototypes 
-Ibuild/temp.linux-armv7l-3.1-pydebug/libffi/include 
-Ibuild/temp.linux-armv7l-3.1-pydebug/libffi 
-I/home/doko/buildarea/3.1.klose-linux-arm/build/Modules/_ctypes/libffi/src -I. 
-I./Include -I/usr/local/include -IInclude 
-I/home/doko/buildarea/3.1.klose-linux-arm/build -c 
/home/doko/buildarea/3.1.klose-linux-arm/build/Modules/_ctypes/callproc.c -o 
build/temp.linux-armv7l-3.1-pydebug/home/doko/buildarea/3.1.klose-linux-arm/build/Modules/_ctypes/callproc.o
gcc -pthread -fPIC -fno-strict-aliasing -g -O2 -g -Wall -Wstrict-prototypes 
-Ibuild/temp.linux-armv7l-3.1-pydebug/libffi/include 
-Ibuild/temp.linux-armv7l-3.1-pydebug/libffi 
-I/home/doko/buildarea/3.1.klose-linux-arm/build/Modules/_ctypes/libffi/src -I. 
-I./Include -I/usr/local/include -IInclude 
-I/home/doko/buildarea/3.1.klose-linux-arm/build -c 
/home/doko/buildarea/3.1.klose-linux-arm/build/Modules/_ctypes/stgdict.c -o 
build/temp.linux-armv7l-3.1-pydebug/home/doko/buildarea/3.1.klose-linux-arm/build/Modules/_ctypes/stgdict.o
gcc -pthread -fPIC -fno-strict-aliasing -g -O2 -g -Wall -Wstrict-prototypes 
-Ibuild/temp.linux-armv7l-3.1-pydebug/libffi/include 
-Ibuild/temp.linux-armv7l-3.1-pydebug/libffi 
-I/home/doko/buildarea/3.1.klose-linux-arm/build/Modules/_ctypes/libffi/src -I. 
-I./Include -I/usr/local/include -IInclude 
-I/home/doko/buildarea/3.1.klose-linux-arm/build -c 
/home/doko/buildarea/3.1.klose-linux-arm/build/Modules/_ctypes/cfield.c -o 
build/temp.linux-armv7l-3.1-pydebug/home/doko/buildarea/3.1.klose-linux-arm/build/Modules/_ctypes/cfield.o
gcc -pthread -fPIC -fno-strict-aliasing -g -O2 -g -Wall -Wstrict-prototypes 
-Ibuild/temp.linux-armv7l-3.1-pydebug/libffi/include 
-Ibuild/temp.linux-armv7l-3.1-pydebug/libffi 
-I/home/doko/buildarea/3.1.klose-linux-arm/build/Modules/_ctypes/libffi/src -I. 
-I./Include -I/usr/local/include -IInclude 
-I/home/doko/buildarea/3.1.klose-linux-arm/build -c 
/home/doko/buildarea/3.1.klose-linux-arm/build/Modules/_ctypes/malloc_closure.c 
-o 
build/temp.linux-armv7l-3.1-pydebug/home/doko/buildarea/3.1.klose-linux-arm/build/Modules/_ctypes/malloc_closure.o
gcc -pthread -fPIC -fno-strict-aliasing -g -O2 -g -Wall -Wstrict-prototypes 
-Ibuild/temp.linux-armv7l-3.1-pydebug/libffi/include 
-Ibuild/temp.linux-armv7l-3.1-pydebug/libffi 
-I/home/doko/buildarea/3.1.klose-linux-arm/build/Modules/_ctypes/libffi/src -I. 
-I./Include -I/usr/local/include -IInclude 
-I/home/doko/buildarea/3.1.klose-linux-arm/build -c 
/home/doko/buildarea/3.1.klose-linux-arm/build/Modules/_ctypes/libffi/src/prep_cif.c
 -o 
build/temp.linux-armv7l-3.1-pydebug/home/doko/buildarea/3.1.klose-linux-arm/build/Modules/_ctypes/libffi/src/prep_cif.o
gcc -pthread -fPIC -fno-strict-aliasing -g -O2 -g -Wall -Wstrict-prototypes 
-Ibuild/temp.linux-armv7l-3.1-pydebug/libffi/include 
-Ibuild/temp.linux-armv7l-3.1-pydebug/libffi 
-I/home/doko/buildarea/3.1.klose-linux-arm/build/Modules/_ctypes/libffi/src 

[issue7319] Silence DeprecationWarning by default

2010-04-25 Thread Benjamin Peterson

Benjamin Peterson benja...@python.org added the comment:

Brett, by the way, you had spurious changes in subprocess in your last commit.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7319
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8531] libffi: selected processor does not support stfeqd/ldfd (ARMv7Thumb)

2010-04-25 Thread STINNER Victor

STINNER Victor victor.stin...@haypocalc.com added the comment:

Python 3.x doesn't have this issue because it uses libffi 3.0.9 and this 
version was fixed for ARMv7. Python 3.1 uses libffi 3.0.5.

The buildbot should install libffi 3.0.9 and set system libffi path to Python 
configure script.

... or fix test_ascii_formatd: skip the test if _ctypes module is missing.

See #8142 (libffi update to 3.0.9).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8531
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8531] test_ascii_formatd fails if ctypes module is missing

2010-04-25 Thread STINNER Victor

STINNER Victor victor.stin...@haypocalc.com added the comment:

This issue is specific to Python 3.1: test_ascii_formatd starts with 
PyOS_ascii_formatd is deprecated and not called from anywhere in Python 
itself. So this module is the only place it gets tested. Test that it works, 
and test that it's deprecated.

I prefer to fix the test than the buildbot because _ctypes compilation may fail 
on different CPU/OS, ctypes should not be mandatory.

Fixed by r80495.

--
status: open - closed
title: libffi: selected processor does not support stfeqd/ldfd (ARMv7Thumb) - 
test_ascii_formatd fails if ctypes module is missing

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8531
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue7319] Silence DeprecationWarning by default

2010-04-25 Thread Brett Cannon

Brett Cannon br...@python.org added the comment:

I'm tired of svn. I swear I made sure I didn't list that file in what was 
changed.

Anyway, it's been reverted in r80496.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7319
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8531] test_ascii_formatd fails if ctypes module is missing

2010-04-25 Thread Matthias Klose

Matthias Klose d...@debian.org added the comment:

libffi 3.0.9 is installed. the buildbot master needs to pass 
--with-system-libffi to configure.

--
nosy: +doko

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8531
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8514] Create fsencode() and fsdecode() functions in os.path

2010-04-25 Thread STINNER Victor

STINNER Victor victor.stin...@haypocalc.com added the comment:

Update path: rename fs_encode/fs_decode to fsencode/fsdecode.

--
title: Create fs_encode() and fs_decode() functions in os.path - Create 
fsencode() and fsdecode() functions in os.path
Added file: http://bugs.python.org/file17082/os_path_fs_encode_decode-2.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8514
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8514] Create fsencode() and fsdecode() functions in os.path

2010-04-25 Thread STINNER Victor

Changes by STINNER Victor victor.stin...@haypocalc.com:


Removed file: http://bugs.python.org/file17061/os_path_fs_encode_decode.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8514
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8514] Create fsencode() and fsdecode() functions in os.path

2010-04-25 Thread STINNER Victor

STINNER Victor victor.stin...@haypocalc.com added the comment:

Oops, Update path: I mean Update patch ;-)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8514
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8531] test_ascii_formatd fails if ctypes module is missing

2010-04-25 Thread STINNER Victor

STINNER Victor victor.stin...@haypocalc.com added the comment:

 libffi 3.0.9 is installed. the buildbot master needs to pass 
 --with-system-libffi to configure.

I fixed this issue by fixing my initial problem: test_ascii_formatd failure. 
Would you like to contact buildbot master owner to change the configuration? 
(are you the owner?) I don't want to work on this anymore.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8531
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue6462] bsddb3 intermittent test failures

2010-04-25 Thread STINNER Victor

STINNER Victor victor.stin...@haypocalc.com added the comment:

Is the following error related to this issue?

http://www.python.org/dev/buildbot/builders/x86 Windows7 
2.6/builds/256/steps/test/logs/stdio

test test_bsddb3 failed -- Traceback (most recent call last):
  File 
D:\cygwin\home\db3l\buildarea\2.6.bolen-windows7\build\lib\bsddb\test\test_lock.py,
 line 127, in test03_lock_timeout
self.assertTrue((end_time-start_time) = 0.0999)
AssertionError

--
nosy: +haypo

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6462
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue4171] SSL handshake fails after TCP connection in getpeername()

2010-04-25 Thread Antoine Pitrou

Antoine Pitrou pit...@free.fr added the comment:

What happens if you remove the call to settimeout()?
Also, it would be nice if you could try with the latest py3k checkout. There's 
a couple of fixes for do_handshake there (including timeout issues).

--
nosy: +pitrou
priority:  - normal
versions: +Python 3.1, Python 3.2 -Python 3.0

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4171
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue7672] _ssl module overwrites existing thread safety callbacks

2010-04-25 Thread Antoine Pitrou

Antoine Pitrou pit...@free.fr added the comment:

I don't think asking Python extensions to be safe in the face of brutal DLL 
unloading is wise. I don't think any of our C extensions tries to respect such 
an use case. We could still try to support the use case of some application 
registering its own callbacks before importing _ssl.

As for the patch, it looks wrong since CRYPTO_get_locking_callback() is 
compared to NULL a second time after the callback has been set to something 
non-NULL.

--
nosy: +pitrou
priority:  - normal
title: _ssl module causes segfault - _ssl module overwrites existing thread 
safety callbacks
versions: +Python 2.7, Python 3.1, Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7672
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8523] shutil.rmtree and os.listdir cannot recover on error conditions

2010-04-25 Thread R. David Murray

R. David Murray rdmur...@bitdance.com added the comment:

If solution 1 is acceptable in the general case, then I think a better fix  
would look like this:

try:
names = os.listdir(path)
except os.error, err:
onerror(os.listdir, path, sys.exc_info())
return

That is, this is another case in which we can't continue even if onerror 
returns.  However, onerror is free to correct the problem and then call rmtree. 
 (The danger, of course, is infinite recursion, but I don't think it is our 
responsibility to protect the author of an onerror handler from that potential 
mistake.)

By analogy to the other place rmtree returns after an onerror call, the above 
fix does fix a real bug, regardless of the disposition of the feature request, 
since currently if onerror returns we get a name error.

--
components: +Library (Lib) -Extension Modules
keywords: +easy
nosy: +r.david.murray
priority:  - low
stage:  - unit test needed
type:  - behavior
versions: +Python 2.6, Python 2.7, Python 3.1, Python 3.2 -Python 2.5

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8523
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8532] Refinements to Python 3 New GIL

2010-04-25 Thread David Beazley

New submission from David Beazley d...@dabeaz.com:

The attached patch makes two simple refinements to the new GIL implemented in 
Python 3.2.   Each is briefly described below.

1. Changed mechanism for thread time expiration

In the current implementation, threads perform a timed-wait on a condition 
variable.  If time expires and no thread switches have occurred, the currently 
running thread is forced to drop the GIL.

In the patch, timeouts are now performed by a special GIL monitor thread.  
This thread runs independently of Python and simply handles time expiration.  
Basically, it records the number of thread switches, sleeps for a specified 
interval (5ms), and then looks at the number of thread switches again.  If no 
switches occurred, it forces the currently running thread to drop the GIL.

With this monitor thread, it is no longer necessary to perform any timed 
condition variable waits.  This approach has a few subtle benefits.  First, 
threads no longer sit in a wait/timeout cycle when trying to get the GIL (so, 
there is less overhead).   Second, you get FIFO scheduling of threads.  When 
time expires, the thread that has been waiting the longest on the condition 
variable runs next.  Generally, you want this.

2. A very simple two-level priority mechanism

A new attribute 'cpu_bound' is added to the PyThreadState structure.  If a 
thread is ever forced to drop the GIL, this attribute is simply set True (1).  
If a thread gives up the GIL voluntarily, it is set back to False (0).  This 
attribute is used to set up simple scheduling (described next).

There are now two separate condition variables (gil_cpu_cond) and (gil_io_cond) 
that separate waiting threads according to their cpu_bound attribute setting.  
CPU-bound threads wait on gil_cpu_cond whereas I/O-bound threads wait on 
gil_io_cond. 

Using the two condition variables, the following scheduling rules are enforced:

   - If there are any waiting I/O bound threads, they are always signaled 
first, before any CPU-bound threads.
   - If an I/O bound thread wants the GIL, but a CPU-bound thread is running, 
the CPU-bound thread is immediately forced to drop the GIL.
   - If a CPU-bound thread wants the GIL, but another CPU-bound thread is 
running, the running thread is immediately forced to drop the GIL if its time 
period has already expired.

Results
---
This patch gives excellent results for both the ccbench test and all of my 
previous I/O bound tests.  Here is the output:

== CPython 3.2a0.0 (py3k:80470:80497M) ==
== i386 Darwin on 'i386' ==

--- Throughput ---

Pi calculation (Python)

threads=1: 871 iterations/s.
threads=2: 844 ( 96 %)
threads=3: 838 ( 96 %)
threads=4: 826 ( 94 %)

regular expression (C)

threads=1: 367 iterations/s.
threads=2: 345 ( 94 %)
threads=3: 339 ( 92 %)
threads=4: 327 ( 89 %)

bz2 compression (C)

threads=1: 384 iterations/s.
threads=2: 728 ( 189 %)
threads=3: 695 ( 180 %)
threads=4: 707 ( 184 %)

--- Latency ---

Background CPU task: Pi calculation (Python)

CPU threads=0: 0 ms. (std dev: 0 ms.)
CPU threads=1: 0 ms. (std dev: 0 ms.)
CPU threads=2: 1 ms. (std dev: 2 ms.)
CPU threads=3: 0 ms. (std dev: 1 ms.)
CPU threads=4: 0 ms. (std dev: 1 ms.)

Background CPU task: regular expression (C)

CPU threads=0: 0 ms. (std dev: 0 ms.)
CPU threads=1: 2 ms. (std dev: 1 ms.)
CPU threads=2: 1 ms. (std dev: 1 ms.)
CPU threads=3: 1 ms. (std dev: 1 ms.)
CPU threads=4: 2 ms. (std dev: 1 ms.)

Background CPU task: bz2 compression (C)

CPU threads=0: 0 ms. (std dev: 0 ms.)
CPU threads=1: 0 ms. (std dev: 2 ms.)
CPU threads=2: 2 ms. (std dev: 3 ms.)
CPU threads=3: 0 ms. (std dev: 1 ms.)
CPU threads=4: 0 ms. (std dev: 1 ms.)

--- I/O bandwidth ---

Background CPU task: Pi calculation (Python)

CPU threads=0: 5850.9 packets/s.
CPU threads=1: 5246.8 ( 89 %)
CPU threads=2: 4228.9 ( 72 %)
CPU threads=3: 4222.8 ( 72 %)
CPU threads=4: 2959.5 ( 50 %)

Particular attention should be given to tests involving I/O performance.  In 
particular, here are the results of the I/O bandwidth test using the unmodified 
GIL:

--- I/O bandwidth ---

Background CPU task: Pi calculation (Python)

CPU threads=0: 6007.1 packets/s.
CPU threads=1: 189.0 ( 3 %)
CPU threads=2: 19.7 ( 0 %)
CPU threads=3: 19.7 ( 0 %)
CPU threads=4: 5.1 ( 0 %)

Other Benefits
--
This patch does not involve any complicated libraries, platform specific 
functionality, low-level lock twiddling, or mathematically complex priority 
scheduling algorithms.  Emphasize: The code is simple.

Negative Aspects

This modification might introduce a starvation effect where CPU-bound threads 
never get to run if there is an extremely heavy load of I/O-bound threads 
competing for the GIL.

Is starvation a real problem or a theoretical problem?  Hard to say. Would need 
study.

--
components: Interpreter Core
files: gil.patch
keywords: patch
messages: 104192
nosy: dabeaz
severity: normal
status: open
title: Refinements to Python 3 New GIL
type: 

[issue8532] Refinements to Python 3 New GIL

2010-04-25 Thread David Beazley

David Beazley d...@dabeaz.com added the comment:

Can't decide whether this should be attached to Issue 7946 or not.
I will also post it there.  (Feel free to close this issue if you want to keep 
7946 alive).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8532
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue7946] Convoy effect with I/O bound threads and New GIL

2010-04-25 Thread David Beazley

David Beazley d...@dabeaz.com added the comment:

The attached patch makes two simple refinements to the new GIL implemented in 
Python 3.2.   Each is briefly described below.

1. Changed mechanism for thread time expiration

In the current implementation, threads perform a timed-wait on a condition 
variable.  If time expires and no thread switches have occurred, the currently 
running thread is forced to drop the GIL.

In the patch, timeouts are now performed by a special GIL monitor thread.  
This thread runs independently of Python and simply handles time expiration.  
Basically, it records the number of thread switches, sleeps for a specified 
interval (5ms), and then looks at the number of thread switches again.  If no 
switches occurred, it forces the currently running thread to drop the GIL.

With this monitor thread, it is no longer necessary to perform any timed 
condition variable waits.  This approach has a few subtle benefits.  First, 
threads no longer sit in a wait/timeout cycle when trying to get the GIL (so, 
there is less overhead).   Second, you get FIFO scheduling of threads.  When 
time expires, the thread that has been waiting the longest on the condition 
variable runs next.  Generally, you want this.

2. A very simple two-level priority mechanism

A new attribute 'cpu_bound' is added to the PyThreadState structure.  If a 
thread is ever forced to drop the GIL, this attribute is simply set True (1).  
If a thread gives up the GIL voluntarily, it is set back to False (0).  This 
attribute is used to set up simple scheduling (described next).

There are now two separate condition variables (gil_cpu_cond) and (gil_io_cond) 
that separate waiting threads according to their cpu_bound attribute setting.  
CPU-bound threads wait on gil_cpu_cond whereas I/O-bound threads wait on 
gil_io_cond. 

Using the two condition variables, the following scheduling rules are enforced:

   - If there are any waiting I/O bound threads, they are always signaled 
first, before any CPU-bound threads.
   - If an I/O bound thread wants the GIL, but a CPU-bound thread is running, 
the CPU-bound thread is immediately forced to drop the GIL.
   - If a CPU-bound thread wants the GIL, but another CPU-bound thread is 
running, the running thread is immediately forced to drop the GIL if its time 
period has already expired.

Results
---
This patch gives excellent results for both the ccbench test and all of my 
previous I/O bound tests.  Here is the output:

== CPython 3.2a0.0 (py3k:80470:80497M) ==
== i386 Darwin on 'i386' ==

--- Throughput ---

Pi calculation (Python)

threads=1: 871 iterations/s.
threads=2: 844 ( 96 %)
threads=3: 838 ( 96 %)
threads=4: 826 ( 94 %)

regular expression (C)

threads=1: 367 iterations/s.
threads=2: 345 ( 94 %)
threads=3: 339 ( 92 %)
threads=4: 327 ( 89 %)

bz2 compression (C)

threads=1: 384 iterations/s.
threads=2: 728 ( 189 %)
threads=3: 695 ( 180 %)
threads=4: 707 ( 184 %)

--- Latency ---

Background CPU task: Pi calculation (Python)

CPU threads=0: 0 ms. (std dev: 0 ms.)
CPU threads=1: 0 ms. (std dev: 0 ms.)
CPU threads=2: 1 ms. (std dev: 2 ms.)
CPU threads=3: 0 ms. (std dev: 1 ms.)
CPU threads=4: 0 ms. (std dev: 1 ms.)

Background CPU task: regular expression (C)

CPU threads=0: 0 ms. (std dev: 0 ms.)
CPU threads=1: 2 ms. (std dev: 1 ms.)
CPU threads=2: 1 ms. (std dev: 1 ms.)
CPU threads=3: 1 ms. (std dev: 1 ms.)
CPU threads=4: 2 ms. (std dev: 1 ms.)

Background CPU task: bz2 compression (C)

CPU threads=0: 0 ms. (std dev: 0 ms.)
CPU threads=1: 0 ms. (std dev: 2 ms.)
CPU threads=2: 2 ms. (std dev: 3 ms.)
CPU threads=3: 0 ms. (std dev: 1 ms.)
CPU threads=4: 0 ms. (std dev: 1 ms.)

--- I/O bandwidth ---

Background CPU task: Pi calculation (Python)

CPU threads=0: 5850.9 packets/s.
CPU threads=1: 5246.8 ( 89 %)
CPU threads=2: 4228.9 ( 72 %)
CPU threads=3: 4222.8 ( 72 %)
CPU threads=4: 2959.5 ( 50 %)

Particular attention should be given to tests involving I/O performance.  In 
particular, here are the results of the I/O bandwidth test using the unmodified 
GIL:

--- I/O bandwidth ---

Background CPU task: Pi calculation (Python)

CPU threads=0: 6007.1 packets/s.
CPU threads=1: 189.0 ( 3 %)
CPU threads=2: 19.7 ( 0 %)
CPU threads=3: 19.7 ( 0 %)
CPU threads=4: 5.1 ( 0 %)

Other Benefits
--
This patch does not involve any complicated libraries, platform specific 
functionality, low-level lock twiddling, or mathematically complex priority 
scheduling algorithms.  Emphasize: The code is simple.

Negative Aspects

This modification might introduce a starvation effect where CPU-bound threads 
never get to run if there is an extremely heavy load of I/O-bound threads 
competing for the GIL.

Comparison to BFS
-
Still need to test. Would be curious.

--
Added file: http://bugs.python.org/file17084/dabeaz_gil.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7946

[issue8532] Refinements to Python 3 New GIL

2010-04-25 Thread Alex

Changes by Alex alex.gay...@gmail.com:


--
nosy: +alex

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8532
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue7946] Convoy effect with I/O bound threads and New GIL

2010-04-25 Thread David Beazley

David Beazley d...@dabeaz.com added the comment:

One comment on that patch I just submitted. Basically, it's an attempt to make 
an extremely simple tweak to the GIL that fixes most of the problems discussed 
here in an extremely simple manner.  I don't have any special religious 
attachment to it though.  Would love to see a BFS comparison.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7946
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8532] Refinements to Python 3 New GIL

2010-04-25 Thread Brian Curtin

Changes by Brian Curtin cur...@acm.org:


--
nosy: +nirai
stage:  - patch review

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8532
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue4171] SSL handshake fails after TCP connection in getpeername()

2010-04-25 Thread Dmitry Dvoinikov

Dmitry Dvoinikov dmi...@targeted.org added the comment:

The problem does not reproduce in 3.1.1 nor in 3.1.2
(either x86 or x64).

Antoine Pitrou пишет:
 Antoine Pitrou pit...@free.fr added the comment:
 
 What happens if you remove the call to settimeout()?
 Also, it would be nice if you could try with the latest py3k checkout. 
 There's a couple of fixes for do_handshake there (including timeout issues).
 
 --
 nosy: +pitrou
 priority:  - normal
 versions: +Python 3.1, Python 3.2 -Python 3.0
 
 ___
 Python tracker rep...@bugs.python.org
 http://bugs.python.org/issue4171
 ___


--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4171
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue7946] Convoy effect with I/O bound threads and New GIL

2010-04-25 Thread David Beazley

David Beazley d...@dabeaz.com added the comment:

Here is the result of running the writes.py test with the patch I submitted.   
This is on OS-X.

bash-3.2$ ./python.exe writes.py
t1 2.83990693092 0
t2 3.27937912941 0
t1 5.54346394539 1
t2 6.68237304688 1
t1 8.9648039341 2
t2 9.60041999817 2
t1 12.1856160164 3
t2 12.5866689682 3
t1 15.3869640827 4
t2 15.7042851448 4
t1 18.4115200043 5
t2 18.5771169662 5
t2 21.4922711849 6
t1 21.6835460663 6
t2 24.6117911339 7
t1 24.9126679897 7
t1 27.1683580875 8
t2 28.2728791237 8
t1 29.4513950348 9
t1 32.2438161373 10
t2 32.5283250809 9
t1 34.8905010223 11
t2 36.0952250957 10
t1 38.109760046 12
t2 39.3465380669 11
t1 41.5758800507 13
t2 42.587772131 12
t1 45.1536290646 14
t2 45.8339021206 13
t1 48.6495029926 15
t2 49.1581180096 14
t1 51.5414950848 16
t2 52.6768190861 15
t1 54.818582058 17
t2 56.1163961887 16
t1 58.1549630165 18
t2 59.6944830418 17
t1 61.4515309334 19
t2 62.7685520649 18
t1 64.3223180771 20
t2 65.8158640862 19
65.8578810692

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7946
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue7946] Convoy effect with I/O bound threads and New GIL

2010-04-25 Thread Gregory P. Smith

Gregory P. Smith g...@krypto.org added the comment:

Nice dabeaz.

One potential concern with dabeaz_gil.patch 2010-04-25 21:13 is that it 
appears to always leave the gil_monitor thread running.  This is bad on 
mobile/embedded platforms where waking up at regular intervals prevents 
advanced sleep states and wastes power/battery.  (practical example: the OLPC 
project has run into this issue in other code in the past)

Could this be modified so that gil_monitor stops looping (blocks) so long as 
there are only IO bound Python threads running or while no python thread owns 
the GIL?

In that situation a multithreaded python process that has either reverted to 
one thread or has all threads blocked in IO would be truly idle rather than 
leaving the gil_monitor polling.

--
nosy: +gregory.p.smith

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7946
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8532] Refinements to Python 3 New GIL

2010-04-25 Thread Gregory P. Smith

Gregory P. Smith g...@krypto.org added the comment:

since the other discussion is ongoing on issue7946 i posted my comment there.

--
nosy: +gregory.p.smith

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8532
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8532] Refinements to Python 3 New GIL

2010-04-25 Thread Georg Brandl

Changes by Georg Brandl ge...@python.org:


--
assignee:  - pitrou
nosy: +pitrou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8532
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue8514] Create fsencode() and fsdecode() functions in os.path

2010-04-25 Thread Gregory P. Smith

Gregory P. Smith g...@krypto.org added the comment:

i'm +0.7 on fsencode/fsdecode going into os.path.

My bikeshed 0.7?  They're also useful for dealing with environment variables 
which are not strictly filesystem (fs) related but also suffer from the same 
issue requiring surrogate escape.  But other than just calling these os.encode 
and os.decode I don't have any brilliant alternate naming suggestions.  
thoughts?  I could easily live with os.path.fsencode/fsdecode, I just wanted to 
point the other use out.

--
nosy: +gregory.p.smith

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8514
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com