[issue4195] Regression for executing packages

[Nick Coghlan]

Nick Coghlan <[EMAIL PROTECTED]> added the comment:

Added revised version of patch with test cases and documentation
updates, as well as fixing a potential recursion issue with pathological
packages where __main__ was also a package)

--
keywords: +patch
Added file: 
http://bugs.python.org/file11885/issue4195_runpy_package_support.diff

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4012] Minor errors in multiprocessing docs

[Erick Tryzelaar]

Erick Tryzelaar <[EMAIL PROTECTED]> added the comment:

fyi, the multiprocessing docs also refer to applyAync, but the function 
was renamed to apply_async. This is also in the python 3.0 docs.

--
nosy: +erickt
versions: +Python 3.0

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4202] Multiple interpreters and readline module hook functions.

[Graham Dumpleton]

New submission from Graham Dumpleton <[EMAIL PROTECTED]>:

Because the readline module uses PyGILState_Ensure() to facilitate triggering 
callbacks into Python code, this would make the ability to use the hook 
functions incompatible with use in sub interpreters.

If this is the case, then that readline module cannot be used in sub 
interpreters should be documented if not already.

Better still, attempts to register hooks from sub interpreters should result in 
an exception. Further, when in use, in sub interpreter, callback hooks should 
also not be called if defined, because if defined they would be the hooks from 
the main interpreter since variables holding the hooks are static C variables 
and shared across all interpreters.

This issue derived from reading of code only and not tested in real program.

--
components: Interpreter Core
messages: 75201
nosy: grahamd
severity: normal
status: open
title: Multiple interpreters and readline module hook functions.
versions: Python 2.5, Python 3.0

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4200] atexit module not safe in Python 3.0 with multiple interpreters

[Graham Dumpleton]

Graham Dumpleton <[EMAIL PROTECTED]> added the comment:

I wouldn't be concerned about mod_python as likelihood that it will be ported 
is 
very low and even if it was it would be a long long way in the future. There is 
too 
much in mod_python itself unrelated to Python 3.0 that needs to be fixed before 
a 
port to Python 3.0 should be considered.

As to mod_wsgi, I can work around it in the short term by installing an 
'atexit' 
module replacement in sub interpreters and avoid the problem.

As to a solution, yes, using PyInterpreterState would seem the most logical 
place, 
however there is a lot more to it than that.

Prior to Python 3.0, any callbacks registered with atexit module in sub 
interpreters 
weren't called anyway. This is because Py_EndInterpreter() didn't trigger them, 
nor 
did Py_Finalize(). The latter is in Python 3.0 at the moment, but as pointed 
out 
that is a problem in itself.

So, although one may register sub interpreter atexit callbacks against 
PyInterpreterState, what would be done with them. A decision would need to be 
made 
as to whether Py_EndInterpreter() should trigger them, or whether the status 
quo be 
maintained and nothing done with them.

In the short term, ie., for Python 3.0.0, the simplest thing to do may be to 
have 
functions of atexit module silently not actually do anything for sub 
interpreters.

The only place this would probably cause a problem would be for mod_wsgi where 
it 
was itself calling sys.exitfunc() on sub interpreters to ensure they were run. 
Since 
mod_wsgi has to change for Python 3.0 anyway, to call atexit._run_exitfuncs, 
with a 
bit more work mod_wsgi can just replace atexit module altogether in sub 
interpreter 
context and have mod_wsgi track the callback functions and trigger them.

By having atexit module ignore stuff for sub interpreters, at least for now 
avoid 
problem of callbacks against sub interpreters being execute by Py_Finalize() in 
main 
interpreter context.

And no I haven't looked at how PEP 3121 has changed things in Python 3.0. Up 
till 
now I hadn't seen any problems to suggest I may need to look at it.

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4197] Doctest module does not work with zipped packages

[Alexander Belopolsky]

Changes by Alexander Belopolsky <[EMAIL PROTECTED]>:


--
versions: +Python 2.5.3, Python 2.6

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4201] Pdb cannot access source code in zipped packages.

[Alexander Belopolsky]

Changes by Alexander Belopolsky <[EMAIL PROTECTED]>:


--
components: +Library (Lib)
type:  -> behavior
versions: +Python 2.5.3

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4200] atexit module not safe in Python 3.0 with multiple interpreters

[Christian Heimes]

Christian Heimes <[EMAIL PROTECTED]> added the comment:

Ouch! We totally forgot about sub interpreters during the
reimplementation of the atexit module. 

In my opinion it's a big bug for systems like mod_python and mod_wsgi.
However it's too late to fix it for 3.0.0. Perhaps we can fix it in
3.0.1. We always knew that Python 3.0 is going to have some bugs that
can only be revealed by 3rd party code. I've added Barry to the nosy
list in order to get his opinion as release manager. 

Do you have a suggestion how the problem should be solved? I'd store the
atexit_callback information in PyInterpreterState instead of the module.
Maybe Martin's PEP 3121 could come into play to rescue us.

By the way PEP 3121 ... does the new module initialization code play
nice with multiple sub interpreters?

--
nosy: +barry, christian.heimes
priority:  -> critical
versions: +Python 3.1

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4015] [patch] make installed scripts executable on windows

[Mark Hammond]

Mark Hammond <[EMAIL PROTECTED]> added the comment:

I can see how this might be useful, but I agree it should not happen by
default, at least until it has been out for a while and feedback is
clear that people do want it by default.

I'd also like to find a way to pass all args, not just the first 9 -
that may well end up a source of bugs for people.  It might also be
worth considering that setuptools has the ability to create a true
executable from a script by using a stub executable.  Wouldn't that be
better still?  In the short term, maybe we should keep this
functionality in setuptools etc and let it filter back to distutils as
it proves itself?

--
nosy: +mhammond

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4199] add shorthand global and nonlocal statements

[Benjamin Peterson]

Changes by Benjamin Peterson <[EMAIL PROTECTED]>:


Added file: http://bugs.python.org/file11884/global_nonlocal_shorthand.patch

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4201] Pdb cannot access source code in zipped packages.

[Alexander Belopolsky]

Changes by Alexander Belopolsky <[EMAIL PROTECTED]>:


--
keywords: +patch
Added file: http://bugs.python.org/file11883/pdb.patch

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4201] Pdb cannot access source code in zipped packages.

[Alexander Belopolsky]

New submission from Alexander Belopolsky <[EMAIL PROTECTED]>:

With attached test.zip and svn revision 67006,

$ ./python.exe test.zip 
> /Users/sasha/Work/python-svn/trunk/test.zip/__main__.py(2)f()
(Pdb) l
[EOF]

With pdb.patch:

$ ./python.exe test.zip 
> /Users/sasha/Work/python-svn/trunk/test.zip/__main__.py(2)f()
-> pass
(Pdb) l
  1 def f():
  2  -> pass
  3 
  4 import pdb
  5 pdb.runcall(f)
  6 
  7 
[EOF]

--
files: test.zip
messages: 75197
nosy: belopolsky
severity: normal
status: open
title: Pdb cannot access source code in zipped packages.
Added file: http://bugs.python.org/file11882/test.zip

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2320] Race condition in subprocess using stdin

[Ben Creech]

Ben Creech <[EMAIL PROTECTED]> added the comment:

You can trigger this bug without use of stdin=subprocess.PIPE, or for
that matter any subprocess.PIPE's, although that makes it worse.  (So
the name for this issue may be overly specific.)

I have seen this happening intermittently in our parallel job manager,
and verified the lockup via gdb.

I've attached a regression test that should reliably trigger it without
use of any pipes for stdin, stderr, or stdout.  By kludging
subprocess.py with an extra "sleep", we can reliably trigger the race
condition.

The test launches two children, each from a thread.  Since there are no
calls to Popen.wait(), the main process should exit quickly, but it
never does.

Indeed, setting close_fds=True will correct the problem, although there
may be rare instances wherein the user wants the child to inherit some
file descriptors (for instance, this is how you usually use UNIX domain
sockets).

The problem is, as hinted at by Adam Olsen, as follows.

Say we have two threads, A and B, concurrently executing
subprocess.Popen.__init__.

In subprocess.Popen._execute_child, thread B is sitting between the
following lines:
errpipe_read, errpipe_write = os.pipe()
self._set_cloexec_flag(errpipe_write)

... when thread A calls fork().

Thus, thread A's child inherits both pipes, and never closes either one
when it calls execv.

Until thread A's child exits, that errpipe_write handle created in
thread B will remain open.

So, thread B will not return from the "os.read(errpipe_read...)" line,
and thus Popen.__init__ until thread A's child exits.  That is surely a bug.

On the other hand, it is hard to fix.  Ludwig Hähne suggested wrapping
the pipe creation and cloexec in a mutex.  As indicated by Gregory P.
Smith, that will not work reliably, as fork() could be called in one
thread while another thread is between those operations.  The mutex
would have to block the fork from happening during the atomic
create/cloexec operations.

Additionally, a parent could be creating a stdin output pipe end and
stdout/stderr input pipe ends for a particular subprocess.  Due to the
pooled nature of POSIX file descriptors, if another subprocess is
launched in a separate thread, it will inherit those pipe ends!  That
could cause unforeseen artifacts such as Popen.stdout hanging after
termination of the process that stdout was intended (because ANOTHER
process also inherited those pipe ends, and is still running).

Anyways, like Mr. Olson said, this is a mess.  close_fds=True is
probably the best option.  Perhaps it would be best to add an argument
allowing for a list of FD's to NOT close, in case the user is
intentionally trying to pass on additional file descriptors (eg UNIX
domain sockets).

I would vote for a change to set close_fds=True by default, and adding
an explicit list of inherited_fds.  This is based on the logic that
setting close_fds=False is unsafe and should not be default.

If that is too extreme, foolish users like myself at least need a huge
warning in the documentation, and an inherited_fds parameter would still
be useful.

--
nosy: +bpcreech
Added file: http://bugs.python.org/file11881/subprocess_race_condition.py

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4199] add shorthand global and nonlocal statements

[Benjamin Peterson]

Changes by Benjamin Peterson <[EMAIL PROTECTED]>:


Removed file: 
http://bugs.python.org/file11880/global_nonlocal_short_assign.patch

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4200] atexit module not safe in Python 3.0 with multiple interpreters

[Graham Dumpleton]

New submission from Graham Dumpleton <[EMAIL PROTECTED]>:

In Python 3.0 the atexit module was translated from Python code to C code.

Prior to Python 3.0, because it was implemented at C code level, the list of 
callbacks was specific to each sub interpreter. In Python 3.0 that appears to 
no longer be the case, with the list of callbacks being maintained globally as 
static C variable across all interpreters.

The end result is that if a sub interpreter registers an atexit callback, that 
callback will be called within the context of the main interpreter on call of 
Py_Finalize(), and not in context of the sub interpreter against which it was 
registered.

Various problems could ensue from this depending on whether or not the sub 
interpreter had also since been destroyed.

Still need to validate the above, but from visual inspection looks to be a 
problem. Issue found because mod_wsgi will trigger atexit callbacks for a sub 
interpreter when the sub interpreter is being destroyed. This is because Python 
prior to 3.0 only did it from the main interpreter. In Python 3.0, this all 
seems to get messed up now as no isolation between callbacks for each 
interpreter.

For mod_wsgi case, since it is explicitly triggering atexit callbacks for sub 
interpreter, in doing so it is actually calling all registered atexit callbacks 
across all interpreters for each sub interpreter being destroyed. They then 
again get called for Py_Finalize().

Even if mod_wsgi weren't doing this, still a problem that Py_Finalize() calling 
atexit callbacks in context of main interpreter which were actually associated 
with sub interpreter.

For mod_wsgi, will probably end up installing own 'atexit' module variant in 
sub interpreters to ensure separation.

--
components: Interpreter Core
messages: 75195
nosy: grahamd
severity: normal
status: open
title: atexit module not safe in Python 3.0 with multiple interpreters
type: behavior
versions: Python 3.0

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4194] Miserable subprocess.Popen performance

[Winfried Plappert]

Changes by Winfried Plappert <[EMAIL PROTECTED]>:


--
type:  -> performance
versions: +Python 2.4, Python 2.5, Python 2.6

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC

[Koen van de Sande]

Koen van de Sande <[EMAIL PROTECTED]> added the comment:

Ah, I wasn't thinking it through. It is fine for all .pyd of course, 
because they don't have any extra dependencies. I was thinking of DLLs 
with extra dependencies (of which there is one, TK85.dll). There, 
leaving out the manifest will break it. But indeed, the patch is fine 
for .pyd building. My mistake!

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4199] add shorthand global and nonlocal statements

[Benjamin Peterson]

New submission from Benjamin Peterson <[EMAIL PROTECTED]>:

PEP 3104 says that the nonlocal and global statements should allow a
shorthand. ("global x; x = 3" == "global x = 3") This patch implements that.

--
components: Interpreter Core
files: global_nonlocal_short_assign.patch
keywords: patch
messages: 75193
nosy: benjamin.peterson
priority: high
severity: normal
status: open
title: add shorthand global and nonlocal statements
versions: Python 3.1
Added file: http://bugs.python.org/file11880/global_nonlocal_short_assign.patch

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4196] library documentation errors

[Benjamin Peterson]

Benjamin Peterson <[EMAIL PROTECTED]> added the comment:

Thanks for the report! Fixed in r67011.

--
nosy: +benjamin.peterson
resolution:  -> fixed
status: open -> closed

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4197] Doctest module does not work with zipped packages

[Alexander Belopolsky]

Alexander Belopolsky <[EMAIL PROTECTED]> added the comment:

I don't think there is an easy way to fix source file name reporting 
without injecting __file__ = '/' into the 
module namespace is zipimporter, but I don't know if any code relies on 
__file__ set to None.

Setting __file__ to (in the test case) testmodule.zip/__main__.py will 
eliminate the need for the first patch and the second patch will only need 
to pass globs to linecache.getlines() to fix the line number issue.

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4194] Miserable subprocess.Popen performance

[Winfried Plappert]

Winfried Plappert <[EMAIL PROTECTED]> added the comment:

Yes, I can confirm that the performance is lousy on Solaris.

Solaris-9/Python 2.5.1:
time with os.popen :  0.124045133591
time with subprocess.Popen :  1.60335588455

Solaris-9/Python 2.6:
time with os.popen :  0.115752220154
time with subprocess.Popen :  1.61768198013

Solaris-10/Python 2.4.4:
time with os.popen :  0.124130010605
time with subprocess.Popen :  1.45624995232

Solaris-10/Python 2.6:
time with os.popen :  0.113790988922
time with subprocess.Popen :  1.42739701271

All machines are idendital in processor speed: v240.

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4197] Doctest module does not work with zipped packages

[Alexander Belopolsky]

Alexander Belopolsky <[EMAIL PROTECTED]> added the comment:

The next patch, doctest-1.patch addresses the line number issue, but the 
file name is still wrong.

$ ./python.exe testmodule.zip 
**
File "__main__", line 2, in __main__.c
Failed example:
'line 2'
Expected nothing
Got:
'line 2'
**
1 items had failures:
   1 of   1 in __main__.c
***Test Failed*** 1 failures.

Added file: http://bugs.python.org/file11879/doctest-1.patch

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4197] Doctest module does not work with zipped packages

[Alexander Belopolsky]

Changes by Alexander Belopolsky <[EMAIL PROTECTED]>:


--
keywords: +patch
Added file: http://bugs.python.org/file11878/doctest.patch

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4197] Doctest module does not work with zipped packages

[Alexander Belopolsky]

Alexander Belopolsky <[EMAIL PROTECTED]> added the comment:

Attached patch addresses the crash, but two issues remain:

$ ./python.exe testmodule.zip 
**
File "__main__", line ?, in __main__.c
Failed example:
'line 2'
Expected nothing
Got:
'line 2'
**
1 items had failures:
   1 of   1 in __main__.c
***Test Failed*** 1 failures.


The line number for the failed example cannot be found and file name is 
reported as __main__.c instead of __main__.py

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4198] os.path.normcase gets fooled on windows with mapped linux network drive

[dawidjoubert]

New submission from dawidjoubert <[EMAIL PROTECTED]>:

The documentation for os.path.normcase reads the case gets normalized
based on the file system
http://docs.python.org/dev/library/os.path.html

Current documentation:
Normalize the case of a pathname. On Unix, this returns the path
unchanged; on case-insensitive filesystems, it converts the path to
lowercase. On Windows, it also converts forward slashes to backward slashes.

Problem:
When mounting a network drive (via Samba) onto windows the mapped
network drive will claim to be an NTFS network (on Windows XP). Where
the network drive is actually a Samba share this can cause problems with
case sensitivity versus case insensitivity.

More:
The Windows based file systems such as FAT, FAT32 and NTFS are as a
matter of fact case preservant but case insensitive. That is a file
saved with the name 'MyName' will retain its name as 'MyName' and a ls
(dir) command will returnr the file as 'MyName' however access to
'Myname' or 'myname' or 'MYNAME' will all still access the same file.

Solution:
I suggest we drop converting the case to lower case where the file
systems FAT, FAT32 and NTFS are involved

--
components: Windows
messages: 75187
nosy: dawidjoubert
severity: normal
status: open
title: os.path.normcase gets fooled on windows with mapped linux network drive
type: behavior
versions: Python 2.7

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4197] Doctest module does not work with zipped packages

[Alexander Belopolsky]

New submission from Alexander Belopolsky <[EMAIL PROTECTED]>:

With attached teestmodule.zip and svn revision 67006,

$ ./python.exe testmodule.zip 
Traceback (most recent call last):
  File "/Users/sasha/Work/python-svn/trunk/Lib/runpy.py", line 121, in 
_run_module_as_main
"__main__", fname, loader, pkg_name)
  File "/Users/sasha/Work/python-svn/trunk/Lib/runpy.py", line 34, in 
_run_code
exec code in run_globals
  File "testmodule.zip/__main__.py", line 7, in 
  File "/Users/sasha/Work/python-svn/trunk/Lib/doctest.py", line 1817, 
in testmod
for test in finder.find(m, name, globs=globs, 
extraglobs=extraglobs):
  File "/Users/sasha/Work/python-svn/trunk/Lib/doctest.py", line 842, in 
find
self._find(tests, obj, name, module, source_lines, globs, {})
  File "/Users/sasha/Work/python-svn/trunk/Lib/doctest.py", line 884, in 
_find
test = self._get_test(obj, name, module, globs, source_lines)
  File "/Users/sasha/Work/python-svn/trunk/Lib/doctest.py", line 965, in 
_get_test
if filename[-4:] in (".pyc", ".pyo"):
TypeError: 'NoneType' object is unsubscriptable

--
components: Library (Lib)
files: testmodule.zip
messages: 75186
nosy: belopolsky
severity: normal
status: open
title: Doctest module does not work with zipped packages
type: crash
versions: Python 2.7
Added file: http://bugs.python.org/file11877/testmodule.zip

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3238] backport python 3.0 language functionality to python 2.6 by adding 7 opcodes to ceval.c

[Martin v. Löwis]

Martin v. Löwis <[EMAIL PROTECTED]> added the comment:

> Added file: http://bugs.python.org/file11875/mario.jpg

Please don't use the bug tracker to upload unrelated code
(such as Numpy code - Numpy is not part of Python core).
I'm tempted to mark all these images as spam.

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4194] Miserable subprocess.Popen performance

[Winfried Plappert]

Winfried Plappert <[EMAIL PROTECTED]> added the comment:

Here are my figures from a different processor on Linux (Ubuntu):

[EMAIL PROTECTED]:~/python$ python2.5 popentest.py 
time with os.popen :  0.0478880405426
time with subprocess.Popen :  0.0596849918365
[EMAIL PROTECTED]:~/python$ python2.6 popentest.py 
time with os.popen :  0.0481159687042
time with subprocess.Popen :  0.0592260360718
uname -a: ... 2.6.24-19-generic #1 SMP Wed Aug 20 22:56:21 UTC 2008 i686
GNU/Linux

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC

[Martin v. Löwis]

Martin v. Löwis <[EMAIL PROTECTED]> added the comment:

> What would be the best way to integrate a search/replace
> operation like this in the Python build process? Is the manifest 
> generated as a seperate file here first as well, or is it directly 
> embedded?

What's wrong with Andre Heider's patch?

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4194] Miserable subprocess.Popen performance

[Skip Montanaro]

Skip Montanaro <[EMAIL PROTECTED]> added the comment:

I don't expect Python3 to be all that great io performance-wise yet.  
Still, for me on the Mac os.popen beats subprocess.Popen pretty handily:

  % python3.0 popentest.py
  time with os.popen :  0.874988
  time with subprocess.Popen :  2.112976

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4194] Miserable subprocess.Popen performance

[David W. Lambert]

David W. Lambert <[EMAIL PROTECTED]> added the comment:

cygwin Python 2.5.1  (similar)
time with os.popen :  0.43818802
time with subprocess.Popen :  0.36161035

linux python 2.4.2 (similar)
time with os.popen :  0.0344939231873
time with subprocess.Popen :  0.0354421138763

linux python3K (both awful)
time with os.popen :  0.715471029282
time with subprocess.Popen :  1.58627915382

--
nosy: +LambertDW

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4171] SSL handshake fails after TCP connection in getpeername()

[Bill Janssen]

Changes by Bill Janssen <[EMAIL PROTECTED]>:


--
assignee:  -> janssen

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4194] Miserable subprocess.Popen performance

[Skip Montanaro]

Skip Montanaro <[EMAIL PROTECTED]> added the comment:

Good suggestion Sameer.  I tried it out with Python 2.5 on a Linux host 
here and saw essentially identical results for the two alternatives (~ 
0.08s).

S

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4194] Miserable subprocess.Popen performance

[Sameer]

Sameer <[EMAIL PROTECTED]> added the comment:

The subprocess module does different things depending on whether the
systems are Windows or Posix. This could explain the bad performance on
Solaris and the Mac and the good performance on Windows. 

If this is correct, then we should see poor performance on Linux also.
Also, I don't have a box to test that out.

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4194] Miserable subprocess.Popen performance

[Skip Montanaro]

Skip Montanaro <[EMAIL PROTECTED]> added the comment:

Good question.  I don't think it's MacOSX-specific.  The original problem 
raised its ugly head on Solaris 10.  I don't have quite as many versions 
of Python available there, but the relative performance is still bad, 
roughly 0.05 for os.popen and 0.82 for subprocess.Popen.  Not as bad as on 
the Mac, but certainly not as good as what you saw.

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4194] Miserable subprocess.Popen performance

[Winfried Plappert]

Winfried Plappert <[EMAIL PROTECTED]> added the comment:

Hi Skip,

I find different measurements om Windows/XP: I copied the script and ran
it under Python 2.5.2 and Python 2.6 (as downloaded from
http://python.org/ftp/python/2.6/python-2.6.msi):

[EMAIL PROTECTED] /cygdrive/e/tmp $ python ./timing.py # 2.5.2
time with os.popen :  0.29631335
time with subprocess.Popen :  0.20368665

in other words: identical or subprocess.Popen even faster.

and it is getting even better with Python2.6:

[EMAIL PROTECTED] /cygdrive/e/tmp $ python26 timing.py # 2.6
time with os.popen :  0.26632425
time with subprocess.Popen :  0.1873624

So could that be a Mac OS issue?

--
nosy: +wplappert

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4196] library documentation errors

[David W. Lambert]

New submission from David W. Lambert <[EMAIL PROTECTED]>:

4 problems noted here.

Thanks, Dave.

1,2---problems 1 & 2 concern py<3k references:

http://docs.python.org/dev/3.0/library/multiprocessing.html


1) "apply(func[, args[, kwds]])
Equivalent of the apply() builtin function. It blocks till the result 
is ready."

Of course there isn't any longer an "apply" builtin.


2) "imap(func, iterable[, chunksize])
An equivalent of itertools.imap()."

Likewise, there's no "itertools.imap".  Perhaps use 
itertools.some_sort_of_truncating_wrapper(itertools.zip_longest(...))?


This brings us to the itertools problem.


3,4---problems 3 & 4 concern "izip" in: 

http://docs.python.org/dev/3.0/library/itertools.html

This mentions "izip_longest()", but there isn't an "izip_anything" in 
py3k itertools.  Searching the page for the string "izip" produces two 
separate occurrences.  izip_longest -> zip_longest but I don't know 
how to fix the other one.

--
assignee: georg.brandl
components: Documentation
messages: 75176
nosy: LambertDW, georg.brandl
severity: normal
status: open
title: library documentation errors
versions: Python 3.0

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4195] Regression for executing packages

[vajda]

New submission from vajda <[EMAIL PROTECTED]>:

Copy of #2751.

Having discussed this with Nick a bit more over email, he suggested that
package execution could actually be properly supported by looking for a
__main__ module inside the package and executing it instead. This is
consistent with the way .zip archives are executed and would neatly
resolve this problem. I attached a simple patch implementing this.

Nick also said that fixing this might be considered a new feature and
might have to wait until 2.7. Given that this worked on Python 2.5,
albeit unintentionally, I'd argue that this is closer to a fix of a
somewhat buggy Python 2.5 feature than a new feature in 2.6.

--
components: Interpreter Core
files: patch.2751
messages: 75175
nosy: benjamin.peterson, ncoghlan, vajda
severity: normal
status: open
title: Regression for executing packages
type: behavior
versions: Python 2.6
Added file: http://bugs.python.org/file11876/patch.2751

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4194] Miserable subprocess.Popen performance

[Sameer]

Changes by Sameer <[EMAIL PROTECTED]>:


--
nosy: +sameerd

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3238] backport python 3.0 language functionality to python 2.6 by adding 7 opcodes to ceval.c

[kai zhu]

Changes by kai zhu <[EMAIL PROTECTED]>:


Added file: http://bugs.python.org/file11875/mario.jpg

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3238] backport python 3.0 language functionality to python 2.6 by adding 7 opcodes to ceval.c

[kai zhu]

Changes by kai zhu <[EMAIL PROTECTED]>:


Added file: http://bugs.python.org/file11874/img2txt.py

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3238] backport python 3.0 language functionality to python 2.6 by adding 7 opcodes to ceval.c

[kai zhu]

Changes by kai zhu <[EMAIL PROTECTED]>:


Added file: http://bugs.python.org/file11873/img2txt 3dplot screenshot.gif

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3238] backport python 3.0 language functionality to python 2.6 by adding 7 opcodes to ceval.c

[kai zhu]

kai zhu <[EMAIL PROTECTED]> added the comment:

post fyi:

here's moderately complex python-3.0 script (plus curry extension) i
wrote using numpy, PIL, & scipy.weave 2.6 extensions.

it takes a jpeg, gif... image file & outputs it in colorized ascii art.
the actual purpose is for colorized 3-d scientific plottin in text
terminal (screenshots of image conversion & 3d plots in putty terminal
included)

usage:
python -c "import py3to2; import img2txt; img2txt.img2txt.test()"
python -c "import py3to2; import img2txt; img2txt.tplot3d.test()"

Added file: http://bugs.python.org/file11872/img2txt mario.jpg screenshot.gif

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4194] Miserable subprocess.Popen performance

[Skip Montanaro]

New submission from Skip Montanaro <[EMAIL PROTECTED]>:

I noticed a colleague at work today checked in a change to his code to
switch back from subprocess.Popen to os.popen.  I asked him about it and he
reported that subprocess.Popen was about 10x slower than os.popen.  I asked
him for a simple test case, which is attached.  Here are my results with
Python 2.4 through 2.7 (aka CVS HEAD):

tmp% python2.4 popentest.py 
time with os.popen :  0.09
time with subprocess.Popen :  2.27
tmp% python2.5 popentest.py 
time with os.popen :  0.03
time with subprocess.Popen :  1.52
tmp% python2.6 popentest.py 
time with os.popen :  0.038824
time with subprocess.Popen :  1.517056
tmp% python2.7 popentest.py 
time with os.popen :  0.033746
time with subprocess.Popen :  1.512331

These times are on my Mac laptop, all writing to the local disk.  It seems
there was a bit of improvement in the 2.5 release but that it is still
miserably slow when compared with os.popen.

His original test used time.clock() as the timer.  I changed to time.time()
but got essentially the same result.

Skip

--
files: popentest.py
messages: 75173
nosy: skip.montanaro
severity: normal
status: open
title: Miserable subprocess.Popen performance
Added file: http://bugs.python.org/file11871/popentest.py

___
Python tracker <[EMAIL PROTECTED]>

___import os, subprocess, time

os_filename = "tmp_os"
sub_filename = "tmp_subprocess"

def write_file(filename): # write out a reasonably big file
   fh = file(filename, 'w')
   for i in xrange(10):
   print >>fh, i
   fh.close()

write_file(os_filename)
write_file(sub_filename)

start = time.time()
read_handle = os.popen('cat %s' % os_filename)
a = [i for i in read_handle]
print "time with os.popen : ", time.time() - start

start = time.time()
read_handle = subprocess.Popen(['cat', sub_filename], 
stdout=subprocess.PIPE).stdout
a = [i for i in read_handle]
print "time with subprocess.Popen : ", time.time() - start

os.unlink(os_filename)
os.unlink(sub_filename)
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4193] Multiprocessing example

[David W. Lambert]

New submission from David W. Lambert <[EMAIL PROTECTED]>:

http://docs.python.org/dev/3.0/library/multiprocessing.html

I'm sure the examples have been thoughtfully contrived.  Still, this 
seems instructive without adding much complexity.  I'd change the 
first "trivial example" to

###
from multiprocessing import Process
import os

def info(title):
print(title)
print('module name:',__name__)
print('parent process:',os.getppid())
print('process id:',os.getpid())
print()

def f(name):
info('function f')
print('hello', name)

if __name__ == '__main__':
info('main line')
p = Process(target=f, args=('bob',))
p.start()
p.join()
###

with output similar to


main line
module name: __main__
parent process: 12926
process id: 17002

function f
module name: __main__
parent process: 17002
process id: 17004

hello bob

--
assignee: georg.brandl
components: Documentation
messages: 75172
nosy: LambertDW, georg.brandl
severity: normal
status: open
title: Multiprocessing example
type: feature request
versions: Python 3.0

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4136] merge json library with simplejson 2.0.4

[Bob Ippolito]

Changes by Bob Ippolito <[EMAIL PROTECTED]>:


--
title: merge json library with simplejson 2.0.3 -> merge json library with 
simplejson 2.0.4

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4136] merge json library with simplejson 2.0.3

[Bob Ippolito]

Bob Ippolito <[EMAIL PROTECTED]> added the comment:

Attached is a new diff, one byte fix to the float parser when parsing JSON 
documents that are just a float (also a test and a version bump).

Added file: http://bugs.python.org/file11870/json_issue4136_r67009.diff

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2983] Ttk support for Tkinter

[Brett Cannon]

Brett Cannon <[EMAIL PROTECTED]> added the comment:

On Fri, Oct 24, 2008 at 5:27 AM, Guilherme Polo <[EMAIL PROTECTED]> wrote:
>
> Guilherme Polo <[EMAIL PROTECTED]> added the comment:
>
> Now that the new branches are around I guess we could consider adding it
> then ? I will help with anything.
>

Yes, we can consider adding this now.

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4192] Subprocess error with I/O redirection to Pipes

[Pascal Chambon]

New submission from Pascal Chambon <[EMAIL PROTECTED]>:

I've created on my desktop a file flooder.py containing just the following:
##
import sys, os

progress=open(r"C:\Users\v-pascha\Desktop\STDERR.txt","w")

for i in range(101):
   print str(i)*20
   progress.write( str(i)+"\n" )
   
progress.close()
##


and a file receiver.py containing just :



import sys, os, subprocess

#os.system("pause")
subprocess.Popen(r"python
C:\Users\v-pascha\Desktop\flooder.py",stdin=subprocess.PIPE,stdout=subprocess.PIPE)
#,stdin=subprocess.PIPE,stdout=subprocess.PIPE
#os.system("pause")

#

And when I launch receiver.py, I get a crash with that (not explicit)
message :
"close failed in file object destructor:
Error in sys.excepthook:

Original exception was:"

The crash doesn't happen if I don't redirect the stdout and stdin of the
child process. So it seems something weird happens when subprocess tries
to redirect the child's I/O to the PIPEs...

--
components: Library (Lib), Windows
messages: 75169
nosy: pakal
severity: normal
status: open
title: Subprocess error with I/O redirection to Pipes
type: crash
versions: Python 2.6

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4177] Crash in MIMEText on FreeBSD

[Sérgio Surkamp]

Sérgio Surkamp <[EMAIL PROTECTED]> added the comment:

When I first saw the problem, the email system queue was stopped about 2
days (weekend) :-(

The email system control the number of open threads, so I wasn't opening
new threads too and issuing many warnings about it on logs

Anyway, already installed the ptrace tool and Ill start debuging when I
came back from launch

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4177] Crash in MIMEText on FreeBSD

[STINNER Victor]

STINNER Victor <[EMAIL PROTECTED]> added the comment:

> Usually 100%. But saw it with more (using both CPU's), I think that mean
> more then one thread "freezed".

Does the program finish its job after 10 minutes or 1 hour? Using all the CPU 
doesn't mean that Python is frozen, it's the opposite: Python is working hard 
to compute the result :-)

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4177] Crash in MIMEText on FreeBSD

[Sérgio Surkamp]

Sérgio Surkamp <[EMAIL PROTECTED]> added the comment:

- FreeBSD version?

FreeBSD 7.0-RELEASE

 - CPU, memory?

CPU: 2 x Pentium III 1.133 GHz
Memory: 512 Mbytes

 - Full Python version?

Python 2.5.2 (r252:60911, Oct  2 2008, 10:03:50) 
[GCC 4.2.1 20070719  [FreeBSD]] on freebsd7

> On "freeze", the process uses 0% or 100% of the CPU time? You can use
the strace program to trace Python activity during the freeze.

Usually 100%. But saw it with more (using both CPU's), I think that mean
more then one thread "freezed".

I will download your trace program and do some tests with it. Ill try to
collect some informations using GDB too.

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4177] Crash in MIMEText on FreeBSD

[STINNER Victor]

STINNER Victor <[EMAIL PROTECTED]> added the comment:

> The original text block is about 5 to 9 Mbytes (...), it freezes 
> building the MIMEText object. Actually no MemoryError isn't raised,
> just a sudden freeze of the running thread.

Can you give more details about the freeze?
 - FreeBSD version?
 - CPU, memory?
 - Full Python version?

On "freeze", the process uses 0% or 100% of the CPU time? You can use the 
strace program to trace Python activity during the freeze.

Your might try my clone of strace, strace.py, which works on FreeBSD without 
the Linux emulation (but on FreeBSD, only i386 is supported):
   http://python-ptrace.hachoir.org/trac

> Unfortunately the test script submited does not do the same behavior,
> maybe some other things are freezing the software instead of raise the
> MemoryError.

You can try the isolate the bug? Remove some code, disable functions, etc.

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4177] Crash in MIMEText on FreeBSD

[Sérgio Surkamp]

Sérgio Surkamp <[EMAIL PROTECTED]> added the comment:

> Your text file is ~40 MB. Python may allocate mutiple objects bigger 
than 40 MB to create the email content. The algorithm should be 
changed to work on a stream (process small chunks, eg. 4 KB) instead 
of manipule the full text in memory (+40,000 KB).

The original text block is about 5 to 9 Mbytes - its a server generated
report by pflogsum. When it came to our mailing list processing program
(wrote by someone else in Python), it freezes building the MIMEText
object. Actually no MemoryError isn't raised, just a sudden freeze of
the running thread.

Unfortunately the test script submited does not do the same behavior,
maybe some other things are freezing the software instead of raise the
MemoryError. I have checked for blocks of try: ... except ...: pass that
could hide the problem, but found nothing.

I have already limited the size on Postfix, but the strange thing is why
this happens on FreeBSD and don't on Linux.

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2983] Ttk support for Tkinter

[Guilherme Polo]

Guilherme Polo <[EMAIL PROTECTED]> added the comment:

Now that the new branches are around I guess we could consider adding it
then ? I will help with anything.

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue639266] Tkinter sliently discards all Tcl errors

[Guilherme Polo]

Guilherme Polo <[EMAIL PROTECTED]> added the comment:

Uhm, long time without discussion but I'm hoping someone interested may
read this.

You have to provide a "bgerror" command to Tcl, so it will get invoked
when a background error happens. Either _tkinter.c could create one for
an interpreter is created or Tkinter.py could create one when a Tk
instance is created. The command will be invoked with a single argument,
the background error message.


If you want to play with it:

import Tkinter

def bgerr_handler(msg):
print msg, "<<"

root = Tkinter.Tk()
root.tk.createcommand("bgerror", bgerr_handler)
btn = Tkinter.Button(command="expr 1/0")
btn.pack()
root.mainloop()


To get some form of traceback you would need to play with inspect and
traceback modules. But in both cases (in the example above and using
inspect) an exception isn't raised at all, and if you try raising then
Tcl will think the bgerror handler failed and will tell you that. For
this reason I would prefer to leave to _tkinter.c to handle this.

--
nosy: +gpolo

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4184] Remove use of private attributes in smtpd

[Doug Hellmann]

Changes by Doug Hellmann <[EMAIL PROTECTED]>:


--
nosy: +doughellmann

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2980] Pickle stream for unicode object may contain non-ASCII characters.

[Dan Dibagh]

Dan Dibagh <[EMAIL PROTECTED]> added the comment:

> Which PEP specifically? PEP 263 only mentions the unicode-escape
> encoding in its problem statement, i.e. as a pre-existing thing.
> It doesn't specify it, nor does it give a rationale for why it behaves
> the way it does.

PEP 100 and PEP 263. What I looked for was a description of the
functional intention and a technical definition of raw unicode escape.
The term "raw" tends to have different meanings depending on the context
in which it appears. PEP 263 is of interest in the overall understanding
of the intention of raw unicode escape. If raw unicode escape is to
convert from python source into unicode strings then the decoding of raw
unicode escape strings depends on the source code encoding. Then perhaps
it would give an idea what the encoding part is supposed to do... PEP
100 is of interest for the technical description. It describes the
section "unicode constructors" as the definition.

> What code are you looking at, and where do you find it difficult to
> follow it? Maybe you get confused between the "unicode-escape" codec,
> and the "raw-unicode-escape" codec, also.

Since it is the issue with non-ASCII characters in pickle output I look
at, it is raw-unicode-escape being in focus. For the decoding bit the
distinction between unicode-escape and raw-unicode-escape is very clear. 

I look at the function PyUnicode_EncodeRawUnicodeEscape in
Objects/unicodeobject.c. At the point of the comment "/* Copy everything
else as-is */", given the perceived intentions of the encoding type, I
try to figure out why there isn't a "/* Map non-printable US ASCII to
'\xhh' */" section like in the unicodeescape_string function. The
background in older pythons you explained is essentially what I guessed.

> The raw-unicode-escape codec? It was designed to support parsing of
> Python 2.0 source code, and of "raw" unicode strings (ur"") in
> particular. In Python 2.0, you only needed to escape characters above
> U+0100; Latin-1 characters didn't need escaping. Python, itself, only
> relied on the decoding directory. That the codec choses not to escape
> Latin-1 characters on encoding is an arbitrary choice (I guess); it's
> still symmetric with decoding.

I suppose you mean symmetric with decoding as long as you stick to the
latin-1 character set, as raw unicode escaping isn't a one-to-one mapping.

When PEP 263 came into the picture, wouldn't it have made sense to
change PyUnicode_EncodeRawUnicodeEscape to produce ASCII-only output, or
perhaps output conforming to the current default encoding? Given the
intention of the raw unicode escape, encoding something with it means
producing python source code. But it is in latin-1 while the rest of
Python has moved on to use ASCII by default or whatever being configured
in the source. I tried to put shine on that problem in my previous example.

> Even though the choice was arbitrary, you shouldn't change it now,
> because people may rely on how this codec works.

> Applications might rely on what was implemented rather than what was
> specified. If they had implemented their own pickle readers, such
> readers might break if the pickle format is changed. In principle, 
> even the old pickle readers of Python 2.0..2.6 might break if the
>format changes in 2.7 - we would have to go back and check that they don't
> break (although I do believe that they would work fine).

Then let me ask: How far reaching is the aim to maintain compatibility
with programs which depends on Python internals? Even if the internal
thing is a bug and the thing which depends on the bug is also a bug?
Maybe it is a provoking question, let me explain. The question(s)
applies to some extent to the workings of the codec but it is really the
pickle problem I think of. In the case of older Python releases, it is
just a matter of testing, just as you say. It is boring and perhaps
tedious but there is nothing special which prevents it from being done.
If there are many versions there ought to be a way to write a program
which does it automatically. 

In the case of those who have implemented their own pickle readers, the
source and the comments in pickletools.py clearly states that unicode
strings are raw unicode escaped in format 0. Now raw unicode escape
isn't a canonical format. The letter A can be represented either as
\u0041 or as itself as A. If a hypothetical implementor gets the idea
that characters in the range 0-255 cannot be represented by \u00xx
sequences then the fact that pickle replaces \ with \u005c and \n with
\u000a should give a hint that he is wrong. So if characters in the
range 128-255 gets escaped with \u00xx any pickle reader should handle
it. I've tried to come up with some sensible way to write a pickle
implemenation which fails to understand \u00xx characters without
calling it a bug. I cannot. Can you? So it seems that the worry for
changing protocol 0 is buggy programs depending on a pickle bug.

In the other end of the spectru

[issue4177] Crash in MIMEText on FreeBSD

[STINNER Victor]

STINNER Victor <[EMAIL PROTECTED]> added the comment:

Your example work here on:
 - Linux, i386, 2 Go of memory, Python 2.5
 - FreeBSD in Qemu, i386, 512 MB of memory, Python 2.5

> The program just eat all the CPU and with high memory usage or raise 
a MemoryError

Yes, it takes one minute or more to finish. If there is not enough 
memory, Python raises a MemoryError. The behaviour is correct: Python 
doesn't crash, it's just slow.

Your text file is ~40 MB. Python may allocate mutiple objects bigger 
than 40 MB to create the email content. The algorithm should be 
changed to work on a stream (process small chunks, eg. 4 KB) instead 
of manipule the full text in memory (+40,000 KB).

Why do you try to send 40 MB by email? Use FTP or another protocol :-p 
Or use another encoding (base64) to attach the text to the email.

--
nosy: +haypo

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC

[Koen van de Sande]

Koen van de Sande <[EMAIL PROTECTED]> added the comment:

OK, so the define is not going to work. For Python extensions built 
through distutils, the msvc9compiler_stripruntimes.diff patch will 
work: it will remove just the dependency on the VC90runtimes. It will 
leave other assembly dependencies if they are there.

For the .pyd's and .dll's in the DLLs folder: I have opened them in a 
hex editor, and just by hand replaced the  entry for 
the VC90runtimes in each file with spaces. I can confirm that this 
works, also for _tkinter, for which tk85.dll has an additional 
dependency on Windows Common Controls (though this one was available in 
my WinSXS). What would be the best way to integrate a search/replace 
operation like this in the Python build process? Is the manifest 
generated as a seperate file here first as well, or is it directly 
embedded?

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4177] Crash in MIMEText on FreeBSD

[Roumen Petrov]

Roumen Petrov <[EMAIL PROTECTED]> added the comment:

what about data segment and stack size limits ?

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3714] nntplib module broken by str to unicode conversion

[STINNER Victor]

STINNER Victor <[EMAIL PROTECTED]> added the comment:

As I did for POP3 (poplib) and IMAP4 (imaplib), nntplib should use 
bytes instead of characters because the charset may be different for 
each message and nntplib is unable to retreive the charset from the 
email header.

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3518] multiprocessing: BaseManager.from_address documented but doesn't exist

[Mart Sõmermaa]

Mart Sõmermaa <[EMAIL PROTECTED]> added the comment:

I propose we add the following to that section as well.

If you need to provide access to a queue from both local and remote
processes, use `multiprocessing.Queue` in the server:

>>> from multiprocessing import Process, Queue
>>> from multiprocessing.managers import BaseManager
>>> class Worker(Process):
... def __init__(self, q):
... self.q = q
... super(Worker, self).__init__()
... def run(self):
... self.q.put('local hello')
... 
>>> q = Queue()
>>> w = Worker(q)
>>> w.start()
>>> class QueueManager(BaseManager): pass
... 
>>> QueueManager.register('getQueue', callable=lambda: q)
>>> m = QueueManager(address=('', 5), authkey='abracadabra')
>>> s = m.get_server()
>>> s.serve_forever()

Use it in the client as shown above:

>>> from multiprocessing.managers import BaseManager
>>> class QueueManager(BaseManager): pass
... 
>>> QueueManager.register('getQueue')
>>> m = QueueManager(address=('localhost', 5), authkey='abracadabra')
>>> m.connect()
>>> q = m.getQueue()
>>> q.get()
'local hello'
>>> q.put('remote hello')

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3714] nntplib module broken by str to unicode conversion

[Barry A. Warsaw]

Changes by Barry A. Warsaw <[EMAIL PROTECTED]>:


--
priority: deferred blocker -> release blocker

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC

[Andre Heider]

Andre Heider <[EMAIL PROTECTED]> added the comment:

I tried to use _CRT_NOFORCE_MANIFEST but i couldn't get it working.
There're some infos about this approach at
http://blog.m-ri.de/index.php/2008/05/06/hotfix-fuer-usemsprivateassembliesh-und-vc-2008/
, but even with the mentioned "__declspec(selectany) int
_forceCRTManifest[RTM];" statement, i wasn't able to suppress the
manifest dependency :(

Microsoft, in its infinite wisdom, decided to compile the CRT itself
with those #pragma's enabled. That puts the "/manifestdependency" linker
arguments in msvcrt.lib. When you link against that file you get that
CRT manifest entry, whether you like it or not :(
Looks like _CRT_NOFORCE_MANIFEST only works if you build your own CRT
without the #pragma's.
Of course that only applies to the dynamic version of the CRT, when
using /MT the manifest hell just vanishes.

I'm unsure what the best way to fix this issue for python is...

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4191] urlparse normalize URL path

[monk.e.boy]

New submission from monk.e.boy <[EMAIL PROTECTED]>:

Hi,

  The way urljoin works is a bit funky, equivalent paths do not get
cleaned in a consistent way:


import urlparse
import posixpath

print urlparse.urljoin('http://www.example.com', '///')
print urlparse.urljoin('http://www.example.com/', '///')
print urlparse.urljoin('http://www.example.com///', '///')
print urlparse.urljoin('http://www.example.com///', '//')
print urlparse.urljoin('http://www.example.com///', '/')
print urlparse.urljoin('http://www.example.com///', '')
print
# the above should reduce down to:
print posixpath.normpath('///')
print
print urlparse.urljoin('http://www.example.com///', '.')
print urlparse.urljoin('http://www.example.com///', '/.')
print urlparse.urljoin('http://www.example.com///', './')
print urlparse.urljoin('http://www.example.com///', '/.')
print
print posixpath.normpath('/.')
print
print urlparse.urljoin('http://www.example.com///', '..')
print urlparse.urljoin('http://www.example.com', '/a/../a/')
print urlparse.urljoin('http://www.example.com', '../')
print urlparse.urljoin('http://www.example.com', 'a/../a/')
print urlparse.urljoin('http://www.example.com', 'a/../a/./')
print urlparse.urljoin('http://www.example.com/a/../a/', '../a/./../a/')
print urlparse.urljoin('http://www.example.com/a/../a/', '/../a/./../a/')

The results of the above code are:

http://www.example.com/
http://www.example.com/
http://www.example.com/
http://www.example.com///
http://www.example.com/
http://www.example.com///

/

http://www.example.com///
http://www.example.com/.
http://www.example.com///
http://www.example.com/.

/

http://www.example.com
http://www.example.com/.
http://www.example.com
http://www.example.com/.

http://www.example.com//
http://www.example.com/a/../a/
http://www.example.com/../
http://www.example.com/a/
http://www.example.com/a/
http://www.example.com/a/
http://www.example.com/../a/./../a/

Sometimes the path is cleaned, sometimes it is not. When it is cleaned,
the cleaning process is not perfect.

The bit of code that is causing problems is commented:

  # XXX The stuff below is bogus in various ways...

If I may be so bold, I would like to see this URL cleaning code stripped
from urljoin.

A new method/function could be added that cleans a URL. It could have a
'mimic browser' option, because a browser *will* follow URLs like:
http://example.com/../../../ (see this non-bug
http://bugs.python.org/issue2583 )

The URL cleaner could use some of the code from "posixpath". Shorter
URLs would be preferred over longer (e.g: http://example.com preferred
to http://example.com/ )

Thanks,

monk.e.boy

--
messages: 75154
nosy: monk.e.boy
severity: normal
status: open
title: urlparse normalize URL path
versions: Python 2.6

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com