[Python-Dev] Issue #11051: system calls per import
Hi, Antoine Pitrou noticed that Python 3.2 tries a lot of filenames to load a module: http://bugs.python.org/issue11051 Python 3.1 does already test many filenames, but with Python 3.2, it is even worse. For each directory in sys.path, it tries 9 suffixes: '', '.cpython-32m.so', 'module.cpython-32m.so', '.abi3.so', 'module.abi3.so', '.so', 'module.so', '.py', '.pyc'. I don't understand why it tests so much .so suffixes. And why it does test with and without "module". Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #11051: system calls per import
> Python 3.1 does already test many filenames, but with Python 3.2, it is > even worse. > > For each directory in sys.path, it tries 9 suffixes: '', > '.cpython-32m.so', 'module.cpython-32m.so', '.abi3.so', > 'module.abi3.so', '.so', 'module.so', '.py', '.pyc'. > > I don't understand why it tests so much .so suffixes. And why it does > test with and without "module". The many extensions have been specified in PEP 3149. The PEP also specifies # This "tag" will appear between the module base name and the operation # file system extension for shared libraries. which apparently meant that the existing mechanism is extended to add the tag. The support for both the "short extension" (i.e. ".so") and "long extension" (i.e. "module.so") goes back to r4297 (Python 1.1), when the short extension was added as an alternative to the long extension. The original module suffix was defined in r3518 when dynamic extension modules got supported, as either "module.so" (SUN_SHLIB) or "module.o" (dl_loadmod, apparently Irix). Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #11051: system calls per import
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Am 30.01.2011 09:56, schrieb Victor Stinner: > Hi, > > Antoine Pitrou noticed that Python 3.2 tries a lot of filenames to load > a module: > http://bugs.python.org/issue11051 > > Python 3.1 does already test many filenames, but with Python 3.2, it is > even worse. > > For each directory in sys.path, it tries 9 suffixes: '', > '.cpython-32m.so', 'module.cpython-32m.so', '.abi3.so', > 'module.abi3.so', '.so', 'module.so', '.py', '.pyc'. '' is not really a suffix, but a test for a package directory. > I don't understand why it tests so much .so suffixes. Because of PEP 3149 and PEP 384. > And why it does test with and without "module". Because it always did (there's a thing called backwards compatibility.) This is of course probably the obvious one to start a deprecation process. Georg -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk1FLnEACgkQN9GcIYhpnLApaACdGDe9qVlZNVHRF92yTqYnYFIp hjIAn34YqvMy8fy7pcz0qAlS/WhRWR4G =1b9C -END PGP SIGNATURE- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #11051: system calls per import
On Sun, Jan 30, 2011 at 7:25 PM, Georg Brandl wrote: >> And why it does test with and without "module". > > Because it always did (there's a thing called backwards compatibility.) > > This is of course probably the obvious one to start a deprecation process. But why do we check the long suffix for the *new* extension module naming variants from PEP 3149 and PEP 384? Those are completely new, so there's no backwards compatibility argument there. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #11051: system calls per import
Le dimanche 30 janvier 2011 à 22:52 +1000, Nick Coghlan a écrit :
> On Sun, Jan 30, 2011 at 7:25 PM, Georg Brandl wrote:
> >> And why it does test with and without "module".
> >
> > Because it always did (there's a thing called backwards compatibility.)
> >
> > This is of course probably the obvious one to start a deprecation process.
>
> But why do we check the long suffix for the *new* extension module
> naming variants from PEP 3149 and PEP 384? Those are completely new,
> so there's no backwards compatibility argument there.
My implicit question was: can we limit the number of tested suffixes? I
see two candidates: remove 'module.cpython-32m.so' ('.cpython-32m.so'
should be enough) and 'module.abi3.so' ('.abi3.so' should be enough).
And the real question is: should we change that before 3.2 final? If we
don't change that in 3.2, it will be harder to change it later (but it
is still possible).
Limit the number of suffixes is maybe not the right solution to limit
the number of system calls at startup. We can imagine alternatives:
* remember the last filename when loading a module and retry this
filename first
* specify that a module is a Python system module and should only be
loaded from "system directories"
* specify the module type (directory, .py file, dynamic library, ...)
when loading a module
* (or a least remember the module type and retry this type first)
* etc.
We should find a compromise between speed (limit the number of system
calls) and the usability of Python modules.
Victor
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #11051: system calls per import
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Am 30.01.2011 17:35, schrieb Victor Stinner:
> Le dimanche 30 janvier 2011 à 22:52 +1000, Nick Coghlan a écrit :
>> On Sun, Jan 30, 2011 at 7:25 PM, Georg Brandl wrote:
And why it does test with and without "module".
>>>
>>> Because it always did (there's a thing called backwards compatibility.)
>>>
>>> This is of course probably the obvious one to start a deprecation process.
>>
>> But why do we check the long suffix for the *new* extension module
>> naming variants from PEP 3149 and PEP 384? Those are completely new,
>> so there's no backwards compatibility argument there.
>
> My implicit question was: can we limit the number of tested suffixes? I
> see two candidates: remove 'module.cpython-32m.so' ('.cpython-32m.so'
> should be enough) and 'module.abi3.so' ('.abi3.so' should be enough).
>
> And the real question is: should we change that before 3.2 final?
We most definitely shouldn't.
Georg
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.17 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk1FltgACgkQN9GcIYhpnLDquwCfZH+jtM6nsXz4Iyi2XrhpDKBH
+6IAnA4Be/CWQhiQ9hq1VqGH2ent7say
=e1d5
-END PGP SIGNATURE-
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #11051: system calls per import
On Sun, Jan 30, 2011 at 11:35 AM, Victor Stinner wrote: .. > We should find a compromise between speed (limit the number of system > calls) and the usability of Python modules. Do you have measurements that show python spending significant time on failing open calls? ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Stable buildbots
On 23 November 2010 23:18, David Bolen wrote: > Trent Nelson writes: > >> That's interesting. (That kill_python.exe doesn't kill the wedged >> processes, but pskill does.) kill_python is pretty simple, it just >> calls TerminateProcess() after acquiring a handle with the relevant >> PROCESS_TERMINATE access right. (...) >> >> Are you calling pskill with the -t flag? i.e. kill process and all >> dependents? That might be the ticket, especially if killing the child >> process that wedged select() is waiting on causes it to return, and >> thus, makes it killable. > > Nope, just "pskill python_d". Haven't bothered to check the pskill > source but I'm assuming it's just a basic TerminateProcess. Ideally my > quickest workaround would just be to replace the kill_python in the > buildbot tools script with that command but of course they could get > updated on checkouts and I'm not arguing it's generally appropriate enough > to belong in the source. After a long, long time (:-(), I'm finally getting a chance to look at this. I've patched buildbot as mentioned earlier in the thread, but I don't see where I should put the pskill command to make it work. At the moment, I have scheduled tasks to pskill python_d and vsjitdebugger. The python_d one runs daily and the debugger one hourly. (I daren't kill python_d too often, or I'll start killing in-progress tests, I assume). The vsjitdebugger one is there because I think it solves the CRT popup issue (I'll add the autoit script as well, but as I'm running as a service, I'm not sure the popup will alwats be visible for the autoit script to pick up...) Presumably, you're inserting a pskill command somewhere into the actual build process. I don't know much about buildbot, but I thought that was controlled by the master and/or the Python build scripts, neither of which I can change. If I want to add a pskill command just after a build/test has run (i.e., about where kill_python runs at the moment) how do I do that? Thanks, Paul. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #11051: system calls per import
Am 30.01.2011 17:54, schrieb Alexander Belopolsky: > On Sun, Jan 30, 2011 at 11:35 AM, Victor Stinner > wrote: > .. >> We should find a compromise between speed (limit the number of system >> calls) and the usability of Python modules. > > Do you have measurements that show python spending significant time on > failing open calls? No; past measurements always showed that this is insignificant, probably thanks to operating system caching the relevant directory blocks (so it doesn't really matter whether you make one or ten lookups per directory; my guess is that it matters more if you look into ten directories instead of one). Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Stable buildbots
Paul Moore writes:
> Presumably, you're inserting a pskill command somewhere into the
> actual build process. I don't know much about buildbot, but I thought
> that was controlled by the master and/or the Python build scripts,
> neither of which I can change.
>
> If I want to add a pskill command just after a build/test has run
> (i.e., about where kill_python runs at the moment) how do I do that?
I haven't been able to - as you say there's no good way to hook into
the build process in real time as the changes have to be external or
they'll get zapped on the next checkout. I suppose you could rapidly
try to monitor the output of the build slave log file, but then you
risk killing a process from a next step if you miss something or are
too slow. And I've had cases (after long periods of continuous
runtime) where the build slave log stops being generated even while
the slave is running fine.
Anyway, in the absence of changes to the build tree, I finally gave up
and now run an external script (see below) that whacks any python_d
process it finds running for more than 2 hours (arbitrary choice). I
considered trying to dig deeper to identify processes with no logical
test parent (more similar to the build kill_python itself), but
decided it was too much effort for the minimal extra gain. So not
terribly different from your once a day pskill, though as you say if
you arbitrarily kill all python_d processes at any given point in
time, you risk interrupting an active test.
So the AutoIt script covers pop-ups and the script below cleans up
hung processes. On the subject of pop-ups, I'm not sure but if you
find your service not showing them try enabling the "Allow service to
interact with the desktop" option in the service definition. In my
experience though if a service can't perform a UI interaction, the
interaction just fails, so I wouldn't expect the process to get stuck
in that case.
Anyway, in my case the kill script itself is Cygwin/bash based, but
using the pstools tools, and conceptually just kills (pskill) any
python_d process identified as having been running for 2 or more hours
of wall time (via pslist):
- - - - - - - - - - - - - - - - - - - - - - - - -
#!/bin/sh
#
# kill_python.sh
#
# Quick 'n dirty script to watch for python_d processes that exceed a few
# hours of runtime, then kill then assuming they're hung
#
PROC="python_d"
TIMEOUT="2"
while [ 1 ]; do
echo "`date` Checking..."
PIDS=`pslist 2>&1 | grep "^$PROC" | awk -v TIMEOUT=$TIMEOUT
'{split($NF,fields,":"); if (int(fields[1]) >= int(TIMEOUT)) {print $2}}'`
if [ "$PIDS" ]; then
echo = `date`
for pid in $PIDS; do
pslist $pid 2>&1 | grep "^$PROC"
pskill $pid
done
echo =
fi
sleep 300
done
- - - - - - - - - - - - - - - - - - - - - - - - -
It's a kludge, but as you say, for us to impose this on the build
slave side requires it to be outside of the build tree. I've been
running it for about a month now and it seems to be doing the job. I
run a similar script on OSX (my Tiger slave also sometimes sees stuck
processes, though they just burn CPU rather than interfere with
tests), but there I can identify stranded python_d processes if they
are owned by init, so the script can react more quickly.
I'm pretty sure the best long term fix is to move the kill processing
into the clean script (as per issue 9973) rather than where it
currently is in the build script, but so far I don't think the idea
has been able to attract the interest of anyone who can actually
commit such a change. (See also the Dec continuation of this thread -
http://www.mail-archive.com/[email protected]/msg54389.html)
I had also created issue 10641 from when I thought I found a problem
with kill_python, but that turned out incorrect, and in subsequent
tests kill_python in the build tree always worked, so the core issue
seems to always be the failure to run it at all as opposed to it not
working.
For now though, these two external "monitors" seem to have helped
contain the number of manual operations I have to do on my two Windows
slaves. (Though recently I've begun seeing two new sorts of pop-ups
under Windows 7 but both related to memory, so I think I just need to
give my VM a little more memory)
-- David
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Stable buildbots
Hello, > I'm pretty sure the best long term fix is to move the kill processing > into the clean script (as per issue 9973) rather than where it > currently is in the build script, but so far I don't think the idea > has been able to attract the interest of anyone who can actually > commit such a change. Thanks for bringing my attention on this. I've added a comment on that issue. If you say this should improve things, there's probably no reason not to commit such a patch. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Stable buildbots
On 30 January 2011 20:50, David Bolen wrote: > I haven't been able to - as you say there's no good way to hook into > the build process in real time as the changes have to be external or > they'll get zapped on the next checkout. I suppose you could rapidly > try to monitor the output of the build slave log file, but then you > risk killing a process from a next step if you miss something or are > too slow. And I've had cases (after long periods of continuous > runtime) where the build slave log stops being generated even while > the slave is running fine. OK, sounds like I hadn't missed anything, then, which is good in some sense :-) > For now though, these two external "monitors" seem to have helped > contain the number of manual operations I have to do on my two Windows > slaves. (Though recently I've begun seeing two new sorts of pop-ups > under Windows 7 but both related to memory, so I think I just need to > give my VM a little more memory) Yes, my (somewhat more simplistic) kill scripts had done some good as well. Having said that, http://bugs.python.org/issue9931 is currently stopping my buildslave (at least if I run it as a service), so it's a bit of a moot point at the moment... (One thing that might be good is if there were a means in the buildslave architecture to deliberately disable a test temporarily, if it's known to fail - I know ignoring errors isn't a good thing in general, but OTOH, having a slave effectively dead for months because of a known issue isn't a lot of help, either :-() Thanks for the reply. Paul. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #11051: system calls per import
Victor Stinner wrote: Limit the number of suffixes is maybe not the right solution to limit the number of system calls at startup. We can imagine alternatives: * remember the last filename when loading a module and retry this filename first * specify that a module is a Python system module and should only be loaded from "system directories" * specify the module type (directory, .py file, dynamic library, ...) when loading a module * (or a least remember the module type and retry this type first) * etc. Maybe also * Read and cache the directory contents and search it ourselves instead of making a system call for every possible name. -- Greg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] r88273 - python/branches/py3k/Doc/whatsnew/3.2.rst
On Jan 30, 2011, at 8:21 PM, eli.bendersky wrote: Please use the open tracker item and do not edit the document directly. Raymond ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #11051: system calls per import
> Maybe also > >* Read and cache the directory contents and search it ourselves > instead of making a system call for every possible name. I wouldn't do that - I would expect that this is actually slower than making the system calls, because the system might get away with not reading the entire directory (whereas it will have to when we explicitly ask for that). Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
