Re: Restricting import file lookup for pyd, dll, ...
Hello, It's been almost two months since I last investigated this issue, so now I wish to revive this conversation. To answer some of the questions raised by contributors [Gabriel Genellina] Try to shorten the PYTHONPATH to the really required directories (deleting those locations that don't make sense). [Bernard] I customized the PYTHONPATH using a pth file. The pth file contains this: \\Linuxserver\ANIMATION\XSI\WORKGROUP_ANIMATION\Data\Scripts \\Linuxserver\ANIMATION\DB\MT\MT_WORKGROUP\Data\Scripts \\Linuxserver\ANIMATION\DB\TS\TS_WORKGROUP\Data\Scripts \\Linuxserver\ANIMATION\FARM\PYTHON\RELEASE That's it. I could hardly shorten these paths, unless the ressources exposed through these paths were copied locally on every computer. Atm I do not have management tools at my disposal to handle such a setup. When print the paths: C:\WINDOWS\system32\python24.zip C:\Documents and Settings\blebel C:\Python24\DLLs C:\Python24\lib C:\Python24\lib\plat-win C:\Python24\lib\lib-tk C:\Python24 d:\bernard\work\workgroups\workgroup_animation\data\scripts d:\bernard\work\workgroups\mt_workgroup\data\scripts d:\bernard\work\workgroups\ts_workgroup\data\scripts \\Linuxserver\ANIMATION\FARM\PYTHON\RELEASE c:\users\blebel\Softimage\XSI_5.11\Data\Scripts C:\Python24\lib\site-packages C:\Python24\lib\site-packages\win32 C:\Python24\lib\site-packages\win32\lib C:\Python24\lib\site-packages\Pythonwin If by shortening the PYTHONPATH you meant having less paths, I hardly see how I could do less than what I do with the pth. All these paths seem to be built in. Now what would be REALLY nice is to have the ability to specify in the paths the location of *specific files*. Is this possible? What happens is that Python is visiting all kinds of locations to find some files, like os, ntpath, and many more. If I could tell Python right away where are these files located, I hope I could gain something. [Fredrik Lundh] a plain Python 2.4 interpreter can start, execute a command, and shut down in about 0.13 seconds on my machine. 2.5 does the same thing in 0.10 seconds. [Bernard] The problem is not firing up the standalone Python interpreter. The problem is firing the application that embeds a Python interpreter. As explained above, when I start this application, many files are looked in many different places. All PYTHONPATH locations are traversed, where pyd-dll-py-pyw-pyc files are searched. Many of these files are searched in highly improbable locations. [Fredrik Lundh] are you sure you're benchmarking *Python's* start up time, and not the time it takes to load all the modules used by your application, or the time it takes for filemon to print all those 4500 requests to the monitor window? [Bernard] The Filemon overhead consistently clocks at 1 second, not matter the test I perform. Since I ran all tests with Filemon running, I feel safe to ignore this constant. Now, as to the first sentence, to be fair, no, I'm not exactly sure. I don't know all the details of the Python's embedding in this software, and this knowledge is out of my reach atm. Here is the relevant Filemon log: http://www.bernardlebel.com/img_remote/3D/XSI/python/Filemon_XSIPython.LOG (710k file) [Magnus Lycka] Sounds like a broken (networked?) file system. The only time I've had that kind of problems with python startup is when I've had really slow anti-virus programs that scanned all the files I opened. But then it wasn't file requests that mattered, but actually opening them... It still wasn't anywhere near 9 seconds though... [Bernard] This is not entirely impossible, but I get similar results on every computer I perform this test. By every computer, I mean different locations, with different hardwares and different network setups. Even when the files are all local on the computer I get this. Thanks Bernard On 10/28/06, Fredrik Lundh [EMAIL PROTECTED] wrote: Magnus Lycka wrote: That's because I'm using Python through another application, via the pywin32 extensions. When that other application starts, it performs several thousands of file requests (we're talking 4,500, roughly) in the Python installation, locations where there are Python files, and in some other locations that don't make sense. This adds considerable time to the startup time of the application, we're talking between 2 and 9 seconds. Sounds like a broken (networked?) file system. The only time I've had that kind of problems with python startup is when I've had really slow anti-virus programs that scanned all the files I opened. But then it wasn't file requests that mattered, but actually opening them... It still wasn't anywhere near 9 seconds though... if anyone finds out more about this issue, feel free to add a note to this FAQ entry: http://www.effbot.org/pyfaq/why-does-python-sometimes-take-so-long-to-start.htm /F -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: Restricting import file lookup for pyd, dll, ...
Oops, sorry for the inconsistency. The pth file rather looks like this: d:\bernard\work\workgroups\workgroup_animation\data\scripts d:\bernard\work\workgroups\mt_workgroup\data\scripts d:\bernard\work\workgroups\ts_workgroup\data\scripts \\Linuxserver\ANIMATION\FARM\PYTHON\RELEASE c:\users\blebel\Softimage\XSI_5.11\Data\Scripts Bernard On 12/11/06, Bernard Lebel [EMAIL PROTECTED] wrote: Hello, It's been almost two months since I last investigated this issue, so now I wish to revive this conversation. To answer some of the questions raised by contributors [Gabriel Genellina] Try to shorten the PYTHONPATH to the really required directories (deleting those locations that don't make sense). [Bernard] I customized the PYTHONPATH using a pth file. The pth file contains this: \\Linuxserver\ANIMATION\XSI\WORKGROUP_ANIMATION\Data\Scripts \\Linuxserver\ANIMATION\DB\MT\MT_WORKGROUP\Data\Scripts \\Linuxserver\ANIMATION\DB\TS\TS_WORKGROUP\Data\Scripts \\Linuxserver\ANIMATION\FARM\PYTHON\RELEASE That's it. I could hardly shorten these paths, unless the ressources exposed through these paths were copied locally on every computer. Atm I do not have management tools at my disposal to handle such a setup. When print the paths: C:\WINDOWS\system32\python24.zip C:\Documents and Settings\blebel C:\Python24\DLLs C:\Python24\lib C:\Python24\lib\plat-win C:\Python24\lib\lib-tk C:\Python24 d:\bernard\work\workgroups\workgroup_animation\data\scripts d:\bernard\work\workgroups\mt_workgroup\data\scripts d:\bernard\work\workgroups\ts_workgroup\data\scripts \\Linuxserver\ANIMATION\FARM\PYTHON\RELEASE c:\users\blebel\Softimage\XSI_5.11\Data\Scripts C:\Python24\lib\site-packages C:\Python24\lib\site-packages\win32 C:\Python24\lib\site-packages\win32\lib C:\Python24\lib\site-packages\Pythonwin If by shortening the PYTHONPATH you meant having less paths, I hardly see how I could do less than what I do with the pth. All these paths seem to be built in. Now what would be REALLY nice is to have the ability to specify in the paths the location of *specific files*. Is this possible? What happens is that Python is visiting all kinds of locations to find some files, like os, ntpath, and many more. If I could tell Python right away where are these files located, I hope I could gain something. [Fredrik Lundh] a plain Python 2.4 interpreter can start, execute a command, and shut down in about 0.13 seconds on my machine. 2.5 does the same thing in 0.10 seconds. [Bernard] The problem is not firing up the standalone Python interpreter. The problem is firing the application that embeds a Python interpreter. As explained above, when I start this application, many files are looked in many different places. All PYTHONPATH locations are traversed, where pyd-dll-py-pyw-pyc files are searched. Many of these files are searched in highly improbable locations. [Fredrik Lundh] are you sure you're benchmarking *Python's* start up time, and not the time it takes to load all the modules used by your application, or the time it takes for filemon to print all those 4500 requests to the monitor window? [Bernard] The Filemon overhead consistently clocks at 1 second, not matter the test I perform. Since I ran all tests with Filemon running, I feel safe to ignore this constant. Now, as to the first sentence, to be fair, no, I'm not exactly sure. I don't know all the details of the Python's embedding in this software, and this knowledge is out of my reach atm. Here is the relevant Filemon log: http://www.bernardlebel.com/img_remote/3D/XSI/python/Filemon_XSIPython.LOG (710k file) [Magnus Lycka] Sounds like a broken (networked?) file system. The only time I've had that kind of problems with python startup is when I've had really slow anti-virus programs that scanned all the files I opened. But then it wasn't file requests that mattered, but actually opening them... It still wasn't anywhere near 9 seconds though... [Bernard] This is not entirely impossible, but I get similar results on every computer I perform this test. By every computer, I mean different locations, with different hardwares and different network setups. Even when the files are all local on the computer I get this. Thanks Bernard On 10/28/06, Fredrik Lundh [EMAIL PROTECTED] wrote: Magnus Lycka wrote: That's because I'm using Python through another application, via the pywin32 extensions. When that other application starts, it performs several thousands of file requests (we're talking 4,500, roughly) in the Python installation, locations where there are Python files, and in some other locations that don't make sense. This adds considerable time to the startup time of the application, we're talking between 2 and 9 seconds. Sounds like a broken (networked?) file system. The only time I've had that kind of problems with python startup is when I've had
Re: Restricting import file lookup for pyd, dll, ...
At Monday 11/12/2006 19:11, Bernard Lebel wrote: If by shortening the PYTHONPATH you meant having less paths, I hardly see how I could do less than what I do with the pth. All these paths seem to be built in. I mean, delete the ones that actually don't exist. sys.path is a simple list; you can traverse it (perhaps backwards is easier) in site.py or sitecustomize.py and delete unexistant entries. Moving the network related paths (\\linuxserver\...) towards the end may help (if you don't get conflicts by finding the wrong module in the wrong place) [Fredrik Lundh] a plain Python 2.4 interpreter can start, execute a command, and shut down in about 0.13 seconds on my machine. 2.5 does the same thing in 0.10 seconds. [Bernard] The problem is not firing up the standalone Python interpreter. The problem is firing the application that embeds a Python interpreter. As explained above, when I start this application, many files are looked in many different places. All PYTHONPATH locations are traversed, where pyd-dll-py-pyw-pyc files are searched. Many of these files are searched in highly improbable locations. But the whole process takes about 3sec - as seen on your log. (enabling ms resolution on FileMon may help too, because the timing is now too much rough) [Fredrik Lundh] are you sure you're benchmarking *Python's* start up time, and not the time it takes to load all the modules used by your application, or the time it takes for filemon to print all those 4500 requests to the monitor window? [Bernard] The Filemon overhead consistently clocks at 1 second, not matter the test I perform. Since I ran all tests with Filemon running, I feel safe to ignore this constant. Now, as to the first sentence, to be fair, no, I'm not exactly sure. I don't know all the details of the Python's embedding in this software, and this knowledge is out of my reach atm. Have you tried running the interpreter alone? Not your embedded application. Try invoking python with the -v flag. For an embedded application, define PYTHONVERBOSE=1 in the environment before running. Perhaps it's not Python which slows down your startup, but other parts of the app. -- Gabriel Genellina Softlab SRL __ Correo Yahoo! Espacio para todos tus mensajes, antivirus y antispam ¡gratis! ¡Abrí tu cuenta ya! - http://correo.yahoo.com.ar -- http://mail.python.org/mailman/listinfo/python-list
Re: Restricting import file lookup for pyd, dll, ...
Magnus Lycka wrote: That's because I'm using Python through another application, via the pywin32 extensions. When that other application starts, it performs several thousands of file requests (we're talking 4,500, roughly) in the Python installation, locations where there are Python files, and in some other locations that don't make sense. This adds considerable time to the startup time of the application, we're talking between 2 and 9 seconds. Sounds like a broken (networked?) file system. The only time I've had that kind of problems with python startup is when I've had really slow anti-virus programs that scanned all the files I opened. But then it wasn't file requests that mattered, but actually opening them... It still wasn't anywhere near 9 seconds though... if anyone finds out more about this issue, feel free to add a note to this FAQ entry: http://www.effbot.org/pyfaq/why-does-python-sometimes-take-so-long-to-start.htm /F -- http://mail.python.org/mailman/listinfo/python-list
Re: Restricting import file lookup for pyd, dll, ...
Bernard Lebel wrote: Hi, That's because I'm using Python through another application, via the pywin32 extensions. When that other application starts, it performs several thousands of file requests (we're talking 4,500, roughly) in the Python installation, locations where there are Python files, and in some other locations that don't make sense. This adds considerable time to the startup time of the application, we're talking between 2 and 9 seconds. Sounds like a broken (networked?) file system. The only time I've had that kind of problems with python startup is when I've had really slow anti-virus programs that scanned all the files I opened. But then it wasn't file requests that mattered, but actually opening them... It still wasn't anywhere near 9 seconds though... -- http://mail.python.org/mailman/listinfo/python-list
Restricting import file lookup for pyd, dll, ...
Hello, Running Python 2.4.0 on Windows. I have run a series of tests using Filemon (SysInternals), which logs the file system requests performed by the computer. By default, when I import a module in Python, Python looks for pyd, dll, py, then pyw and finally pyc file. In our company we do not use pyd and dll for Python stuff. Is there any chance to disable Python from looking at these? Perhaps a flag we can change in Python? Or would it be doable by recompiling the sources? Any advice is welcomed Bernard -- http://mail.python.org/mailman/listinfo/python-list
Re: Restricting import file lookup for pyd, dll, ...
At Thursday 19/10/2006 18:38, Bernard Lebel wrote: By default, when I import a module in Python, Python looks for pyd, dll, py, then pyw and finally pyc file. In our company we do not use pyd and dll for Python stuff. Is there any chance to disable Python from looking at these? Perhaps a flag we can change in Python? Or would it be doable by recompiling the sources? You can't; part of the standard library are .pyd/.dll files. Those existence checks should be fairly fast - why are you worried about them? -- Gabriel Genellina Softlab SRL __ Correo Yahoo! Espacio para todos tus mensajes, antivirus y antispam ¡gratis! ¡Abrí tu cuenta ya! - http://correo.yahoo.com.ar -- http://mail.python.org/mailman/listinfo/python-list
Re: Restricting import file lookup for pyd, dll, ...
Hi, That's because I'm using Python through another application, via the pywin32 extensions. When that other application starts, it performs several thousands of file requests (we're talking 4,500, roughly) in the Python installation, locations where there are Python files, and in some other locations that don't make sense. This adds considerable time to the startup time of the application, we're talking between 2 and 9 seconds. This a problem with the application, not Python. But I'm looking for ways to minimize this overhead so the users of this application waste less time waiting after the startup. Thanks Bernard On 10/19/06, Gabriel Genellina [EMAIL PROTECTED] wrote: You can't; part of the standard library are .pyd/.dll files. Those existence checks should be fairly fast - why are you worried about them? -- http://mail.python.org/mailman/listinfo/python-list
Re: Restricting import file lookup for pyd, dll, ...
At Thursday 19/10/2006 22:38, Bernard Lebel wrote: That's because I'm using Python through another application, via the pywin32 extensions. When that other application starts, it performs several thousands of file requests (we're talking 4,500, roughly) in the Python installation, locations where there are Python files, and in some other locations that don't make sense. This adds considerable time to the startup time of the application, we're talking between 2 and 9 seconds. This a problem with the application, not Python. But I'm looking for ways to minimize this overhead so the users of this application waste less time waiting after the startup. Try to shorten the PYTHONPATH to the really required directories (deleting those locations that don't make sense). -- Gabriel Genellina Softlab SRL __ Correo Yahoo! Espacio para todos tus mensajes, antivirus y antispam ¡gratis! ¡Abrí tu cuenta ya! - http://correo.yahoo.com.ar -- http://mail.python.org/mailman/listinfo/python-list
Re: Restricting import file lookup for pyd, dll, ...
I'm affraid this step has already been done. I was not being pedantic when I said locations that don't make sense, I *really* meant it. You would have to see the Filemon log! Anyway, then is there a way to tell Python not to look for pyd/dll and others in certain locations? In locations were you expect not to have such files for example? Thanks Bernard On 10/19/06, Gabriel Genellina [EMAIL PROTECTED] wrote: At Thursday 19/10/2006 22:38, Bernard Lebel wrote: That's because I'm using Python through another application, via the pywin32 extensions. When that other application starts, it performs several thousands of file requests (we're talking 4,500, roughly) in the Python installation, locations where there are Python files, and in some other locations that don't make sense. This adds considerable time to the startup time of the application, we're talking between 2 and 9 seconds. This a problem with the application, not Python. But I'm looking for ways to minimize this overhead so the users of this application waste less time waiting after the startup. Try to shorten the PYTHONPATH to the really required directories (deleting those locations that don't make sense). -- Gabriel Genellina Softlab SRL __ Correo Yahoo! Espacio para todos tus mensajes, antivirus y antispam ¡gratis! ¡Abrí tu cuenta ya! - http://correo.yahoo.com.ar -- http://mail.python.org/mailman/listinfo/python-list
Re: Restricting import file lookup for pyd, dll, ...
At Thursday 19/10/2006 23:19, Bernard Lebel wrote: I'm affraid this step has already been done. I was not being pedantic when I said locations that don't make sense, I *really* meant it. You would have to see the Filemon log! Python searches modules along the directories in sys.path - if you see it looking at w:\foo\bar\baz I bet that such directory is included there. Well, it might not be a directory but a zip file, or you might have installed some import hooks (have you?) but on the most common cases, a simple scan along sys.path is performed. Anyway, then is there a way to tell Python not to look for pyd/dll and others in certain locations? In locations were you expect not to have such files for example? I'm not really sure, but I don't think it could be done. The OS does a similar scan when looking for needed libraries at program startup. Are you sure that *this* behavior is what causes your slow startup? What OS/filesystem are you using? Are you loading your application across the network, using a file share, samba or something like that? -- Gabriel Genellina Softlab SRL __ Correo Yahoo! Espacio para todos tus mensajes, antivirus y antispam ¡gratis! ¡Abrí tu cuenta ya! - http://correo.yahoo.com.ar -- http://mail.python.org/mailman/listinfo/python-list
Re: Restricting import file lookup for pyd, dll, ...
Bernard Lebel wrote: That's because I'm using Python through another application, via the pywin32 extensions. When that other application starts, it performs several thousands of file requests (we're talking 4,500, roughly) in the Python installation, locations where there are Python files, and in some other locations that don't make sense. This adds considerable time to the startup time of the application, we're talking between 2 and 9 seconds. a plain Python 2.4 interpreter can start, execute a command, and shut down in about 0.13 seconds on my machine. 2.5 does the same thing in 0.10 seconds. are you sure you're benchmarking *Python's* start up time, and not the time it takes to load all the modules used by your application, or the time it takes for filemon to print all those 4500 requests to the monitor window? /F -- http://mail.python.org/mailman/listinfo/python-list
