Re: Imports and memory usage
On Mon, Jul 20, 2009 at 07:07:08AM -0700, Pavel Skvazh wrote: I was going to ask this for ages now. For example we've got a controller. import os class MyController(BaseController): def do_stuff(): import os os.open(...) def do_more(): import os os.do(...) The question being, what's the best practice here for this exactly in pylons. PEP-8: imports at top level http://www.python.org/dev/peps/pep-0008/ When the server is launched, the top level import is done, making it availible for everyone in the module. I think the import of a controller is not done eagerly; controllers are imported when the first request is routed to a particular controller. If I'll only import os in the method, won't it cause more work to import it every single time the method is run? Yes, but is 1 ms of extra work really that important? m...@platonas:~ $ python -m timeit 'import os' 100 loops, best of 3: 0.972 usec per loop On the other hand, won't it probably be more efficient to import only for the time, the method that actually uses this import works and then (i believe) garbage collection will free the memory until the next time it'll be used. No; once a module is imported it stays in memory. The reference from sys.modules[module_name] keeps it alive. You can forcefully unload it with del sys.modules['os'], but I don't think the memory savings (if any) will be worth the trouble. Always measure when optimizing. Never make assumptions. I do understand that this question is only loosely related to pylons, but I'm new to python and don't know know where I can find this low level behaivior explained, especially in relation to Pylons and WSGI apps. If there're sources out their, I'd really appreciate if you'll point me their. I'd love to rtfm :) This presentation is very good for understanding the low-level behaviour of Python: http://video.google.com/videoplay?docid=7760178035196894549 I don't recall if it dives into the import mechanism and garbage collection; it's been a while since I watched it. Cheers! Marius Gedminas -- Ways in which you can avoid starting a flame war on a Debian mailing list. The subtle: * Write in your TODO list that you definitely need to reply to that mail -- Enrico Zini signature.asc Description: Digital signature
Re: Imports and memory usage
On subsequent imports of a module, Python will look to see if it's already loaded and, if it is, will hand you back a reference to it-- the module won't be reloaded. Repeatedly importing a module inside a method will cause a slight performance hit, but one that you would likely never notice. The bad thing about importing within functions or methods (or anywhere other than at the very top of a module) is that it's hard to tell what a module's dependencies are. You should only do that under rare circumstances. As far as garbage collection, I'm not sure that will happen with modules, since there's a reference in sys.modules. To boil this down to some succint advice: always place imports at the top of a module unless you have a very good reason to place them elsewhere, and don't worry about the performance impact of imports expect in special cases. On Jul 20, 7:07 am, Pavel Skvazh pavel.skv...@gmail.com wrote: I was going to ask this for ages now. For example we've got a controller. import os class MyController(BaseController): def do_stuff(): import os os.open(...) def do_more(): import os os.do(...) The question being, what's the best practice here for this exactly in pylons. When the server is launched, the top level import is done, making it availible for everyone in the module. If I'll only import os in the method, won't it cause more work to import it every single time the method is run? On the other hand, won't it probably be more efficient to import only for the time, the method that actually uses this import works and then (i believe) garbage collection will free the memory until the next time it'll be used. I do understand that this question is only loosely related to pylons, but I'm new to python and don't know know where I can find this low level behaivior explained, especially in relation to Pylons and WSGI apps. If there're sources out their, I'd really appreciate if you'll point me their. I'd love to rtfm :) Thanks a lot! --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to pylons-discuss+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: mod_wsgi and auth_kit: Login doesn't work
[Fri Jul 17 19:01:43 2009] [error] [client 192.168.1.100] TypeError: sequence of string values expected, value of type literal found, referer:http://192.168.1.100/ This seems like the common error in middleware to not return strings but probably a number in your case. You could try to wrap different middleware (AurhKit?) with wsgiref.validate to check it's conformance to the WSGI standard. http://docs.python.org/library/wsgiref.html#module-wsgiref.validate --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to pylons-discuss+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: Imports and memory usage
On Mon, Jul 20, 2009 at 7:07 AM, Pavel Skvazhpavel.skv...@gmail.com wrote: def do_more(): import os os.do(...) Don't do this. The question being, what's the best practice here for this exactly in pylons. When the server is launched, the top level import is done, making it availible for everyone in the module. If I'll only import os in the method, won't it cause more work to import it every single time the method is run? It has little to do with Pylons specifically, so much as it has to do with Python. Modules don't get GC'ed unless you do special tricks to unload them. After the first import, all you're effectively doing is re-establishing the namespace reference to point to the module object already stored in sys.modules. You're simply not saving memory doing this. And, in fact, you're costing yourself performance. See: http://wiki.python.org/moin/PythonSpeed/PerformanceTips#ImportStatementOverhead This type of late importing is actually costly (as well as generally ineffective), so I'd just avoid it. Your goal in importing should be to establish what you need to reference in a module that you're writing, so generally do it at the module level. Let Python take care of the rest. -- Kyle. www.kylev.com --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to pylons-discuss+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: Imports and memory usage
2009/7/20 Pavel Skvazh pavel.skv...@gmail.com I was going to ask this for ages now. For example we've got a controller. import os class MyController(BaseController): def do_stuff(): import os os.open(...) def do_more(): import os os.do(...) The question being, what's the best practice here for this exactly in pylons. When the server is launched, the top level import is done, making it availible for everyone in the module. If I'll only import os in the method, won't it cause more work to import it every single time the method is run? They are both basically equivalent. When you do an import os, the first thing it checks is sys.modules['os'], and if it's there it just returns that module object. This is pretty cheap, and no objects are created. You might even gain the time back because now os is a local variable, and those are slightly faster than globals (if you did import at the global scope). The only time memory is effected is if you wouldn't have imported os at all. For a more obscure module this might work, though generally it just means your application warms up -- everything gets imported eventually, but just not all at once, trading off startup time for a little delay when you do first load the module. -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to pylons-discuss+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
Re: mod_wsgi and auth_kit: Login doesn't work
On Jul 21, 3:10 am, Damjan gdam...@gmail.com wrote: [Fri Jul 17 19:01:43 2009] [error] [client 192.168.1.100] TypeError: sequence of string values expected, value of type literal found, referer:http://192.168.1.100/ This seems like the common error in middleware to not return strings but probably a number in your case. You could try to wrap different middleware (AurhKit?) with wsgiref.validate to check it's conformance to the WSGI standard. http://docs.python.org/library/wsgiref.html#module-wsgiref.validate FWIW, this isn't a problem with response headers, where returning types other than strings is often a problem, but in the type of values yielded by the returned iterable, or as contained in the list returned. They should be strings and instead it is returning something else. I am not actually sure that wsgiref.validate properly checks that the iterable yields strings, so it may not pick it up correctly. At least my tests indicate that returning a non string from the iterable just gets passed straight through wsgiref.validate and still chokes on mod_wsgi. This could be viewed as a bug/deficiency in wsgiref.validate. Doesn't seem to be fixed in Python 3.1 either. Graham --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups pylons-discuss group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to pylons-discuss+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~--~~~~--~~--~--~---
