Re: I thought I understood how import worked...
On 08Aug2012 14:14, Ben Finney wrote: | Cameron Simpson writes: | > All of you are saying "two names for the same module", and variations | > thereof. And that is why the doco confuses. | > | > I would expect less confusion if the above example were described as | > _two_ modules, with the same source code. | | That's not true though, is it? It's the same module object with two | different references, I thought. No. They're two in-memory module instantiations with distinct module names. ISTR that when this got me I had imported the a module in one place with its full name and elsewhere with a bad relative reference; I forget the details now. But anyway, that got me the same module code loaded twice, distinctly named. | Also, even if what you say were true, “source code” implies the module | was loaded from source code, when Python allows loading modules with no | source code available. So that implication just seems to be inviting | different confusion. Please let's nail one thing first. We can always quibble about the term "source code" to prefer "source code or byte code" or something of that ilki, later. But the confusion comes from calling these things the "same module". Which breaks the "modules are only loaded once" mental image. The confusion would often be avoided if the doco took the line that it is two modules because they were obtained using two names. Their original might be the same physical/logical place (such as a module source code file), but the end result is two modules. The point is that the "module" term would be better if it referred to the in-memory Python module, and didn't do double time as meaning the (typically) source file from which the Python code comes. Cheers, -- Cameron Simpson We had the experience, but missed the meaning. - T.S. Eliot -- http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
In article <87hasehvfu@benfinney.id.au>, Ben Finney wrote: > Cameron Simpson writes: > > > All of you are saying "two names for the same module", and variations > > thereof. And that is why the doco confuses. > > > > I would expect less confusion if the above example were described as > > _two_ modules, with the same source code. > > That's not true though, is it? It's the same module object with two > different references, I thought. Nope. I modified my test case to print out the id of the module: ('broken', ) 140608299115512 ('foo.broken', ) 140608299116232 > Also, even if what you say were true, âsource codeâ implies the module > was loaded from source code, when Python allows loading modules with no > source code available. This is true. In fact, when I first started chasing this down, one import was finding the .py file, and the other the .pyc file (created during the first import). I originally assumed that the .py/.pyc distinction was the critical piece of the puzzle (and went down a rathole exploring that). Then I went down a different rathole when I noticed that one code path was doing "import settings", while the other was doing "__import__(module_name)", thinking import and __import__ were somehow doing different things. -- http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
Roy Smith wrote: So, it appears that you *can* import a module twice, if you refer to it by different names! This is surprising. It means that having non-idempotent code which is executed at import time is a Bad Thing. Not exactly, it means that one module is different from another if its path is different. That means you need to be extra careful about how you reference a module. Content is not used to discriminate modules. JM -- http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
On 2012-08-08 06:14, Ben Finney wrote: Cameron Simpson writes: All of you are saying "two names for the same module", and variations thereof. And that is why the doco confuses. I would expect less confusion if the above example were described as _two_ modules, with the same source code. That's not true though, is it? It's the same module object with two different references, I thought. They are not the same. Proof: $ mkdir test $ cd test $ touch __init__.py $ touch m.py $ cd .. $ python Python 2.7.3 (default, Apr 20 2012, 22:39:59) [GCC 4.6.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.path.append('test') >>> import m >>> from test import m >>> import m >>> from test import m as m2 >>> m is m2 False >>> m.a = 3 >>> m2.a Traceback (most recent call last): File "", line 1, in AttributeError: 'module' object has no attribute 'a' So it is still true that top level code gets executed only once, when the module is first imported. The trick is that a module is not a file. It is a module object that is created from a file, with a name. If you change the name, then you create ("import") a new module. You can also use the reload() function to execute module level code again, but it won't create a new module object. It will just update the contents of the very same module object: What is more interesting is how the reload() function works: Python 2.7.3 (default, Apr 20 2012, 22:39:59) [GCC 4.6.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import test.m >>> a = test.m >>> import os >>> test.m is a True >>> os.system("echo \"import sys\" >> test/m.py") 0 >>> reload(test.m) # Updates the module object >>> test.m is a # They are still the same True >>> a.sys # So a.sys is a exist >>> -- http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
Cameron Simpson writes: > All of you are saying "two names for the same module", and variations > thereof. And that is why the doco confuses. > > I would expect less confusion if the above example were described as > _two_ modules, with the same source code. That's not true though, is it? It's the same module object with two different references, I thought. Also, even if what you say were true, “source code” implies the module was loaded from source code, when Python allows loading modules with no source code available. So that implication just seems to be inviting different confusion. -- \ “I'm not a bad guy! I work hard, and I love my kids. So why | `\ should I spend half my Sunday hearing about how I'm going to | _o__)Hell?” —Homer Simpson | Ben Finney -- http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
In article , Cameron Simpson wrote: > This, I think, is a core issue in this misunderstanding. (I got bitten > by this too, maybe a year ago. My error, and I'm glad to have improved > my understanding.) > > All of you are saying "two names for the same module", and variations > thereof. And that is why the doco confuses. > > I would expect less confusion if the above example were described as > _two_ modules, with the same source code. +1 -- http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
On 07Aug2012 13:52, Steven D'Aprano wrote: | On Tue, 07 Aug 2012 09:18:26 -0400, Roy Smith wrote: | > I thought modules could not get imported twice. The first time they get | > imported, they're cached, and the second import just gets you a | > reference to the original. Playing around, however, I see that it's | > possible to import a module twice if you refer to it by different names. | | Yes. You've found a Python gotcha. [...] | > $ cat try.py | > import broken | > import foo.broken | | Which are two names for the same module. [...] This, I think, is a core issue in this misunderstanding. (I got bitten by this too, maybe a year ago. My error, and I'm glad to have improved my understanding.) All of you are saying "two names for the same module", and variations thereof. And that is why the doco confuses. I would expect less confusion if the above example were described as _two_ modules, with the same source code. Make it clear that these are _two_ modules (because they have two names), who merely happen to have been obtained from the same "physical" filesystem object due to path search effects i.e. change the doco wording to describe a module as the in-memory result of reading a "file" found from an import name. So I think I'm arguing for a small change in terminology in the doco with no change in Python semantics. Is a module a set of files on the disc, or an in-memory Python notion with a name? I would argue for the latter. With such a change, the "a module can't be imported twice" would then be true (barring hacking around in sys.modules between imports). Cheers, -- Cameron Simpson As you can see, unraveling even a small part of 'sendmail' can introduce more complexity than answers.- Brian Costales, _sendmail_ -- http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
On Tue, 07 Aug 2012 08:25:43 -0700, Roy Smith wrote: > On Tuesday, August 7, 2012 9:52:59 AM UTC-4, Steven D'Aprano wrote: > >> In general, you should avoid non-idempotent code. You should doubly >> avoid it during imports, and triply avoid it on days ending with Y. You seem to have accidentally deleted my smiley. > I don't understand your aversion to non-idempotent code as a general > rule. Most code is non-idempotent. That doesn't necessarily make it a good thing. Most code is also buggy. > Surely you're not saying we should never write: > foo += 1 > > or > my_list.pop() > > ??? Of course not. I'm not going so far as to say that we should always, without exception, write purely functional code. I like my list.append as much as anyone :) But at the level of larger code units, functions and modules, it is a useful property to have where possible. A function is free to increment an integer, or pop items from a list, as much as it likes -- so long as they are *local* to the function, and get reset to their initial state each time the function is called with the same arguments. I realise that many problems are most easily satisfied by non-idempotent tactics. "Customer orders widget" is not naturally idempotent, since if the customer does it twice, they get two widgets, not one. But such behaviour should be limited to the parts of your code which must be non- idempotent. In short, non-idempotent code is hard to get right, hard to test, and hard to debug, so we should use as little of it as possible. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
On 07/08/2012 14:18, Roy Smith wrote: I've been tracking down some weird import problems we've been having with django. Our settings.py file is getting imported twice. It has some non-idempotent code in it, and we blow up on the second import. I thought modules could not get imported twice. The first time they get imported, they're cached, and the second import just gets you a reference to the original. Playing around, however, I see that it's possible to import a module twice if you refer to it by different names. Here's a small-ish test case which demonstrates what I'm talking about (python 2.6.5): In directory /home/roy/play/import/foo, I've got: __init__.py (empty file) try.py broken.py $ cat broken.py print __file__ $ cat try.py import broken import foo.broken import sys for m in sys.modules.items(): if m[0].endswith('broken'): print m And when I run try.py (with foo as the current directory): $ PYTHONPATH=/home/roy/play/import python try.py /home/roy/play/import/foo/broken.pyc /home/roy/play/import/foo/broken.pyc ('broken', ) ('foo.broken', ) So, it appears that you *can* import a module twice, if you refer to it by different names! This is surprising. It means that having non-idempotent code which is executed at import time is a Bad Thing. It also means that you could have multiple copies of a module's global namespace, depending on how your users imported the module. Which is kind of mind-blowing. Maybe not directly applicable to what you're saying, but Brett Cannon ought to know something about the import mechanism. I believe he's been working on it on and off for several years. See http://docs.python.org/dev/whatsnew/3.3.html for a starter on the gory details. -- Cheers. Mark Lawrence. -- http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
On 8/7/2012 11:32 AM, Roy Smith wrote: On Tuesday, August 7, 2012 9:55:16 AM UTC-4, Ben Finney wrote: The tutorial is misleading on this. It it says plainly: A module can contain executable statements as well as function definitions. […] They are executed only the *first* time the module is imported somewhere. The last sentence should be more like "They are executed only the *first* time the module is imported anywhere with a particular name. (One should avoid importing a module under different names.) http://docs.python.org/tutorial/modules.html> [1] In fact function definitions are also ‘statements’ that are ‘executed’; the execution of a module-level function enters the function name in the module’s global symbol table. I think what it's supposed to say is "... the execution of a module-level def statement ..." right Care to file a documentation bug http://bugs.python.org/> describing this? Sure, once I understand how it's really supposed to work :-) You don't need a final solution to file. Anyway, I think the change above might be enough. -- Terry Jan Reedy -- http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
On 8/7/2012 9:28 AM, Ramchandra Apte wrote: I don't think the modules are actually imported twice. This is incorrect as Roy's original unposted example showed. Modify one of the two copies and it will be more obvious. PS. I agree with Mark about top posting. I often just glance as such postings rather that go look to find out the context. However, this one is wrong on its own ;-). -- Terry Jan Reedy -- http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
On Aug 7, 2012 8:41 AM, "Roy Smith" wrote: > > On Tuesday, August 7, 2012 9:55:16 AM UTC-4, Ben Finney wrote: > > > The tutorial is misleading on this. It it says plainly: > > > > A module can contain executable statements as well as function > > definitions. […] They are executed only the *first* time the module > > is imported somewhere. > > > > http://docs.python.org/tutorial/modules.html> > > That's more than misleading. It's plain wrong. The example I gave demonstrates the "print __file__" statement getting executed twice. > > The footnote to that is wrong too: > > > [1] In fact function definitions are also ‘statements’ that are ‘executed’; the execution of a > > module-level function enters the function name in the module’s global symbol table. > > I think what it's supposed to say is "... the execution of a module-level def statement ..." > > > Care to file a documentation bug http://bugs.python.org/> > > describing this? > > Sure, once I understand how it's really supposed to work :-) > > -- Each module only gets imported once. But if the same module can be accessed as two different names, python doesn't recognize that they are the same module. Along the same lines, if you create a symlink to the file, you'll be able to import the same module twice. -- http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
Roy Smith writes: >> In general, you should avoid non-idempotent code. > I don't understand your aversion to non-idempotent code as a general > rule. Most code is non-idempotent. Surely you're not saying we > should never write: foo += 1 > or my_list.pop() > ??? I don't think "in general avoid" means the same thing as "never write". One of the tenets of the functional-programming movement is that it is in fact reasonable to write in a style that avoids "foo += 1" and "my_list.pop()" most of the time, leading to cleaner, more reliable code. In Python it's not possible to get rid of ALL of the data mutation without horrendous contortions, but it's pretty easy (and IMHO of worthwhile benefit) to avoid quite a lot of it. -- http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
On Tuesday, August 7, 2012 9:55:16 AM UTC-4, Ben Finney wrote: > The tutorial is misleading on this. It it says plainly: > > A module can contain executable statements as well as function > definitions. […] They are executed only the *first* time the module > is imported somewhere. > > http://docs.python.org/tutorial/modules.html> That's more than misleading. It's plain wrong. The example I gave demonstrates the "print __file__" statement getting executed twice. The footnote to that is wrong too: > [1] In fact function definitions are also ‘statements’ that are ‘executed’; > the execution of a > module-level function enters the function name in the module’s global symbol > table. I think what it's supposed to say is "... the execution of a module-level def statement ..." > Care to file a documentation bug http://bugs.python.org/> > describing this? Sure, once I understand how it's really supposed to work :-) -- http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
On Tuesday, August 7, 2012 9:52:59 AM UTC-4, Steven D'Aprano wrote: > In general, you should avoid non-idempotent code. You should > doubly avoid it during imports, and triply avoid it on days ending with Y. I don't understand your aversion to non-idempotent code as a general rule. Most code is non-idempotent. Surely you're not saying we should never write: >>> foo += 1 or >>> my_list.pop() ??? Making top-level module code idempotent, I can understand (given this new-found revelation that modules aren't really singletons), but you seem to be arguing something stronger and more general. -- http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
On 2012-08-07 15:55, Ben Finney wrote: Roy Smith writes: So, it appears that you *can* import a module twice, if you refer to it by different names! This is surprising. The tutorial is misleading on this. It it says plainly: A module can contain executable statements as well as function definitions. […] They are executed only the *first* time the module is imported somewhere. http://docs.python.org/tutorial/modules.html> but it doesn't make clear that a module can exist in the ‘sys.modules’ list multiple times under different names. sys.modules is a dict. But yes, there can be multiple "instances" of the same module loaded. What I do with bigger projects is that I always use absolute module names. For example, when I develop a project called "project1" that has several sub packages, then I always do these kinds of imports: from project1.package1.subpackage2.submodule3 import * from project1.package1.subpackage2 import submodule3 from project1.package1.subpackage2.submodule3 import some_class Even from a source file that is inside project1.package1.subpackage2, I tend to import them the same way. This makes sure that every module is imported under the same package path. You just need to make sure that the main project has a unique name (which is usually the case) and that it is on your sys path (which is usually the case, especially when the script is started in the project's directory). The cost is that you have to type more. The benefit is that you can be sure that you are importing the thing that you want to import, and there will be no multiple imports for the same module. Mabye somebody will give method that works even better. For small projects without sub-packages, it is not a problem. Best, Laszlo -- http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
On 07/08/2012 14:28, Ramchandra Apte wrote: I don't think the modules are actually imported twice. The entry is just doubled;that's all Please don't top post, this is the third time of asking. -- Cheers. Mark Lawrence. -- http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
Roy Smith writes: > So, it appears that you *can* import a module twice, if you refer to > it by different names! This is surprising. The tutorial is misleading on this. It it says plainly: A module can contain executable statements as well as function definitions. […] They are executed only the *first* time the module is imported somewhere. http://docs.python.org/tutorial/modules.html> but it doesn't make clear that a module can exist in the ‘sys.modules’ list multiple times under different names. Care to file a documentation bug http://bugs.python.org/> describing this? > It means that having non-idempotent code which is executed at import > time is a Bad Thing. This is true whether or not the above about module imports is true. A well-designed module should have top level code that performs idempotent actions. Be thankful that you've discovered this, and apply it well :-) -- \ “See, in my line of work you gotta keep repeating things over | `\ and over and over again, for the truth to sink in; to kinda | _o__) catapult the propaganda.” —George W. Bush, 2005-05 | Ben Finney -- http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
On Tue, 07 Aug 2012 09:18:26 -0400, Roy Smith wrote: > I thought modules could not get imported twice. The first time they get > imported, they're cached, and the second import just gets you a > reference to the original. Playing around, however, I see that it's > possible to import a module twice if you refer to it by different names. Yes. You've found a Python gotcha. The most common example of this is when a single file acts as both an importable module, and as a runnable script. When run as a script, it is known as "__main__". When imported, it is known by the file name. Unless you take care, it is easy to end up with the module imported twice. The usual advice is "never have one module used as both script and importable module". I think *never* is a bit strong, but if you do so, you need to take extra care. > Here's a small-ish test case which demonstrates what I'm talking about > (python 2.6.5): > > In directory /home/roy/play/import/foo, I've got: > > __init__.py (empty file) > try.py > broken.py Aside: calling a module "try.py" is asking for trouble, because you can't do this: import try > $ cat broken.py > print __file__ > > > $ cat try.py > import broken > import foo.broken Which are two names for the same module. > So, it appears that you *can* import a module twice, if you refer to it > by different names! This is surprising. It means that having > non-idempotent code which is executed at import time is a Bad Thing. Well yes. In general, you should avoid non-idempotent code. You should doubly avoid it during imports, and triply avoid it on days ending with Y. The rest of the time, it is perfectly safe to have non-idempotent code. :) I kid, of course, but only half. Side-effects are bad, non-idempotent code is bad, and you should avoid them as much as possible, and unless you have no other reasonable choice. > It also means that you could have multiple copies of a module's global > namespace, depending on how your users imported the module. Which is > kind of mind-blowing. Oh that part is trivial. Module namespaces are just dicts, there's nothing special about them. py> import math # for example py> import copy py> namespace = copy.deepcopy(math.__dict__) py> math.__dict__ == namespace True py> math.__dict__ is namespace False It are modules which should be special, and Python tries really hard to ensure that they are singletons. (Multitons?) But not superhumanly hard. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
I don't think the modules are actually imported twice. The entry is just doubled;that's all On 7 August 2012 18:48, Roy Smith wrote: > I've been tracking down some weird import problems we've been having with > django. Our settings.py file is getting imported twice. It has some > non-idempotent code in it, and we blow up on the second import. > > I thought modules could not get imported twice. The first time they get > imported, they're cached, and the second import just gets you a reference > to the > original. Playing around, however, I see that it's possible to import a > module > twice if you refer to it by different names. Here's a small-ish test case > which > demonstrates what I'm talking about (python 2.6.5): > > In directory /home/roy/play/import/foo, I've got: > > __init__.py (empty file) > try.py > broken.py > > > $ cat broken.py > print __file__ > > > $ cat try.py > import broken > import foo.broken > > import sys > for m in sys.modules.items(): > if m[0].endswith('broken'): > print m > > > And when I run try.py (with foo as the current directory): > > $ PYTHONPATH=/home/roy/play/import python try.py > /home/roy/play/import/foo/broken.pyc > /home/roy/play/import/foo/broken.pyc > ('broken', ) > ('foo.broken', '/home/roy/play/import/foo/broken.pyc'>) > > > So, it appears that you *can* import a module twice, if you refer to it by > different names! This is surprising. It means that having non-idempotent > code > which is executed at import time is a Bad Thing. > > It also means that you could have multiple copies of a module's global > namespace, depending on how your users imported the module. Which is kind > of > mind-blowing. > -- > http://mail.python.org/mailman/listinfo/python-list > -- http://mail.python.org/mailman/listinfo/python-list
I thought I understood how import worked...
I've been tracking down some weird import problems we've been having with django. Our settings.py file is getting imported twice. It has some non-idempotent code in it, and we blow up on the second import. I thought modules could not get imported twice. The first time they get imported, they're cached, and the second import just gets you a reference to the original. Playing around, however, I see that it's possible to import a module twice if you refer to it by different names. Here's a small-ish test case which demonstrates what I'm talking about (python 2.6.5): In directory /home/roy/play/import/foo, I've got: __init__.py (empty file) try.py broken.py $ cat broken.py print __file__ $ cat try.py import broken import foo.broken import sys for m in sys.modules.items(): if m[0].endswith('broken'): print m And when I run try.py (with foo as the current directory): $ PYTHONPATH=/home/roy/play/import python try.py /home/roy/play/import/foo/broken.pyc /home/roy/play/import/foo/broken.pyc ('broken', ) ('foo.broken', ) So, it appears that you *can* import a module twice, if you refer to it by different names! This is surprising. It means that having non-idempotent code which is executed at import time is a Bad Thing. It also means that you could have multiple copies of a module's global namespace, depending on how your users imported the module. Which is kind of mind-blowing. -- http://mail.python.org/mailman/listinfo/python-list