Re: I thought I understood how import worked...
On 2012-08-08 06:14, Ben Finney wrote:
Cameron Simpson c...@zip.com.au writes:
All of you are saying two names for the same module, and variations
thereof. And that is why the doco confuses.
I would expect less confusion if the above example were described as
_two_ modules, with the same source code.
That's not true though, is it? It's the same module object with two
different references, I thought.
They are not the same. Proof:
$ mkdir test
$ cd test
$ touch __init__.py
$ touch m.py
$ cd ..
$ python
Python 2.7.3 (default, Apr 20 2012, 22:39:59)
[GCC 4.6.3] on linux2
Type help, copyright, credits or license for more information.
import sys
sys.path.append('test')
import m
from test import m
import m
from test import m as m2
m is m2
False
m.a = 3
m2.a
Traceback (most recent call last):
File stdin, line 1, in module
AttributeError: 'module' object has no attribute 'a'
So it is still true that top level code gets executed only once, when
the module is first imported. The trick is that a module is not a file.
It is a module object that is created from a file, with a name. If you
change the name, then you create (import) a new module.
You can also use the reload() function to execute module level code
again, but it won't create a new module object. It will just update the
contents of the very same module object:
What is more interesting is how the reload() function works:
Python 2.7.3 (default, Apr 20 2012, 22:39:59)
[GCC 4.6.3] on linux2
Type help, copyright, credits or license for more information.
import test.m
a = test.m
import os
test.m is a
True
os.system(echo \import sys\ test/m.py)
0
reload(test.m) # Updates the module object
module 'test.m' from 'test/m.py'
test.m is a # They are still the same
True
a.sys # So a.sys is a exist
module 'sys' (built-in)
--
http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
Roy Smith wrote: So, it appears that you *can* import a module twice, if you refer to it by different names! This is surprising. It means that having non-idempotent code which is executed at import time is a Bad Thing. Not exactly, it means that one module is different from another if its path is different. That means you need to be extra careful about how you reference a module. Content is not used to discriminate modules. JM -- http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
In article 87hasehvfu@benfinney.id.au,
Ben Finney ben+pyt...@benfinney.id.au wrote:
Cameron Simpson c...@zip.com.au writes:
All of you are saying two names for the same module, and variations
thereof. And that is why the doco confuses.
I would expect less confusion if the above example were described as
_two_ modules, with the same source code.
That's not true though, is it? It's the same module object with two
different references, I thought.
Nope. I modified my test case to print out the id of the module:
('broken', module 'broken' from
'/home/roy/play/import/foo/broken.pyc') 140608299115512
('foo.broken', module 'foo.broken' from
'/home/roy/play/import/foo/broken.pyc') 140608299116232
Also, even if what you say were true, âsource codeâ implies the module
was loaded from source code, when Python allows loading modules with no
source code available.
This is true. In fact, when I first started chasing this down, one
import was finding the .py file, and the other the .pyc file (created
during the first import). I originally assumed that the .py/.pyc
distinction was the critical piece of the puzzle (and went down a
rathole exploring that).
Then I went down a different rathole when I noticed that one code path
was doing import settings, while the other was doing
__import__(module_name), thinking import and __import__ were somehow
doing different things.
--
http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
On 08Aug2012 14:14, Ben Finney ben+pyt...@benfinney.id.au wrote: | Cameron Simpson c...@zip.com.au writes: | All of you are saying two names for the same module, and variations | thereof. And that is why the doco confuses. | | I would expect less confusion if the above example were described as | _two_ modules, with the same source code. | | That's not true though, is it? It's the same module object with two | different references, I thought. No. They're two in-memory module instantiations with distinct module names. ISTR that when this got me I had imported the a module in one place with its full name and elsewhere with a bad relative reference; I forget the details now. But anyway, that got me the same module code loaded twice, distinctly named. | Also, even if what you say were true, “source code” implies the module | was loaded from source code, when Python allows loading modules with no | source code available. So that implication just seems to be inviting | different confusion. Please let's nail one thing first. We can always quibble about the term source code to prefer source code or byte code or something of that ilki, later. But the confusion comes from calling these things the same module. Which breaks the modules are only loaded once mental image. The confusion would often be avoided if the doco took the line that it is two modules because they were obtained using two names. Their original might be the same physical/logical place (such as a module source code file), but the end result is two modules. The point is that the module term would be better if it referred to the in-memory Python module, and didn't do double time as meaning the (typically) source file from which the Python code comes. Cheers, -- Cameron Simpson c...@zip.com.au We had the experience, but missed the meaning. - T.S. Eliot -- http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
I don't think the modules are actually imported twice. The entry is just
doubled;that's all
On 7 August 2012 18:48, Roy Smith r...@panix.com wrote:
I've been tracking down some weird import problems we've been having with
django. Our settings.py file is getting imported twice. It has some
non-idempotent code in it, and we blow up on the second import.
I thought modules could not get imported twice. The first time they get
imported, they're cached, and the second import just gets you a reference
to the
original. Playing around, however, I see that it's possible to import a
module
twice if you refer to it by different names. Here's a small-ish test case
which
demonstrates what I'm talking about (python 2.6.5):
In directory /home/roy/play/import/foo, I've got:
__init__.py (empty file)
try.py
broken.py
$ cat broken.py
print __file__
$ cat try.py
import broken
import foo.broken
import sys
for m in sys.modules.items():
if m[0].endswith('broken'):
print m
And when I run try.py (with foo as the current directory):
$ PYTHONPATH=/home/roy/play/import python try.py
/home/roy/play/import/foo/broken.pyc
/home/roy/play/import/foo/broken.pyc
('broken', module 'broken' from '/home/roy/play/import/foo/broken.pyc')
('foo.broken', module 'foo.broken' from
'/home/roy/play/import/foo/broken.pyc')
So, it appears that you *can* import a module twice, if you refer to it by
different names! This is surprising. It means that having non-idempotent
code
which is executed at import time is a Bad Thing.
It also means that you could have multiple copies of a module's global
namespace, depending on how your users imported the module. Which is kind
of
mind-blowing.
--
http://mail.python.org/mailman/listinfo/python-list
--
http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
On Tue, 07 Aug 2012 09:18:26 -0400, Roy Smith wrote: I thought modules could not get imported twice. The first time they get imported, they're cached, and the second import just gets you a reference to the original. Playing around, however, I see that it's possible to import a module twice if you refer to it by different names. Yes. You've found a Python gotcha. The most common example of this is when a single file acts as both an importable module, and as a runnable script. When run as a script, it is known as __main__. When imported, it is known by the file name. Unless you take care, it is easy to end up with the module imported twice. The usual advice is never have one module used as both script and importable module. I think *never* is a bit strong, but if you do so, you need to take extra care. Here's a small-ish test case which demonstrates what I'm talking about (python 2.6.5): In directory /home/roy/play/import/foo, I've got: __init__.py (empty file) try.py broken.py Aside: calling a module try.py is asking for trouble, because you can't do this: import try $ cat broken.py print __file__ $ cat try.py import broken import foo.broken Which are two names for the same module. So, it appears that you *can* import a module twice, if you refer to it by different names! This is surprising. It means that having non-idempotent code which is executed at import time is a Bad Thing. Well yes. In general, you should avoid non-idempotent code. You should doubly avoid it during imports, and triply avoid it on days ending with Y. The rest of the time, it is perfectly safe to have non-idempotent code. :) I kid, of course, but only half. Side-effects are bad, non-idempotent code is bad, and you should avoid them as much as possible, and unless you have no other reasonable choice. It also means that you could have multiple copies of a module's global namespace, depending on how your users imported the module. Which is kind of mind-blowing. Oh that part is trivial. Module namespaces are just dicts, there's nothing special about them. py import math # for example py import copy py namespace = copy.deepcopy(math.__dict__) py math.__dict__ == namespace True py math.__dict__ is namespace False It are modules which should be special, and Python tries really hard to ensure that they are singletons. (Multitons?) But not superhumanly hard. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
Roy Smith r...@panix.com writes: So, it appears that you *can* import a module twice, if you refer to it by different names! This is surprising. The tutorial is misleading on this. It it says plainly: A module can contain executable statements as well as function definitions. […] They are executed only the *first* time the module is imported somewhere. URL:http://docs.python.org/tutorial/modules.html but it doesn't make clear that a module can exist in the ‘sys.modules’ list multiple times under different names. Care to file a documentation bug URL:http://bugs.python.org/ describing this? It means that having non-idempotent code which is executed at import time is a Bad Thing. This is true whether or not the above about module imports is true. A well-designed module should have top level code that performs idempotent actions. Be thankful that you've discovered this, and apply it well :-) -- \ “See, in my line of work you gotta keep repeating things over | `\ and over and over again, for the truth to sink in; to kinda | _o__) catapult the propaganda.” —George W. Bush, 2005-05 | Ben Finney -- http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
On 07/08/2012 14:28, Ramchandra Apte wrote: I don't think the modules are actually imported twice. The entry is just doubled;that's all Please don't top post, this is the third time of asking. -- Cheers. Mark Lawrence. -- http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
On 2012-08-07 15:55, Ben Finney wrote: Roy Smith r...@panix.com writes: So, it appears that you *can* import a module twice, if you refer to it by different names! This is surprising. The tutorial is misleading on this. It it says plainly: A module can contain executable statements as well as function definitions. […] They are executed only the *first* time the module is imported somewhere. URL:http://docs.python.org/tutorial/modules.html but it doesn't make clear that a module can exist in the ‘sys.modules’ list multiple times under different names. sys.modules is a dict. But yes, there can be multiple instances of the same module loaded. What I do with bigger projects is that I always use absolute module names. For example, when I develop a project called project1 that has several sub packages, then I always do these kinds of imports: from project1.package1.subpackage2.submodule3 import * from project1.package1.subpackage2 import submodule3 from project1.package1.subpackage2.submodule3 import some_class Even from a source file that is inside project1.package1.subpackage2, I tend to import them the same way. This makes sure that every module is imported under the same package path. You just need to make sure that the main project has a unique name (which is usually the case) and that it is on your sys path (which is usually the case, especially when the script is started in the project's directory). The cost is that you have to type more. The benefit is that you can be sure that you are importing the thing that you want to import, and there will be no multiple imports for the same module. Mabye somebody will give method that works even better. For small projects without sub-packages, it is not a problem. Best, Laszlo -- http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
On Tuesday, August 7, 2012 9:52:59 AM UTC-4, Steven D'Aprano wrote: In general, you should avoid non-idempotent code. You should doubly avoid it during imports, and triply avoid it on days ending with Y. I don't understand your aversion to non-idempotent code as a general rule. Most code is non-idempotent. Surely you're not saying we should never write: foo += 1 or my_list.pop() ??? Making top-level module code idempotent, I can understand (given this new-found revelation that modules aren't really singletons), but you seem to be arguing something stronger and more general. -- http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
On Tuesday, August 7, 2012 9:55:16 AM UTC-4, Ben Finney wrote: The tutorial is misleading on this. It it says plainly: A module can contain executable statements as well as function definitions. […] They are executed only the *first* time the module is imported somewhere. URL:http://docs.python.org/tutorial/modules.html That's more than misleading. It's plain wrong. The example I gave demonstrates the print __file__ statement getting executed twice. The footnote to that is wrong too: [1] In fact function definitions are also ‘statements’ that are ‘executed’; the execution of a module-level function enters the function name in the module’s global symbol table. I think what it's supposed to say is ... the execution of a module-level def statement ... Care to file a documentation bug URL:http://bugs.python.org/ describing this? Sure, once I understand how it's really supposed to work :-) -- http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
Roy Smith r...@panix.com writes: In general, you should avoid non-idempotent code. I don't understand your aversion to non-idempotent code as a general rule. Most code is non-idempotent. Surely you're not saying we should never write: foo += 1 or my_list.pop() ??? I don't think in general avoid means the same thing as never write. One of the tenets of the functional-programming movement is that it is in fact reasonable to write in a style that avoids foo += 1 and my_list.pop() most of the time, leading to cleaner, more reliable code. In Python it's not possible to get rid of ALL of the data mutation without horrendous contortions, but it's pretty easy (and IMHO of worthwhile benefit) to avoid quite a lot of it. -- http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
On Aug 7, 2012 8:41 AM, Roy Smith r...@panix.com wrote: On Tuesday, August 7, 2012 9:55:16 AM UTC-4, Ben Finney wrote: The tutorial is misleading on this. It it says plainly: A module can contain executable statements as well as function definitions. […] They are executed only the *first* time the module is imported somewhere. URL:http://docs.python.org/tutorial/modules.html That's more than misleading. It's plain wrong. The example I gave demonstrates the print __file__ statement getting executed twice. The footnote to that is wrong too: [1] In fact function definitions are also ‘statements’ that are ‘executed’; the execution of a module-level function enters the function name in the module’s global symbol table. I think what it's supposed to say is ... the execution of a module-level def statement ... Care to file a documentation bug URL:http://bugs.python.org/ describing this? Sure, once I understand how it's really supposed to work :-) -- Each module only gets imported once. But if the same module can be accessed as two different names, python doesn't recognize that they are the same module. Along the same lines, if you create a symlink to the file, you'll be able to import the same module twice. -- http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
On 8/7/2012 9:28 AM, Ramchandra Apte wrote: I don't think the modules are actually imported twice. This is incorrect as Roy's original unposted example showed. Modify one of the two copies and it will be more obvious. PS. I agree with Mark about top posting. I often just glance as such postings rather that go look to find out the context. However, this one is wrong on its own ;-). -- Terry Jan Reedy -- http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
On 8/7/2012 11:32 AM, Roy Smith wrote: On Tuesday, August 7, 2012 9:55:16 AM UTC-4, Ben Finney wrote: The tutorial is misleading on this. It it says plainly: A module can contain executable statements as well as function definitions. […] They are executed only the *first* time the module is imported somewhere. The last sentence should be more like They are executed only the *first* time the module is imported anywhere with a particular name. (One should avoid importing a module under different names.) URL:http://docs.python.org/tutorial/modules.html [1] In fact function definitions are also ‘statements’ that are ‘executed’; the execution of a module-level function enters the function name in the module’s global symbol table. I think what it's supposed to say is ... the execution of a module-level def statement ... right Care to file a documentation bug URL:http://bugs.python.org/ describing this? Sure, once I understand how it's really supposed to work :-) You don't need a final solution to file. Anyway, I think the change above might be enough. -- Terry Jan Reedy -- http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
On 07/08/2012 14:18, Roy Smith wrote:
I've been tracking down some weird import problems we've been having with
django. Our settings.py file is getting imported twice. It has some
non-idempotent code in it, and we blow up on the second import.
I thought modules could not get imported twice. The first time they get
imported, they're cached, and the second import just gets you a reference to the
original. Playing around, however, I see that it's possible to import a module
twice if you refer to it by different names. Here's a small-ish test case which
demonstrates what I'm talking about (python 2.6.5):
In directory /home/roy/play/import/foo, I've got:
__init__.py (empty file)
try.py
broken.py
$ cat broken.py
print __file__
$ cat try.py
import broken
import foo.broken
import sys
for m in sys.modules.items():
if m[0].endswith('broken'):
print m
And when I run try.py (with foo as the current directory):
$ PYTHONPATH=/home/roy/play/import python try.py
/home/roy/play/import/foo/broken.pyc
/home/roy/play/import/foo/broken.pyc
('broken', module 'broken' from '/home/roy/play/import/foo/broken.pyc')
('foo.broken', module 'foo.broken' from
'/home/roy/play/import/foo/broken.pyc')
So, it appears that you *can* import a module twice, if you refer to it by
different names! This is surprising. It means that having non-idempotent code
which is executed at import time is a Bad Thing.
It also means that you could have multiple copies of a module's global
namespace, depending on how your users imported the module. Which is kind of
mind-blowing.
Maybe not directly applicable to what you're saying, but Brett Cannon
ought to know something about the import mechanism. I believe he's been
working on it on and off for several years. See
http://docs.python.org/dev/whatsnew/3.3.html for a starter on the gory
details.
--
Cheers.
Mark Lawrence.
--
http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
On Tue, 07 Aug 2012 08:25:43 -0700, Roy Smith wrote: On Tuesday, August 7, 2012 9:52:59 AM UTC-4, Steven D'Aprano wrote: In general, you should avoid non-idempotent code. You should doubly avoid it during imports, and triply avoid it on days ending with Y. You seem to have accidentally deleted my smiley. I don't understand your aversion to non-idempotent code as a general rule. Most code is non-idempotent. That doesn't necessarily make it a good thing. Most code is also buggy. Surely you're not saying we should never write: foo += 1 or my_list.pop() ??? Of course not. I'm not going so far as to say that we should always, without exception, write purely functional code. I like my list.append as much as anyone :) But at the level of larger code units, functions and modules, it is a useful property to have where possible. A function is free to increment an integer, or pop items from a list, as much as it likes -- so long as they are *local* to the function, and get reset to their initial state each time the function is called with the same arguments. I realise that many problems are most easily satisfied by non-idempotent tactics. Customer orders widget is not naturally idempotent, since if the customer does it twice, they get two widgets, not one. But such behaviour should be limited to the parts of your code which must be non- idempotent. In short, non-idempotent code is hard to get right, hard to test, and hard to debug, so we should use as little of it as possible. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
On 07Aug2012 13:52, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: | On Tue, 07 Aug 2012 09:18:26 -0400, Roy Smith wrote: | I thought modules could not get imported twice. The first time they get | imported, they're cached, and the second import just gets you a | reference to the original. Playing around, however, I see that it's | possible to import a module twice if you refer to it by different names. | | Yes. You've found a Python gotcha. [...] | $ cat try.py | import broken | import foo.broken | | Which are two names for the same module. [...] This, I think, is a core issue in this misunderstanding. (I got bitten by this too, maybe a year ago. My error, and I'm glad to have improved my understanding.) All of you are saying two names for the same module, and variations thereof. And that is why the doco confuses. I would expect less confusion if the above example were described as _two_ modules, with the same source code. Make it clear that these are _two_ modules (because they have two names), who merely happen to have been obtained from the same physical filesystem object due to path search effects i.e. change the doco wording to describe a module as the in-memory result of reading a file found from an import name. So I think I'm arguing for a small change in terminology in the doco with no change in Python semantics. Is a module a set of files on the disc, or an in-memory Python notion with a name? I would argue for the latter. With such a change, the a module can't be imported twice would then be true (barring hacking around in sys.modules between imports). Cheers, -- Cameron Simpson c...@zip.com.au As you can see, unraveling even a small part of 'sendmail' can introduce more complexity than answers.- Brian Costales, _sendmail_ -- http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
In article mailman.3071.1344380066.4697.python-l...@python.org, Cameron Simpson c...@zip.com.au wrote: This, I think, is a core issue in this misunderstanding. (I got bitten by this too, maybe a year ago. My error, and I'm glad to have improved my understanding.) All of you are saying two names for the same module, and variations thereof. And that is why the doco confuses. I would expect less confusion if the above example were described as _two_ modules, with the same source code. +1 -- http://mail.python.org/mailman/listinfo/python-list
Re: I thought I understood how import worked...
Cameron Simpson c...@zip.com.au writes: All of you are saying two names for the same module, and variations thereof. And that is why the doco confuses. I would expect less confusion if the above example were described as _two_ modules, with the same source code. That's not true though, is it? It's the same module object with two different references, I thought. Also, even if what you say were true, “source code” implies the module was loaded from source code, when Python allows loading modules with no source code available. So that implication just seems to be inviting different confusion. -- \ “I'm not a bad guy! I work hard, and I love my kids. So why | `\ should I spend half my Sunday hearing about how I'm going to | _o__)Hell?” —Homer Simpson | Ben Finney -- http://mail.python.org/mailman/listinfo/python-list
