Re: caught in the import web again
On Tue, 18 Nov 2014 00:00:42 -0700, Michael Torrie wrote: On 11/17/2014 03:45 PM, Steven D'Aprano wrote: Circular dependencies are not just a problem in Python, they are a problem throughout most of software design. Personally I find that duck typing eliminates a lot of the circular dependency problems. Class A doesn't necessarily have to know about Class B to work with instances of Class B. All that has to be known is what methods are going to be available on those instances. The interface pattern can help here if you wanted a bit more safety (like what Zope provides). Either way it breaks up the dependency cycle nicely. If Class A needs to instantiate instances of Class B (which depends on Class A), then it could call a factory method to do so, perhaps set up by a third party after both modules have been imported. There are lots of methods of avoiding the issue entirely. In the C or C++ world, I find that signalling frameworks can also be used to break a dependency cycle. For example, if Class A needs to call a method on an instance of Class B (which in turn depends on Class A), that could be done instead as a signal, which is later connected to Class B's method after everything is successfully instantiated. Also makes testing a bit easier, because you can simply hook up the signals to a test harness. A bunch of useful thoughts. Thanks, folks! -- https://mail.python.org/mailman/listinfo/python-list
Re: caught in the import web again
On Mon, Nov 17, 2014 at 6:20 PM, Dave Angel da...@davea.name wrote: Ian Kelly ian.g.ke...@gmail.com Wrote in message: On Mon, Nov 17, 2014 at 3:17 PM, Dave Angel da...@davea.name wrote: In a module that might get tangled in a cycle, avoid global code that depends on other modules. Instead of putting such initialization at top level, put inside a function that gets called after all suspect imports are completed. (That function has global keywords ). If the problem is that one of those modules would import the current module, then after all suspect imports are completed basically means after the current module has finished importing. So what would be responsible for calling such a function? If one builds a set of modules that cannot avoid recursive imports, then one can ask the code that imports it to make a call after importing. Or all that is needed to avoid exposing the mess is that the top level not be part of the loops. This would make the module less reusable though, since the first other module to import it in one program might still be used in another program but not be the first other module to import it. Or I suppose for simplicity it could be required that all importers call the initialization function, and have the function return immediately if it's been called previously. However, I don't think there's any way to do this without ending up with smelly code. -- https://mail.python.org/mailman/listinfo/python-list
Re: caught in the import web again
Ian Kelly ian.g.ke...@gmail.com Wrote in message: On Mon, Nov 17, 2014 at 6:20 PM, Dave Angel da...@davea.name wrote: Ian Kelly ian.g.ke...@gmail.com Wrote in message: On Mon, Nov 17, 2014 at 3:17 PM, Dave Angel da...@davea.name wrote: In a module that might get tangled in a cycle, avoid global code that depends on other modules. Instead of putting such initialization at top level, put inside a function that gets called after all suspect imports are completed. (That function has global keywords ). If the problem is that one of those modules would import the current module, then after all suspect imports are completed basically means after the current module has finished importing. So what would be responsible for calling such a function? If one builds a set of modules that cannot avoid recursive imports, then one can ask the code that imports it to make a call after importing. Or all that is needed to avoid exposing the mess is that the top level not be part of the loops. This would make the module less reusable though, since the first other module to import it in one program might still be used in another program but not be the first other module to import it. Or I suppose for simplicity it could be required that all importers call the initialization function, and have the function return immediately if it's been called previously. However, I don't think there's any way to do this without ending up with smelly code. The smelly code happened when the designer can't break the cycle. I understood the question to be then what. Two-stage initialization is sometimes needed in any language and I was just showing it's possible in Python. I once worked on (and then fixed) a build system that could not complete a build from clean. It needed some pieces from a previous build in order to get to the point where it was ready to build those pieces. Recursive depencies at compile and link time. -- DaveA -- https://mail.python.org/mailman/listinfo/python-list
Re: caught in the import web again
On Wed, Nov 19, 2014 at 6:09 AM, Dave Angel da...@davea.name wrote: I once worked on (and then fixed) a build system that could not complete a build from clean. It needed some pieces from a previous build in order to get to the point where it was ready to build those pieces. Recursive depencies at compile and link time. Sometimes the solution to that is to add intermediate files to the source code archive. Pike can't be built from source control unless you already have a reasonably recent Pike installed, but the exported .tar.gz archive includes all the generated files, so you can build that, then go to the absolute latest from source control. There are always ways around a circular dependency. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: caught in the import web again
Chris Angelico ros...@gmail.com Wrote in message: On Wed, Nov 19, 2014 at 6:09 AM, Dave Angel da...@davea.name wrote: I once worked on (and then fixed) a build system that could not complete a build from clean. It needed some pieces from a previous build in order to get to the point where it was ready to build those pieces. Recursive depencies at compile and link time. Sometimes the solution to that is to add intermediate files to the source code archive. Pike can't be built from source control unless you already have a reasonably recent Pike installed, but the exported .tar.gz archive includes all the generated files, so you can build that, then go to the absolute latest from source control. There are always ways around a circular dependency. Exactly. In the case I was talking about there was a somewhat related problem. The build broke every time, and the fixups were very manual. Nobody even knew if the results depended on the order of applying the fixups. The build lady spent about two days each week fixing stuff up. -- DaveA -- https://mail.python.org/mailman/listinfo/python-list
Re: caught in the import web again
On Mon, 17 Nov 2014 08:08:40 +0100, dieter wrote: Charles T. Smith cts.private.ya...@gmail.com writes: ... Are others equally frustrated by this or is there a trick or principle that I'm missing. At this point, I guess the way I'll have to proceed is to put every class in its own file, no matter how small. Hopefully that takes care of the problem. Recursive imports are inherently difficult. It is best if a careful design avoids it (note that a cycle free dependency structure is much easier to handle than a cyclic one -- not only for Python). If you think such a cycle free module structure is impossible in your case, then try to delay module related operations: * consider local imports (imports inside a function). They delay the import until the function is called rather than execute it at module import time. * prefer import module over from module import ... This may delay the module access; perhaps (but not necessarily) sufficiently such that the accessed object is already defined. Neither of these approaches is complete - you may still get cyclic import dependencies and related problems. Usually, you need a careful module dependency design to avoid cyclic dependencies altogether. I compared perl's object-oriented philosophy with python's and didn't like how perl *requires* you to put everything into its own file. Now it looks like python does, too, implicitly. If you put everything in a single file, then you do not get recursive imports (but you may still have other problems related to cyclic dependencies). Thus, Python's difficulties with recursive imports do not force you to put everything in its own file. If you think about it, you will recognize that cyclic dependencies are inherently difficult (e.g. chicken - egg type problems) and that the way Python handles cyclic import dependencies is rather natural (and understandable). Well, I guess that's the definitive answer... the tips for delaying import are good, I'll try to leverage them. I was hoping there would be a way to have python postpone evaluation similar to C's forward references. -- https://mail.python.org/mailman/listinfo/python-list
Re: caught in the import web again
Charles T. Smith cts.private.ya...@gmail.com Wrote in message: Well, I guess that's the definitive answer... the tips for delaying import are good, I'll try to leverage them. I was hoping there would be a way to have python postpone evaluation similar to C's forward references. In a module that might get tangled in a cycle, avoid global code that depends on other modules. Instead of putting such initialization at top level, put inside a function that gets called after all suspect imports are completed. (That function has global keywords ). Similarly avoid class attribute initialization at class level, if such initialization references higher level modules. -- DaveA -- https://mail.python.org/mailman/listinfo/python-list
Re: caught in the import web again
Charles T. Smith wrote: Yes, we're talking about recursive imports here. It's a complex, object- oriented system with big classes and little classes that are strongly interrelated. Well, there's your problem right there. You're working with a complex, highly coupled code-base. Alarms bells should be ringing in your head. That sort of highly-interdependent code base is a nightmare to maintain at the best of times. Recursive imports are just the start of the problem. Circular dependencies are not just a problem in Python, they are a problem throughout most of software design. https://en.wikipedia.org/wiki/Acyclic_dependencies_principle http://stackoverflow.com/questions/1897537/why-are-circular-references-considered-harmful I can get the imports configured properly so everything works but if I make a little change to the code, then suddenly all my imports are broken and I must painstakingly go through the whole structure, reorganizing them. The problems you are having with recursive imports are the symptom of a deeper design problem. If you can, you should try to refactor your code so as to have fewer mutual dependencies and less coupling. Are others equally frustrated by this or is there a trick or principle that I'm missing. Probably, and yes. Personally, I try very hard to avoid recursive imports as a matter of principle, even when I can manage them safely, so personally I'm never frustrated by them. But there are a few tricks you can use to reduce the pain a little. Think of these as pain-killers. All they are doing is masking the symptoms of an underlying problem, namely excessive coupling. You can often delay imports to when a function or method is called. Although this is not best practice, it does have its uses. Instead of this recursive import: # spam relies on this module, but this module relies on spam import spam class K: def __init__(self): self.eggs = spam.stuff() you can write this: class K: def __init__(self): import spam self.eggs = spam.stuff() That delays importing spam until after this module is already fully initialised. If you're worried about efficiency, don't be, only the very first import of spam will be expensive, the others will be very fast. Another way to avoid recursive imports is to merge the code from both modules into a single module. Note that this is the complete opposite of the conclusion you draw below. Google for avoiding circular imports python (and similar) and you will find lots of discussions about this. Leave out the python and you will find the same thing for other languages. At this point, I guess the way I'll have to proceed is to put every class in its own file, no matter how small. Hopefully that takes care of the problem. I don't think that's going to help. If anything, it will make it worse. Suppose you have two classes, A and B, which require each other: class A: def __init__(self): self.thing = B() class B(A): pass There's a circular dependency right there, but no import is needed so it is comparatively easy to manage. Now move them into separate modules, and you've turned a manageable circular dependency within a single module into a painful recursive import. Python is not Java, nor Perl, and if you're putting every class into its own file, you are doing it wrong. I compared perl's object-oriented philosophy with python's and didn't like how perl *requires* you to put everything into its own file. Now it looks like python does, too, implicitly. You are mistaken about that. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: caught in the import web again
On Mon, Nov 17, 2014 at 3:17 PM, Dave Angel da...@davea.name wrote: Charles T. Smith cts.private.ya...@gmail.com Wrote in message: Well, I guess that's the definitive answer... the tips for delaying import are good, I'll try to leverage them. I was hoping there would be a way to have python postpone evaluation similar to C's forward references. In a module that might get tangled in a cycle, avoid global code that depends on other modules. Instead of putting such initialization at top level, put inside a function that gets called after all suspect imports are completed. (That function has global keywords ). If the problem is that one of those modules would import the current module, then after all suspect imports are completed basically means after the current module has finished importing. So what would be responsible for calling such a function? -- https://mail.python.org/mailman/listinfo/python-list
Re: caught in the import web again
Ian Kelly ian.g.ke...@gmail.com Wrote in message: On Mon, Nov 17, 2014 at 3:17 PM, Dave Angel da...@davea.name wrote: In a module that might get tangled in a cycle, avoid global code that depends on other modules. Instead of putting such initialization at top level, put inside a function that gets called after all suspect imports are completed. (That function has global keywords ). If the problem is that one of those modules would import the current module, then after all suspect imports are completed basically means after the current module has finished importing. So what would be responsible for calling such a function? If one builds a set of modules that cannot avoid recursive imports, then one can ask the code that imports it to make a call after importing. Or all that is needed to avoid exposing the mess is that the top level not be part of the loops. -- DaveA -- https://mail.python.org/mailman/listinfo/python-list
Re: caught in the import web again
On Monday, November 17, 2014 4:46:05 PM UTC-6, Steven D'Aprano wrote: [...] Python is not Java, nor Perl, and if you're putting every class into its own file, you are doing it wrong. Stop making these gross generalizations. Just because Java *REQUIRES* that you only have one class per file does not mean that having one class per file is *BAD*, or *WRONG* -- quite the contrary! You might as well join the endless hordes that parrot off such ridiculous bile as: * All republicans are racists! or: * All democrats are whiners! or: * All politicians are filthy nasty liars! ...that last one might be true though! There are times when i have only one class in a file, but there are other times when i have multiple classes per file BUT NOT MANY TIMES! AND NOT MANY CLASSES! The number of classes per file is a very subjective matter indeed, but in my opinion, the number should be kept as low as possible, because, as the number of lines in a file increases, your mental grasp of the contents decreases. I don't care who you *THINK* you are, but your mental focus is *NOT* infinite! One of the problems inherent in large file editing is the fact that since you are dealing with multiple namespaces, there is great possibilities of different namespaces re-using the same symbols. So for instance, if your searching in class A for symbolX, and symbolX also exists in class M, you're going to find yourself 1000 lines away from the scope you intended to search, and now you have to trog back up to your original spot! BOY I LOVE TROGGING! YES, YES, i know some of you have fancy editing tools with scoped search abilities, but even *IF* you use such a tool, you still must define the scope in some way --by selecting a block of text or choosing from a predefined set of scopes-- but in any event, you're just doing new work (scoped search rules) to avoid doing old work (flipping between windows), and although there is arguably no net gain or loss between the two techniques *DIRECTLY*, the person trying to wrangle a large file is more likely to loose focus. IT'S NOT A ZERO SUM GAME, SO *YOU* LOOSE! The one class per file rule handles the scoped search *IMPLICITLY* for me. When i'm in a file that contains only one class, i know that i could never acidentally edit the wrong symbol in *ANOTHER* class, and I don't need to define a search area, no, all i need to do to search, is to, well, *SEARCH*. WHAT A NOVEL IDEA! Combining multiple classes into a single file is like combining multiple drinking glasses into a single unit. Imagine a wine glass, a beer mug, and a soda can joined with a tight fitting rubber-band -- sure, you can carry the unit comfortably with one hand, and you might impress your drunk friends at the local pub with your new beverage dispenser, however, try drinking from the beer mug without getting drenched with wine or sticky soda. And even if you *DID* manage to drink the beer without spilling the other two in the process, you would have to expend so much *EXTRA* mental focus during the process that you've undermined the usefulness of the combination. Most times, the simplest solution is the best. JUST DRINK FROM ONE GLASS AT A TIME YOU IDIOT! -- https://mail.python.org/mailman/listinfo/python-list
Re: caught in the import web again
On Tue, Nov 18, 2014 at 1:17 PM, Rick Johnson rantingrickjohn...@gmail.com wrote: BOY I LOVE TROGGING! For consistency, you should say GOVE there. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: caught in the import web again
On 11/17/2014 03:45 PM, Steven D'Aprano wrote: Circular dependencies are not just a problem in Python, they are a problem throughout most of software design. Personally I find that duck typing eliminates a lot of the circular dependency problems. Class A doesn't necessarily have to know about Class B to work with instances of Class B. All that has to be known is what methods are going to be available on those instances. The interface pattern can help here if you wanted a bit more safety (like what Zope provides). Either way it breaks up the dependency cycle nicely. If Class A needs to instantiate instances of Class B (which depends on Class A), then it could call a factory method to do so, perhaps set up by a third party after both modules have been imported. There are lots of methods of avoiding the issue entirely. In the C or C++ world, I find that signalling frameworks can also be used to break a dependency cycle. For example, if Class A needs to call a method on an instance of Class B (which in turn depends on Class A), that could be done instead as a signal, which is later connected to Class B's method after everything is successfully instantiated. Also makes testing a bit easier, because you can simply hook up the signals to a test harness. -- https://mail.python.org/mailman/listinfo/python-list
Re: caught in the import web again
On Mon, 17 Nov 2014 18:17:13 -0800, Rick Johnson wrote: BOY I LOVE TROGGING! Yes, we've noticed. http://www.urbandictionary.com/define.php?term=trogging -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: caught in the import web again
On Sun, 16 Nov 2014 08:14:05 +0100, dieter wrote: Charles T. Smith cts.private.ya...@gmail.com writes: Now, I'm getting these errors: ImportError: cannot import name ... and AttributeError: 'module' object has no attribute ... (what is 'module'?) Is there a way to resolve this without having to restructure my code every couple of days? I thought using imports of the form: from module import symbol was the right way to avoid these hassles... I have not noticed your previous posts regarding import problems -- thus, I may repeats things already said. Usually, Python imports are straight forward and do not lead to surprises. There are two exceptions: recursive imports and imports in separate threads. Let's look at the problem areas in turn. Recursive imports. In this case you have module A which imports module B (maybe indirectly via a sequence of intervening imports) which imports module A. In this, import means either import ... or from ... import ... (it does not matter). When module B tries to import module A, then A is not yet complete (as during the execution of A's initialization code, it started to import B (maybe indirectly) and the following initialization code has not yet been executed). As a consequence, you may get AttributeError (or ImportError) when you try to access things from A. To avoid problems like this, try to avoid recursive imports. One way to do this, are local imports -- i.e. imports inside a function. This way, the import happens when the functions is called which (hopefully) happens after module initialization. Yes, we're talking about recursive imports here. It's a complex, object- oriented system with big classes and little classes that are strongly interrelated. I can get the imports configured properly so everything works but if I make a little change to the code, then suddenly all my imports are broken and I must painstakingly go through the whole structure, reorganizing them. Are others equally frustrated by this or is there a trick or principle that I'm missing. At this point, I guess the way I'll have to proceed is to put every class in its own file, no matter how small. Hopefully that takes care of the problem. I compared perl's object-oriented philosophy with python's and didn't like how perl *requires* you to put everything into its own file. Now it looks like python does, too, implicitly. cts www.creative-telcom-solutions.de -- https://mail.python.org/mailman/listinfo/python-list
Re: caught in the import web again
On Sun, Nov 16, 2014 at 7:53 PM, Charles T. Smith cts.private.ya...@gmail.com wrote: Yes, we're talking about recursive imports here. It's a complex, object- oriented system with big classes and little classes that are strongly interrelated. I can get the imports configured properly so everything works but if I make a little change to the code, then suddenly all my imports are broken and I must painstakingly go through the whole structure, reorganizing them. So reorganize *once*, such that there are no recursive imports. Reorganize your code so things don't actually import one another. Remember, you can pass an instance of some class to an instance of another class that has no knowledge of the first class - like this: # file1.py class Foo: def some_method(self, some_obj): some_obj.some_other_method() # file2.py class Bar: def some_other_method(self): return 42 # main.py import file1, file2 f=file1.Foo() b=file2.Bar() f.some_method(b) This works just fine, and neither file1 nor file2 needs to import the other. Recursive imports are almost never necessary. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: caught in the import web again
Charles T. Smith cts.private.ya...@gmail.com writes: ... Are others equally frustrated by this or is there a trick or principle that I'm missing. At this point, I guess the way I'll have to proceed is to put every class in its own file, no matter how small. Hopefully that takes care of the problem. Recursive imports are inherently difficult. It is best if a careful design avoids it (note that a cycle free dependency structure is much easier to handle than a cyclic one -- not only for Python). If you think such a cycle free module structure is impossible in your case, then try to delay module related operations: * consider local imports (imports inside a function). They delay the import until the function is called rather than execute it at module import time. * prefer import module over from module import ... This may delay the module access; perhaps (but not necessarily) sufficiently such that the accessed object is already defined. Neither of these approaches is complete - you may still get cyclic import dependencies and related problems. Usually, you need a careful module dependency design to avoid cyclic dependencies altogether. I compared perl's object-oriented philosophy with python's and didn't like how perl *requires* you to put everything into its own file. Now it looks like python does, too, implicitly. If you put everything in a single file, then you do not get recursive imports (but you may still have other problems related to cyclic dependencies). Thus, Python's difficulties with recursive imports do not force you to put everything in its own file. If you think about it, you will recognize that cyclic dependencies are inherently difficult (e.g. chicken - egg type problems) and that the way Python handles cyclic import dependencies is rather natural (and understandable). -- https://mail.python.org/mailman/listinfo/python-list
Re: caught in the import web again
Charles T. Smith cts.private.ya...@gmail.com writes: Now, I'm getting these errors: Please reduce the problem to a minimal, complete example demonstrating the behaviour URL:http://sscce.org/ so that you can show us exactly what's happening. AttributeError: 'module' object has no attribute ... (what is 'module'?) The type of the object which doesn't have the attribute. It's saying you're tring to access an attribute on a particular object, and that object (which is of type ‘module’, hence it is a “'module' object”) doesn't have that attribute. Is there a way to resolve this without having to restructure my code every couple of days? Not knowing what the actual problem is, it's difficult to say. Please come up with a (contrived, if you like) minimal clear example of what's happening – i.e. make it from scratch, without irrelevant parts from the rest of your program, but ensure it still does what you're confused by – and present it here. -- \ “It's a terrible paradox that most charities are driven by | `\ religious belief.… if you think altruism without Jesus is not | _o__) altruism, then you're a dick.” —Tim Minchin, 2010-11-28 | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: caught in the import web again
On Sat, 15 Nov 2014 22:52:33 +, Charles T. Smith wrote: Now, I'm getting these errors: ImportError: cannot import name ... and AttributeError: 'module' object has no attribute ... It would be useful to know what you're actually trying to import and what the complete error messages are. For example, are these imports of code that you've written yourself, or part of the standard library modules, or third party modules. -- Denis McMahon, denismfmcma...@gmail.com -- https://mail.python.org/mailman/listinfo/python-list
Re: caught in the import web again
Charles T. Smith cts.private.ya...@gmail.com writes: Now, I'm getting these errors: ImportError: cannot import name ... and AttributeError: 'module' object has no attribute ... (what is 'module'?) Is there a way to resolve this without having to restructure my code every couple of days? I thought using imports of the form: from module import symbol was the right way to avoid these hassles... I have not noticed your previous posts regarding import problems -- thus, I may repeats things already said. Usually, Python imports are straight forward and do not lead to surprises. There are two exceptions: recursive imports and imports in separate threads. Let's look at the problem areas in turn. Recursive imports. In this case you have module A which imports module B (maybe indirectly via a sequence of intervening imports) which imports module A. In this, import means either import ... or from ... import ... (it does not matter). When module B tries to import module A, then A is not yet complete (as during the execution of A's initialization code, it started to import B (maybe indirectly) and the following initialization code has not yet been executed). As a consequence, you may get AttributeError (or ImportError) when you try to access things from A. To avoid problems like this, try to avoid recursive imports. One way to do this, are local imports -- i.e. imports inside a function. This way, the import happens when the functions is called which (hopefully) happens after module initialization. Imports and threads. The import machinery changes global data structures (e.g. sys.modules). To protect those changes, it uses a lock, held while an import is in progress. When, during an import, a separate thread tries to import something, it blocks - waiting for the import lock to be released. In case, the importing thread waits on this thread, then the system deadlocks. To avoid this: do not start threads during imports. -- https://mail.python.org/mailman/listinfo/python-list
