Re: Late initialization using __getattribute__
bukzor a écrit : On Sep 3, 1:02 pm, Bruno Desthuilliers [EMAIL PROTECTED] wrote: bukzor a écrit : (snip) Thanks for the reply. Just to see it not work, I tried to remove __getattribute__ from LateInitMixIn, but couldn't get it to work. ??? Sorry, I don't get what you mean... Since you said __getattribute__ is an attribute of the class, I tried to run (something to the effect of) delattr(self.__class__, __getattribute__), Ah, ok. Definitively *not* a thing to do... but it threw an AttributeError. You'd have to call it on the mixin class itself, not on a subclass. My Base class is a C class (_mysql.connection from MySQLdb) that sometimes segfaults if you try to use it before it's fully initialized, ... I have used MySQLdb for years on more than a dozen linux distribs, and never had such a problem. Is this a known bug ? Or is there something wrong with your setup ? I'm trying to optimize my number of connections Connection pooling anyone ? IIRC, this is something that already exists in quite a couple libs / frameworks. Is there a reason you try to roll your own ? by not fully initializing (read: not connecting) my connection until it's used in some way. Of course the maintainer didn't envision this (mis)use, so the object sometimes throws bad errors until it's fully initialized. Ok. Of course the *correct* way to do this is to be more careful about when I create connections, but I think I should be able to get this to work, and it (would be) much easier to do it The Right Way once it works. Please pardon me for repeating the same thing over and over, but messing with __getattribute__ is certainly not the way to go. so unfortunately I think I need to use __getattribute__ to do this. I'm doing all this just to make the connection not actually connect until used. I may be dumb, but I don't get how this is supposed to solve your problem. But anyway : there's a known design pattern for what you're trying to do, that doesn't require mixins nor messing with __getattribute__ (which, I repeat, is more often than not something you *don't* want to do). The name of the design pattern is proxy. I strongly suggest that you 1/ try to cure the real problem instead of hacking around and 2/ read about the proxy design pattern. My 2 cents... I like the idea of mix-ins, but can't figure out how to make a proxy work that way. You mean, how to use a proxy for lazy initialization ? Heck, that's the exact use case in the GoF. For a long time I had a proxy class that added five or six features on top of the MySQLdb package, but it wasn't configurable enough, and I'm working on splitting each feature into its own MixIn class. As an aside, this is working for me pretty well. The reconnect method (inheritied from a Reconnectable mixin) uses several of the object's attributes, so I need to set _inited beforehand so that I don't get into an infinite __getattribute__ loop. What I'd *really* like to do is remove __getattribute__ from the object at that point. You can't. Or, more exactly, all you can do is remove __getattribute__ from the mixin class - but then the mixin class won't work anymore. I don't mean to be condescendant, but it looks like you don't have a clear understanding of Python's object model here - else you wouldn't even consider doing such a thing. FWIW, I posted a solution based on the __getattr__ hook, which did work - at least for the specs implied by your code snippet. -- http://mail.python.org/mailman/listinfo/python-list
Re: Late initialization using __getattribute__
so unfortunately I think I need to use __getattribute__ to do this. I'm doing all this just to make the connection not actually connect until used. I may be dumb, but I don't get how this is supposed to solve your problem. But anyway : there's a known design pattern for what you're trying to do, that doesn't require mixins nor messing with __getattribute__ (which, I repeat, is more often than not something you *don't* want to do). The name of the design pattern is proxy. I strongly suggest that you 1/ try to cure the real problem instead of hacking around and 2/ read about the proxy design pattern. My 2 cents... I like the idea of mix-ins, but can't figure out how to make a proxy work that way. You mean, how to use a proxy for lazy initialization ? Heck, that's the exact use case in the GoF. I mean, how to make a MixIn class that uses the proxy pattern. I'd like to be able to do something like this: class SuperCursor(FeatureOneMixIn, FeatureTwoMixin, ..., VanillaCursor): pass This works with my current implementation. After thinking about it more, I think I've got such a thing written. I had to use inspect.getmro and new.classobj to do it, but it works and it honors the usual mro (as far as I can tell). I've put the code at the bottom to (try to) maintain readability. For a long time I had a proxy class that added five or six features on top of the MySQLdb package, but it wasn't configurable enough, and I'm working on splitting each feature into its own MixIn class. As an aside, this is working for me pretty well. The reconnect method (inheritied from a Reconnectable mixin) uses several of the object's attributes, so I need to set _inited beforehand so that I don't get into an infinite __getattribute__ loop. What I'd *really* like to do is remove __getattribute__ from the object at that point. You can't. Or, more exactly, all you can do is remove __getattribute__ from the mixin class - but then the mixin class won't work anymore. I don't mean to be condescendant, but it looks like you don't have a clear understanding of Python's object model here - else you wouldn't even consider doing such a thing. FWIW, I posted a solution based on the __getattr__ hook, which did work - at least for the specs implied by your code snippet. My toy example turned out to be not the best representation of the problem. The base class has attributes that exist but either throw errors or segfault if used before reconnect() is called. This means that I need to capture more than just the attributes that would throw AttributeError. #CODE class Base(object): def __init__(self, *args, **kwargs): self.args = args self.kwargs = kwargs def __str__(self): return Base object created with %s %s % (self.args, self.kwargs) class MixIn2(object): def __str__(self): return MixIn2 with %s % super(MixIn2, self).__str__() class MixIn1(object): def __str__(self): return MixIn1 with %s % super(MixIn1, self).__str__() class ProxyMixIn(object): def __init__(self, *args, **kwargs): self.__proxied = None self.__args = args self.__kwargs = kwargs def __getattr__(self, attr): print Getting, attr try: return getattr(self.__proxied, attr) except AttributeError: if self.__proxied: raise else: from inspect import getmro mro = getmro(self.__class__) mro = mro[list(mro).index(ProxyMixIn) + 1:] print Proxied mro, mro from new import classobj self.__proxied = classobj(Proxied, mro, globals()) (*self.__args, **self.__kwargs) return getattr(self.__proxied, attr) def __str__(self): return Proxy of %s % super(ProxyMixIn, self).__str__() class Proxy(MixIn1, ProxyMixIn, MixIn2, Base): pass def main(): p = Proxy(1,2,3, one=1, two=2) print p main() #OUTPUT## Getting args Proxied mro (class '__main__.MixIn2', class '__main__.Base', type 'object') Getting kwargs MixIn1 with Proxy of MixIn2 with Base object created with (1, 2, 3) {'two': 2, 'one': 1} -- http://mail.python.org/mailman/listinfo/python-list
Re: Late initialization using __getattribute__
On Sep 4, 12:36 pm, bukzor [EMAIL PROTECTED] wrote: so unfortunately I think I need to use __getattribute__ to do this. I'm doing all this just to make the connection not actually connect until used. I may be dumb, but I don't get how this is supposed to solve your problem. But anyway : there's a known design pattern for what you're trying to do, that doesn't require mixins nor messing with __getattribute__ (which, I repeat, is more often than not something you *don't* want to do). The name of the design pattern is proxy. I strongly suggest that you 1/ try to cure the real problem instead of hacking around and 2/ read about the proxy design pattern. My 2 cents... I like the idea of mix-ins, but can't figure out how to make a proxy work that way. You mean, how to use a proxy for lazy initialization ? Heck, that's the exact use case in the GoF. I mean, how to make a MixIn class that uses the proxy pattern. You don't. That's not how proxies work. I'd like to be able to do something like this: class SuperCursor(FeatureOneMixIn, FeatureTwoMixin, ..., VanillaCursor): pass Why does it have to look like that? A good programmer lets the code look however it has to look to most effectively do it's job. With a proxy, the base class isn't a base class but a member. Here is a very simple example: class SuperCursor(object): def __init__(self): self._cursor = VanillaCursor() self._connected = False def __getattr__(self,attr): if not self._connected: self._cursor.connect() self._connected = True return getattr(self._cursor,attr) cursor = SuperCursor() That doens't use a mixin, but why should it? Carl Banks -- http://mail.python.org/mailman/listinfo/python-list
Re: Late initialization using __getattribute__
I'd like to be able to do something like this: class SuperCursor(FeatureOneMixIn, FeatureTwoMixin, ..., VanillaCursor): pass Why does it have to look like that? A good programmer lets the code look however it has to look to most effectively do it's job. With a proxy, the base class isn't a base class but a member. Here is a very simple example: class SuperCursor(object): def __init__(self): self._cursor = VanillaCursor() self._connected = False def __getattr__(self,attr): if not self._connected: self._cursor.connect() self._connected = True return getattr(self._cursor,attr) cursor = SuperCursor() That doesn't use a mixin, but why should it? The point of using a mixin is to not limit myself to inheriting from VanillaCursor. I want to put this on top of various subclasses of the vanilla cursor, like TimeLimitedCursor or RetryingCursor. I have four other mixins that operate this way, so it's desirable to keep this one in line with that. -- http://mail.python.org/mailman/listinfo/python-list
Re: Late initialization using __getattribute__
bukzor a écrit : so unfortunately I think I need to use __getattribute__ to do this. I'm doing all this just to make the connection not actually connect until used. I may be dumb, but I don't get how this is supposed to solve your problem. But anyway : there's a known design pattern for what you're trying to do, that doesn't require mixins nor messing with __getattribute__ (which, I repeat, is more often than not something you *don't* want to do). The name of the design pattern is proxy. I strongly suggest that you 1/ try to cure the real problem instead of hacking around and 2/ read about the proxy design pattern. My 2 cents... I like the idea of mix-ins, but can't figure out how to make a proxy work that way. You mean, how to use a proxy for lazy initialization ? Heck, that's the exact use case in the GoF. I mean, how to make a MixIn class that uses the proxy pattern. That's not how proxies work. They use composition/delegation, not MI. I'd like to be able to do something like this: class SuperCursor(FeatureOneMixIn, FeatureTwoMixin, ..., VanillaCursor): pass Yuck. Sorry, but it reminds me of the worst MI abuses in Zope2. (snip) My toy example turned out to be not the best representation of the problem. This is often the case with toy examples. The base class has attributes that exist but either throw errors or segfault if used before reconnect() is called. Then these attributes should probably call reconnect() by themselves. Decorators and/or computed attributes may help here. -- http://mail.python.org/mailman/listinfo/python-list
Re: Late initialization using __getattribute__
On Sep 4, 3:38 pm, bukzor [EMAIL PROTECTED] wrote: The point of using a mixin is to not limit myself to inheriting from VanillaCursor. I want to put this on top of various subclasses of the vanilla cursor, like TimeLimitedCursor or RetryingCursor. I have four other mixins that operate this way, so it's desirable to keep this one in line with that. http://www.bartleby.com/59/3/foolishconsi.html I think that desire is hurting you more than it's helping. It's fine to be consistent for consistency's sake, but you are taking consistency to an unhealthy extreme. A mixin is simply the wrong tool to do this with. My advice: either use a proxy, or manage your connections better. Carl Banks -- http://mail.python.org/mailman/listinfo/python-list
Re: Late initialization using __getattribute__
bukzor a écrit : I want to make a MixIn class that waits to initialize its super- classes until an attribute of the object is accessed. Not generally useful, but desirable in my case. I've written this, and it works, but would like to take any suggestions you guys have. You shouldn't mess with __getattribute__ unless you really know what you're doing and are ok to suffer the constant performance hit you'll get. Remember that __getattribute__ actually *is* the implementation of attribute lookup rules, and is called on each and every attribute lookup. Your below snippet would be much better using __getattr__ (which is only called as a last resort). I've commented out the delattr call because it throws an AttributeError (although I don't know why). __getattribute__ belongs to the class, not to the instance. delattr() only removes instance attributes. You'd have to remove __getattribute__ from the LateInitMixIn class object itself, but then it would break the whole thing. class LateInitMixIn(object): def __init__(self): print LateInit initialization self.inited = False def __getattribute__(self, attr): print Doing __getattribute__ getattr = lambda attr:object.__getattribute__(self, attr) if not getattr(inited): super(LateInitMixIn, self).__init__() setattr(self, inited, True) #delattr(self, __getattribute__) return getattr(attr) Here's another possible implementation (which doesn't solve all problems, cf below) using __getattr__: class LateInitMixin(object): def __init__(self): print not yet self.__initialized = False def __getattr__(self, name): if self.__initialized: raise AttributeError( object %s has no attribute '%s' % (type(self), name) ) super(LateInitMixin, self).__init__() self.__initialized = True return getattr(self, name) class Base(object): def __init__(self): print yet self.base = True class LateInit(LateInitMixin, Base): pass def main(): print shouldn't init S = LateInit() print should init print S.base if __name__==__main__: main() Ok, now, the other problem : what if Base.__init__ expects args ? -- http://mail.python.org/mailman/listinfo/python-list
Re: Late initialization using __getattribute__
On Sep 3, 12:19 pm, Bruno Desthuilliers [EMAIL PROTECTED] wrote: bukzor a écrit : I want to make a MixIn class that waits to initialize its super- classes until an attribute of the object is accessed. Not generally useful, but desirable in my case. I've written this, and it works, but would like to take any suggestions you guys have. You shouldn't mess with __getattribute__ unless you really know what you're doing and are ok to suffer the constant performance hit you'll get. Remember that __getattribute__ actually *is* the implementation of attribute lookup rules, and is called on each and every attribute lookup. Your below snippet would be much better using __getattr__ (which is only called as a last resort). I've commented out the delattr call because it throws an AttributeError (although I don't know why). __getattribute__ belongs to the class, not to the instance. delattr() only removes instance attributes. You'd have to remove __getattribute__ from the LateInitMixIn class object itself, but then it would break the whole thing. class LateInitMixIn(object): def __init__(self): print LateInit initialization self.inited = False def __getattribute__(self, attr): print Doing __getattribute__ getattr = lambda attr:object.__getattribute__(self, attr) if not getattr(inited): super(LateInitMixIn, self).__init__() setattr(self, inited, True) #delattr(self, __getattribute__) return getattr(attr) Here's another possible implementation (which doesn't solve all problems, cf below) using __getattr__: class LateInitMixin(object): def __init__(self): print not yet self.__initialized = False def __getattr__(self, name): if self.__initialized: raise AttributeError( object %s has no attribute '%s' % (type(self), name) ) super(LateInitMixin, self).__init__() self.__initialized = True return getattr(self, name) class Base(object): def __init__(self): print yet self.base = True class LateInit(LateInitMixin, Base): pass def main(): print shouldn't init S = LateInit() print should init print S.base if __name__==__main__: main() Ok, now, the other problem : what if Base.__init__ expects args ? Thanks for the reply. Just to see it not work, I tried to remove __getattribute__ from LateInitMixIn, but couldn't get it to work. My Base class is a C class (_mysql.connection from MySQLdb) that sometimes segfaults if you try to use it before it's fully initialized, so unfortunately I think I need to use __getattribute__ to do this. I'm doing all this just to make the connection not actually connect until used. -- http://mail.python.org/mailman/listinfo/python-list
Re: Late initialization using __getattribute__
bukzor a écrit : (snip) Thanks for the reply. Just to see it not work, I tried to remove __getattribute__ from LateInitMixIn, but couldn't get it to work. ??? Sorry, I don't get what you mean... My Base class is a C class (_mysql.connection from MySQLdb) that sometimes segfaults if you try to use it before it's fully initialized, Then don't use it before it's fully initialized. patient: doctor, when I do this, it hurts doctor : then don't do it !-) More seriously, I have used MySQLdb for years on more than a dozen linux distribs, and never had such a problem. Is this a known bug ? Or is there something wrong with your setup ? so unfortunately I think I need to use __getattribute__ to do this. I'm doing all this just to make the connection not actually connect until used. I may be dumb, but I don't get how this is supposed to solve your problem. But anyway : there's a known design pattern for what you're trying to do, that doesn't require mixins nor messing with __getattribute__ (which, I repeat, is more often than not something you *don't* want to do). The name of the design pattern is proxy. I strongly suggest that you 1/ try to cure the real problem instead of hacking around and 2/ read about the proxy design pattern. My 2 cents... -- http://mail.python.org/mailman/listinfo/python-list
Re: Late initialization using __getattribute__
On Sep 3, 1:02 pm, Bruno Desthuilliers [EMAIL PROTECTED] wrote: bukzor a écrit : (snip) Thanks for the reply. Just to see it not work, I tried to remove __getattribute__ from LateInitMixIn, but couldn't get it to work. ??? Sorry, I don't get what you mean... Since you said __getattribute__ is an attribute of the class, I tried to run (something to the effect of) delattr(self.__class__, __getattribute__), but it threw an AttributeError. My Base class is a C class (_mysql.connection from MySQLdb) that sometimes segfaults if you try to use it before it's fully initialized, ... I have used MySQLdb for years on more than a dozen linux distribs, and never had such a problem. Is this a known bug ? Or is there something wrong with your setup ? I'm trying to optimize my number of connections by not fully initializing (read: not connecting) my connection until it's used in some way. Of course the maintainer didn't envision this (mis)use, so the object sometimes throws bad errors until it's fully initialized. Of course the *correct* way to do this is to be more careful about when I create connections, but I think I should be able to get this to work, and it (would be) much easier to do it The Right Way once it works. so unfortunately I think I need to use __getattribute__ to do this. I'm doing all this just to make the connection not actually connect until used. I may be dumb, but I don't get how this is supposed to solve your problem. But anyway : there's a known design pattern for what you're trying to do, that doesn't require mixins nor messing with __getattribute__ (which, I repeat, is more often than not something you *don't* want to do). The name of the design pattern is proxy. I strongly suggest that you 1/ try to cure the real problem instead of hacking around and 2/ read about the proxy design pattern. My 2 cents... I like the idea of mix-ins, but can't figure out how to make a proxy work that way. For a long time I had a proxy class that added five or six features on top of the MySQLdb package, but it wasn't configurable enough, and I'm working on splitting each feature into its own MixIn class. As an aside, this is working for me pretty well. The reconnect method (inheritied from a Reconnectable mixin) uses several of the object's attributes, so I need to set _inited beforehand so that I don't get into an infinite __getattribute__ loop. What I'd *really* like to do is remove __getattribute__ from the object at that point. def __getattribute__(self, attr): connect if it would otherwise cause an error. getattr = lambda attr:object.__getattribute__(self, attr) if not getattr(_inited): print connecting. setattr(self, _inited, True) getattr(reconnect)() return getattr(attr) Thanks for your help, --Buck -- http://mail.python.org/mailman/listinfo/python-list
Re: Late initialization using __getattribute__
On Sep 4, 12:26 am, bukzor [EMAIL PROTECTED] wrote: I'm trying to optimize my number of connections by not fully initializing (read: not connecting) my connection until it's used in some way. I had the same use case and I solved with a simple property. Here is the code I have for pymssql: @property def conn(self): if self._conn is None: self._conn = _mssql.connect(self.host, self.user,self.passwd) self._conn.select_db(self.dbname) return self._conn The connection is really instantiate only when you call self.conn to perform the query, not when you instantiate the class. -- http://mail.python.org/mailman/listinfo/python-list