RE: [ZODB-Dev] zodb connection question
[ Tim Peters wrote:] > [Jürgen Herrmann] >> hmm, as it seemed quite impossible the way i wanted it, i almost dropped >> it from my wishlist. now you say, it's doable... > > It would be possible to add new official APIs to ZODB to supply some > notion > of the collection of all modified objects, at the level ZODB "sees" > objects. > "Of course" that would be doable, given enough work (which I sketched). > >> i already had hacked into zodb.connection, > > That's part of what would be needed, yes. > >> the problem was that the objects maintained there >> (self._registered_objects) are not wrapped for acquisition, thus my >> reindexing method did not find the catalog to register with. how would >> you solve this problem? > > For the entirety of what you want to do, I'd pursue Jim's suggestion and > forget about working at the ZODB level. You didn't actually want "all > modified objects" to begin with: that appeared to be a hack, an indirect > way to get at what you do want, which is a very specific subset of > application objects that need to be reindexed. Jim suggested some > principled ways to accomplish that without hacking ZODB internals. i'd have the objects sorted out, that simply don't implement the necessary callback function. this way only the "interesting" objects would take part in that dance. knowing all modified objects just seemed an easy way to get where i wanted. > >> to answer your question above: a point-in-time list would be enough for >> me, just like to reindex all modified "model" objects. that should only >> change indexes. > > If we were to add official APIs to ZODB, we'd have to consider what > everyone > might want. Point-in-time is probably easiest to code, anyway. > hmm, after several posts to the zope list and now to the zodb-dev list also, i guess that no one actually needs these hooks. everyone that replied to me, suggested not to try it this way. now i wrote an index manager, that keeps track of registered objects - still i have to register objects in set methods and methods that otherwise change the state of objects but all the other stuff is the way i want it now. in short: probably to much work if noone wants to use it afterwards (except me :))) regards, juergen herrmann ___ >> XLhost.de - eXperts in Linux hosting << Juergen Herrmann Weiherweg 10, 93051 Regensburg, Germany Fon: +49 (0)700 XLHOSTDE [0700 95467833] Fax: +49 (0)721 151 463027 ICQ: 27139974 - IRC: [EMAIL PROTECTED] WEB: http://www.XLhost.de ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
RE: [ZODB-Dev] zodb connection question
[Jürgen Herrmann] > hmm, as it seemed quite impossible the way i wanted it, i almost dropped > it from my wishlist. now you say, it's doable... It would be possible to add new official APIs to ZODB to supply some notion of the collection of all modified objects, at the level ZODB "sees" objects. "Of course" that would be doable, given enough work (which I sketched). > i already had hacked into zodb.connection, That's part of what would be needed, yes. > the problem was that the objects maintained there > (self._registered_objects) are not wrapped for acquisition, thus my > reindexing method did not find the catalog to register with. how would > you solve this problem? For the entirety of what you want to do, I'd pursue Jim's suggestion and forget about working at the ZODB level. You didn't actually want "all modified objects" to begin with: that appeared to be a hack, an indirect way to get at what you do want, which is a very specific subset of application objects that need to be reindexed. Jim suggested some principled ways to accomplish that without hacking ZODB internals. > to answer your question above: a point-in-time list would be enough for > me, just like to reindex all modified "model" objects. that should only > change indexes. If we were to add official APIs to ZODB, we'd have to consider what everyone might want. Point-in-time is probably easiest to code, anyway. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
RE: [ZODB-Dev] zodb connection question
[ Tim Peters wrote:] > [Jürgen Herrmann] > ... >> so, what i need is a way to have my mechanism called before transaction >> commits (would be possible to use the hooks provided by zodb 3.4 for >> that, sure) and cycle through all modified objects (that's where the >> hooks in 3.4 are not enough for me, they don't provide any way to access >> a list of modified objects). > > Note that this isn't just a matter of exposing something to you: ZODB 3.4 > has no list of modified objects, not even internally. Data managers (like > ZODB.Connection) register with transactions in 3.3+, and data managers are > responsible for keeping track of their own modified objects. Any number > of > data managers may register with a transaction, and each keeps track of its > own details. > > So doing what you want would require at least two changes: > > 1. Adding a new method to the IDataManager interface, to deliver some >notion of modified objects under the control of the data manager. >Of course data managers would need to grow implementations of that too. >Since you're presumably going to be changing even more objects based on >the stream of modified objects returned, this can be tricky (do you >get a point-in-time list, ignoring further changes? or an iterator >that includes later changes? if the latter, is it OK to see an object >more than once in the stream? like that). > > 2. Adding a related new method to ITransaction, to combine the notions of >modified objects from all registered data managers. > > That's all doable, but it's hard to see it taking priority over other > work. hmm, as it seemed quite impossible the way i wanted it, i almost dropped it from my wishlist. now you say, it's doable... i already had hacked into zodb.connection, the problem was that the objects maintained there (self._registered_objects) are not wrapped for acquisition, thus my reindexing method did not find the catalog to register with. how would you solve this problem? to answer your question above: a point-in-time list would be enough for me, just like to reindex all modified "model" objects. that should only change indexes. regards, juergen herrmann ___ >> XLhost.de - eXperts in Linux hosting << Juergen Herrmann Weiherweg 10, 93051 Regensburg, Germany Fon: +49 (0)700 XLHOSTDE [0700 95467833] Fax: +49 (0)721 151 463027 ICQ: 27139974 - IRC: [EMAIL PROTECTED] WEB: http://www.XLhost.de ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
RE: [ZODB-Dev] zodb connection question
[Jürgen Herrmann] ... > so, what i need is a way to have my mechanism called before transaction > commits (would be possible to use the hooks provided by zodb 3.4 for > that, sure) and cycle through all modified objects (that's where the > hooks in 3.4 are not enough for me, they don't provide any way to access > a list of modified objects). Note that this isn't just a matter of exposing something to you: ZODB 3.4 has no list of modified objects, not even internally. Data managers (like ZODB.Connection) register with transactions in 3.3+, and data managers are responsible for keeping track of their own modified objects. Any number of data managers may register with a transaction, and each keeps track of its own details. So doing what you want would require at least two changes: 1. Adding a new method to the IDataManager interface, to deliver some notion of modified objects under the control of the data manager. Of course data managers would need to grow implementations of that too. Since you're presumably going to be changing even more objects based on the stream of modified objects returned, this can be tricky (do you get a point-in-time list, ignoring further changes? or an iterator that includes later changes? if the latter, is it OK to see an object more than once in the stream? like that). 2. Adding a related new method to ITransaction, to combine the notions of modified objects from all registered data managers. That's all doable, but it's hard to see it taking priority over other work. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zodb connection question
Jürgen Herrmann wrote: [ Jim Fulton wrote:] Jürgen Herrmann wrote: [ Jim Fulton wrote:] Jürgen Herrmann wrote: hi all! i'm trying to form a patch that will result in a method "_before_commit()" being called on each modified object in a transaction (if that method exists of course) right before commit. main sense is to automate/delay (re)cataloging. first i looked at the Transaction class, as there have been heavy modifications to it from zodb 3.2 to 3.4 Patching is generally a bad idea. I suggest creatinga new catalog (by subclassing or adapting an existing one) that: - queues updates - registers a before-commit callback with the TM on the first update in a transaction - processes the queue in the callback I did this recently for with Zope 3's catalog and it worked very well. (I happend to use subclassing and would use adaptation if I were to do it again.) Jim hi jim! thanks for your reply, i already thought about such a solution and discarded it because it would still be necessary to call a method on an object to recatalog it. this step (the programmer's responsibility) i want to eliminate for several reasons. Which are? i have developed a framework that nicely handles many2many relations, heavily uml based, python code generation for attribute getter/setters, code generation for methods (the def only), automatic index creation... in short, i want to make everything as easy for the programmer (atm only me :) first i had the code generation framework write out an explicit self.reindex_object() at the end of each setter, ugly. it also had the drawback that setting 3 attributes caused the object to reindex itself 3 times. not the best for performance. If you are generating code, there there should be no problem generating the calls to reindex, or, with Zope 2.8, you can use the Zope 3 event system. which should be cleaner. You can avoid the cost of indexing on each modification by creating a catalog with the queuing strategy I describer earlier. ... if zodb is way to low for my approach, where do i get a possibility to get notified about a imminent transaction commit? and how could i find out about the modified objects? You use events. Have your generated code generate events, which include the content-wrapped objects, when an object is modified. Create an object, such as a specialized catalog or some other object that maintains a modified- object collection. This object tracks which objects have been modified, collapsing modification events for the same object. For the sake of discussion, call this object the "modification buffer". Provide a subscriber for modification events that registers objects with the modification buffer. When the buffer gets the first object (in a transaction), it registers itself to be called by the transaction machinery. When it is called, it processes (e.g. catalogs) the objects in its buffer and clears the buffer. another question: would it be possible to wrap the objects to enable acquisition in my first approach? No. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zodb connection question
[ Jim Fulton wrote:] > Jürgen Herrmann wrote: >> [ Jim Fulton wrote:] >> >>>Jürgen Herrmann wrote: >>> hi all! i'm trying to form a patch that will result in a method "_before_commit()" being called on each modified object in a transaction (if that method exists of course) right before commit. main sense is to automate/delay (re)cataloging. first i looked at the Transaction class, as there have been heavy modifications to it from zodb 3.2 to 3.4 >>> >>>Patching is generally a bad idea. >>> >>>I suggest creatinga new catalog (by subclassing or adapting an existing >>>one) >>>that: >>> >>>- queues updates >>> >>>- registers a before-commit callback with the TM on the first update >>> in a transaction >>> >>>- processes the queue in the callback >>> >>>I did this recently for with Zope 3's catalog and it worked very well. >>>(I happend to use subclassing and would use adaptation if I were to >>>do it again.) >>> >>>Jim >> >> >> hi jim! >> >> thanks for your reply, i already thought about such a solution and >> discarded it because it would still be necessary to call a method >> on an object to recatalog it. this step (the programmer's >> responsibility) >> i want to eliminate for several reasons. > > Which are? i have developed a framework that nicely handles many2many relations, heavily uml based, python code generation for attribute getter/setters, code generation for methods (the def only), automatic index creation... in short, i want to make everything as easy for the programmer (atm only me :) first i had the code generation framework write out an explicit self.reindex_object() at the end of each setter, ugly. it also had the drawback that setting 3 attributes caused the object to reindex itself 3 times. not the best for performance. > >> so, what i need is a way to have my mechanism called before transaction >> commits (would be possible to use the hooks provided by zodb 3.4 for >> that, sure) and cycle through all modified objects (that's where the >> hooks in 3.4 are not enough for me, they don't provide any way to access >> a list of modified objects). > > I would oppose a per-object callback mechanism. OTOH, I would not > oppose providing enough hooks to allow you to implement such a mechanism > yourself without patching ZODB. Note however, that such a mechanism might > not help you anyway, as discussed below. > >> as i showed in my first post, the callback to modified objects works >> as expected, but in the called method i'm not able to acquire the >> responsible Catalog for the modified object. > > That's because objects registered are not acquisition wrapped. > >> up to what point can i modify objects in a transaction? why doesn't >> acquisition on my objects work in Connection::tpc_begin() ??? > > The objects are not acquisition wrapped. > > IMO, ZODB is way too low a level to do what you want to do. > > Note that what you are trying to do would be easier in Zope 3 > because Zope 3 relies far less on wrappers for acquisition. > > Jim oh well, unfortunately i began with that project about 8months ago. that time zope (x)3 was still not recommended to users on the zope mailinglist, so i went for zope2. anyways, there must be a solution to that problem. if zodb is way to low for my approach, where do i get a possibility to get notified about a imminent transaction commit? and how could i find out about the modified objects? another question: would it be possible to wrap the objects to enable acquisition in my first approach? guess i'd have to know the container for that, right (which is no problem, because i have one distinct container folder for each class)? if that works somehow, i'd bug you further to provide me with that hooks you described above :) thanks in advance for your replies... juergen herrmann ___ >> XLhost.de - eXperts in Linux hosting << Juergen Herrmann Weiherweg 10, 93051 Regensburg, Germany Fon: +49 (0)700 XLHOSTDE [0700 95467833] Fax: +49 (0)721 151 463027 ICQ: 27139974 - IRC: [EMAIL PROTECTED] WEB: http://www.XLhost.de ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zodb connection question
Jürgen Herrmann wrote: [ Jim Fulton wrote:] Jürgen Herrmann wrote: hi all! i'm trying to form a patch that will result in a method "_before_commit()" being called on each modified object in a transaction (if that method exists of course) right before commit. main sense is to automate/delay (re)cataloging. first i looked at the Transaction class, as there have been heavy modifications to it from zodb 3.2 to 3.4 Patching is generally a bad idea. I suggest creatinga new catalog (by subclassing or adapting an existing one) that: - queues updates - registers a before-commit callback with the TM on the first update in a transaction - processes the queue in the callback I did this recently for with Zope 3's catalog and it worked very well. (I happend to use subclassing and would use adaptation if I were to do it again.) Jim hi jim! thanks for your reply, i already thought about such a solution and discarded it because it would still be necessary to call a method on an object to recatalog it. this step (the programmer's responsibility) i want to eliminate for several reasons. Which are? so, what i need is a way to have my mechanism called before transaction commits (would be possible to use the hooks provided by zodb 3.4 for that, sure) and cycle through all modified objects (that's where the hooks in 3.4 are not enough for me, they don't provide any way to access a list of modified objects). I would oppose a per-object callback mechanism. OTOH, I would not oppose providing enough hooks to allow you to implement such a mechanism yourself without patching ZODB. Note however, that such a mechanism might not help you anyway, as discussed below. as i showed in my first post, the callback to modified objects works as expected, but in the called method i'm not able to acquire the responsible Catalog for the modified object. That's because objects registered are not acquisition wrapped. up to what point can i modify objects in a transaction? why doesn't acquisition on my objects work in Connection::tpc_begin() ??? The objects are not acquisition wrapped. IMO, ZODB is way too low a level to do what you want to do. Note that what you are trying to do would be easier in Zope 3 because Zope 3 relies far less on wrappers for acquisition. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zodb connection question
[ Jim Fulton wrote:] > Jürgen Herrmann wrote: >> hi all! >> >> i'm trying to form a patch that will result in a method >> "_before_commit()" >> being called on each modified object in a transaction (if that method >> exists of course) right before commit. >> main sense is to automate/delay (re)cataloging. >> >> first i looked at the Transaction class, as there have been heavy >> modifications to it from zodb 3.2 to 3.4 > > Patching is generally a bad idea. > > I suggest creatinga new catalog (by subclassing or adapting an existing > one) > that: > > - queues updates > > - registers a before-commit callback with the TM on the first update >in a transaction > > - processes the queue in the callback > > I did this recently for with Zope 3's catalog and it worked very well. > (I happend to use subclassing and would use adaptation if I were to > do it again.) > > Jim hi jim! thanks for your reply, i already thought about such a solution and discarded it because it would still be necessary to call a method on an object to recatalog it. this step (the programmer's responsibility) i want to eliminate for several reasons. so, what i need is a way to have my mechanism called before transaction commits (would be possible to use the hooks provided by zodb 3.4 for that, sure) and cycle through all modified objects (that's where the hooks in 3.4 are not enough for me, they don't provide any way to access a list of modified objects). as i showed in my first post, the callback to modified objects works as expected, but in the called method i'm not able to acquire the responsible Catalog for the modified object. up to what point can i modify objects in a transaction? why doesn't acquisition on my objects work in Connection::tpc_begin() ??? regards, juergen herrmann ___ >> XLhost.de - eXperts in Linux hosting << Juergen Herrmann Weiherweg 10, 93051 Regensburg, Germany Fon: +49 (0)700 XLHOSTDE [0700 95467833] Fax: +49 (0)721 151 463027 ICQ: 27139974 - IRC: [EMAIL PROTECTED] WEB: http://www.XLhost.de ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zodb connection question
Jürgen Herrmann wrote: hi all! i'm trying to form a patch that will result in a method "_before_commit()" being called on each modified object in a transaction (if that method exists of course) right before commit. main sense is to automate/delay (re)cataloging. first i looked at the Transaction class, as there have been heavy modifications to it from zodb 3.2 to 3.4 Patching is generally a bad idea. I suggest creatinga new catalog (by subclassing or adapting an existing one) that: - queues updates - registers a before-commit callback with the TM on the first update in a transaction - processes the queue in the callback I did this recently for with Zope 3's catalog and it worked very well. (I happend to use subclassing and would use adaptation if I were to do it again.) Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
