Re: [ZODB-Dev] ConflictErrors won't clear
Jeremy Hylton wrote: If you don't run an asyncore mainloop, you'll be responsible for manually sync-ing all the storages/connections that asyncore would handle automatically. Tim guessed wrong ;-) Apart from the odd .sych() call on the various connections, what other housekeeping does this asyncore thread do? In fact, where should I hunt to see the source for this? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
RE: [ZODB-Dev] ConflictErrors won't clear
... [Chris Withers] >> I'd really prefer not to do that unless absolutely necessary: >> http://mail.zope.org/pipermail/zodb-dev/2004-June/007554.html [Jeremy Hylton] > It sounds like the answer here it so avoid fork, rather than asyncore. Good advice in general to avoid both <0.5 wink>. Still, people frightened by that link should read the whole thread, on zope-dev. POSIX fork() doesn't clone threads to begin with, so the scenario described in the original message doesn't apply even in the presence of forking on most boxes. Dieter eventually came to that conclusion too: http://www.mail-archive.com/zope-dev@zope.org/msg16809.html Discussion in "[EMAIL PROTECTED]" suggests that this problem cannot occur as described with a POSIX fork implementation. I observed the problem in a different setup and concluded from the Linux "fork" manual page only that Zope's "asyncore.mainloop" thread will suffer from the same problem. I did not observe the behaviour in this setup and it is well possible that it cannot occur > If you don't run an asyncore mainloop, you'll be responsible for > manually sync-ing all the storages/connections that asyncore would > handle automatically. Worth repeating. Worth guessing Chris's next question too : http://mail.zope.org/pipermail/zodb-dev/2003-September/005930.html ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
RE: [ZODB-Dev] ConflictErrors won't clear
[Jeremy Hylton] >> It's mentioned in the documentation -- see section 3.4 ZEO programming >> notes -- and it's been discussed on this list many, many times. [Chris Withers] > Where are these notes? http://www.zope.org/Wikis/ZODB/FrontPage/guide/index.html Then click on section "3.4 ZEO Programming Notes". >> It sounds like the simplest approach for your application is to do like >> Zope and start a separate thread that runs an asyncore mainloop. Then >> your application threads will see updates when the commit and abort >> transactions, just like the would in Zope. > I'd really prefer not to do that unless absolutely necessary: > http://mail.zope.org/pipermail/zodb-dev/2004-June/007554.html You should read that entire thread, but on zope-dev, to which follow-ups were set and where almost all discussion took place. But I'm betting your app doesn't call fork() to begin which, in which case nothing in that thread has any relevance to your app. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ConflictErrors won't clear
On 4/15/05, Chris Withers <[EMAIL PROTECTED]> wrote: > Jeremy Hylton wrote: > > > > It's mentioned in the documentation -- see section 3.4 ZEO programming > > notes -- and it's been discussed on this list many, many times. > > Where are these notes? In the ZODB & ZEO programming guide that's packaged with ZODB and available in the Wiki. Perhaps you've read it before? > > It sounds like the simplest approach for your application is to do > > like Zope and start a separate thread that runs an asyncore mainloop. > > Then your application threads will see updates when the commit and > > abort transactions, just like the would in Zope. > > I'd really prefer not to do that unless absolutely necessary: > http://mail.zope.org/pipermail/zodb-dev/2004-June/007554.html It sounds like the answer here it so avoid fork, rather than asyncore. If you don't run an asyncore mainloop, you'll be responsible for manually sync-ing all the storages/connections that asyncore would handle automatically. Jeremy ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ConflictErrors won't clear
Jeremy Hylton wrote: It's mentioned in the documentation -- see section 3.4 ZEO programming notes -- and it's been discussed on this list many, many times. Where are these notes? It sounds like the simplest approach for your application is to do like Zope and start a separate thread that runs an asyncore mainloop. Then your application threads will see updates when the commit and abort transactions, just like the would in Zope. I'd really prefer not to do that unless absolutely necessary: http://mail.zope.org/pipermail/zodb-dev/2004-June/007554.html cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ConflictErrors won't clear
On 4/15/05, Chris Withers <[EMAIL PROTECTED]> wrote: > > Short course: A ZEO client needs to run an asyncore mainloop if it wants to > > get invalidations processed "by magic". Alternatives include calling > > sync(), or closing and (re)opening the connection, at appropriate times. > > Hurm, this really should be in bright flashing neon somewhere. I the 5 > years this process has been used, I've never known that ZEO don't work > right when there's no asyncore loop :-( It's mentioned in the documentation -- see section 3.4 ZEO programming notes -- and it's been discussed on this list many, many times. It sounds like the simplest approach for your application is to do like Zope and start a separate thread that runs an asyncore mainloop. Then your application threads will see updates when the commit and abort transactions, just like the would in Zope. Jeremy ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ConflictErrors won't clear
Christian Robottom Reis wrote: It is the official way to call sync(), yes. I don't know how you do it in Zope, but the connection is what DB.open() returns, and is also attached to persisted objects as the _p_jar attribute. Hmm, does anyone know the "right" way to get this in Zope? If I do someobj._p_jar.sync() on an object, will that synch all objects in that storage? It seems pretty nasty to me, how many connections will I have open if I have two mounted storages? How can I find them all and sync them all? Isn't there a nicer, more official way of synching all open connections? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ConflictErrors won't clear
On Fri, Apr 15, 2005 at 02:39:03PM +0100, Chris Withers wrote: > >>Okay, where in the above should I be calling sync()? > >>Where do I get sync from? get_transaction() doesn't have a synch > >>attribute.. > > > >On the Connection, of course > > And how do I get hold of the connection? Is that the "official way"? It is the official way to call sync(), yes. I don't know how you do it in Zope, but the connection is what DB.open() returns, and is also attached to persisted objects as the _p_jar attribute. > Does this change if I have several storages mounted with the stuff that > ships with Zope 2.7? Possibly; I'd imagine each storage had its own Connection. Take care, -- Christian Robottom Reis | http://async.com.br/~kiko/ | [+55 16] 3361 2331 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ConflictErrors won't clear
Christian Robottom Reis wrote: On Fri, Apr 15, 2005 at 12:57:07PM +0100, Chris Withers wrote: Okay, where in the above should I be calling sync()? Where do I get sync from? get_transaction() doesn't have a synch attribute.. On the Connection, of course And how do I get hold of the connection? Is that the "official way"? Does this change if I have several storages mounted with the stuff that ships with Zope 2.7? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ConflictErrors won't clear
On Fri, Apr 15, 2005 at 12:57:07PM +0100, Chris Withers wrote: > Okay, where in the above should I be calling sync()? > Where do I get sync from? get_transaction() doesn't have a synch attribute.. On the Connection, of course Take care, -- Christian Robottom Reis | http://async.com.br/~kiko/ | [+55 16] 3361 2331 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ConflictErrors won't clear
Tim Peters wrote: The non-zope client has logic that looks roughly like: for work in queue: try: get_transaction().begin() # do work, change zodb objects, etc get_transaction().commit() except ConflictError: get_transaction().abort() queue.append(work) except: get_transaction().abort() Okay, where in the above should I be calling sync()? Where do I get sync from? get_transaction() doesn't have a synch attribute.. You mean apart from that `queue` grows without bound <0.9 wink>? Well, I'm hoping that things won't conflicterror EVERY time Type this at Google: site:mail.zope.org zodb-dev asyncore mainloop Short course: A ZEO client needs to run an asyncore mainloop if it wants to get invalidations processed "by magic". Alternatives include calling sync(), or closing and (re)opening the connection, at appropriate times. Hurm, this really should be in bright flashing neon somewhere. I the 5 years this process has been used, I've never known that ZEO don't work right when there's no asyncore loop :-( Chris - who wants nothing to do with an asyncore loop, ever... PS: Do the guys working on non-asyncore servers for Z3 know about this limitation? -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
RE: [ZODB-Dev] ConflictErrors won't clear
[Chris Withers] > I have a non-zope zeo client that pumps data into a storage server for > later consumption by a zope zeo client. Everything is Zope 2.7.5. > > The non-zope client has logic that looks roughly like: > > for work in queue: >try: > get_transaction().begin() > # do work, change zodb objects, etc > get_transaction().commit() > >except ConflictError: > get_transaction().abort() > queue.append(work) > >except: > get_transaction().abort() > > Does anything look amiss there? You mean apart from that `queue` grows without bound <0.9 wink>? > The reason I ask is that it works fine, unless someone changes something > via the zope zeo client resulting in a conflict error on the non-zope > zeo client. When that happens, the work where the conflict occurs gets > put back in the queue, but seems to conflict again next time round, even > though the change on the zope zeo client is long since committed and > passed. The "work" effectively gets stuck in an endless loop of being > retried and then conflicterror'ing :-( > > Any ideas why that could be? Type this at Google: site:mail.zope.org zodb-dev asyncore mainloop Short course: A ZEO client needs to run an asyncore mainloop if it wants to get invalidations processed "by magic". Alternatives include calling sync(), or closing and (re)opening the connection, at appropriate times. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ConflictErrors won't clear
I had a problem like that, and I had to explicitely sync() the connection before begin(), I think. Florent Chris Withers <[EMAIL PROTECTED]> wrote: > Hi there, > > I have a non-zope zeo client that pumps data into a storage server for > later consumption by a zope zeo client. Everything is Zope 2.7.5. > > The non-zope client has logic that looks roughly like: > > for work in queue: >try: > > get_transaction().begin() > > # do work, change zodb objects, etc > > get_transaction().commit() > >except ConflictError: > > get_transaction().abort() > > queue.append(work) > >except: > > get_transaction().abort() > > Does anything look amiss there? > > The reason I ask is that it works fine, unless someone changes something > via the zope zeo client resulting in a conflict error on the non-zope > zeo client. When that happens, the work where the conflict occurs gets > put back in the queue, but seems to conflict again next time round, even > though the change on the zope zeo client is long since committed and > passed. The "work" effectively gets stuck in an endless loop of being > retried and then conflicterror'ing :-( > > Any ideas why that could be? > > cheers, > > Chris > > -- > Simplistix - Content Management, Zope & Python Consulting > - http://www.simplistix.co.uk > ___ > For more information about ZODB, see the ZODB Wiki: > http://www.zope.org/Wikis/ZODB/ > > ZODB-Dev mailing list - ZODB-Dev@zope.org > http://mail.zope.org/mailman/listinfo/zodb-dev > -- Florent Guillaume, Nuxeo (Paris, France) CTO, Director of R&D +33 1 40 33 71 59 http://nuxeo.com [EMAIL PROTECTED] ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] ConflictErrors won't clear
Hi there, I have a non-zope zeo client that pumps data into a storage server for later consumption by a zope zeo client. Everything is Zope 2.7.5. The non-zope client has logic that looks roughly like: for work in queue: try: get_transaction().begin() # do work, change zodb objects, etc get_transaction().commit() except ConflictError: get_transaction().abort() queue.append(work) except: get_transaction().abort() Does anything look amiss there? The reason I ask is that it works fine, unless someone changes something via the zope zeo client resulting in a conflict error on the non-zope zeo client. When that happens, the work where the conflict occurs gets put back in the queue, but seems to conflict again next time round, even though the change on the zope zeo client is long since committed and passed. The "work" effectively gets stuck in an endless loop of being retried and then conflicterror'ing :-( Any ideas why that could be? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev