Re: [Zope-dev] Bulletproof ZCatalog proposal

2001-06-08 Thread Phillip J. Eby

At 04:58 PM 6/8/01 -0400, Shane Hathaway wrote:
On Thursday 07 June 2001 21:28, Phillip J. Eby wrote:
  Upon being told to perform a transaction or subtransaction commit,
  the transaction would notify all the ruleAgents, and then all the
  indexingAgents.  Objects could still subscribe to either queue while
  this notifying is taking place.  (So that triggered actions could
  cause indexish objects to register as indexingAgents, not to mention
  causing updated objects to fire additional triggers.)
 
  Once all agents in a queue are notified, that queue should be cleared
  so that notifications are on a per-subtransaction basis.  Once both
  queues are cleared, normal transaction behavior goes forward.

Would you say this would occur before tpc_begin() messages or just
after?

Before.  I'm saying this would take place immediately at the start of the 
existing Transaction.commit() method.


  Hm.  That's simpler than I thought it was going to be.  Shoot, I can
  even see how to implement it as a runtime patch, that would've been
  simpler than all the shenanigans ZPatterns goes through to fake out
  the transaction machinery...  and it's a better
  implementation.  Ah well.  At the time I wanted to avoid trying to
  convince Jim to add machinery to transactions just for ZPatterns,
  given that ZPatterns wasn't a particularly established product at the
  time.
 
  Let me know if you guys put something like this in, though, and I'll
  definitely look at reworking ZPatterns to use the mechanism.  It
  could potentially cut a lot of code out and improve the robustness at
  the same time.

I don't foresee us adding this capability right away since we need to
understand it better and it only applies to one working product and a
hypothetical product.  I'd suggest you go ahead with the runtime patch.

I think I'll wait until ZPatterns works with Zope 2.4, unless it becomes 
necessary otherwise.  Replacing what I've got now is a pretty significant 
undertaking, and risk-prone to boot, so I don't want to delay the finishing 
of ZPatterns' support for Zope 2.3.x.

Basically, the runtime patch would be to replace 
ZODB.Transaction.Transaction with a subclass that implemented the 
notification queues in the commit() method.  There'd be a few other little 
extensions needed, like clearing the queues on any abort operation, and 
adding registerIndex() and registerRule() or some such methods.


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Bulletproof ZCatalog proposal

2001-06-07 Thread Toby Dickenson

On Thu, 31 May 2001 10:36:55 -0400 (EDT), Shane Hathaway
[EMAIL PROTECTED] wrote:

On Thu, 31 May 2001, Toby Dickenson wrote:

 On Thu, 31 May 2001 10:03:31 -0400 (EDT), Shane Hathaway
 [EMAIL PROTECTED] wrote:
 Right now ZCatalog randomly generates ConflictErrors even if there are no
 conflicts in the data being indexed.  It's quite rare, however, and
 there's machinery that retries the entire transaction up to three times
 when it happens.

 If the current mechanism is reliable, and efficient (given that
 conflict/retries are rare) then I dont understand the motivation

The system becomes less reliable as server load increases.

 Large catalog
updates (where every object is reindexed) also generate a lot of
conflicts.

Is that last bit true? I thought 'Update Catalog' created *new*
indexes. There might be a conflict on the root catalog opbject, but
inside the index BTrees.



Toby Dickenson
[EMAIL PROTECTED]

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Bulletproof ZCatalog proposal

2001-06-07 Thread Toby Dickenson

My last post didnt make much sense. This time ill include all the
words in my sentence

 Large catalog
updates (where every object is reindexed) also generate a lot of
conflicts.

Is that last bit true? I thought 'Update Catalog' created *new*
indexes. There might be a conflict on the root catalog object, but
not on the index BTrees.


Toby Dickenson
[EMAIL PROTECTED]

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Bulletproof ZCatalog proposal

2001-06-07 Thread Shane Hathaway

On Thursday 07 June 2001 07:43, Toby Dickenson wrote:
  Large catalog
 updates (where every object is reindexed) also generate a lot of
 conflicts.

 Is that last bit true? I thought 'Update Catalog' created *new*
 indexes. There might be a conflict on the root catalog object, but
 not on the index BTrees.

I should say large catalog updates tend to generate a conflict.  It 
really doesn't matter how many conflicts there are.  Within a single 
transaction, 1 conflict is as bad as 100.

One thing I didn't make clear in the proposal is that I'm interested in 
repurposing ZCatalog as a general ZODB indexing mechanism and 
essentially moving it down from the application layer to the database 
layer.  Catalogs would be updated automatically (in a lazy fashion to 
avoid performance penalties).  There would potentially be a large 
number of catalogs, though, so the number of conflicts would increase 
significantly.

But I think I have a solution for all of the issues in conflict 
resolution.  If you, or anyone else, is also interested in this, please 
show support (if only by saying please do this!) :-)  I can't work on 
it unless I have a reason to.

Shane

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Bulletproof ZCatalog proposal

2001-06-07 Thread Chris Withers

Shane Hathaway wrote:
 
 But I think I have a solution for all of the issues in conflict
 resolution.  If you, or anyone else, is also interested in this, please
 show support (if only by saying please do this!) :-)  I can't work on
 it unless I have a reason to.

Please do this! :-)

I'm particularly interested in ZCatalog becoming a standard ZODB feature, and I
know there are otehrs on ZODB-dev who are dying for this right now...

cheers,

Chris

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Bulletproof ZCatalog proposal

2001-06-07 Thread Erik Enge

On Thu, 7 Jun 2001, Shane Hathaway wrote:

 It really doesn't matter how many conflicts there are.  Within a
 single transaction, 1 conflict is as bad as 100.

Why?
 
 But I think I have a solution for all of the issues in conflict 
 resolution. 

Whee!

 If you, or anyone else, is also interested in this, please show
 support (if only by saying please do this!) :-)  I can't work on it
 unless I have a reason to.

Please do this!


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Bulletproof ZCatalog proposal

2001-06-07 Thread John D . Heintz

Me too! Me too!

John

On Thursday 07 June 2001 08:49, Chris Withers wrote:
 Shane Hathaway wrote:
  But I think I have a solution for all of the issues in conflict
  resolution.  If you, or anyone else, is also interested in this, please
  show support (if only by saying please do this!) :-)  I can't work on
  it unless I have a reason to.

 Please do this! :-)

 I'm particularly interested in ZCatalog becoming a standard ZODB feature,
 and I know there are otehrs on ZODB-dev who are dying for this right now...

 cheers,

 Chris

 ___
 Zope-Dev maillist  -  [EMAIL PROTECTED]
 http://lists.zope.org/mailman/listinfo/zope-dev
 **  No cross posts or HTML encoding!  **
 (Related lists -
  http://lists.zope.org/mailman/listinfo/zope-announce
  http://lists.zope.org/mailman/listinfo/zope )

-- 
. . . . . . . . . . . . . . . . . . . . . . . .

John D. Heintz | Senior Engineer

1016 La Posada Dr. | Suite 240 | Austin TX 78752
T 512.633.1198 | [EMAIL PROTECTED]

w w w . d a t a c h a n n e l . c o m

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Bulletproof ZCatalog proposal

2001-06-07 Thread Dan L. Pierson



--On Thursday, June 07, 2001 09:34:54 AM -0400 Shane Hathaway 
[EMAIL PROTECTED] wrote:

 But I think I have a solution for all of the issues in conflict
 resolution.  If you, or anyone else, is also interested in this, please
 show support (if only by saying please do this!) :-)  I can't work on
 it unless I have a reason to.

Please do this!  It is the sort of thing that can save us and others a lot
of time by keeping use of Zope simple and stable.





___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Bulletproof ZCatalog proposal

2001-06-07 Thread Shane Hathaway

On Thursday 07 June 2001 11:51, Erik Enge wrote:
 On Thu, 7 Jun 2001, Shane Hathaway wrote:
  It really doesn't matter how many conflicts there are.  Within a
  single transaction, 1 conflict is as bad as 100.

 Why?

Because in Zope it means the whole request is processed again (which 
can lead to strange behavior if non-transactional data is changed by 
the request).  In other apps it might mean messy exception handling.

  If you, or anyone else, is also interested in this, please show
  support (if only by saying please do this!) :-)  I can't work on
  it unless I have a reason to.

 Please do this!

I should have stated that it also needs to be a priority.  Within DC 
we've been using better project management and it's getting harder to 
do projects just because we might think they'll be useful for someone.

Shane

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Bulletproof ZCatalog proposal

2001-06-07 Thread Shane Hathaway

On Thursday 07 June 2001 12:17, Phillip J. Eby wrote:
 At 09:34 AM 6/7/01 -0400, Shane Hathaway wrote:
 One thing I didn't make clear in the proposal is that I'm interested
  in repurposing ZCatalog as a general ZODB indexing mechanism and
  essentially moving it down from the application layer to the
  database layer.  Catalogs would be updated automatically (in a lazy
  fashion to avoid performance penalties).  There would potentially
  be a large number of catalogs, though, so the number of conflicts
  would increase significantly.
 
 But I think I have a solution for all of the issues in conflict
 resolution.  If you, or anyone else, is also interested in this,
  please show support (if only by saying please do this!) :-)  I
  can't work on it unless I have a reason to.

 Sounds cool.  At one time Ty and I thought we might do something like
 this by adding to a linked list of pending updates which the
 catalog would use to alter its search results, until the list got
 too long, at which point it would post the actual updates.

That sounds like what I have in mind.

 The
 only catch was that this would still produce conflicts at the head
 end of the linked list.  :(  Of course, that was in the days before
 ZODB conflict resolution.  Nowadays, you could probably implement it
 as a simple sequence object with the conflict resolution method
 implemented.  But then there'd be the question of how to resolve
 conflicts if more than one thread or ZEO client decided to apply the
 queued changes...  a 100 conflicts vs 1 situation.  Ugh.

I was thinking the queue would only last the duration of a transaction 
and that the queue would be thread-specific.

 Anyway, I'd be really interested in seeing a solution for all the
 issues in conflict resolution.

I was thinking that certain types of objects would be committed by the 
transaction manager before all others.  In this case, the catalog (or a 
special object in the catalog) would be committed first.  It would 
resolve all conflicts in the contained indices before they occur by 
replaying the changes in the persisted queues from the transaction 
history, then setting the _p_serial attributes to convince the storage 
that the conflicts have already been resolved.

 Will it help with arguments, too? 
 How about world peace?  ;)

Okay, maybe we haven't solved that. :-)

Shane

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Bulletproof ZCatalog proposal

2001-06-07 Thread Phillip J. Eby

At 02:07 PM 6/7/01 -0400, Shane Hathaway wrote:
On Thursday 07 June 2001 12:17, Phillip J. Eby wrote:

 The
 only catch was that this would still produce conflicts at the head
 end of the linked list.  :(  Of course, that was in the days before
 ZODB conflict resolution.  Nowadays, you could probably implement it
 as a simple sequence object with the conflict resolution method
 implemented.  But then there'd be the question of how to resolve
 conflicts if more than one thread or ZEO client decided to apply the
 queued changes...  a 100 conflicts vs 1 situation.  Ugh.

I was thinking the queue would only last the duration of a transaction 
and that the queue would be thread-specific.

That's not hard to do - you could just toss an object into the transaction
queue to do it during transaction commit, similar to what ZPatterns does in
a more general way.  (Note that ZPatterns users can already get the
benefits of deferring catalog changes until the end of a (sub)transaction,
by using triggers to implement automatic cataloging).


 Anyway, I'd be really interested in seeing a solution for all the
 issues in conflict resolution.

I was thinking that certain types of objects would be committed by the 
transaction manager before all others.  In this case, the catalog (or a 
special object in the catalog) would be committed first.  It would 
resolve all conflicts in the contained indices before they occur by 
replaying the changes in the persisted queues from the transaction 
history, then setting the _p_serial attributes to convince the storage 
that the conflicts have already been resolved.


Hm.  Sounds to me like what you actually want is for the transaction
manager to do this *after* everything else, rather than before.  Thus, you
would catch any changes which occur *during* transaction commit - such as
commit-phase cataloging (as some folks do with ZPatterns currently).

That is, in ZPatterns one can specify triggers such as:

WHEN OBJECT DELETED, CHANGED CALL
someCatalog.manage_uncatalog(self.absolute_url(1))
WHEN OBJECT ADDED, CHANGED CALL
someCatalog.manage_catalog(self,self.absolute_url(1))

Which will be executed during transaction commit if the conditions apply.
If you have the catalog process its queue *before* other objects commit, it
will not have received these calls yet.  This will end up making the
catalog try to commit itself a second time, which will fail with a conflict
error - 100% of the time.  (And retry won't help, because the transaction
is conflicting with itself.)

Of course, this issue could be fixed by minor adjustment to the ZODB API
and implementation such that it is permissible to store() an object more
than once during a transaction's commit phase without creating a
ConflictError.


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Bulletproof ZCatalog proposal

2001-06-07 Thread Shane Hathaway

Phillip J. Eby wrote:
 That is, in ZPatterns one can specify triggers such as:
 
 WHEN OBJECT DELETED, CHANGED CALL
 someCatalog.manage_uncatalog(self.absolute_url(1))
 WHEN OBJECT ADDED, CHANGED CALL
 someCatalog.manage_catalog(self,self.absolute_url(1))

After I read this again I realized what you were saying.  This
capability of ZPatterns is very brittle, don't you think?  If the
catalog is updated manually before the special ZPatterns object is added
to the queue, the behavior is undefined AFAIK--either the later changes
to the catalog will be ignored, will cause a conflict, or some objects
will be written twice in the same transaction.

However, if we could specify transaction commit priorities, and the
ZPatterns update came first, auto-indexing came second, and everything
else followed, I think it would work.  Or perhaps ZPatterns should be
able to register things that occur *before* the two-phase commit
protocol.

Shane

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Bulletproof ZCatalog proposal

2001-06-07 Thread Shane Hathaway

On Thu, 7 Jun 2001, Phillip J. Eby wrote:

 I was thinking that certain types of objects would be committed by the
 transaction manager before all others.  In this case, the catalog (or a
 special object in the catalog) would be committed first.  It would
 resolve all conflicts in the contained indices before they occur by
 replaying the changes in the persisted queues from the transaction
 history, then setting the _p_serial attributes to convince the storage
 that the conflicts have already been resolved.

 Hm.  Sounds to me like what you actually want is for the transaction
 manager to do this *after* everything else, rather than before.  Thus, you
 would catch any changes which occur *during* transaction commit - such as
 commit-phase cataloging (as some folks do with ZPatterns currently).

Maybe I didn't explain this clearly enough.  Let me write some quick
pseudocode:


class Catalog (Persistent):
  finished_changes = None   # Mapping: {path - (object, adding)}
  unfinished_changes = None # Same as above
  tid = None# Transaction ID

  def catalogObject(self, ob, path):
unf = self.unfinished_changes
if unf is None: self.unfinished_changes = unf = {}
unf[path] = (ob, 1)

  def uncatalogObject(self, path):
unf = self.unfinished_changes
if unf is None: self.unfinished_changes = unf = {}
unf[path] = (ob, 0)

  def searchResults(self, ...):
self.finishChanges()
# ... Perform search ...
return results

  def finishChanges(self):
unf = self.unfinished_changes
if unf is not None:
  fin = self.finished_changes
  if fin is None or self._p_serial != self.tid:
# Create finished_changes if not yet created
# and clear it if we're in a different transaction
# from the last time finished_changes was changed.
self.finished_changes = fin = {}
self.tid = self._p_serial
  for path, (ob, adding) in unf.items():
if adding: self.addToIndexes(ob, path)
else: self.removeFromIndexes(path)
fin[path] = (ob, adding)
  self.unfinished_changes = None

  def __getstate__(self):
# Called during transaction commit.
self.finishChanges()
return Persistent.__getstate__(self)

  def _p_priority(self):
# Causes this object to be added to the *top*
# of the list of objects to commit rather than the
# bottom.  (Just an idea.)
return 1

  def _p_resolveConflict(self, old, committed, newstate):
'''
Apply the changes in self.finished_changes to
committed and return the result.
'''


This does mean that _p_resolveConflict() might be called frequently, but
(potentially) it would never fail because of conflicts.

Now, this doesn't provide any automatic cataloging, which is what I think
you're suggesting.  I think automatic reindexing, a good idea, is mostly
independent of bulletproofing and lazifying the catalog.

To achieve automatic indexing, I was thinking that a special attribute
would be added to cataloged objects.  It would contain the OIDs of the
catalogs interested in the object.  Transaction.register() would look for
this attribute and invoke catalogObject().  Of course, that wouldn't quite
work because the object might change again within the transaction and the
transaction manager wouldn't be told about the second and further changes.
But I'm sure there's a good way to compensate for this, such as making the
catalog scan for later changes before calling searchResults().
(cPersistence might need to assist.)

Shane


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Bulletproof ZCatalog proposal

2001-06-07 Thread Phillip J. Eby

At 07:07 PM 6/7/01 -0400, Shane Hathaway wrote:
Phillip J. Eby wrote:
 That is, in ZPatterns one can specify triggers such as:
 
 WHEN OBJECT DELETED, CHANGED CALL
 someCatalog.manage_uncatalog(self.absolute_url(1))
 WHEN OBJECT ADDED, CHANGED CALL
 someCatalog.manage_catalog(self,self.absolute_url(1))

After I read this again I realized what you were saying.  This
capability of ZPatterns is very brittle, don't you think?

Yep.  That's why I've previously described ZPatterns as a hack.  :)


If the
catalog is updated manually before the special ZPatterns object is added
to the queue, the behavior is undefined AFAIK--either the later changes
to the catalog will be ignored, will cause a conflict, or some objects
will be written twice in the same transaction.

True.  But this behavior is avoidable through the use of subtransaction
commits, in the event that someone has to have transactions which update a
ZCatalog directly.  Usually, when someone is using catalogs with ZPatterns,
they use triggers to do all the updates and don't touch the catalog manually.

Note that I'm not saying this still isn't a hack.  But it's the best I
could do without either fixing the multi-commit issue in ZODB, or with some
kind of priority scheme.


However, if we could specify transaction commit priorities, and the
ZPatterns update came first, auto-indexing came second, and everything
else followed, I think it would work.  Or perhaps ZPatterns should be
able to register things that occur *before* the two-phase commit
protocol.

Yep.  One of the last two times I spoke with Jim in person (either the
January DC visit or IPC 8, I forget which), he said something about it
maybe being a good idea to have some kind of priority system like that.
I'd love to see something like it exist, if it would make some of
ZPatterns' hackery unnecessary.

The implementation could consist of two subscription queues: ruleAgents and
indexingAgents.  ZCatalog would register in indexingAgents, and ZPatterns
objects would register in one or the other, usually ruleAgents.  (I can
think of some circumstances where it would be nice to use the
indexingAgents queue, but right now ZPatterns apps have to work around this
by defining their rules in execution priority order.)

Upon being told to perform a transaction or subtransaction commit, the
transaction would notify all the ruleAgents, and then all the
indexingAgents.  Objects could still subscribe to either queue while this
notifying is taking place.  (So that triggered actions could cause indexish
objects to register as indexingAgents, not to mention causing updated
objects to fire additional triggers.)

Once all agents in a queue are notified, that queue should be cleared so
that notifications are on a per-subtransaction basis.  Once both queues are
cleared, normal transaction behavior goes forward.

Hm.  That's simpler than I thought it was going to be.  Shoot, I can even
see how to implement it as a runtime patch, that would've been simpler than
all the shenanigans ZPatterns goes through to fake out the transaction
machinery...  and it's a better
implementation.  Ah well.  At the time I wanted to avoid trying to convince
Jim to add machinery to transactions just for ZPatterns, given that
ZPatterns wasn't a particularly established product at the time.

Let me know if you guys put something like this in, though, and I'll
definitely look at reworking ZPatterns to use the mechanism.  It could
potentially cut a lot of code out and improve the robustness at the same time.


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Bulletproof ZCatalog proposal

2001-05-31 Thread Erik Enge

On Wed, 30 May 2001, Shane Hathaway wrote:

 I have written a proposal.

So, if I understand correctly, when this is implemented, will the ZCatalog
always resolve conflicts and never raise ConflictErrors?  If there are no
ConflictErrors in the main object system?

I think I'm missing the point here.


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Bulletproof ZCatalog proposal

2001-05-31 Thread Toby Dickenson

On Thu, 31 May 2001 10:03:31 -0400 (EDT), Shane Hathaway
[EMAIL PROTECTED] wrote:

On Thu, 31 May 2001, Erik Enge wrote:

 On Wed, 30 May 2001, Shane Hathaway wrote:

  I have written a proposal.

 So, if I understand correctly, when this is implemented, will the ZCatalog
 always resolve conflicts and never raise ConflictErrors?  If there are no
 ConflictErrors in the main object system?

 I think I'm missing the point here.

How did you miss the point?  That's exactly the point! :-)

Right now ZCatalog randomly generates ConflictErrors even if there are no
conflicts in the data being indexed.  It's quite rare, however, and
there's machinery that retries the entire transaction up to three times
when it happens. 

If the current mechanism is reliable, and efficient (given that
conflict/retries are rare) then I dont understand the motivation

 But I want it to be more reliable, especially so that
ZCatalog can be used in ZODB applications outside Zope.

Is it because other non-Zope applications might not be so proficient
at performing retries?


Toby Dickenson
[EMAIL PROTECTED]

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Bulletproof ZCatalog proposal

2001-05-31 Thread Shane Hathaway

On Thu, 31 May 2001, Toby Dickenson wrote:

 On Thu, 31 May 2001 10:03:31 -0400 (EDT), Shane Hathaway
 [EMAIL PROTECTED] wrote:
 Right now ZCatalog randomly generates ConflictErrors even if there are no
 conflicts in the data being indexed.  It's quite rare, however, and
 there's machinery that retries the entire transaction up to three times
 when it happens.

 If the current mechanism is reliable, and efficient (given that
 conflict/retries are rare) then I dont understand the motivation

The system becomes less reliable as server load increases.  Large catalog
updates (where every object is reindexed) also generate a lot of
conflicts.

  But I want it to be more reliable, especially so that
 ZCatalog can be used in ZODB applications outside Zope.

 Is it because other non-Zope applications might not be so proficient
 at performing retries?

Yes.  Long-running transactions are especially problematic.

Shane


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Bulletproof ZCatalog proposal

2001-05-31 Thread Erik Enge

On Thu, 31 May 2001, Shane Hathaway wrote:

 How did you miss the point?  That's exactly the point! :-)

Oh :)
 
 Right now ZCatalog randomly generates ConflictErrors even if there are
 no conflicts in the data being indexed.

This is why I think I've missed a/the point: wouldn't it be better to
figure out why ZCatalog randomly generates these ConflictErrors instead of
making them disappear using other means?


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Bulletproof ZCatalog proposal

2001-05-31 Thread Shane Hathaway

On Thu, 31 May 2001, Erik Enge wrote:

 On Thu, 31 May 2001, Shane Hathaway wrote:

  How did you miss the point?  That's exactly the point! :-)

 Oh :)

  Right now ZCatalog randomly generates ConflictErrors even if there are
  no conflicts in the data being indexed.

 This is why I think I've missed a/the point: wouldn't it be better to
 figure out why ZCatalog randomly generates these ConflictErrors instead of
 making them disappear using other means?

We know why they happen.  They are only *apparently* random--they actually
make perfect sense.  (Maybe I didn't make that clear.)  If you
concurrently modify objects that happen to be placed near enough each
other in one of the indexes then a conflict will occur.  We have a partial
solution in place which tries to merge the buckets that make up the index,
but this solution can't solve the whole problem.

Think about what SQL databases do.  You can add entries to a table
simultaneously from different connections.  As long as the contents of
the entries don't conflict, there will not be any kind of conflict in the
indexes either.

ZCatalog is to ZODB what indexes are to SQL databases.  I'd like ZCatalog
to have the same level of reliability.

Shane


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )