Re: [ZODB-Dev] Automating retry management

2010-05-11 Thread Jim Fulton
On Tue, May 11, 2010 at 11:35 AM, Laurence Rowe  wrote:
> On 11 May 2010 15:08, Jim Fulton  wrote:
>> On Tue, May 11, 2010 at 8:38 AM, Benji York  wrote:
>>> On Tue, May 11, 2010 at 7:34 AM, Jim Fulton  wrote:
 [...] The best I've been
 able to come up with is something like:

    t = ZODB.transaction(3)
    while t.trying:
        with t:
            ... transaction body ...
>>>
>>> I think you could get this to work:
>>>
>>> for transaction in ZODB.retries(3):
>>>    with transaction:
>>>        ... transaction body ...
>>>
>>> ZODB.retries would return an iterator that would raise StopIteration on
>>> the next go-round if the previously yielded context manager exited
>>> without a ConflictError.
>>
>> This is an improvement. It's still unsatisfying, but I don't think I'm going 
>> to
>> get satisfaction. :)
>>
>> BTW, if I do something like this, I think I'll add a retry exception to
>> the transaction package and have ZODB.POSException.ConflictError
>> extend it so I can add the retry automation to the transaction package.
>
> The repoze.retry package lets you configure a list of exceptions.
> http://pypi.python.org/pypi/repoze.retry
> http://svn.repoze.org/repoze.retry/trunk/repoze/retry/__init__.py
>
>  Though it seems inspecting the error text is required for most sql
> database errors to know if they are retryable, as ZPsycoPGDA does:
>
>  188                 except (psycopg2.ProgrammingError,
> psycopg2.IntegrityError), e:
>  189                     if e.args[0].find("concurrent update") > -1:
>  190                         raise ConflictError
>
> (https://dndg.it/cgi-bin/gitweb.cgi?p=public/psycopg2.git;a=blob;f=ZPsycopgDA/db.py)
>
> For PostgreSQL it should be sufficient to catch these errors and raise
> Retry during tpc_vote.
>
> For databases which do not provide MVCC in the same way as PostgreSQL,
> concurrency errors could be manifested at any point in the
> transaction. Even Oracle can raise an error during a long running
> transaction when insufficient rollback space is available, resulting
> in what is essentially a read conflict error. Such errors could not be
> caught by a data manager and reraised as a Retry exception.
>
> I think it might be useful to add an optional method to data managers
> that is queried by the retry automation machinery to see if an
> exception should potentially be retried. Perhaps this would best be
> accomplished in two steps:
>
> 1. Add an optional property to data managers called ``retryable``.
> This is a list of potentially retryable exceptions. When a data
> manager is added to the transaction, the transaction's list of
> retryable exceptions is extended by the joining data managers list of
> retryable exceptions.
>
> t = transaction.begin()
> try:
>    application()
> except t.retryable, e:
>    t.retry(e):
>
> 2. t.retry(e) is then checks with each registered data manager if that
> particular exceptions is retryable, and if so raises Retry.
>
> def retry(self, e):
>    for datamanager in self._resources:
>        try:
>            retry = datamanager.retry
>        except AttributeError:
>            continue
>        if isinstance(e, datamanager.retryable):
>            datamanager.retry(e) # dm may raise Retry here

Thanks.

I don't think we need 1 and 2.
I'm inclined to go with 2.

Jim

-- 
Jim Fulton
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Automating retry management

2010-05-11 Thread Laurence Rowe
On 11 May 2010 15:08, Jim Fulton  wrote:
> On Tue, May 11, 2010 at 8:38 AM, Benji York  wrote:
>> On Tue, May 11, 2010 at 7:34 AM, Jim Fulton  wrote:
>>> [...] The best I've been
>>> able to come up with is something like:
>>>
>>>    t = ZODB.transaction(3)
>>>    while t.trying:
>>>        with t:
>>>            ... transaction body ...
>>
>> I think you could get this to work:
>>
>> for transaction in ZODB.retries(3):
>>    with transaction:
>>        ... transaction body ...
>>
>> ZODB.retries would return an iterator that would raise StopIteration on
>> the next go-round if the previously yielded context manager exited
>> without a ConflictError.
>
> This is an improvement. It's still unsatisfying, but I don't think I'm going 
> to
> get satisfaction. :)
>
> BTW, if I do something like this, I think I'll add a retry exception to
> the transaction package and have ZODB.POSException.ConflictError
> extend it so I can add the retry automation to the transaction package.

The repoze.retry package lets you configure a list of exceptions.
http://pypi.python.org/pypi/repoze.retry
http://svn.repoze.org/repoze.retry/trunk/repoze/retry/__init__.py

 Though it seems inspecting the error text is required for most sql
database errors to know if they are retryable, as ZPsycoPGDA does:

 188 except (psycopg2.ProgrammingError,
psycopg2.IntegrityError), e:
 189 if e.args[0].find("concurrent update") > -1:
 190 raise ConflictError

(https://dndg.it/cgi-bin/gitweb.cgi?p=public/psycopg2.git;a=blob;f=ZPsycopgDA/db.py)

For PostgreSQL it should be sufficient to catch these errors and raise
Retry during tpc_vote.

For databases which do not provide MVCC in the same way as PostgreSQL,
concurrency errors could be manifested at any point in the
transaction. Even Oracle can raise an error during a long running
transaction when insufficient rollback space is available, resulting
in what is essentially a read conflict error. Such errors could not be
caught by a data manager and reraised as a Retry exception.

I think it might be useful to add an optional method to data managers
that is queried by the retry automation machinery to see if an
exception should potentially be retried. Perhaps this would best be
accomplished in two steps:

1. Add an optional property to data managers called ``retryable``.
This is a list of potentially retryable exceptions. When a data
manager is added to the transaction, the transaction's list of
retryable exceptions is extended by the joining data managers list of
retryable exceptions.

t = transaction.begin()
try:
application()
except t.retryable, e:
t.retry(e):

2. t.retry(e) is then checks with each registered data manager if that
particular exceptions is retryable, and if so raises Retry.

def retry(self, e):
for datamanager in self._resources:
try:
retry = datamanager.retry
except AttributeError:
continue
if isinstance(e, datamanager.retryable):
datamanager.retry(e) # dm may raise Retry here

Laurence
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Automating retry management

2010-05-11 Thread Nitro
Am 11.05.2010, 17:08 Uhr, schrieb Nitro :

> Am 11.05.2010, 16:01 Uhr, schrieb Jim Fulton :
>
>> This wouldn't work.  You would need to re-execute the suite
>> for each retry. It's not enough to just keep committing the same
>> transaction. (There are other details wrong with the code above,
>> but they are fixable.)  Python doesn't provide a way to keep
>> executing the suite.
>
> You are right.
>
> The only thing I could come up with was something like below, using a
> decorator instead of a context.
>
> -Matthias
>
> @doTransaction(count = 5)
> def storeData():
> ... store data here ...
>
> def doTransaction(transaction = None, count = 3):
>  def decorator(func):
>  def do():
>  for i in range(1+count):
>  try:
>  func()
>  except:
>  transaction.abort()
>  raise
>  try:
>  transaction.commit()
>  except ConflictError:
>  if i == count:
>  raise
>  else:
>  return
>  return do

This should read return do(), i.e. the decorator should directly execute  
the storeData function.

All in all I think Benji's proposal looks better :-)

-Matthias
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Automating retry management

2010-05-11 Thread Nitro
Am 11.05.2010, 16:01 Uhr, schrieb Jim Fulton :

> This wouldn't work.  You would need to re-execute the suite
> for each retry. It's not enough to just keep committing the same
> transaction. (There are other details wrong with the code above,
> but they are fixable.)  Python doesn't provide a way to keep
> executing the suite.

You are right.

The only thing I could come up with was something like below, using a  
decorator instead of a context.

-Matthias

@doTransaction(count = 5)
def storeData():
... store data here ...

def doTransaction(transaction = None, count = 3):
 def decorator(func):
 def do():
 for i in range(1+count):
 try:
 func()
 except:
 transaction.abort()
 raise
 try:
 transaction.commit()
 except ConflictError:
 if i == count:
 raise
 else:
 return
 return do
 return decorator

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Automating retry management

2010-05-11 Thread Benji York
On Tue, May 11, 2010 at 10:08 AM, Jim Fulton  wrote:
> This is an improvement. It's still unsatisfying, but I don't think I'm going 
> to
> get satisfaction. :)

Given that PEP 343 explicitly mentions *not* supporting an auto retry
construct, I should think not. :)
-- 
Benji York
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Automating retry management

2010-05-11 Thread Jim Fulton
On Tue, May 11, 2010 at 8:38 AM, Benji York  wrote:
> On Tue, May 11, 2010 at 7:34 AM, Jim Fulton  wrote:
>> [...] The best I've been
>> able to come up with is something like:
>>
>>    t = ZODB.transaction(3)
>>    while t.trying:
>>        with t:
>>            ... transaction body ...
>
> I think you could get this to work:
>
> for transaction in ZODB.retries(3):
>    with transaction:
>        ... transaction body ...
>
> ZODB.retries would return an iterator that would raise StopIteration on
> the next go-round if the previously yielded context manager exited
> without a ConflictError.

This is an improvement. It's still unsatisfying, but I don't think I'm going to
get satisfaction. :)

BTW, if I do something like this, I think I'll add a retry exception to
the transaction package and have ZODB.POSException.ConflictError
extend it so I can add the retry automation to the transaction package.

Jim

-- 
Jim Fulton
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Automating retry management

2010-05-11 Thread Jim Fulton
On Tue, May 11, 2010 at 7:52 AM, Nitro  wrote:
> I'm already using custom transaction/savepoint context managers in my
> code. I use them like

...
> Now you could probably extend this to look like
>
> class TransactionContext(object):
>     def __init__(self, txn = None, retryCount = 3):
>         if txn is None:
>             txn = transaction.get()
>         self.txn = txn
>         self.retryCount = retryCount
>
>     def __enter__(self):
>         return self.txn
>
>     def __exit__(self, t, v, tb):
>         if t is not None:
>             self.txn.abort()
>         else:
>             for i in range(self.retryCount):
>                 try:
>                     self.txn.commit()
>                 except ConflictError as exc2:
>                     exc = exc2
>                 else:
>                     return
>             raise exc
>
> The looping/except part could probably look nicer. Use case looks like:
>
> with TransactionContext(mytransaction, retryCount = 5):
>     db.root['sp_test'] = 'init'
>
> Does this look similar to what you were looking for?

This wouldn't work.  You would need to re-execute the suite
for each retry. It's not enough to just keep committing the same
transaction. (There are other details wrong with the code above,
but they are fixable.)  Python doesn't provide a way to keep
executing the suite.

Jim

-- 
Jim Fulton
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Automating retry management

2010-05-11 Thread Benji York
On Tue, May 11, 2010 at 7:34 AM, Jim Fulton  wrote:
> [...] The best I've been
> able to come up with is something like:
>
>    t = ZODB.transaction(3)
>    while t.trying:
>        with t:
>            ... transaction body ...

I think you could get this to work:

for transaction in ZODB.retries(3):
with transaction:
... transaction body ...

ZODB.retries would return an iterator that would raise StopIteration on
the next go-round if the previously yielded context manager exited
without a ConflictError.
-- 
Benji York
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Automating retry management

2010-05-11 Thread Nitro
>  def __exit__(self, t, v, tb):
>  if t is not None:
>  self.txn.abort()
>  else:
>  for i in range(self.retryCount):

Oops, bug here. It should read range(1 + self.retryCount). It should  
probably have unittests anyway :-)

-Matthias
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Automating retry management

2010-05-11 Thread Nitro
I'm already using custom transaction/savepoint context managers in my  
code. I use them like

with TransactionContext():
 db.root['sp_test'] = 'init'


with SavepointContext():
 db.root['sp_test'] = 'saved'

On of the context managers:

class TransactionContext(object):
 def __init__(self, txn = None):
 if txn is None:
 txn = transaction.get()
 self.txn = txn

 def __enter__(self):
 return self.txn

 def __exit__(self, t, v, tb):
 if t is not None:
 self.txn.abort()
 else:
 self.txn.commit()


Now you could probably extend this to look like

class TransactionContext(object):
 def __init__(self, txn = None, retryCount = 3):
 if txn is None:
 txn = transaction.get()
 self.txn = txn
 self.retryCount = retryCount

 def __enter__(self):
 return self.txn

 def __exit__(self, t, v, tb):
 if t is not None:
 self.txn.abort()
 else:
 for i in range(self.retryCount):
 try:
 self.txn.commit()
 except ConflictError as exc2:
 exc = exc2
 else:
 return
 raise exc

The looping/except part could probably look nicer. Use case looks like:

with TransactionContext(mytransaction, retryCount = 5):
 db.root['sp_test'] = 'init'

Does this look similar to what you were looking for?

-Matthias

For completeness, here's my savepoint manager:

class SavepointContext(object):
 def __enter__(self, txn = None):
 if txn is None:
 txn = transaction.get()
 self.savepoint = txn.savepoint()
 return self.savepoint

 def __exit__(self, type, value, traceback):
 if type is not None:
 self.savepoint.rollback()
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev


[ZODB-Dev] Automating retry management

2010-05-11 Thread Jim Fulton
So I'm about to update the transaction package and this gives me an
opportunity to do something I've been meaning to do for a while, which
is to add support for the Python with statement:

with transaction:
... transaction body ...

and:

with some_transaction_manager as t:
... transaction body, accesses current transaction as t ...

This looks really great, IMO, but there's a major piece missing, which
is dealing with transient transaction failures due to conflicts.  If
using an optimistic transaction mechanism, like ZODB's, you have to
deal with conflict errors.  If using a lock-based transaction
mechanism, you'd have to deal with deadlock detection. In either case,
you have to detect conflicts and retry the transaction.

I wonder how other Python database interfaces deal with this.  (I just
skimmed the DBI v2 spec and didn't see anything.) What happens, for
example, if there are conflicting writes in a Postgress or Oracle
application? I assume that some sort of exception is raised.

I also wonder how this situation could be handled elegantly.  To deal
with conflicts, (assuming transaction had with statement support)
you'd end up with:

tries = 0
while 1:
try:
with transaction:
conn.root.x = 1
except ZODB.POSExeption.ConflictError:
tries += 1
if tries > 3:
raise

Yuck!  (Although it's better than it would be without transaction
with statement support.) In web applications, we generally don't see
the retry management because the framework takes care of it for
us. That is until we write a script to do something outside of a web
application.

This would be easier to automate if Python let us write custom looping
structures or allowed full anonymous functions.  The best I've been
able to come up with is something like:

t = ZODB.transaction(3)
while t.trying:
with t:
... transaction body ...

Here the transaction function returns an object that:

- keeps track of how many times it's tried and manages a "trying"
  attribute that is true while we haven't given up or suceeded, and

- is a context manager that takes care of transaction boundaries and
  updates the trying attr depending on transaction outcome.

This version is better than the one with the try/except version,
but isn't entirely satisfying. :)

Does anyone have any better ideas?

I use a ZODB function, because ConflictError is ZODB specific.  It
would be nice if this could be standardized, so that a mechanism could
be defined by the transaction package.

Jim

--
Jim Fulton
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev