I sometimes use a separate database connection with autocommit for logging, but I do it in a request finalizer. I also do it in a separate database because the database becomes large and other applications use it too.
On Sat, Jan 28, 2017 at 12:43 PM, Jonathan Vanasco <[email protected]> wrote: > > On Saturday, January 28, 2017 at 2:06:11 PM UTC-5, Mike Orr wrote: >> >> >> How are people synchronizing something transactional with something >> non-transactional, especially a database record with a filesystem >> file. > > > I deal with this a lot in the area of S3 file archiving. > > I abort the transaction if the s3 upload fails. Usually I can delete the > file via a except/finally clause -- but there are still edge cases where I > could end up with a stray because the cleanup activity can fail. > > To handle that, I use a second SqlAlchemy connection in autocommit mode for > bookkeeping, then run periodic tasks to reconcile issues later on. > > from memory, it looks something like this (psuedocde!) > > try: > # transactional record > objectRecord = {"id":id, "filename": filename} > dbSessionTransaction.add(objectRecord) > dbSessionTransaction.flush() > > # autocommit log > objectLog = {"id": id, "status": "prep"} > dbSessionAutocommit.add(objectLog) > dbSessionAutocommit.flush() > > # mark a partial upload > objectLog.status = 'uploading' > dbSessionAutocommit.flush() > > boto.upload_file(filename) > > # mark uploaded > objectLog.status = 'uploaded' > dbSessionAutocommit.flush() > > # mark the file > objectRecord.status = 'uploaded' > objectRecord.timestamp_upload = datetime() > dbSessionTransaction.commit() > > except: > # mark the autocommit as "deletion-start" > # try to delete the file > # mark the autocommit as "deleted" > > I periodically look through the autocommit logs to see if anything is stuck > and make sure deleted items are not in s3. > > it's definitely overkill, but it has so far caught all the (rare) edge > cases. > > -- > You received this message because you are subscribed to the Google Groups > "pylons-discuss" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/pylons-discuss/00c9a773-ac1b-40c7-9198-d46132f4c17e%40googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. -- Mike Orr <[email protected]> -- You received this message because you are subscribed to the Google Groups "pylons-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/CAH9f%3Dur0Lgs4t60SqsMjuo1y7JVjBkk7n9cvUh%2BN9E-TK06Kxw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
