On Mon, Jul 30, 2012 at 9:01 PM, Steven D'Aprano <st...@pearwood.info>wrote:
> If you want to be robust, it is best not to try to beat the database. That > means you should write to the database as soon as you can, as often as you > need to, and let the database do what it does best: reliable transaction > storage. Any decent database will guarantee ACID compliance (atomicity, > consistency, isolation, durability). > > But sometimes you need to compromise on robustness for speed or > convenience. > > > [...] > > This is how the program was originally structured, but we found >> performance >> problems with in. >> It's using Django and an Oracle DB, which is notoriously bad for single >> record read/writes. So, I >> figured why don't we append them all to an array, and write them out at >> the >> end of the program. >> > > I suggest you periodically check the item log, and if there are more than > (say) 20 items, you write them to the database and clear the temporary > list. That way, the number of item logs should never get too large, and the > performance shouldn't suffer too greatly. You will need to tweak that > number to be more or less depending on how poorly the Oracle DB performs. > > Then, at the very end of the program, you write whatever items are left in > a finally clause, or atexit. That way, the finally clause should be nice > and speedy and the user is unlikely to hit Ctrl-C a second time. > > I think that is the process I'm going to take. It doesn't seem as elegant, but I think it's a good compromise. > > This will be the most robust as it will >>> also work for cases where the program is terminated without the use of >>> the keyboard (i.e. kill -9, task manager, computer reboot, etc.) but >>> >> > That unfortunately is not so. kill -9 does *not* send a signal or raise an > exception. It just kills the process dead, instantly and without warning. > > Likewise for cases when somebody trips over the power cord and disconnects > the server from the power. (Really paranoid sys admins insist on having two > power supplies connected to two different UPSes for their servers.) > > > > > it might also slow the program down. You could also try writing the >>> item log to a local file using pickle or csv. >>> >> > Better is to use the logging module rather than to reinvent the wheel. > > Do your item logs actually need to go in the database? Perhaps you could > just write them to a local log file and leave the database for more > critical records. > > Yes for reprocessing. :( -Tino
_______________________________________________ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
