On 6/15/06, Kevin Dangoor <[EMAIL PROTECTED]> wrote:
This is a nice recipe. I'll add it to the docs, if that's okay with you.
On Jun 15, 2006, at 6:56 AM, Richard Clark wrote:
>
> Basically a short writeup of some interesting stuff I've had to do for
> a project I've been working on recently. If people find this
> interesting, I'll see if I can post some actual code listings.
>
> Creating a background batch processing thread.
>
>
> Our objective here is to have a single thread running in the
> background
> that performs operations that may run longer than we might want to
> have
> the user sit around for. There are a couple of possible areas for
> this,
> certain type sof credit card processing, large batch operations,
> interfaces with (potentially unavailable) external apis etc.
>
> The general pattern works like this:
>
> First up, create a module to place your batch processing stuff in.
> I'll
> call mine "processor" (processor.py in your standard turbogears app
> dir)
>
> Within this, we need three things. We need a the thread function
> itself, we need a singleton to allow us to identify and share
> resources
> with the main thread, and we need a startup function to make it
> easy to
> start the thread up.
>
> First up, the singleton, this is what ties everything together and
> makes it possible to provide notification to the thread.
>
> class Singleton(object):
> def __init__(self):
> self.event = threading.Event()
> self.thread = None
> self.event.set()
>
> singleton = Singleton()
>
> Lets start with a really stupid processor:
>
> def processingThread():
> """ Thread to process domains """
> while True:
> singleton.event.wait(30) # Wait up to 30 seconds for a
> new event
> log.debug("Doing processing")
>
> And finally, the startup:
>
> def start():
> if singleton.thread and singleton.thread.isAlive():
> """ Bail out if we're already running """
> return
>
> log.debug("Starting processing")
> singleton.thread = threading.Thread(target=processingThread)
> singleton.thread.setDaemon(True) # Make sure the thread shuts
> down when the main program does
> singleton.thread.start()
>
>
> Ok, that's almost it really, we just need to tie it into our program.
> First up, in controllers.py we want to start() it on startup:
>
> import processor
> import turbogears
>
> def myStartup():
> processor.start()
>
> turbogears.startup.call_on_startup.append(myStartup)
>
> Adding it to call_on_startup means we'll get called whenever
> turbgoears
> starts up (duh).
>
> Now what we have is a processing thread being launched that will wait
> up to 30 seconds for a new event, then try processing anyway. At the
> moment it doesn't do anything, lets modify the thread a bit:
>
> def processingThread():
> """ Thread to change creditcards from waiting to paid """
> # This part is important, it gives us a new connection for
> this
> # thread, otherwise your thread will get messy db access
> from sqlobject.util.threadinglocal import local as
> threading_local
> hub.threadingLocal = threading_local()
>
> while True:
> singleton.event.wait(300)
> log.debug("Processing creditcards")
> toCharge = Creditcard.selectBy(status="waiting")
> for creditcard in toCharge:
> hub.begin() # Begin transaction
> log.debug("Processing creditcard %d" %
> creditcard.id
> creditcard.status = "paid"
> hub.commit()
>
> Ok! Now we have a (kinda) useful thing. Once every 5 minutes it'll
> process the creditcard table looking for CC's that are waiting to be
> charged, and it'll mark them paid.
>
> But wait, 5 minutes is annoying. That's what that event thing is for.
> The 5 minutes is a safety mark, our cleanup point.
>
> What you should do, is any time you've added new creditcards in there
> as waiting, simply do:
>
> import processor
> processor.singleton.event.set()
>
> This will set the event, and cause the wait(300) to break out early.
> That way it'll most likely be processed immediately, but worst case
> it'll happen in 5 minutes.
>
> Even better, with this mechanism (rather than, say, using scheduler),
> you don't run the risk of having the processor execute simultaneously,
> if it takes longer than 5 minutes to process the creditcards, it
> simply
> won't get back around to waiting. When it does, if there are any new
> events, it'll go check again.
>
> Other uses include things like a thread which maintains a jabber
> connection and lets you log messages into a conference (done that,
> good
> fun :) multiplayer logs with comments!).
>
>
> >
--
Kevin Dangoor
TurboGears / Zesty News
email: [EMAIL PROTECTED]
company: http://www.BlazingThings.com
blog: http://www.BlueSkyOnMars.com
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "TurboGears" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at http://groups.google.com/group/turbogears
-~----------~----~----~----~------~----~------~--~---
- [TurboGears] Re: Creating a background batch processing ... Eric Larson
- [TurboGears] Re: Creating a background batch proces... Richard Clark
- [TurboGears] Re: Creating a background batch proces... Jorge Vargas
- [TurboGears] Re: Creating a background batch pr... Richard Clark
- [TurboGears] Re: Creating a background batc... [EMAIL PROTECTED]
- [TurboGears] Re: Creating a background batc... Andrew Grover
- [TurboGears] Re: Creating a background ... Richard Clark
