In parts of my app, some needed values take a long time to generate,
so I'm farming the generation of these values out to Scheduled tasks
and designing with the expectation that some values just won't be
ready to include in a page. When ready, the generated values are then
incorporated into the page on subsequent requests or by ajax calls
from the page.
Almost always, I want to cache these generated values. I thought this
seemed a common enough need that a mechanism should be incorporated
with the task Scheduler. Here's a start at what this might look like:
def bgCache(key, db, task, args=None, time_expire=60 * 60 * 24,
cache=cache.ram):
'''
cache argument expects either cache.ram or cache.disk
If the value was sitting in the cache, return it from the cache.
If the value is not in the cache, check if task was previously run
and has completed, in which case get the output value from the task
db.
Otherwise, schedule the task and return None to indicate that the
value is not ready. TODO use a special sentinel value to indicate no
value? Maybe just throw an exception?
'''
val = cache(key, lambda: None, time_expire=time_expire)
if val == None:
# see if task has been completed and get value
val = getTaskOutput(task):
if val == None:
scheduleTask(task, args) # TODO don't schedule redundant
tasks
else:
cache(key, None)
cache(key, lambda: val, time_expire=time_expire)
return val
So calling bgCache('key'...) returns the value for 'key' if it's
already in cache or if the task to generate it has completed;
otherwise, the task is scheduled and None returned. (So None couldn't
be a valid generated value. Arguably, some other special sentinel
value should be used to indicate that the generated value is not ready
so that None could be used as a valid generated value.)
To make this work generically, the input and output to the task would
be passed through the db as JSON. If I'm not wrong, this would serve
most Scheduler use cases while obviating any custom task db tables.
Of course, the precise details and API need some thought. Perhaps
bgCache could instead be a Scheduler method called 'cache'. Any
thoughts?
