I am working with a third-party web-scraping application that also
uses twisted (Scrapy) and would like for it to do its thing in a
transaction along with a database action. This can be done with
runInteraction, but Scrapy uses a non-blocking call when it does its
thing. This actually doesn't even return a deferred, but you pass in
the callback function you would like it to call when finished.
Is there some way for runInteraction to wait for this deferred to be
finished before ending the transaction?
Or could something like this work, where the thread gets paused until
it completes?:
def interaction(txn):
finished = {'test':False, 'failed': False}
def _cb(r):
finished['test'] = True
def _err(failure):
finished['failed'] = True
Spider.crawl(_cb, _err)
# do database stuff here
while not finished['test'] and not finished['failed']:
time.sleep(0.25)
if finished['failed']:
raise Exception('Web scraping failed.')
d = dbpool.runInteraction(interaction)
Thanks for a look,
Landreville
_______________________________________________
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
