hi, I am struggling to understand how to leverage python's multiprocessing module in a while loop. the examples I have found seem to assume it is known ahead of time how many items need to be processed.
specifically, I am reading from an external queue. I currently process items one at a time until the queue is empty. I wrote a wrapper function to handle connecting to the queue, pulling the next message, and deleting it when I am done. ideally, I'd like to modify this wrapper function to take an additional argument (processes) to specify the number of messages to process simultaneously. I've attached a script that captures what I am doing now. unfortunately, the external queue object is not publicly accessible and I'm not quite sure how to set up a local object that would support testing. any suggestions would be most welcome. thanks, Johan
#!/usr/bin/env python import boto import logging LOG_FMT = '%(asctime)s - %(levelname)s - %(module)s.%(funcName)s - %(message)s' QUEUE = 'my-queue' logger = logging.getLogger(__name__) # read from an external queue; items may be added during processing def consume_queue(queue_name): conn = boto.connect_sqs() q = conn.get_queue(queue_name) m = q.read() while m is not None: yield m q.delete_message(m) logger.debug('message deleted') m = q.read() # high variability in message processing times (seconds - hours) def handle_message(message) s = message.get_body() logger.info(s) def main(): for message in consume_queue(QUEUE): handle_message(message) if __name__ == '__main__': logging.basicConfig(level=logging.DEBUG, format=LOG_FMT) main()
-- https://mail.python.org/mailman/listinfo/python-list