First off, always process in the smallest increment you can. So don't grab EVERY row, just grab ONE row and process it, then grab the next row and process it... That allows you to run multiple, concurrent processors and each one can grab the data they want.
Secondly.... there are many queing services already written, http://en.wikipedia.org/wiki/Message_queue As an oddball idea, instead of using a database store, to add an item to the queue, json encode it and email it to a special address on the processing server(s)....then configure your mail server to trigger a PHP script for each email received and process it from there. Then your email server will give you queueing for free.. On Sun, Feb 14, 2010 at 8:49 PM, Anthony Papillion <papill...@gmail.com>wrote: > Hello Everyone, > > I'm designing a system that will work on a schedule. Users will submit data > for processing into the database and then, every minute, a PHP script will > pass through the db looking for unprocessed rows (marked pending) and > process them. > > The problem is, I may eventually have a few million records to process at a > time. Each record could take anywhere from a few seconds to a few minutes to > perform the required operations on. My concern is making sure that the > script, on the next scheduled pass, doesn't grab the records currently being > processed and start processing them again. > > Right now, I'm thinking of accomplishing this by updating a 'status' field > in the database. So unprocessed records would have a status of 'pending', > records being processed would have a status of 'processing' and completly > processed record will have a status of 'complete'. > > For some reason, I see this as ugly but that's the only way I can think of > making sure that records aren't duplicatly processed. So when I select > records to process, I'm ONLY selecting one's with the status of 'pending' > which means they are new, unprocessed. > > Is there a better, more eleqent way of doing this or is this pretty much > it? > > Thanks! > Anthony Papillion > > _______________________________________________ > New York PHP Users Group Community Talk Mailing List > http://lists.nyphp.org/mailman/listinfo/talk > > http://www.nyphp.org/Show-Participation > -- ---- Hudson Valley Sudbury School What GPL is for application users Our school is for students Help your children grow, change, and learn Let your child direct, control, amend Check out http://www.sudburyschool.org
_______________________________________________ New York PHP Users Group Community Talk Mailing List http://lists.nyphp.org/mailman/listinfo/talk http://www.nyphp.org/Show-Participation
