Anthony, I'd have to actually look, but you may be able to kick the process off with a system command and grab the process id at the same time. Or kick the process off with some unique id that will help you grab the exact process id after the fact. One of the other guys, may have a more practice approach to this.
Michele _____ From: talk-boun...@lists.nyphp.org [mailto:talk-boun...@lists.nyphp.org] On Behalf Of Michele Waldman Sent: Sunday, February 14, 2010 9:11 PM To: 'NYPHP Talk' Subject: Re: [nyphp-talk] Best way to accomplish this task Anthony, I'd actually have to put some real thought into it. I'm watching tv. But, I think a timestamp would actually be ugly. Some random id doesn't really tell you anything. When I work with shell scripts, I actually capture the process id. Then if the records are unprocessed and the process id is not running, then I'd reprocess. For some backend jobs, I use php. Other times, I use bash or some other scripting language. That's actually what I believe the correct process is. As jobs grow time can grow as well, time stamps are not reliable. Michele _____ From: talk-boun...@lists.nyphp.org [mailto:talk-boun...@lists.nyphp.org] On Behalf Of Anthony Papillion Sent: Sunday, February 14, 2010 9:00 PM To: NYPHP Talk Subject: Re: [nyphp-talk] Best way to accomplish this task Michele, Thank you for your input. I'm glad I seem to be on the right track here and I'm curious about what others might think of this method of walking the database too. I hadn't thought about the additional logic part in the event of an error. How would you propose that be handled? Perhaps a 'current time' field that's updated after each operation? We could assume that a 'current time' that's more than, say 15 minutes, is an indication of the row needing to be processed even though the status might be 'processing'. What do you think about that? Thanks, Anthony ----- Original Message ----- From: Michele Waldman To: 'NYPHP Talk' Sent: Sunday, February 14, 2010 7:55 PM Subject: Re: [nyphp-talk] Best way to accomplish this task As I was reading, I was thinking that would be my approach. I'm not seeing anything ugly about it. Records being processed should have a status as such. You might have some additional logic, that picks up records in the event one of the processes fail for some reason. Sever goes down or some one accidently kills the process, etc. We'll see what the other folks think. Michele From: talk-boun...@lists.nyphp.org [mailto:talk-boun...@lists.nyphp.org] On Behalf Of Anthony Papillion Sent: Sunday, February 14, 2010 8:50 PM To: 'NYPHP Talk' Subject: [nyphp-talk] Best way to accomplish this task Hello Everyone, I'm designing a system that will work on a schedule. Users will submit data for processing into the database and then, every minute, a PHP script will pass through the db looking for unprocessed rows (marked pending) and process them. The problem is, I may eventually have a few million records to process at a time. Each record could take anywhere from a few seconds to a few minutes to perform the required operations on. My concern is making sure that the script, on the next scheduled pass, doesn't grab the records currently being processed and start processing them again. Right now, I'm thinking of accomplishing this by updating a 'status' field in the database. So unprocessed records would have a status of 'pending', records being processed would have a status of 'processing' and completly processed record will have a status of 'complete'. For some reason, I see this as ugly but that's the only way I can think of making sure that records aren't duplicatly processed. So when I select records to process, I'm ONLY selecting one's with the status of 'pending' which means they are new, unprocessed. Is there a better, more eleqent way of doing this or is this pretty much it? Thanks! Anthony Papillion _______________________________________________ New York PHP Users Group Community Talk Mailing List http://lists.nyphp.org/mailman/listinfo/talk http://www.nyphp.org/Show-Participation
_______________________________________________ New York PHP Users Group Community Talk Mailing List http://lists.nyphp.org/mailman/listinfo/talk http://www.nyphp.org/Show-Participation
