On Mon, Feb 15, 2010 at 6:23 AM, Leam Hall <l...@reuel.net> wrote: > > What I don't understand is why you need to have a cron job to deal with the > user data. Why not have your processing script called when the user submits? > This keeps your script from having to go through the entire database to find > uncommitted changes. If you're going to have a gazillion row database aren't > you going to be spending a lot of time on queries just to find those not > committed? > > Disconnecting the submission and the processing is helpful in a lot of situations. Sometimes you have a small amount of data that takes a lot of processing. For example an ETL ( http://en.wikipedia.org/wiki/Extract,_transform,_load) job that has a lot of validation. If I am an expediting warehouse and I receive pick and ship orders from online storefronts via XML files, I will want to make sure my inventory database says I can fill the pick and ship orders. I might receive these XML file via a web service that does basic schema validation and returns a receipt of some kind.Those XML files can go into a queue (be it a loading table, directory in a file system, MSMQ/ActiveMQ/TibcoQ or other technology). At some point the files will be processed, pick and ship orders put in another queue, a status message prepared for the storefront that submitted the order.
This disconnect provides you several advantages. For one, "threading" gets easier when you have a disconnect between submission and receiving. If submission order is not that important, you can scale your submission and processing ends separately. Second, your submitters don't have to wait for you to process their data. They can wait for the report and deal with the problems later. Third, you can schedule database writes to not occur during peak read times. Regards, Justin Dearing
_______________________________________________ New York PHP Users Group Community Talk Mailing List http://lists.nyphp.org/mailman/listinfo/talk http://www.nyphp.org/Show-Participation
