Re: Better manage bulk update process

2009-11-13 Thread Agha Mehdi
there are other tasks that run at the same time too but none that would interact with the same tables. One thing I did notice though that there is only one task that runs at 8:00 PM (which is this one) and there are 4 different tasks that run at 8:00 AM (including this task). Could that be an

Re: Better manage bulk update process

2009-11-13 Thread Agha Mehdi
I also just thought of something. I am using the following setting; Frequency: Daily every 12 hours start time: 11:00 PM. Do I need to specify End time too? On Fri, Nov 13, 2009 at 8:58 AM, Agha Mehdi aghaime...@gmail.com wrote: there are other tasks that run at the same time too but none

RE: Better manage bulk update process

2009-11-13 Thread DURETTE, STEVEN J (ATTASIAIT)
Subject: Re: Better manage bulk update process I also just thought of something. I am using the following setting; Frequency: Daily every 12 hours start time: 11:00 PM. Do I need to specify End time too? On Fri, Nov 13, 2009 at 8:58 AM, Agha Mehdi aghaime...@gmail.com wrote: there are other tasks

Re: Better manage bulk update process

2009-11-13 Thread Jason Fisher
No, you shouldn't need to specify end, but make sure there's a timeout sufficient for the process to complete. ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion

Re: Better manage bulk update process

2009-11-13 Thread Maureen
Do any of the other tasks use the tables you are deleting or updating? On Fri, Nov 13, 2009 at 8:58 AM, Agha Mehdi aghaime...@gmail.com wrote: there are other tasks that run at the same time too but none that would interact with the same tables. One thing I did notice though that there is

Re: Better manage bulk update process

2009-11-13 Thread Agha Mehdi
nope, the other tasks don't do anything to those tables. so here's what I did. wrapped cftransaction around the process. created two staging tables that hold the process updates. once the process finishes, i call a stored proc, which deletes old data from prod tables, copy data from staging to

Better manage bulk update process

2009-11-12 Thread Agha Mehdi
All, I have a bulk update process for our products catalog. It needs to delete old data before it loads new data. Sometimes it fails to load new data after deleting old data, which causes product catalog to be empty. I don't have CFTransaction wrapped around it but I want to put it in place.

Re: Better manage bulk update process

2009-11-12 Thread Alan Rother
Well... There are a number of techniques for ensuring data integrity. In cases where I'm using CF to manage a bulk import (rather than the database itself) I tend to use a staging database to import the data and then I only transfer it to the production database IF the import passes some sort of

Re: Better manage bulk update process

2009-11-12 Thread Agha Mehdi
Tell me about it. I would take that approach if it was an option. It has to be done through cf on a live production db before tomorrow morning. On Thu, Nov 12, 2009 at 1:02 PM, Alan Rother alan.rot...@gmail.com wrote: Well... There are a number of techniques for ensuring data integrity. In

Re: Better manage bulk update process

2009-11-12 Thread Maureen
I would put the transaction and a catch/try around each insert. Then if it fails, you know exactly where it fails, and can handle the error cleanly. I would also do a record count after the delete to confirm that old data is being cleared, and reset any auto-increment fields, triggers, and

Re: Better manage bulk update process

2009-11-12 Thread Jason Fisher
Just a note that the CFTRANSACTION needs to be around the entire block of CFQUERY statements, or else it does you no good ... cftransaction cftry cfquery DELETE some stuff /cfquery cfquery INSERT some stuff /cfquery cfquery UPDATE some other stuff /cfquery cfcatch type=Database trap your error

Re: Better manage bulk update process

2009-11-12 Thread Alan Rother
Hmm... Ok, any chance you could create Temp tables that match the production ones? Run your import on them, then once complete, table by table TRUNCATE the prod data and then transfer the new data over to the real tables? =] -- Alan Rother Adobe Certified Advanced ColdFusion MX 7 Developer

Re: Better manage bulk update process

2009-11-12 Thread Agha Mehdi
Yup. that's what I am doing now. CFTransaction is an option but it won't do me any good if for some reason the processing just stops without any error. It is strange but that's what's happening. I haven't found any errors in the logs. Also, it gets more interesting as the task runs every 12

Re: Better manage bulk update process

2009-11-12 Thread Maureen
Is there any chance the morning run is conflicting with another task, like a backup, or another import, and just stalling? On Thu, Nov 12, 2009 at 4:29 PM, Agha Mehdi aghaime...@gmail.com wrote: Yup. that's what I am doing now. CFTransaction is an option but it won't do me any good if for