My question is how do I reduce processing time... is there anything I
could be doing better?
I am tasked with doing an import on a file whose size can essentially
be unlimited. We've been able to handle in the 10-15mb range but it
recently ballooned to 100 mb, and its going to get larger. Processing
time seems to be about 66 hours for a 45 mb file and thats a disaster.
For a 14mb file its about 90 minutes.
Whats happening is this: CF is looping over a very large number of
records that are in themselves fairly complex. The more records there
are, the longer the loop over them takes. More records + more time
per record = a seemingly geometric increase in processing time.
The file is read in like so.
<!---
read the imported file
--->
<cflock
name="mates_import"
type="EXCLUSIVE"
timeout="10">
<cffile
action="READ"
file="#variables.mates.srcFile#"
variable="x">
</cflock>
<cfset x =
replaceNoCase(x,"StateOrProvinceCountrySub-DivisionID","StateProvince","ALL")>
<!---
turn the file into a coldfusion xml object
--->
<cfset x=ltrim(trim(x))>
<cfset x=XMLParse(x)>
The above takes only a few seconds. No problem there.
Next I have to read in some header info
<cfscript>:
header.logicalID=x.processvehicleremarketing.applicationarea.sender.logicalID.xmltext;
header.taskID=x.processvehicleremarketing.applicationarea.sender.taskID.xmltext;
header.BODID=x.processvehicleremarketing.applicationarea.BODID.xmltext;
header.created=x.processvehicleremarketing.applicationarea.CreationDateTime.xmltext;
// ...
// and here comes the node with all of the line items in it I'll have
to loop over. This is where all of the speed issues have been traced
to:
variables.mates.boatArrayLen=arrayLen(x.processvehicleremarketing.ProcessVehicleRemarketingDataArea.VehicleRemarketing);
</cfscript>
knowing the array length I can use CFLOOP to look over it and pull
data in where it is then stored in a db.
<cfloop
from="1"
to="#variables.mates.boatArrayLen#"
index="i">
<cfscript>
listings_mates.inHouseListingNumber="M-" &
x.processVehicleRemarketing.processVehicleRemarketingDataArea.vehicleRemarketing[i].vehicleRemarketingHeader.documentIdentificationGroup.documentIdentification.documentID.xmltext;
listings_mates.price=x.processVehicleRemarketing.processVehicleRemarketingDataArea.vehicleRemarketing[i].vehicleRemarketingBoatLineItem.pricingABIE.price.chargeAmount.xmltext;
listings_mates.currency=x.processVehicleRemarketing.processVehicleRemarketingDataArea.vehicleRemarketing[i].vehicleRemarketingBoatLineItem.pricingABIE.price.chargeAmount.xmlAttributes.currencyID;
// there can be more than one of these items so run a loop inside
of the loop
variables.mates.engineArrayLen=arrayLen(x.processVehicleRemarketing.processVehicleRemarketingDataArea.vehicleRemarketing[i].vehicleRemarketingBoatLineItem.VehicleRemarketingEngineLineItem);
ii=0;
do {
ii=ii+1;
listings_mates.engineDesc=x.processVehicleRemarketing.processVehicleRemarketingDataArea.vehicleRemarketing[i].vehicleRemarketingBoatLineItem.VehicleRemarketingEngineLineItem[ii].VehicleRemarketingEngine.modelDescription.xmltext;
listings_mates.engHrs=x.processVehicleRemarketing.processVehicleRemarketingDataArea.vehicleRemarketing[i].vehicleRemarketingBoatLineItem.VehicleRemarketingEngineLineItem[ii].VehicleRemarketingEngine.totalEngineHoursNumeric.xmltext;
} while (ii LTE variables.mates.engineArrayLen);
...
</cfscript>
...
</cfloop>
And so on. A hundred or so fields and a dozen or so loops inside the
main loop, along with a loop or two inside of those. So the very long
variable names get even longer.
As you can see I am pouring the data into a struct, and when done, I
insert it as a db record.
Anyone see a mistake in my methods? Would things speed up if, before
I read it into an xml object, I ran a replace() or three to shorten up
some of those names?
--
--m@Robertson--
Janitor, The Robertson Team
mysecretbase.com
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Order the Adobe Coldfusion Anthology now!
http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion
Archive:
http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:342213
Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/groups/cf-talk/unsubscribe.cfm
