Hi, Matt... I know you might have looked at this link at dev.mysql.com for a solution to processing your data feed, but I thought I'd bring it up, just in case you haven't seen it. Looks like it might be right up your alley.
http://dev.mysql.com/tech-resources/articles/xml-in-mysql5.1-6.0.html#xml-5. 1-in-and-out While I didn't have to process xml files, I do have to process a daily data feed of real estate data in comma-delimited files and insert that info into a MySQL 5 database. At first, I tried processing the files with CF, but it was a *really* slow process using CF, taking almost a minute to process some files. Finally, someone on this list suggested I take a look at the MySQL commands for loading data from files, such as, in your case, "Load_File()", which opens an entire XML document, makes it available as a string, and inserts this string into a table column. In my case, using MySQL's load data infile, the time was reduced to less than a second. Check out the page above for details on usage and syntax. hth, Rick -----Original Message----- From: Matt Robertson [mailto:[email protected]] Sent: Wednesday, February 16, 2011 12:22 PM To: cf-talk Subject: Re: Big XML files processing Really s-l-o-w. Solution? Hey everybody, we're in a bind here and I pitched the idea to the client of bringing in a ringer. Is there someone out there who is interested in building a direct-to-SQL Server or direct-to-mySQL-based solution? Solution would have to, on a scheduled basis, grab a file in a location on the server on its own without CF and process it (we have stuff that retrieves/places the file). Current file size is about 50 mb but must be expected to go to around 100. At 42mb using just CF I am at about 70 seconds per record loop. We need a *significant* improvement in performance. Work would be on a dedicated Windows server. Present box is a fairly powerful Crystaltech Win2003 Server w/4GB RAM and a Xeon 2.8 processor w/4 cores. We'd consider a Win2008, 64-bit installation to improve speed. Looking at a 24-core server at Viviotech w/16gb as a hi-horsepower alternative. Our feed partner is probably going to be able to use a different method of delivery that will reduce file size, but for now we need to plan for the worst and move on it. Anyone interested? You can email me at my for-reals email at matt AT mysecretbase DOT com. -- --m@Robertson-- Janitor, The Robertson Team mysecretbase.com ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| Order the Adobe Coldfusion Anthology now! http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:342338 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/groups/cf-talk/unsubscribe.cfm
