On 4/7/07, mrgmhale <[EMAIL PROTECTED]> wrote: > > So, how are other folks handling large text/csv file imports into remote > updateable views?
You've run into one of the few areas where the BatchUpdate flag on the ODBC connection is actually a good idea. Check out the Hacker's Guide. You're also doing a "bulk update" so the rules we're used to following for online transaction processing are wrong and you need to rethink. Check out: http://www.postgresql.org/docs/8.1/interactive/populate.html Many systems come with a "bulk loader" functionality that basically rips through incoming data and makes it a table, then goes back at the end and cleans things up like record count and creating indexes. That's a much more efficient way to do it than to have each record start a transaction, make room for the record, write the record, update any indexes, clean up, close the transaction and return. > I am certainly curious to see if this is a common issue > with this kind of data processing, or if I am the only one with this issue. There's an entire industry of ETL (Extract-Translate-Load) Tools. Probably the second or third thing written once there were more than two computers in the world. With incompatible formats, of course. > Now for PC tech specs; Those all seem adequate at first glance. You'll want to benchmark and watch some performance indicators to try to figure out where the bottleneck is. -- Ted Roche Ted Roche & Associates, LLC http://www.tedroche.com _______________________________________________ Post Messages to: [email protected] Subscription Maintenance: http://leafe.com/mailman/listinfo/profox OT-free version of this list: http://leafe.com/mailman/listinfo/profoxtech Searchable Archive: http://leafe.com/archives/search/profox This message: http://leafe.com/archives/byMID/profox/[EMAIL PROTECTED] ** All postings, unless explicitly stated otherwise, are the opinions of the author, and do not constitute legal or medical advice. This statement is added to the messages for those lawyers who are too stupid to see the obvious.
