Wow! So THAT is what the BatchUpdate flag is for?!? I completely forgot about it, and I did read every bit of the Hackers Guide for both VFP6 and VFP7 (excellent reference, you and your fellow writers ought to be proud of what you put together). I will look at both that, and PostgreSQL COPY.
As always, your insights are also very much appreciated. Thank you for the time taken in explaining all that away. Regards, Gil > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] Behalf Of Ted Roche > Sent: Saturday, April 07, 2007 6:13 PM > To: [EMAIL PROTECTED] > Subject: Re: PostgreSQL text file import performance > > > On 4/7/07, mrgmhale <[EMAIL PROTECTED]> wrote: > > > > So, how are other folks handling large text/csv file imports into remote > > updateable views? > > You've run into one of the few areas where the BatchUpdate flag on the > ODBC connection is actually a good idea. Check out the Hacker's Guide. > > You're also doing a "bulk update" so the rules we're used to following > for online transaction processing are wrong and you need to rethink. > Check out: > > http://www.postgresql.org/docs/8.1/interactive/populate.html > > Many systems come with a "bulk loader" functionality that basically > rips through incoming data and makes it a table, then goes back at the > end and cleans things up like record count and creating indexes. > That's a much more efficient way to do it than to have each record > start a transaction, make room for the record, write the record, > update any indexes, clean up, close the transaction and return. > > > I am certainly curious to see if this is a common issue > > with this kind of data processing, or if I am the only one with > this issue. > > There's an entire industry of ETL (Extract-Translate-Load) Tools. > Probably the second or third thing written once there were more than > two computers in the world. With incompatible formats, of course. > > > Now for PC tech specs; > > Those all seem adequate at first glance. You'll want to benchmark and > watch some performance indicators to try to figure out where the > bottleneck is. > > > -- > Ted Roche > Ted Roche & Associates, LLC > http://www.tedroche.com > > [excessive quoting removed by server] _______________________________________________ Post Messages to: [email protected] Subscription Maintenance: http://leafe.com/mailman/listinfo/profox OT-free version of this list: http://leafe.com/mailman/listinfo/profoxtech Searchable Archive: http://leafe.com/archives/search/profox This message: http://leafe.com/archives/byMID/profox/[EMAIL PROTECTED] ** All postings, unless explicitly stated otherwise, are the opinions of the author, and do not constitute legal or medical advice. This statement is added to the messages for those lawyers who are too stupid to see the obvious.
