Hi Gil
> Message: 7
> Date: Wed, 2 Apr 2008 10:13:07 -0400
> From: "Gil Hale" <[EMAIL PROTECTED]>
> Subject: > To: [EMAIL PROTECTED]
> Message-ID: <[EMAIL PROTECTED]>
> Content-Type: text/plain; charset="us-ascii"
>
SNIP!!!
>I have no recorded time comparisons, just my gut feel based on years
of watching VFP import records from large csv files (millions of
records at a crack). And, for the record, when I do my imports into a
VFP table I leave any index/order set off, as having a table with an
active index/order will slow down an import process horribly with
large tables as VFP tries to rearrange imported records into the
active sort order...
>
I don't think that's 100% accurate. If you have SET ORDER TO 0, but
still have a CDX, the cdx tags must be updated. If you have SET ORDER
TO something, I don't believe that slows down the APPEND FROM. - Can't
test it right now though.
Regardless - here's something to consider. I hope you can apply this
to PostgreSQL. When importing large tables in I first APPEND the
source into a temporary table first, then do two things. Note the ORs
in the update command. In VFP they stop evaluating once the first one
is met.
UPDATE t ;
SET ;
t.field1 = s.field1, ;
t.field2 = s.field2, ;
t.field3 = s.field3 ;
FROM ;
target t ;
inner join source s ;
on t.pk = s.pk ;
WHERE ;
t.field1 # s.field1 ;
OR t.field2 # s.field2 ;
OR t.field3 # s.field3
INSERT INTO ;
target ;
(;
field1,;
field2,;
field3) ;
SELECT ;
field1,;
field2,;
field3 ;
FROM ;
source s ;
WHERE ;
s.pk NOT IN ;
(SELECT pk FROM target)
This works very well in VFP, because I don't update any indexes doing
the initial import. Then I only update few records (and their indexes)
during the update and append processes. If you could do those commands
directly with the csv files, that might be pretty good.
Mike Yearwood
_______________________________________________
Post Messages to: [email protected]
Subscription Maintenance: http://leafe.com/mailman/listinfo/profox
OT-free version of this list: http://leafe.com/mailman/listinfo/profoxtech
Searchable Archive: http://leafe.com/archives/search/profox
This message: http://leafe.com/archives/byMID/profox/[EMAIL PROTECTED]
** All postings, unless explicitly stated otherwise, are the opinions of the
author, and do not constitute legal or medical advice. This statement is added
to the messages for those lawyers who are too stupid to see the obvious.
