Le mardi 26 février 2008, Joshua D. Drake a écrit :
> > Think 100GB+ of data that's in a CSV or delimited file.  Right now
> > the best import path is with COPY, but it won't execute very fast as
> > a single process.  Splitting the file manually will take a long time
> > (time that could be spend loading instead) and substantially increase
> > disk usage, so the ideal approach would figure out how to load in
> > parallel across all available CPUs against that single file.
>
> You mean load from position? That would be very, very cool.

Did I mention pgloader now does exactly this when configured like this:
 http://pgloader.projects.postgresql.org/dev/pgloader.1.html#_parallel_loading
  section_threads = N
  split_file_reading = True

IIRC, Simon and Greg Smith asked for pgloader to get those parallel loading 
features in order to have some first results and ideas about the performance 
gain, as a first step in the parallel COPY backend implementation design.

Hope this helps,
-- 
dim

Attachment: signature.asc
Description: This is a digitally signed message part.

Reply via email to