Re: [OPEN-ILS-GENERAL] How to have multiple CPU's used, importing a million bib records in EG 2.0?

2011-05-03 Thread Dan Scott
On Tue, May 3, 2011 at 1:39 PM, Repke de Vries wrote: > Hi Dan > > eight piped processes are running fine right now - as per your suggestion. > > However: on tying together with Unix pipe: > >> So there's no delay when you pipe the >> commands; the import into the database begins immediately. > >

Re: [OPEN-ILS-GENERAL] How to have multiple CPU's used, importing a million bib records in EG 2.0?

2011-05-03 Thread Repke de Vries
Hi Dan eight piped processes are running fine right now - as per your suggestion. However: on tying together with Unix pipe: > So there's no delay when you pipe the > commands; the import into the database begins immediately. the evidence is somewhat circumstantial but our observation is that

Re: [OPEN-ILS-GENERAL] How to have multiple CPU's used, importing a million bib records in EG 2.0?

2011-05-02 Thread Dan Scott
Hi Repke: On Mon, May 2, 2011 at 3:40 PM, Repke de Vries wrote: > Hi, > > though we worked out our combination of calling marc2bre followed by > pg_loader followed by psql as successive, separate steps  - we are dead in > the water 'cause PosgreSQL alone (last step) takes four hours for 100K >

[OPEN-ILS-GENERAL] How to have multiple CPU's used, importing a million bib records in EG 2.0?

2011-05-02 Thread Repke de Vries
Hi, though we worked out our combination of calling marc2bre followed by pg_loader followed by psql as successive, separate steps - we are dead in the water 'cause PosgreSQL alone (last step) takes four hours for 100K records: meaning 40 hours for all of our one million bib records. Also does