[OPEN-ILS-GENERAL] ***SPAM*** Importing bib and holdings data into Evergreen 2.4.2

[Linda Jansova]

Dear all,

We have attempted to import bibliographic and holdings data into 
Evergreen 2.4.2 using instructions from chapter 14 
(http://docs.evergreen-ils.org/2.4/_migrating_from_a_legacy_system.html) 
of the official Evergreen documentation.

First, we were unsuccessful in using the create function (step 3):

***

CREATE OR REPLACE FUNCTION staging_importer() RETURNS NULL AS $$
DECLARE stage RECORD;
BEGIN
FOR stage IN SELECT * FROM staging_records_import ORDER BY id LOOP
      INSERT INTO biblio.record_entry (marc, last_xact_id) VALUES (stage.marc, 
'IMPORT');
      UPDATE staging_records_import SET dest = 
currval('biblio.record_entry_id_seq');
   END LOOP;
  END;
  $$ LANGUAGE plpgsql;


***

However, it worked okay when we substituted NULL for VOID as recommended at 
http://comments.gmane.org/gmane.education.libraries.open-ils.general/7820.

Then it took well over 12 hours to import 9460 records. Yet, the import 
was unsuccessful:

***
evergreen=# CREATE TABLE staging_records_import (id BIGSERIAL, dest 
BIGINT, marc TEXT);

evergreen=# CREATE OR REPLACE FUNCTION staging_importer() RETURNS VOID 
AS $$
DECLARE stage RECORD;
BEGIN
FOR stage IN SELECT * FROM staging_records_import ORDER BY id LOOP
      INSERT INTO biblio.record_entry (marc, last_xact_id) VALUES 
(stage.marc, 'IMPORT');
      UPDATE staging_records_import SET dest = 
currval('biblio.record_entry_id_seq');
   END LOOP;
  END;
  $$ LANGUAGE plpgsql;
CREATE FUNCTION


evergreen=# COPY staging_records_import (marc) FROM 
'/home/opensrf/KKB/IMPORT.xml';
COPY 9460


evergreen=# SELECT staging_importer();
ERROR:  could not extend file "base/42287/42883": No space left on device
HINT:  Check free disk space.
CONTEXT:  SQL statement "INSERT INTO metabib.real_full_rec (record, tag, 
ind1, ind2, subfield, value)
        SELECT record, tag, ind1, ind2, subfield, value FROM 
biblio.flatten_marc( bib_id )"
PL/pgSQL function "reingest_metabib_full_rec" line 7 at SQL statement
SQL statement "SELECT metabib.reingest_metabib_full_rec(NEW.id)"
PL/pgSQL function "indexing_ingest_or_delete" line 46 at PERFORM
SQL statement "INSERT INTO biblio.record_entry (marc, last_xact_id) 
VALUES (stage.marc, 'IMPORT')"
PL/pgSQL function "staging_importer" line 5 at SQL statement

***

The statistics of the virtual machine load have showed that the in the 
first hour disk I/O write operation (at about 30 MBps) prevailed, while 
in the rest of the time disk I/O read operations (at about 70 MBps) were 
prevalent.

When we tried to use the import method recommended for previous versions 
(especially version 2.1: 
http://docs.evergreen-ils.org/2.1/html/migrating_records_using_migration_tools.html), 
the import of bibliographic data was successful (and it took only about 
an hour overall). However, when we subsequently wanted to import 
holdings data, the system was not able to fill the column entitled egid 
and therefore we could not import holdings data.

Any ideas how to sort this out? Or which import methods to use for 
Evergreen 2.4.2?

Thanks in advance for any hints!

Linda and Vaclav Jansovi :-)