Hi Chris,
i finished today morning new testing server hosted on my home
server/desktop/router/virthost...
New testing virtual server has this configuration 4GB RAM, 2core, 50GB HDD
64 bit Debian 7, EG 2.4.2
Import currently running over seven hours. 6k5 bib records (with minimal
record complexity) .. DB now consume 6GB HDD space.
Our production server 32bit Debian 6 EG 2.2.2 consume for DB 0.9GB HDD
space (35k bib records, 2 years of history).
Unfortunately i have no information about disk usage from migration to
2.4.2 with use of old migration steps from ESI migration... 6.5k records
has imported to virtual machine hosted on my laptop under 1 hour.
How much clean or dirty is usage of old ESI scripts and importing steps
on new EG (2.4.2) for bib records.
Thanks for advance
Vaclav
On 10/15/2013 01:23 PM, Sharp, Chris wrote:
Hi Vaclav,
6.6 GB sounds a bit tight to me and it appears that it's not enough to
accommodate the import Linda is attempting. Any chance at increasing that
size? I think it will be necessary based on the error Linda shared in her post.
Hope that's helpful,
Chris
----- Original Message -----
From: "Václav Jansa" <[email protected]>
To: [email protected]
Sent: Tuesday, October 15, 2013 6:16:27 AM
Subject: [OPEN-ILS-GENERAL] ***SPAM*** Re: ***SPAM*** Importing bib and
holdings data into Evergreen 2.4.2
Hello Chris,
i reply as a IT part of our Green home team;-)
Its clean EG 2.4.2 installation on 64bit debian-wheezy, virtual
machine
use 10GB image
free space is
opensrf@evergree-nb:~/KKB$ df -h
Filesystem Size Used
Avail
Use% Mounted on
rootfs 9.6G 2.5G
6.6G
28% /
udev 10M 0 10M
0% /dev
tmpfs 202M 216K
202M 1% /run
/dev/disk/by-uuid/e1cc81ee-2958-4d5d-941f-eab6798efe35 9.6G 2.5G
6.6G
28% /
tmpfs 5.0M 0 5.0M
0% /run/lock
tmpfs 497M 0 497M
0% /run/shm
Old migration steps (from ESI migration tools) work for bibliographic
records almost fine. Cca 50 mins for full import (from start SQL
injection to searching in staff OPAC).
It's my second migration to EG (previous in EG 2.0 times).
Thanks for advance
Vaclav Jansa
On 10/15/2013 12:00 PM, Sharp, Chris wrote:
Hi Linda,
This appears to be the problem:
ERROR: could not extend file "base/42287/42883": No space left on
device
HINT: Check free disk space.
You can verify disk space with the bash command 'df -h', but it
looks like you need to increase your disk space before continuing.
Note that this is a PostgreSQL-level issue and is not specific to
Evergreen.
Hope that's helpful!
Chris
----- Original Message -----
From: "Linda Jansova" <[email protected]>
To: "Evergreen Discussion Group"
<[email protected]>
Sent: Tuesday, October 15, 2013 4:33:28 AM
Subject: [OPEN-ILS-GENERAL] ***SPAM*** Importing bib and holdings
data into Evergreen 2.4.2
Dear all,
We have attempted to import bibliographic and holdings data into
Evergreen 2.4.2 using instructions from chapter 14 (
http://docs.evergreen-ils.org/2.4/_migrating_from_a_legacy_system.html
) of the official Evergreen documentation.
First, we were unsuccessful in using the create function (step 3):
***
CREATE OR REPLACE FUNCTION staging_importer() RETURNS NULL AS $$
DECLARE stage RECORD;
BEGIN
FOR stage IN SELECT * FROM staging_records_import ORDER BY id LOOP
INSERT INTO biblio.record_entry (marc, last_xact_id) VALUES
(stage.marc, 'IMPORT');
UPDATE staging_records_import SET dest =
currval('biblio.record_entry_id_seq');
END LOOP;
END;
$$ LANGUAGE plpgsql;
***
However, it worked okay when we substituted NULL for VOID as
recommended at
http://comments.gmane.org/gmane.education.libraries.open-ils.general/7820
. Then it took well over 12 hours to import 9460 records. Yet, the
import was unsuccessful:
***
evergreen=# CREATE TABLE staging_records_import (id BIGSERIAL,
dest
BIGINT, marc TEXT);
evergreen=# CREATE OR REPLACE FUNCTION staging_importer() RETURNS
VOID AS $$
DECLARE stage RECORD;
BEGIN
FOR stage IN SELECT * FROM staging_records_import ORDER BY id LOOP
INSERT INTO biblio.record_entry (marc, last_xact_id) VALUES
(stage.marc, 'IMPORT');
UPDATE staging_records_import SET dest =
currval('biblio.record_entry_id_seq');
END LOOP;
END;
$$ LANGUAGE plpgsql;
CREATE FUNCTION
evergreen=# COPY staging_records_import (marc) FROM
'/home/opensrf/KKB/IMPORT.xml';
COPY 9460
evergreen=# SELECT staging_importer();
ERROR: could not extend file "base/42287/42883": No space left on
device
HINT: Check free disk space.
CONTEXT: SQL statement "INSERT INTO metabib.real_full_rec (record,
tag, ind1, ind2, subfield, value)
SELECT record, tag, ind1, ind2, subfield, value FROM
biblio.flatten_marc( bib_id )"
PL/pgSQL function "reingest_metabib_full_rec" line 7 at SQL
statement
SQL statement "SELECT metabib.reingest_metabib_full_rec(NEW.id)"
PL/pgSQL function "indexing_ingest_or_delete" line 46 at PERFORM
SQL statement "INSERT INTO biblio.record_entry (marc,
last_xact_id)
VALUES (stage.marc, 'IMPORT')"
PL/pgSQL function "staging_importer" line 5 at SQL statement
***
The statistics of the virtual machine load have showed that the in
the first hour disk I/O write operation (at about 30 MBps)
prevailed, while in the rest of the time disk I/O read operations
(at about 70 MBps) were prevalent.
When we tried to use the import method recommended for previous
versions (especially version 2.1:
http://docs.evergreen-ils.org/2.1/html/migrating_records_using_migration_tools.html
), the import of bibliographic data was successful (and it took
only
about an hour overall). However, when we subsequently wanted to
import holdings data, the system was not able to fill the column
entitled egid and therefore we could not import holdings data.
Any ideas how to sort this out? Or which import methods to use for
Evergreen 2.4.2?
Thanks in advance for any hints!
Linda and Vaclav Jansovi :-)
