Re: [OPEN-ILS-DEV] Importing marc records from Sirsi
Ok something weird is going on with this import. I started it on Friday and it was still running on Monday. I came in this morning and it was finished but with an error. Can't locate object method class_name via package HU (perhaps you forgot to load HU?) at pg_loader.pl line 48, line 1648134328. Once I saw that then I decided to try a smaller number of imports at once, that one was 3500. So I had a file with 1000 records in it to try. It started off really slow. I stopped it and tried to restart all of the Evergreen services. I was informed by the services that there wasn't enough free space to start up. So I started checking around. I found out that the osrfsys.log file was almost 120GB in size! It had all of the entries in there from the previous import. So I erased the log file. It automatically created another osrfsys.log file and started putting entries in it, again from the last import. I finally had to kill the perl process so that I could completely erase the log. Now that I have it erased I can't get the router user to connect to the jabber server. Can someone give me some insight as to why this has happened and what I might be doing wrong to cause it to happen? On Mon, Jun 2, 2008 at 12:22 PM, Dan Scott [EMAIL PROTECTED] wrote: 2008/6/2 Robert [EMAIL PROTECTED]: Hey guys, any news on why the copies or volumes might not have copied over? Also, can someone tell me in their experience in importing records what the maximum they imported at once? I tried to import a file that had 3500 records in it over the weekend and it is still running and looks to be hung up. Just out of curiosity. 1) The steps listed for the Gutenberg records get bibliographic records into the system, but no call numbers or copies. That's what the import_demo package tries to demonstrate: http://svn.open-ils.org/trac/ILS-Contrib/wiki/ImportDemo The approach in the import_demo takes you through the steps for getting bib records into the system, then goes beyond that to parse holdings statements directly from the MARC21XML for the bib records and generates call numbers and copies to load into the system. This isn't necessarily the best approach for getting call numbers and copies into your system, but you're going to have to tailor your approach to the system you're working with. 2) The most bib records I have imported in a single file is somewhere around 2 million. This weekend I was importing approximately 360,000 bib records from a single file. Note that you really want to be using the parallel_pg_loader.pl approach (as demonstrated in import_demo) if you're working on a system with memory constraints. -- Dan Scott Laurentian University
Re: [OPEN-ILS-DEV] Importing marc records from Sirsi
Hey guys, any news on why the copies or volumes might not have copied over? Also, can someone tell me in their experience in importing records what the maximum they imported at once? I tried to import a file that had 3500 records in it over the weekend and it is still running and looks to be hung up. Just out of curiosity. On Fri, May 30, 2008 at 1:44 PM, Robert [EMAIL PROTECTED] wrote: Ok that worked for the records themselves but it didn't copy over any volumes or copies or current holds on the items. Is there any way to copy all of this information over at the same time? On Fri, May 30, 2008 at 1:10 PM, Robert [EMAIL PROTECTED] wrote: Oh ok, it's in the /root/Evergreen-ILS-*/Evergreen/src/extras/import/ directory. Thanks for clarifying that Dan. On Fri, May 30, 2008 at 11:58 AM, Dan Scott [EMAIL PROTECTED] wrote: 2008/5/30 Robert [EMAIL PROTECTED]: I exited postgres and went back in and it worked. The import also worked after I exited and went back in. I guess something just got hung up in postgres somewhere. Another quick question though. On step 5 in the Gutenberg example, I don't seem to have the quick_metarecord_map.sql file. Is that something that should be installed initially or something that I just forgot to download? Also I noticed the path in #5 doesn't match where the commands for importing are on the server. The path that it suggests is Evergreen/src/extras/import/quick_metarecord_map.sql the actual path on my server is /root/Evergreen-*/Open-ILS-*/src/extras/import/ but that file isn't in either one of them. Any suggestions on where I can get that file? Robert: I'll add the quick_metarecord_map.sql instruction to the Gutenberg instructions - thanks for pointing out that omission. And the file really does exist in Evergreen/src/extras/import/quick_metarecord_map.sql. When you extract the Evergreen source, two subdirectories are created: Evergreen and OpenILS. Try running: find /root -name quick_metarecord_map.sql -- Dan Scott Laurentian University
Re: [OPEN-ILS-DEV] Importing marc records from Sirsi
I exited postgres and went back in and it worked. The import also worked after I exited and went back in. I guess something just got hung up in postgres somewhere. Another quick question though. On step 5 in the Gutenberg example, I don't seem to have the quick_metarecord_map.sql file. Is that something that should be installed initially or something that I just forgot to download? Also I noticed the path in #5 doesn't match where the commands for importing are on the server. The path that it suggests is Evergreen/src/extras/import/quick_metarecord_map.sql the actual path on my server is /root/Evergreen-*/Open-ILS-*/src/extras/import/ but that file isn't in either one of them. Any suggestions on where I can get that file? On Fri, May 30, 2008 at 8:35 AM, Robert [EMAIL PROTECTED] wrote: I tried that comman and I received this: ERROR: current transaction is aborted, commands ignored until end of transaction block. On Thu, May 29, 2008 at 1:18 PM, Jason Etheridge [EMAIL PROTECTED] wrote: On Thu, May 29, 2008 at 9:18 AM, Robert [EMAIL PROTECTED] wrote: I do already have some records in the system and I did not use the startid option the first time. I did however try the entire process again using the startid option and received the same results. I have 8 different books in the system but some have multiple copies. Should the startid option be the number of actual books or records (I believe this is the right way to go) or should it start with the number after the number of copies? Robert, in psql run this query: select max(id) from biblio.record_entry; Use a higher number than the output from that (I would add 1000). Let us know if this helps. -- Jason Etheridge | VP, Community Support and Advocacy | Equinox Software, Inc. / The Evergreen Experts | phone: 1-877-OPEN-ILS (673-6457) | email: [EMAIL PROTECTED] | web: http://www.esilibrary.com
Re: [OPEN-ILS-DEV] Importing marc records from Sirsi
Ok that worked for the records themselves but it didn't copy over any volumes or copies or current holds on the items. Is there any way to copy all of this information over at the same time? On Fri, May 30, 2008 at 1:10 PM, Robert [EMAIL PROTECTED] wrote: Oh ok, it's in the /root/Evergreen-ILS-*/Evergreen/src/extras/import/ directory. Thanks for clarifying that Dan. On Fri, May 30, 2008 at 11:58 AM, Dan Scott [EMAIL PROTECTED] wrote: 2008/5/30 Robert [EMAIL PROTECTED]: I exited postgres and went back in and it worked. The import also worked after I exited and went back in. I guess something just got hung up in postgres somewhere. Another quick question though. On step 5 in the Gutenberg example, I don't seem to have the quick_metarecord_map.sql file. Is that something that should be installed initially or something that I just forgot to download? Also I noticed the path in #5 doesn't match where the commands for importing are on the server. The path that it suggests is Evergreen/src/extras/import/quick_metarecord_map.sql the actual path on my server is /root/Evergreen-*/Open-ILS-*/src/extras/import/ but that file isn't in either one of them. Any suggestions on where I can get that file? Robert: I'll add the quick_metarecord_map.sql instruction to the Gutenberg instructions - thanks for pointing out that omission. And the file really does exist in Evergreen/src/extras/import/quick_metarecord_map.sql. When you extract the Evergreen source, two subdirectories are created: Evergreen and OpenILS. Try running: find /root -name quick_metarecord_map.sql -- Dan Scott Laurentian University
Re: [OPEN-ILS-DEV] Importing marc records from Sirsi
I do already have some records in the system and I did not use the startid option the first time. I did however try the entire process again using the startid option and received the same results. I have 8 different books in the system but some have multiple copies. Should the startid option be the number of actual books or records (I believe this is the right way to go) or should it start with the number after the number of copies? On Wed, May 28, 2008 at 4:47 PM, Dan Scott [EMAIL PROTECTED] wrote: Robert: 2008/5/28 Robert [EMAIL PROTECTED]: Dan, I did the gutenberg approach this time, sorry didn't notice it near the bottom last time, and I received a .sql file that actually had data in it. I ran the command \i path and I received a lot of errors. Those errors are: evergreen=# \i /root/Desktop/sirsiimport/V001.sql SET BEGIN psql:/root/Desktop/sirsiimport/V001.sql:5: ERROR: duplicate key violates unique constraint record_entry_pkey Do you already have some records in the system? If so, you'll need to change the start ID for the import (--startid option for marc2bre.pl) to avoid generating records that conflict with your existing records. Note that the trunk version of marc2bre.pl automatically sets the start ID to something sane (as long as nobody adds new records to the system before you load your newly processed records). -- Dan Scott Laurentian University
Re: [OPEN-ILS-DEV] Importing marc records from Sirsi
On Thu, May 29, 2008 at 9:18 AM, Robert [EMAIL PROTECTED] wrote: I do already have some records in the system and I did not use the startid option the first time. I did however try the entire process again using the startid option and received the same results. I have 8 different books in the system but some have multiple copies. Should the startid option be the number of actual books or records (I believe this is the right way to go) or should it start with the number after the number of copies? Robert, in psql run this query: select max(id) from biblio.record_entry; Use a higher number than the output from that (I would add 1000). Let us know if this helps. -- Jason Etheridge | VP, Community Support and Advocacy | Equinox Software, Inc. / The Evergreen Experts | phone: 1-877-OPEN-ILS (673-6457) | email: [EMAIL PROTECTED] | web: http://www.esilibrary.com
[OPEN-ILS-DEV] Importing marc records from Sirsi
Now that I have my permission problems out of the way, thanks to Dan, Jason and Mike, I can finally start on something that has been hindering the progression of our testing. I have some records that we exported from Sirsi and looking at the documentation about importing, I was able to run all of the commands to convert the records into the JSON records that Evergreen uses. Unfortunately when I came to the step to create the SQL statements to import the records the only SQL statements that I get are: SET CLIENT_ENCODING TO 'UNICODE'; BEGIN; COMMIT; That is all that is in the sql file. It doesn't import any of the records, obviously. I was wondering if anyone that has successfully imported records from Sirsi could give me some insight on what to do to generate the SQL statements needed to import the records. I am including a copy of the file that I am trying to convert using the commands on the import section in the documentation. The file itself may not be in the correct format orignally to convert it correctly. I am not familiar with all of the formats and could use some insight on that as well. I tried to convert it as if it were in UNICODE format, if that helps. Any help on this subject is greatly appreciated. Again my Evergreen installation is 1.2.1.4 on Debian Etch using 1.2.1.4client. Thank you, Robert V001 Description: Binary data
Re: [OPEN-ILS-DEV] Importing MARC records
2008/5/5 Tom Patterson [EMAIL PROTECTED]: I have a new Evergreen setup and need to import records. I have looked at the information located here: http://open-ils.org/dokuwiki/doku.php?id=evergreen-admin:importing:bibrecords, downloaded the sample scripts and reviewed the README file. My question is whether proceeding along this path is the easiest way to import these records and are there any caveats? Hi Tom: For just a handful of records you could use the Z39.50 import from the client to copy catalogue records - but otherwise, the page sample scripts you pointed to are really the only method I know of importing a large batch of records. So all I can offer are some caveats. If you have a large number of bib records to import, you can just pipe the output of marc2bre.pl | direct_ingest.pl | parallel_pg_loader.pl to speed up the process. As for processing the holdings inside the MARCXML records, if you can separate out the holdings into their own file with a consistent identifying key for bib record - holding, then that's probably a better approach in the long run. You also don't want to have your holdings sitting inside your bib records in the database once they're imported. It will inflate the size of your database with no positive benefits. If your exported bibs do contain the holdings in, say, a 949 field, you can use the --trash parameter in marc2bre.pl to delete that field before it gets imported into the database. -- Dan Scott Laurentian University
