Re: [OPEN-ILS-GENERAL] Evergreen 2.9.3 - 2.10.3 upgrade issues
Hi, The LP Bug is available at https://bugs.launchpad.net/evergreen/+bug/1588543 Kathy On 06/13/2016 01:14 PM, Jason Stephenson wrote: On 06/13/2016 01:08 PM, Jesse McCarty wrote: The interesting part of this process is the two-step process in the 2.9.3-2.10.0 upgrade script where there is intervention needed to exit one screen and continue, and then the fact that part of the script takes all night and then some to finish (The server is a VM with 8 VCPUs, 18GB RAM and resides on SSD storage so it's no slouch). Even on slower hardware of previous upgrades, the records reingest process only took around 4 hours. Yes, that's a "known" issue and Dan Wells has a potential fix for that. http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/dbwells/speed_up_rec_attr_ingest I do not think anyone has made a Launchpad bug for that, yet, however. -- Kathy Lussier Project Coordinator Massachusetts Library Network Cooperative (508) 343-0128 kluss...@masslnc.org Twitter: http://www.twitter.com/kmlussier
Re: [OPEN-ILS-GENERAL] Evergreen 2.9.3 - 2.10.3 upgrade issues
On 06/13/2016 01:08 PM, Jesse McCarty wrote: The interesting part of this process is the two-step process in the 2.9.3-2.10.0 upgrade script where there is intervention needed to exit one screen and continue, and then the fact that part of the script takes all night and then some to finish (The server is a VM with 8 VCPUs, 18GB RAM and resides on SSD storage so it's no slouch). Even on slower hardware of previous upgrades, the records reingest process only took around 4 hours. Yes, that's a "known" issue and Dan Wells has a potential fix for that. http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/dbwells/speed_up_rec_attr_ingest I do not think anyone has made a Launchpad bug for that, yet, however. -- Jason Stephenson Assistant Director for Technology Services Merrimack Valley Library Consortium 4 High ST, Suite 175 North Andover, MA 01845 Phone: 978-557-5891 Email: jstephen...@mvlcstaff.org Please note my new email address.
Re: [OPEN-ILS-GENERAL] Evergreen 2.9.3 - 2.10.3 upgrade issues
Thanks for the information Ben, The server in the old conversation (thanks for link!) was a 2.4.4 test server that was eventually scrapped and I did not perform the upgrade from 2.4.4. Using the old thread, as well as Galen's information, I was able to apply the fix and then the DB upgrade scripts functioned properly and I now have an upgraded and functional test server. Thank you all again! The interesting part of this process is the two-step process in the 2.9.3-2.10.0 upgrade script where there is intervention needed to exit one screen and continue, and then the fact that part of the script takes all night and then some to finish (The server is a VM with 8 VCPUs, 18GB RAM and resides on SSD storage so it's no slouch). Even on slower hardware of previous upgrades, the records reingest process only took around 4 hours. Thanks again everyone! Jesse McCarty City of Burlington IT Technical Assistant -Original Message- From: Open-ils-general [mailto:open-ils-general-boun...@list.georgialibraries.org] On Behalf Of Ben Shum Sent: Thursday, June 02, 2016 4:57 PM To: Evergreen Discussion Group Subject: Re: [OPEN-ILS-GENERAL] Evergreen 2.9.3 - 2.10.3 upgrade issues Hi Jesse, That error for missing metabib_field key 30 means that you do not have an entry in the table with ID of 30 (which is supposed to be LCCN in the stock installation). Looking back in the history, I see you've mentioned some issues with this metabib_field entry before (see http://markmail.org/message/wrdyhmaeenavxk2y) Perhaps it would be a good idea to audit your table's entries and compare it with a stock setup to see the differences (and make adjustments). I would start by checking to see if LCCN is already in the metabib_field table. Something like: SELECT id FROM config.metabib_field WHERE name = 'lccn'; If that gives you the ID for your missing LCCN field, that'll tell you what it is, instead of 30. -- Ben On Thu, Jun 2, 2016 at 7:18 PM, Jesse McCarty wrote: > Thank you Galen, > > Looks like I have something else missing, running the > 0795.schema.z39-batch-fetch-overlay.sql script came up with another error and > did a ROLLBACK: > > ERROR: insert or update on table "z3950_index_field_map" violates foreign > key constraint "z3950_index_field_map_metabib_field_fkey" DETAIL: Key > (metabib_field)=(30) is not present in table "metabib_field". > > Jesse McCarty > City of Burlington > IT Technical Assistant > > > -Original Message- > From: Open-ils-general > [mailto:open-ils-general-boun...@list.georgialibraries.org] On Behalf > Of Galen Charlton > Sent: Thursday, June 02, 2016 10:03 AM > To: Evergreen Discussion Group > > Subject: Re: [OPEN-ILS-GENERAL] Evergreen 2.9.3 - 2.10.3 upgrade > issues > > Hi, > > On Thu, Jun 2, 2016 at 12:28 PM, Jesse McCarty > wrote: >> Could simply running the 2.4.3-2.5.0-upgrade-db.sql prior to the >> current DB scripts work, or would only certain portions of that file >> need to be run? My test server is a VM with snapshots, so reverting >> and re-testing anything is trivial. > > No, a subset should suffice. Specifically, the following steps before > running the main upgrade SQL: > > [1] running the SQL in upgrade/0795.schema.z39-batch-fetch-overlay.sql > (taken from the 2.10.x tarball) > > For example: > > psql> \set eg_version NULL > psql> \i upgrade/0795.schema.z39-batch-fetch-overlay.sql > > [2] following up with just this bit from the 0843 upgrade: > > ALTER TABLE config.z3950_index_field_map DROP CONSTRAINT > z3950_index_field_map_metabib_field_fkey; > ALTER TABLE config.z3950_index_field_map ADD CONSTRAINT > z3950_index_field_map_metabib_field_fkey FOREIGN KEY (metabib_field) > REFERENCES config.metabib_field(id) ON UPDATE CASCADE DEFERRABLE > INITIALLY DEFERRED; > > Apropos of a conversation during the development meeting yesterday, DB > revisions 0841/0842/0843 are a good example of where backporting schema > updates can be tricky. > > Regards, > > Galen > -- > Galen Charlton > Infrastructure and Added Services Manager Equinox Software, Inc. / > Open Your Library > email: g...@esilibrary.com > direct: +1 770-709-5581 > cell: +1 404-984-4366 > skype: gmcharlt > web:http://www.esilibrary.com/ > Supporting Koha and Evergreen: http://koha-community.org & > http://evergreen-ils.org
Re: [OPEN-ILS-GENERAL] Evergreen 2.9.3 - 2.10.3 upgrade issues
Hi Jesse, That error for missing metabib_field key 30 means that you do not have an entry in the table with ID of 30 (which is supposed to be LCCN in the stock installation). Looking back in the history, I see you've mentioned some issues with this metabib_field entry before (see http://markmail.org/message/wrdyhmaeenavxk2y) Perhaps it would be a good idea to audit your table's entries and compare it with a stock setup to see the differences (and make adjustments). I would start by checking to see if LCCN is already in the metabib_field table. Something like: SELECT id FROM config.metabib_field WHERE name = 'lccn'; If that gives you the ID for your missing LCCN field, that'll tell you what it is, instead of 30. -- Ben On Thu, Jun 2, 2016 at 7:18 PM, Jesse McCarty wrote: > Thank you Galen, > > Looks like I have something else missing, running the > 0795.schema.z39-batch-fetch-overlay.sql script came up with another error and > did a ROLLBACK: > > ERROR: insert or update on table "z3950_index_field_map" violates foreign > key constraint "z3950_index_field_map_metabib_field_fkey" DETAIL: Key > (metabib_field)=(30) is not present in table "metabib_field". > > Jesse McCarty > City of Burlington > IT Technical Assistant > > > -Original Message- > From: Open-ils-general > [mailto:open-ils-general-boun...@list.georgialibraries.org] On Behalf Of > Galen Charlton > Sent: Thursday, June 02, 2016 10:03 AM > To: Evergreen Discussion Group > Subject: Re: [OPEN-ILS-GENERAL] Evergreen 2.9.3 - 2.10.3 upgrade issues > > Hi, > > On Thu, Jun 2, 2016 at 12:28 PM, Jesse McCarty > wrote: >> Could simply running the 2.4.3-2.5.0-upgrade-db.sql prior to the >> current DB scripts work, or would only certain portions of that file >> need to be run? My test server is a VM with snapshots, so reverting >> and re-testing anything is trivial. > > No, a subset should suffice. Specifically, the following steps before > running the main upgrade SQL: > > [1] running the SQL in upgrade/0795.schema.z39-batch-fetch-overlay.sql > (taken from the 2.10.x tarball) > > For example: > > psql> \set eg_version NULL > psql> \i upgrade/0795.schema.z39-batch-fetch-overlay.sql > > [2] following up with just this bit from the 0843 upgrade: > > ALTER TABLE config.z3950_index_field_map DROP CONSTRAINT > z3950_index_field_map_metabib_field_fkey; > ALTER TABLE config.z3950_index_field_map ADD CONSTRAINT > z3950_index_field_map_metabib_field_fkey FOREIGN KEY (metabib_field) > REFERENCES config.metabib_field(id) ON UPDATE CASCADE DEFERRABLE INITIALLY > DEFERRED; > > Apropos of a conversation during the development meeting yesterday, DB > revisions 0841/0842/0843 are a good example of where backporting schema > updates can be tricky. > > Regards, > > Galen > -- > Galen Charlton > Infrastructure and Added Services Manager Equinox Software, Inc. / Open Your > Library > email: g...@esilibrary.com > direct: +1 770-709-5581 > cell: +1 404-984-4366 > skype: gmcharlt > web:http://www.esilibrary.com/ > Supporting Koha and Evergreen: http://koha-community.org & > http://evergreen-ils.org
Re: [OPEN-ILS-GENERAL] Evergreen 2.9.3 - 2.10.3 upgrade issues
Thank you Galen, Looks like I have something else missing, running the 0795.schema.z39-batch-fetch-overlay.sql script came up with another error and did a ROLLBACK: ERROR: insert or update on table "z3950_index_field_map" violates foreign key constraint "z3950_index_field_map_metabib_field_fkey" DETAIL: Key (metabib_field)=(30) is not present in table "metabib_field". Jesse McCarty City of Burlington IT Technical Assistant -Original Message- From: Open-ils-general [mailto:open-ils-general-boun...@list.georgialibraries.org] On Behalf Of Galen Charlton Sent: Thursday, June 02, 2016 10:03 AM To: Evergreen Discussion Group Subject: Re: [OPEN-ILS-GENERAL] Evergreen 2.9.3 - 2.10.3 upgrade issues Hi, On Thu, Jun 2, 2016 at 12:28 PM, Jesse McCarty wrote: > Could simply running the 2.4.3-2.5.0-upgrade-db.sql prior to the > current DB scripts work, or would only certain portions of that file > need to be run? My test server is a VM with snapshots, so reverting > and re-testing anything is trivial. No, a subset should suffice. Specifically, the following steps before running the main upgrade SQL: [1] running the SQL in upgrade/0795.schema.z39-batch-fetch-overlay.sql (taken from the 2.10.x tarball) For example: psql> \set eg_version NULL psql> \i upgrade/0795.schema.z39-batch-fetch-overlay.sql [2] following up with just this bit from the 0843 upgrade: ALTER TABLE config.z3950_index_field_map DROP CONSTRAINT z3950_index_field_map_metabib_field_fkey; ALTER TABLE config.z3950_index_field_map ADD CONSTRAINT z3950_index_field_map_metabib_field_fkey FOREIGN KEY (metabib_field) REFERENCES config.metabib_field(id) ON UPDATE CASCADE DEFERRABLE INITIALLY DEFERRED; Apropos of a conversation during the development meeting yesterday, DB revisions 0841/0842/0843 are a good example of where backporting schema updates can be tricky. Regards, Galen -- Galen Charlton Infrastructure and Added Services Manager Equinox Software, Inc. / Open Your Library email: g...@esilibrary.com direct: +1 770-709-5581 cell: +1 404-984-4366 skype: gmcharlt web:http://www.esilibrary.com/ Supporting Koha and Evergreen: http://koha-community.org & http://evergreen-ils.org
Re: [OPEN-ILS-GENERAL] Evergreen 2.9.3 - 2.10.3 upgrade issues
Hi, On Thu, Jun 2, 2016 at 12:28 PM, Jesse McCarty wrote: > Could simply running the 2.4.3-2.5.0-upgrade-db.sql prior to the current DB > scripts work, or would only certain portions of that file need to be run? My > test server is a VM with snapshots, so reverting and re-testing anything is > trivial. No, a subset should suffice. Specifically, the following steps before running the main upgrade SQL: [1] running the SQL in upgrade/0795.schema.z39-batch-fetch-overlay.sql (taken from the 2.10.x tarball) For example: psql> \set eg_version NULL psql> \i upgrade/0795.schema.z39-batch-fetch-overlay.sql [2] following up with just this bit from the 0843 upgrade: ALTER TABLE config.z3950_index_field_map DROP CONSTRAINT z3950_index_field_map_metabib_field_fkey; ALTER TABLE config.z3950_index_field_map ADD CONSTRAINT z3950_index_field_map_metabib_field_fkey FOREIGN KEY (metabib_field) REFERENCES config.metabib_field(id) ON UPDATE CASCADE DEFERRABLE INITIALLY DEFERRED; Apropos of a conversation during the development meeting yesterday, DB revisions 0841/0842/0843 are a good example of where backporting schema updates can be tricky. Regards, Galen -- Galen Charlton Infrastructure and Added Services Manager Equinox Software, Inc. / Open Your Library email: g...@esilibrary.com direct: +1 770-709-5581 cell: +1 404-984-4366 skype: gmcharlt web:http://www.esilibrary.com/ Supporting Koha and Evergreen: http://koha-community.org & http://evergreen-ils.org
Re: [OPEN-ILS-GENERAL] Evergreen 2.9.3 - 2.10.3 upgrade issues
Thanks for the information Josh. We had been running 2.4.4 in the past and skipped over the 2.5 series, upgrading directly to 2.6.3 from 2.4.4 (Our support company performed this upgrade). From 2.6.3 we went to 2.7.3, 2.8.3 and now we are running 2.9.3. Could simply running the 2.4.3-2.5.0-upgrade-db.sql prior to the current DB scripts work, or would only certain portions of that file need to be run? My test server is a VM with snapshots, so reverting and re-testing anything is trivial. Thanks again. Jesse McCarty City of Burlington IT Technical Assistant From: Open-ils-general [mailto:open-ils-general-boun...@list.georgialibraries.org] On Behalf Of Josh Stompro Sent: Thursday, June 02, 2016 8:13 AM To: Evergreen Discussion Group Subject: Re: [OPEN-ILS-GENERAL] Evergreen 2.9.3 - 2.10.3 upgrade issues Hello Jesse, it might help to know the history of your system. What version you started with and how often you have upgraded? The 2.9.3-2.10.0_DBError02.jpg issue is because the config.z3950_index_field_map table doesn't exist. So you probably missed a schema update in the past. Maybe something in the 2.5 upgrade? Looks like the 2.4.3-2.5.0-upgrade-db.sql includes adding that table. Maybe you were on 2.4.4 or greater and missed some updates when you moved to 2.5? Josh Stompro - LARL IT Director From: Open-ils-general [mailto:open-ils-general-boun...@list.georgialibraries.org] On Behalf Of Jesse McCarty Sent: Wednesday, June 01, 2016 6:07 PM To: open-ils-general@list.georgialibraries.org<mailto:open-ils-general@list.georgialibraries.org> Subject: [OPEN-ILS-GENERAL] Evergreen 2.9.3 - 2.10.3 upgrade issues Hello Everyone, I am working on testing our next upgrade of Evergreen (2.9.3 to 2.10.3) and am running into issues with the database upgrade scripts and was wondering if anyone had similar issues or insight into what might be happening. I have attached screen shots of the errors I have been running into for reference. The first error occurs when running the 2.9.3-2.10.0 db upgrade script (2.9.3-2.10.0_DBError01.jpg) and appears to be informational with no impact on the script. The next issue shows up at the end of the 2.9.3-2.10.0 script (2.9.3-2.10.0_DBError02.jpg and 2.9.3-2.10.0_DBError02B.jpg) which appears to undo some of the changes it had made/attempted to make, but still appears to finish some changes. The next issue happens when trying to run the 2.01.0-2.10.1 db upgrade script and this is as far as I got, since it indicates a rollback and no changes made (I did try for kicks to run the 2.10.1-2.10.2 script, which failed as well). I also noticed that the Reingest that is part of the 2.9.3-2.10.0 script gets to a point that requires intervention before conintinueing. The screen hangs with a : at the bottom waiting for some input. I typed in q (like quitting vim) and it then seemed to continue on as normal and I'm not sure if typing q just exited the screen or canceled the work it was going to do. Any help is greatly appreciated, Thanks, Jesse McCarty City of Burlington IT Technical Assistant
Re: [OPEN-ILS-GENERAL] Evergreen 2.9.3 - 2.10.3 upgrade issues
Hello Jesse, it might help to know the history of your system. What version you started with and how often you have upgraded? The 2.9.3-2.10.0_DBError02.jpg issue is because the config.z3950_index_field_map table doesn't exist. So you probably missed a schema update in the past. Maybe something in the 2.5 upgrade? Looks like the 2.4.3-2.5.0-upgrade-db.sql includes adding that table. Maybe you were on 2.4.4 or greater and missed some updates when you moved to 2.5? Josh Stompro - LARL IT Director From: Open-ils-general [mailto:open-ils-general-boun...@list.georgialibraries.org] On Behalf Of Jesse McCarty Sent: Wednesday, June 01, 2016 6:07 PM To: open-ils-general@list.georgialibraries.org Subject: [OPEN-ILS-GENERAL] Evergreen 2.9.3 - 2.10.3 upgrade issues Hello Everyone, I am working on testing our next upgrade of Evergreen (2.9.3 to 2.10.3) and am running into issues with the database upgrade scripts and was wondering if anyone had similar issues or insight into what might be happening. I have attached screen shots of the errors I have been running into for reference. The first error occurs when running the 2.9.3-2.10.0 db upgrade script (2.9.3-2.10.0_DBError01.jpg) and appears to be informational with no impact on the script. The next issue shows up at the end of the 2.9.3-2.10.0 script (2.9.3-2.10.0_DBError02.jpg and 2.9.3-2.10.0_DBError02B.jpg) which appears to undo some of the changes it had made/attempted to make, but still appears to finish some changes. The next issue happens when trying to run the 2.01.0-2.10.1 db upgrade script and this is as far as I got, since it indicates a rollback and no changes made (I did try for kicks to run the 2.10.1-2.10.2 script, which failed as well). I also noticed that the Reingest that is part of the 2.9.3-2.10.0 script gets to a point that requires intervention before conintinueing. The screen hangs with a : at the bottom waiting for some input. I typed in q (like quitting vim) and it then seemed to continue on as normal and I'm not sure if typing q just exited the screen or canceled the work it was going to do. Any help is greatly appreciated, Thanks, Jesse McCarty City of Burlington IT Technical Assistant
