[Archivesspace_Users_Group] Diagnosing issues with ArchivesSpace
Hi Peter & Joshua, Thanks for getting back to me, here’s some replies to your posts. > memory allocation for app & solr ArchivesSpace 3.20 with 35G Solr 8.11.1 with 512M > The plugins that you are using probably aren't the culprit, but they can > add/override index functionality, so listing those out may help as well. These are the plugins that are enabled on our Dev / Test / Prod ArchivesSpace. * nla_staff_spreadsheet_importer <https://github.com/nla/nla_staff_spreadsheet_importer> 3.1 running a fork of https://github.com/hudmol/nla_staff_spreadsheet_importer * nla_accession_events<https://github.com/nla/nla_accession_events> 0.2 running a fork of https://github.com/hudmol/accession_events * as_reftracker<https://github.com/hudmol/as_reftracker> 1.0 * nla_accessions_summary_reports<https://github.com/nla/nla_accessions_summary_reports> 3.0 running a fork of https://github.com/hudmol/accessions_summary_reports * archivesspace_local<https://github.com/nla/archivesspace_local> nla custom plugin only for various AppConfig and other customisation (locales/en.yml) * nla_accession_reports<https://github.com/nla/nla_accession_reports> running a fork of https://github.com/hudmol/nla_accession_reports/ * as_spreadsheet_bulk_updater<https://github.com/hudmol/as_spreadsheet_bulk_updater> 1.5.2 > Couple of things that sprang to mind to check (if you haven't already). Have > you noticed this same behavior in an instance that is not in use? IE have you > set up a clone of your production instance, let it do its initial full index, > and then just let it sit? Do you see errors in the app log that have any > bearing on the problem or pop up around or just before the app goes > unresponsive or OOM? We were seeing identical issues in our dev and test instances of AS though a little less frequent (less concurrent editing users). I have found the cause for our troubles, we upgraded to MySQL 8 in July/August last year, at the time our DBA / Systems Administrators added =UTC to the connection string in our config. I’ve confirmed in our test instance that if edits to a record are started, then stopped (in this case around 2023-05-26 13:27:41 AEDT). While editing, the table correctly incremented the timestamp by 10 seconds (INTERVAL_PERIOD in frontend/app/assets/javascripts/update_monitor.js). However, after the edits were stopped a new row with +10 hours was added to the table. As the frontend / backend continue to sync their copies of the active_edit table, the number of rows continued increasing by 1 with the 10-hr difference. After only a minute or so after exiting the record the table had the following rows and would continue to add new rows with timestamps further into the future. Eventually the active_edits would expire when their timestamp was less than current time – 30 secs (EXPIRE_SECONDS in backend/app/model/active_edit.rb), but it couldn’t keep pace with the rows being created. '17212459','/repositories/2/accessions/5133','mismith','2023-05-28 05:27:41' '17212458','/repositories/2/accessions/5133','mismith','2023-05-27 19:27:41' '17212457','/repositories/2/accessions/5133','mismith','2023-05-27 09:27:41' '17212456','/repositories/2/accessions/5133','mismith','2023-05-26 23:27:41' '17212455','/repositories/2/accessions/5133','mismith','2023-05-26 13:27:41' '17212455','/repositories/2/accessions/5133','mismith','2023-05-26 03:27:41' I’ve confirmed that our production instance has just over 15 million records in the active_edit table with timestamps as far in the future as 2150. We’re planning maintenance to stop ArchivesSpace next week and clear the table, we’ve also updated the time zone in our connection string to the correct time zone =Sydney/Australia (which matches our server timezone now). We’re also going to take ArchivesSpace up to v3.3.1 at the same time. These queries helped when it came to working out system / global time zone settings. SELECT @@GLOBAL.time_zone, @@SESSION.time_zone; SELECT @@system_time_zone; Hopefully, I’ve been able to describe that behaviour in a way that’s able to be understood, but feel free to ask me any further questions. Michael Smith | Software Developer 02 6262 1029 | mism...@nla.gov.au<mailto:mism...@nla.gov.au> | National Library of Australia The National Library of Australia acknowledges Australia’s First Nations Peoples – the First Australians – as the Traditional Owners and Custodians of this land and gives respect to the Elders – past and present – and through them to all Australian Aboriginal and Torres Strait Islander people. ___ Archivesspace_Users_Group mailing list Archivesspace_Users_Group@lyralists.lyrasis.org http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
[Archivesspace_Users_Group] Diagnosing issues with ArchivesSpace
Hello, Our team has been facing recurring issues with our ArchivesSpace setup since October last year, which we've been unable to fully resolve despite concerted efforts. We’re currently running v3.2 on Red Hat Enterprise Linux Server 7.9 (Maipo) and we do have a few custom plugins developed by Hudmol. These don’t appear to be causing the issues that we’re seeing but we haven’t ruled that out yet. The primary problem involves intermittent system slowdowns and shutdowns, requiring frequent reboots to regain functionality. This occurs on average 3-4 times weekly but can sometimes be more frequent. This issue is affecting multiple teams across our organization. The most common symptom of our problem that we are seeing now looks to be a connection pool leak where what looks like indexer threads are holding connections in a closed wait state and preventing them from being used for other requests. This leads to the main page timing out and staff seeing 504 errors, when unresponsive in this manner we usually restart the application. If the application hits an OOM, it will restart itself. Some of the things we’ve attempted so far, * changed default config settings for indexer records per thread, thread count and solr timeout to 10, 2 & 300 * modified archivesspace.sh to increase memory available (ASPACE_JAVA_XMX="-Xmx35g") * disabled both PUI and PUI indexer * application logging to a circular log * changed the garbage collection policies (ASPACE_GC_OPTS="-XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:NewRatio=1 -XX:+ExitOnOutOfMemoryError -XX:+UseGCOverheadLimit") * checked top_containers with empty relationships (0 results) * checked for duplicate event relationships (0 results) * checked for empty indexer state files per record type (0 empty state files) * nightly restarts of the system Any advice with further diagnosis / troubleshooting would be appreciated. If you need additional information about our setup or the issues we're encountering, please let us know. Regards, Michael Smith | Software Developer 02 6262 1029 | mism...@nla.gov.au<mailto:mism...@nla.gov.au> | National Library of Australia The National Library of Australia acknowledges Australia’s First Nations Peoples – the First Australians – as the Traditional Owners and Custodians of this land and gives respect to the Elders – past and present – and through them to all Australian Aboriginal and Torres Strait Islander people. ___ Archivesspace_Users_Group mailing list Archivesspace_Users_Group@lyralists.lyrasis.org http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
Re: [Archivesspace_Users_Group] Looking for ways to best handle object record culturally sensitive restrictions
Hi Naomi, There is a video in the user manual that features as short section on using the notes field for restrictions. The demo starts at the 10.25 mark of the video titled “Adding Instances and Creating Top Containers<https://archivesspace.atlassian.net/wiki/spaces/ArchivesSpaceUserManual/pages/894435365/Managing+Top+Containters>” https://archivesspace.atlassian.net/wiki/spaces/ArchivesSpaceUserManual/pages/915275856/Container+Management Best regards, Michael Smith Archivist Presbyterian Church of Victoria Archives PO Box 9044 South Yarra VIC 3141 michael.sm...@pcv.org.au<mailto:michael.sm...@pcv.org.au> 0425 858 228 Work days – Thursday and Friday [A picture containing shape Description automatically generated] Personal and sensitive information provided by you and collected by the Presbyterian Church of Victoria will be used in conformity with our Privacy Policy (which can be found at www.pcv.org.au<http://www.pcv.org.au/> and www.safechurchpcv.org.au<http://www.safechurchpcv.org.au/> or mailed to you on request). This information is collected for the primary purposes of the Presbyterian Church of Victoria and may be used for any activities conducted or promoted by the PCV. If you do not want your information to be used by us, please do not provide it to us. On 12/11/2022, 5:55 am, "archivesspace_users_group-boun...@lyralists.lyrasis.org<mailto:archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of Franklin, Naomi" mailto:archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of naomi-frank...@utulsa.edu<mailto:naomi-frank...@utulsa.edu>> wrote: Hello, hello. I’m seeking advice for how to best handle archival object record restrictions in ASpace. During a recent consultation of an archive collection, a number of photographs were identified with being culturally sensitive thus needing to be not only restricted to public access, but also internal access. In our efforts to be NAGPRA compliant and to follow the wishes of the Tribe, I need to determine best method(s) of restricting these records based on the options available in ASpace. I cannot find any previous conversations around this topic, and am wondering if others have had to restrict records for similar reasons before or restrict in general. My institution migrated our archive data to ASpace from TMS (The Museum System) a little over a year ago, so we’re still becoming familiar with the ins and outs of ASpace. We have procedures established for handling NAGPRA/Culturally Sensitive materials in TMS, but not yet ASpace. I welcome any thoughts and suggestions. Thanks! Best, Naomi Cataloguing Manager and Associate DBA Gilcrease Museum ___ Archivesspace_Users_Group mailing list Archivesspace_Users_Group@lyralists.lyrasis.org http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
