Re: [Archivesspace_Users_Group] Performance tuning

2022-09-15 Thread Blake Carver 

> ASPACE_JAVA_XMX="-Xmx16144m"

If the site needs 16 gigs of RAM to keep going, something is probably wrong.

> AppConfig[:indexer_records_per_thread] = 25
> AppConfig[:indexer_thread_count] = 7
> AppConfig[:indexer_solr_timeout_seconds] = 300

Messing around with those indexer settings is generally a good idea for sites 
with ALLOTTA records and/or a decent number of people adding new records. "It 
Depends" is about the only answer for those settings. If you have really 
deep/complex resources you'll probably want to change those up. Try slowing it 
down a bit. Try 2/50 or 2/25. You could even slow it WAY down to 1/1. It'll 
take forever to finish, but you can see exactly what it's doing each time.

BUT

If your site was OK and now it's not, maybe something else has gone wrong.

Check your logs. Make sure you're running the log_level on debug to catch 
everything possible.

- Look for ERROR or FATAL, maybe Java out of memory errors?
Sometimes there will be an Error or error but it's usually ERROR

-Is your indexer just running forever? Maybe it's caught in a crashing loop?
 "grep Indexed" do those numbers keep going up and down?
You sould see:
 Indexed 102303 of 102303 archival_object records in repository
Only once, you don't want to see that count going up and up and then suddenly
 Indexed 1 of 102303 archival_object records in repository

- Also check your proxy logs, maybe there's a bad bot hitting the PUI way too 
hard.

- Also just check that the server doesn't have anything new running and using 
up all the resources.

The answer is usually in the logs.


From: archivesspace_users_group-boun...@lyralists.lyrasis.org 
 on behalf of Cowing 
[he], Jared 
Sent: Tuesday, September 13, 2022 12:33 PM
To: archivesspace_users_group@lyralists.lyrasis.org 

Subject: [Archivesspace_Users_Group] Performance tuning

Hi all,

I'm currently working with our campus IT to identify the cause(s) of a 
degradation in performance that we've been seeing. I'm relatively new to our 
library and to ArchivesSpace generally, but I've been told it's been slowing 
bit by bit for a few years, and the problem has escalated just in the past few 
weeks. Neither full re-indexes nor our nightly re-starts seem to help.

I'm aware of this page on 
tuning, 
which has been quite helpful in addition to suggestions already posted to this 
list. We're hopeful that moving to external Solr with our next upgrade will 
also help (currently on 2.7.1), but are still trying other measures just in 
case it doesn't. While we look more into general best practices for tuning our 
hosting environment, I'd also like to check with all of you to see if there are 
common issues that are more specific to ArchivesSpace that we have overlooked 
and should focus our attention on.

Here are a few of our key settings. Our Java memory variables are below. I get 
the sense that they are higher than average, is that so?
ASPACE_JAVA_XMX="-Xmx16144m"
ASPACE_JAVA_XSS="-Xss8m"

Indexer settings from config.rb:
AppConfig[:indexer_records_per_thread] = 25
AppConfig[:indexer_thread_count] = 7
AppConfig[:indexer_solr_timeout_seconds] = 300


And copied from our innodb settings:
innodb_file_per_table
innodb_flush_method=O_DIRECT
innodb_log_file_size=1G
innodb_buffer_pool_size=8G
symbolic-links=0
max_allowed_packet=128M
open_files_limit=9182

I appreciate any tips on what we ought to be looking for. I know it's hard to 
give advice from afar when each institution's situation is different, but 
thought it worth asking in case anything jumped out before we turn to technical 
support.

Thanks,
--

Jared Cowing | Systems Librarian | he/him
WILLIAMS COLLEGE LIBRARIES  | Williamstown, MA | 
(413)597-3061
___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] Missing Japanese charactires in a PUI generated PDF

2022-09-15 Thread Hoffner, Bailey E. 
As a follow-up, we’ve had the same issue with Chinese language characters and, 
the stop-gap workaround we’re using at the moment until we can implement a 
better solution, is to use the the New School Bulk Updater plugin to export a 
spreadsheet for the finding aid which then at least serves as an 
easy-to-generate searchable file outside of AS, even though it’s not the same 
experience as a PDF.

Good luck and we’ll continue to tune into these language-based questions 
related to AS!

-Bailey

Bailey Hoffner, MLIS
Pronouns: she/her or they/them
Metadata and Collections Management Archivist
University of Oklahoma Libraries
bail...@ou.edu

From:  on behalf of 
"Mayo, Dave" 
Reply-To: Archivesspace Users Group 

Date: Wednesday, September 14, 2022 at 10:01 PM
To: Archivesspace Users Group 
Subject: Re: [Archivesspace_Users_Group] Missing Japanese charactires in a PUI 
generated PDF

Hi!

This is something we’ve recently had to deal with – I’m not 100% sure from what 
you’ve posted that it’s the same issue we had, but there are a few issues with 
the PUI’s current PDF generation support that make font handling challenging.

So, first of all – if you’re setting up a fallback hierarchy and the font with 
Japanese characters isn’t in the first position, the PDF generation library 
isn’t seeing it at all.  The flying saucer ipdf library doesn’t support font 
fallback, which is a real problem if you need to support multiple languages.

So, first thing I’d try is making sure that the text in question is _solely_ 
the font supporting Japanese.  If the Japanese characters render, that’ll at 
least verify that that’s the reason.

Our solution, which I’m hoping to work up and submit as a pull request, was to 
replace the existing library with 
https://github.com/danfickle/openhtmltopdf
 - a project based on flying saucer but with several enhancements.  
Implementing it is somewhat complex:

1. Openhtmltopdf and _all dependencies thereof_ need to be provided by putting 
them in the archivesspace/lib directory (the directory the MySQL connector goes 
in during install)

Currently we’re doing this in our dockerfile via:
wget -P /archivesspace/lib 
https://repo1.maven.org/maven2/com/google/zxing/core/3.5.0/core-3.5.0.jar
 && \

wget -P /archivesspace/lib 
https://repo1.maven.org/maven2/junit/junit/4.13.2/junit-4.13.2.jar
 && \

wget -P /archivesspace/lib 
https://repo1.maven.org/maven2/com/openhtmltopdf/openhtmltopdf-core/1.0.10/openhtmltopdf-core-1.0.10.jar
 && \

wget -P /archivesspace/lib 
https://repo1.maven.org/maven2/com/openhtmltopdf/openhtmltopdf-pdfbox/1.0.10/openhtmltopdf-pdfbox-1.0.10.jar
 && \

wget -P /archivesspace/lib 
https://repo1.maven.org/maven2/de/rototor/pdfbox/graphics2d/0.34/graphics2d-0.34.jar
 && \

wget -P /archivesspace/lib 
https://repo1.maven.org/maven2/org/apache/pdfbox/pdfbox/2.0.26/pdfbox-2.0.26.jar
 && \

wget -P /archivesspace/lib 
https://repo1.maven.org/maven2/org/apache/pdfbox/xmpbox/2.0.26/xmpbox-2.0.26.jar
 && \

wget -P /archivesspace/lib 
https://repo1.maven.org/maven2/org/apache/pdfbox/fontbox/2.0.26/fontbox-2.0.26.jar

Re: [Archivesspace_Users_Group] [External] Performance tuning

2022-09-15 Thread Steele, Henry 
One other thing that would be worth checking is your Java version.  ASpace 
needs 1.8 so if you have earlier that could be an issue too

From: Steele, Henry
Sent: Thursday, September 15, 2022 10:39 AM
To: archivesspace_users_group@lyralists.lyrasis.org
Subject: RE: [External] [Archivesspace_Users_Group] Performance tuning

Hey there.  I am definitely not the most qualified person on this listserv to 
answer your question, but I can share what we’ve done in our instances and my 
experience with this, and what I’ve learned over the years.

First the number of indexer threads should be keyed the number of CPU cores you 
have on a given machine.  You list 7—is that how many cores you have?

The Java XMX is the Java heap size.  Your value seems quite high, unless you 
have that much RAM available for the application.  You can check on your server 
by running the command “free” or checking with your IT department.

Another key thing to note if you are running version 2.7.1: there is a bug in 
older versions of ASpace that means that the application does not respect the 
*solr* log level you have in your config file.   I had to find a workaround for 
this by changing the log level at the server level in Java settings for Solr.   
This of course isn’t an issue in recent releases because solr isn’t part of 
ASpace anymore anyway.   But the effect of this bug without altering server 
settings is that the logs are really noisy with solr stuff and this can impact 
performance.  If you think this may be an issue for you, reach out to me and 
I’ll send you a document for how to change the solr log level at the server 
level.

Hope things improve for you!

From: 
archivesspace_users_group-boun...@lyralists.lyrasis.org
 
mailto:archivesspace_users_group-boun...@lyralists.lyrasis.org>>
 On Behalf Of Cowing [he], Jared
Sent: Tuesday, September 13, 2022 12:33 PM
To: 
archivesspace_users_group@lyralists.lyrasis.org
Subject: [External] [Archivesspace_Users_Group] Performance tuning

Hi all,

I'm currently working with our campus IT to identify the cause(s) of a 
degradation in performance that we've been seeing. I'm relatively new to our 
library and to ArchivesSpace generally, but I've been told it's been slowing 
bit by bit for a few years, and the problem has escalated just in the past few 
weeks. Neither full re-indexes nor our nightly re-starts seem to help.

I'm aware of this page on 
tuning, 
which has been quite helpful in addition to suggestions already posted to this 
list. We're hopeful that moving to external Solr with our next upgrade will 
also help (currently on 2.7.1), but are still trying other measures just in 
case it doesn't. While we look more into general best practices for tuning our 
hosting environment, I'd also like to check with all of you to see if there are 
common issues that are more specific to ArchivesSpace that we have overlooked 
and should focus our attention on.

Here are a few of our key settings. Our Java memory variables are below. I get 
the sense that they are higher than average, is that so?
ASPACE_JAVA_XMX="-Xmx16144m"
ASPACE_JAVA_XSS="-Xss8m"

Indexer settings from config.rb:
AppConfig[:indexer_records_per_thread] = 25
AppConfig[:indexer_thread_count] = 7
AppConfig[:indexer_solr_timeout_seconds] = 300


And copied from our innodb settings:
innodb_file_per_table
innodb_flush_method=O_DIRECT
innodb_log_file_size=1G
innodb_buffer_pool_size=8G
symbolic-links=0
max_allowed_packet=128M
open_files_limit=9182

I appreciate any tips on what we ought to be looking for. I know it's hard to 
give advice from afar when each institution's situation is different, but 
thought it worth asking in case anything jumped out before we turn to technical 
support.

Thanks,
--

Jared Cowing | Systems Librarian | he/him
WILLIAMS COLLEGE LIBRARIES  | Williamstown, MA | 
(413)597-3061

Caution: This message originated from outside of the Tufts University 
organization. Please exercise caution when clicking links or opening 
attachments. When in doubt, email the TTS Service Desk at 
i...@tufts.edu or call them directly at 617-627-3376.

___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

Re: [Archivesspace_Users_Group] [External] Performance tuning

2022-09-15 Thread Steele, Henry 
Hey there.  I am definitely not the most qualified person on this listserv to 
answer your question, but I can share what we’ve done in our instances and my 
experience with this, and what I’ve learned over the years.

First the number of indexer threads should be keyed the number of CPU cores you 
have on a given machine.  You list 7—is that how many cores you have?

The Java XMX is the Java heap size.  Your value seems quite high, unless you 
have that much RAM available for the application.  You can check on your server 
by running the command “free” or checking with your IT department.

Another key thing to note if you are running version 2.7.1: there is a bug in 
older versions of ASpace that means that the application does not respect the 
*solr* log level you have in your config file.   I had to find a workaround for 
this by changing the log level at the server level in Java settings for Solr.   
This of course isn’t an issue in recent releases because solr isn’t part of 
ASpace anymore anyway.   But the effect of this bug without altering server 
settings is that the logs are really noisy with solr stuff and this can impact 
performance.  If you think this may be an issue for you, reach out to me and 
I’ll send you a document for how to change the solr log level at the server 
level.

Hope things improve for you!

From: archivesspace_users_group-boun...@lyralists.lyrasis.org 
 On Behalf Of Cowing 
[he], Jared
Sent: Tuesday, September 13, 2022 12:33 PM
To: archivesspace_users_group@lyralists.lyrasis.org
Subject: [External] [Archivesspace_Users_Group] Performance tuning

Hi all,

I'm currently working with our campus IT to identify the cause(s) of a 
degradation in performance that we've been seeing. I'm relatively new to our 
library and to ArchivesSpace generally, but I've been told it's been slowing 
bit by bit for a few years, and the problem has escalated just in the past few 
weeks. Neither full re-indexes nor our nightly re-starts seem to help.

I'm aware of this page on 
tuning, 
which has been quite helpful in addition to suggestions already posted to this 
list. We're hopeful that moving to external Solr with our next upgrade will 
also help (currently on 2.7.1), but are still trying other measures just in 
case it doesn't. While we look more into general best practices for tuning our 
hosting environment, I'd also like to check with all of you to see if there are 
common issues that are more specific to ArchivesSpace that we have overlooked 
and should focus our attention on.

Here are a few of our key settings. Our Java memory variables are below. I get 
the sense that they are higher than average, is that so?
ASPACE_JAVA_XMX="-Xmx16144m"
ASPACE_JAVA_XSS="-Xss8m"

Indexer settings from config.rb:
AppConfig[:indexer_records_per_thread] = 25
AppConfig[:indexer_thread_count] = 7
AppConfig[:indexer_solr_timeout_seconds] = 300


And copied from our innodb settings:
innodb_file_per_table
innodb_flush_method=O_DIRECT
innodb_log_file_size=1G
innodb_buffer_pool_size=8G
symbolic-links=0
max_allowed_packet=128M
open_files_limit=9182

I appreciate any tips on what we ought to be looking for. I know it's hard to 
give advice from afar when each institution's situation is different, but 
thought it worth asking in case anything jumped out before we turn to technical 
support.

Thanks,
--

Jared Cowing | Systems Librarian | he/him
WILLIAMS COLLEGE LIBRARIES  | Williamstown, MA | 
(413)597-3061

Caution: This message originated from outside of the Tufts University 
organization. Please exercise caution when clicking links or opening 
attachments. When in doubt, email the TTS Service Desk at 
i...@tufts.edu or call them directly at 617-627-3376.

___
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group