"No route matches" is just a 404 error, so that's probably not it.
Try that loglevel on debug for a while.
Also, are there outofmemory java errors in the logs or is it just dead?
________________________________
From: archivesspace_users_group-boun...@lyralists.lyrasis.org
<archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of RENTON
Scott <scott.ren...@ed.ac.uk>
Sent: Tuesday, September 7, 2021 11:27 AM
To: Archivesspace Users Group <archivesspace_users_group@lyralists.lyrasis.org>
Subject: Re: [Archivesspace_Users_Group] Connection/login dropping
Apologies, hit send when cutting and pasting.
Prior to the SOLR logging, there is this FATAL, which I think is associated
with either deleted records or the top/subcontainers issue. We see loads of
these in the logs though, and it doesn't seem to stop it functioning. I will
put debug on though and try to get some more info.
F, [2021-08-18T06:20:39.318509 #90658] FATAL -- :
[71fbc902-de38-4056-8574-a9b803620eb3]
F, [2021-08-18T06:20:39.318716 #90658] FATAL -- :
[71fbc902-de38-4056-8574-a9b803620eb3] ActionController::RoutingError (No route
matches [GET] "/repositories/2/archival_objects/21727/request"):
F, [2021-08-18T06:20:39.318771 #90658] FATAL -- :
[71fbc902-de38-4056-8574-a9b803620eb3]
F, [2021-08-18T06:20:39.318825 #90658] FATAL -- :
[71fbc902-de38-4056-8574-a9b803620eb3] actionpack (5.2.4.4)
lib/action_dispatch/middleware/debug_exceptions.rb:65:in `call'
[71fbc902-de38-4056-8574-a9b803620eb3] actionpack (5.2.4.4)
lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
[71fbc902-de38-4056-8574-a9b803620eb3] railties (5.2.4.4)
lib/rails/rack/logger.rb:38:in `call_app'
[71fbc902-de38-4056-8574-a9b803620eb3] railties (5.2.4.4)
lib/rails/rack/logger.rb:26:in `block in call'
[71fbc902-de38-4056-8574-a9b803620eb3] activesupport (5.2.4.4)
lib/active_support/tagged_logging.rb:71:in `block in tagged'
[71fbc902-de38-4056-8574-a9b803620eb3] activesupport (5.2.4.4)
lib/active_support/tagged_logging.rb:28:in `tagged'
[71fbc902-de38-4056-8574-a9b803620eb3] activesupport (5.2.4.4)
lib/active_support/tagged_logging.rb:71:in `tagged'
[71fbc902-de38-4056-8574-a9b803620eb3] railties (5.2.4.4)
lib/rails/rack/logger.rb:26:in `call'
[71fbc902-de38-4056-8574-a9b803620eb3] actionpack (5.2.4.4)
lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
[71fbc902-de38-4056-8574-a9b803620eb3] actionpack (5.2.4.4)
lib/action_dispatch/middleware/request_id.rb:27:in `call'
[71fbc902-de38-4056-8574-a9b803620eb3] rack (2.2.3)
lib/rack/method_override.rb:24:in `call'
[71fbc902-de38-4056-8574-a9b803620eb3] rack (2.2.3) lib/rack/runtime.rb:22:in
`call'
[71fbc902-de38-4056-8574-a9b803620eb3] activesupport (5.2.4.4)
lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
[71fbc902-de38-4056-8574-a9b803620eb3] actionpack (5.2.4.4)
lib/action_dispatch/middleware/executor.rb:14:in `call'
[71fbc902-de38-4056-8574-a9b803620eb3] actionpack (5.2.4.4)
lib/action_dispatch/middleware/static.rb:127:in `call'
[71fbc902-de38-4056-8574-a9b803620eb3] rack (2.2.3) lib/rack/sendfile.rb:110:in
`call'
[71fbc902-de38-4056-8574-a9b803620eb3] railties (5.2.4.4)
lib/rails/engine.rb:524:in `call'
[71fbc902-de38-4056-8574-a9b803620eb3]
uri:classloader:/rack/handler/servlet.rb:22:in `call'
==========
Scott Renton
Digital Library Development & Systems
Floor F East
Argyle House
515219
________________________________
From: archivesspace_users_group-boun...@lyralists.lyrasis.org
<archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of Blake
Carver <blake.car...@lyrasis.org>
Sent: 07 September 2021 16:23
To: Archivesspace Users Group <archivesspace_users_group@lyralists.lyrasis.org>
Subject: Re: [Archivesspace_Users_Group] Connection/login dropping
This email was sent to you by someone outside the University.
You should only click on links or attachments if you are certain that the email
is genuine and the content is safe.
Any ERROR or FATAL?
If not, try bumping up the log_level to debug and see if that will catch
something.
________________________________
From: archivesspace_users_group-boun...@lyralists.lyrasis.org
<archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of RENTON
Scott <scott.ren...@ed.ac.uk>
Sent: Tuesday, September 7, 2021 11:21 AM
To: archivesspace_users_group@lyralists.lyrasis.org
<archivesspace_users_group@lyralists.lyrasis.org>
Subject: Re: [Archivesspace_Users_Group] Connection/login dropping
Thanks Blake.
A mass of solr logging (I am set to ERROR throughout, but still getting plenty
of indexer INFO). Prior to that:
==========
Scott Renton
Digital Library Development & Systems
Floor F East
Argyle House
515219
________________________________
From: archivesspace_users_group-boun...@lyralists.lyrasis.org
<archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of Blake
Carver <blake.car...@lyrasis.org>
Sent: 07 September 2021 15:37
To: archivesspace_users_group@lyralists.lyrasis.org
<archivesspace_users_group@lyralists.lyrasis.org>
Subject: Re: [Archivesspace_Users_Group] Connection/login dropping
This email was sent to you by someone outside the University.
You should only click on links or attachments if you are certain that the email
is genuine and the content is safe.
What's happening in the logs just before that first "Unhanded exception" error?
________________________________
From: archivesspace_users_group-boun...@lyralists.lyrasis.org
<archivesspace_users_group-boun...@lyralists.lyrasis.org> on behalf of RENTON
Scott <scott.ren...@ed.ac.uk>
Sent: Monday, September 6, 2021 9:29 AM
To: archivesspace_users_group@lyralists.lyrasis.org
<archivesspace_users_group@lyralists.lyrasis.org>
Subject: [Archivesspace_Users_Group] Connection/login dropping
Hi all
Our uptime has been a bit flaky the last few weeks- what we're finding is that,
after a certain amount of service, users are unable to login, and if that's not
noticed or acted on, the site generally stops responding some time later. A
restart of the application resolves it.
We're running 2.8.1 against MariaDB 5.5-65, and Java 8. I have tried different
versions of the mysqlconnector, but this does not seem to make a difference
(first noticed on 5.1.39). The MySQL has been tuned to an extent. The VM has
16GB of memory, approximately 10GB of which is apportioned to AS. The MySQL
process was restarted last week, and clearing that has still allowed the issue.
It's pretty hard to spot any particular action which causes this. The logging
we see once it's happened looks like this:
E, [2021-08-18T06:21:01.548843 #90658] ERROR -- : Thread-28310: Unhandled
exception!
E, [2021-08-18T06:21:01.552325 #90658] ERROR -- :
apps.archivesspace.gems.gems.sequel_minus_5_dot_9_dot_0.lib.sequel.model.base.RUBY$method$parent_id$0(/apps/archivesspace/gems/gems/sequel-5.9.0/lib/sequel/model/base.rb:783)
apps.archivesspace.gems.gems.sequel_minus_5_dot_9_dot_0.lib.sequel.model.base.RUBY$method$parent_id$0$__VARARGS__(/apps/archivesspace/gems/gems/sequel-5.9.0/lib/sequel/model/base.rb)
org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:82)
org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)
and will continue to spout "minus_5_dot_9_dot_0" type errors until we reboot.
I have taken some statistics around the numbers of threads running pre and post
bounce. I don't see anything obvious there, and I have left the config.rb value
AppConfig[:db_max_connections] = proc { 20 + (AppConfig[:indexer_thread_count]
* 2) }
as it is.
I'm also aware that we have an issue some others have of a proliferation of
FATAL errors since merging top/subcontainers (items cannot be found), but I do
not know if it is related to this.
However, if anyone recognises this behaviour, and can advise, I'd be grateful
to hear about it.
Cheers
Scott
==========
Scott Renton
Digital Library Development & Systems
Floor F East
Argyle House
515219
The University of Edinburgh is a charitable body, registered in Scotland, with
registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh
Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.
_______________________________________________
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group@lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
