Indexing can also fail at the commit stage, not related to any one
record. That is when ArchivesSpace tells Solr to transfer changes made
in memory to storage. It does that at several points in the indexing
process, but the longest one is at the end of the PUI indexer's run. If,
because you've got a lot of records, or slow storage on your Solr
server, it takes longer it respond than the value of
AppConfig[:indexer_solr_timeout_seconds], it will start all over again,
and potentially go into a loop. The workaround is to increase the timeout.
You might not notice you've got enough records to cause this until you
do a full re-index, or someone edits something linked to most or all
records (e.g. a repository, or a very widely-used subject), triggering
the re-indexing of most of the system's records.
Andrew.
On 10/05/2022 22:06, Blake Carver wrote:
1x1 would mean setting both records_per_thread and thread_count to 1.
Having loglevel on debug and running at 1x1, you'll be able to see
exactly which thing is being indexed as it happens, and when it
crashes, you'll see what it was working through at the time.
PUI will always take longer, and a VERY long time 1x1, but unless
you're sure which indexer is crashing, I'd switch them both up.
You can just `grep Indexed archivesspace.out` after it's running and
watch those numbers. As long as they're going up, all is well.
It is also possible that it will finish without crashing running so
slow as well. I've seen that happen with LARGE records.
------------------------------------------------------------------------
*From:* [email protected]
<[email protected]> on behalf of
Tom Hanstra <[email protected]>
*Sent:* Tuesday, May 10, 2022 4:15 PM
*To:* Archivesspace Users Group
<[email protected]>
*Subject:* Re: [Archivesspace_Users_Group] (re)indexing in 2.8.1
Thanks, Blake.
Turns out we did add quite a few records recently, so maybe there was
something in there that it did not like all that much.
How can you tell which record it is choking on? Is that your "1x1"
suggestion? Or does the DEBUG option make that more clear? I have my
indexing set to:
AppConfig[:indexer_records_per_thread] = 25
AppConfig[:indexer_thread_count] = 2
for both PUI and Staff records. I believe you are suggesting it would
most easily be found using 1 and 1? I can see where that could take a
long time. But it if is going to choke over and over on the same
record, then that may be the best way to address it.
Do you think if I just did staff indexing without PUI, that it would
be identified faster? Or could it pass the staff side but then die on
PUI later?
I hope to try some of these ideas after hours today, so if you can
confirm that I've got the right idea, that would help.
Tom
On Tue, May 10, 2022 at 2:17 PM Blake Carver
<[email protected]> wrote:
> Is this possible?
Short answer, Yes, it's possible your indexer is starting over.
Long answer. This can be tricky to figure out. Something is wrong,
the indexer never wants to do that. Sometimes "something" "bad"
gets into ArchivesSpace and the indexer will just crash and start
over. The problem is the "something" can be anything and the "bad"
can be hard to figure out. The more stuff you have in your DB, the
harder it is to figure out.
First, I'd make sure this is happening. Your logs should make it
obvious. You might see some FATAL errors just before it starts
over. You MIGHT be able to narrow it down from that. That is,
what group of records had that error in the logs? Maybe that
narrows it down enough. You just got lucky!
I don't think I've ever been so lucky. What I'd do next is set
your loglevel to DEBUG and restart. If you're feeling lucky or
just impatient or both, leave the indexer speed as is. You'll get
more details out of the logs and you should be able to narrow it
down better. Ideally, you want to run the indexers at 1x1, which
means it could take forrreeevverrrrr to get back around to the
crash again. If you're lucky, it'll crash on a record, you'll go
look at that record, the problem will be obvious, and there will
be much rejoicing. With it running 1x1 you should see exactly
what's causing the fail. If it's not crashing on the same record
every time.... ugh. That's an even longer answer.
------------------------------------------------------------------------
*From:* [email protected]
<[email protected]> on
behalf of Tom Hanstra <[email protected]>
*Sent:* Tuesday, May 10, 2022 10:23 AM
*To:* Archivesspace Users Group
<[email protected]>
*Subject:* [Archivesspace_Users_Group] (re)indexing in 2.8.1
I don't look at the logs a lot unless there are issues with
ArchivesSpace, so maybe this is something normal. But, after a
restart due to some complaints about database connectivity, it
looks like our ArchivesSpace instance has decided to do a full
reindex. The index log sure looks as if it is starting
from scratch and running through the indexing of both PUI and
Staff indexes.
Is this possible? Is it something that happens periodically and I
just did not notice it? Nothing has changed in my data directory,
so I don't see any reason for indexing to occur. Yet that is what
the logs show.
If it is doing this for some reason, and knowing that we restart
periodically, it seems like we will get into a loop where indexing
just keeps happening all the time. Also, it would be helpful to
understand what caused this to happen.
Any thoughts or experiences from those who have run this for
longer would be appreciated. I'd like to understand if it would be
a good idea to clear the data directory and perform a full index
over the weekend rather than an unexpected and possibly never
ending round in the background.
Thanks,
Tom
--
*Tom Hanstra*
/Sr. Systems Administrator/
[email protected]
_______________________________________________
Archivesspace_Users_Group mailing list
[email protected]
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
--
*Tom Hanstra*
/Sr. Systems Administrator/
[email protected]
_______________________________________________
Archivesspace_Users_Group mailing list
[email protected]
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
_______________________________________________
Archivesspace_Users_Group mailing list
[email protected]
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
