Erik,
Looks like we're also running into this issue.
https://www.mail-archive.com/solr-user@lucene.apache.org/msg153798.html
Is there any think we can do to remedy this besides a node restart, which
causes leader re-election on the good shards which causes them to also
become un-operational?
Gaël:
Many thanks for your writeup. I preserved your and Carlos’ comments in a JIRA:
https://issues.apache.org/jira/browse/SOLR-14679.
How fast you bring the nodes up and down shouldn’t really matter, but if
pausing between bouncing nodes when doing a rolling upgrade keeps you from
having
I think I've come down to the root cause of this mess in our case.
Everything is confirming that the TLOG state is "BUFFERING" rather than
"ACTIVE".
1/ This can be seen with the metrics API as well where we observe:
"TLOG.replay.remaining.bytes":48997506,
"TLOG.replay.remaining.logs":1,
Hello folks,
We see similar behavior from time to time. The main difference seems to be
that you see it while using NRT replication and we see it while using TLOG
replication.
* Solr 7.5.0.
* 1 collection with 12 shards, each with 2 TLOG and 2 PULL replicas.
* 12 machines, each machine hosting
This is a long shot, but look in the overseer queue to see if stuff is stuck.
We ran into that with 6.x.
We restarted the instance that was the overseer and the newly-elected overseer
cleared the queue.
wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/ (my blog)
>
Yes, you should have seen a new tlog after:
- a doc was indexed
- 15 minutes had passed
- another doc was indexed
Well, yes, a leader can be in recovery. It looks like this:
- You’re indexing and docs are written to the tlog.
- Solr un-gracefully shuts down so the segments haven’t been closed.
> Note that for my previous e-mail you’d have to wait 15 minutes after you
> started indexing to see a new tlog and also wait until at least 1,000 new
> document after _that_ before the large tlog went away. I don't think that’s
> your issue though.
Indeed I did wait 15 minutes but not sure
Hmmm, now we’re getting somewhere. Here’s the code block in
DistributedUpdateProcessor
if (ulog == null || ulog.getState() == UpdateLog.State.ACTIVE ||
(cmd.getFlags() & UpdateCommand.REPLAY) != 0) {
super.processCommit(cmd);
} else {
if (log.isInfoEnabled()) {
log.info("Ignoring commit
ext_}",
"Ignoring commit while not ACTIVE" feels strange but maybe It's doesn't mean
what I think it means.
Do you see something strange in there?
Kind regards,
Gaël
De : Erick Erickson
Envoyé : jeudi 23 juillet 2020 13:52
À : solr-user@lucene.apache.org
Objet : Re: tlog keeps growi
Hmmm, this doesn't account for the tlog growth, but a 15 minute hard
commit is excessive and accounts for your down time on restart if Solr
is forcefully shut down. I’d shorten it to a minute or less. You also
shouldn’t have any replay if you shut down your Solr gracefully.
Here’s lots of
changed some collection aliases (I said we have 1 collection, it's not
entirely true: we have 2 collections and we switch to one or the other using
aliases, only one is considered to be "in use" at a time).
Kind regards,
Gaël
De : Erick Erickson
Envoyé : mercredi 22 juillet 2020 22:14
À :
I’m assuming you do not have CDCR configured, correct?
This is weird. Every hard commit should close the current tlog, open a new one
and delete old ones respecting numRecordsToKeep.
Are these NRT replicas or TLOG replicas? That shouldn’t make a lot of
difference, but might be a clue.
Your
Hello,
I'm facing a situation where a transaction log file keeps growing and is never
deleted.
The setup is as follow:
- Solr 8.4.1
- SolrCloud with 2 nodes
- 1 collection, 1 shard
On one of the node I can see the tlog files having the expected behavior, that
is new tlog files being created
13 matches
Mail list logo
