Hi Shawn,
Thanks for the considerations. I also cannot see anything in the logs, which is
unfortunate. I'm trying to see if the autocommits have a direct relationship
with the bad Solr 8 throughput I'm experiencing. I'll start experimenting with
the values!
Regards,
Alex
________________________________
From: Shawn Heisey <apa...@elyograg.org>
Sent: 06 October 2021 16:59
To: users@solr.apache.org <users@solr.apache.org>
Subject: Re: Solr 8 autocommit confirmation
*** External email: use caution ***
On 10/6/21 7:00 AM, Saur, Alexandre (ELS-AMS) wrote:
> I have a quick question regarding Solr 8 autocommit feature. This is my
> configuration:
>
> <autoCommit>
> <maxTime>600000</maxTime>
> <openSearcher>true</openSearcher>
> </autoCommit>
>
> <autoSoftCommit>
> <maxTime>${solr.autoSoftCommit.maxTime:-1}</maxTime>
> </autoSoftCommit>
>
> I can check this is true using the console. However, I cannot see any thread
> doing the autocommit in the logs, nor any metric that gives me the time the
> autocommit is being done.
The client (dovecot) that I have using my personal Solr server issues
commits itself, so I do not have any automatic commits in my log, and at
this time I do not use Solr at work. I have no idea if Solr's logfile
has specific mention of automatic commits. I would hope that it does.
I have done a full reindex several times and I do not see anything that
suggests an automatic commit ever fired. I have 60000 for autoCommit
and 120000 for autoSoftCommit.
If you do a manual commit after every indexing batch, and the batch
takes less than 10 minutes (your autoCommit time) then autoCommit will
not actually do anything, and what I see in my own log suggests that
autoCommit and autoSoftCommit are smart enough to not even run if a
commit is done manually before they fire.
This is what I would do if I were in your shoes, and I might further
reduce the autoSoftCommit, depending on how long commits actually take:
<autoCommit>
<maxTime>60000</maxTime>
<openSearcher>false</openSearcher>
</autoCommit>
<autoSoftCommit>
<maxTime>300000</maxTime>
</autoSoftCommit>
Assuming you don't do manual commits in your indexing software, this autoCommit
would ensure the transaction logs will have less data in them, which is a good
thing, and the autoSoftCommit handles change visibility. Of course, if your
indexing software does its own commits and does them with enough frequency,
then the automatic commits may be essentially unused.
This article is valuable reading. The title says "SolrCloud" but it is also
applicable when NOT in cloud mode:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flucidworks.com%2Fpost%2Funderstanding-transaction-logs-softcommit-and-commit-in-sorlcloud%2F&data=04%7C01%7Ca.saur%40elsevier.com%7Cbb90d89a7de54274fec208d988d9ffac%7C9274ee3f94254109a27f9fb15c10675d%7C0%7C0%7C637691293307772302%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=xVcTSmftEFx3yRb3q%2Bi8GtSUppT0jZx35g4y3cDXLWc%3D&reserved=0
Thanks,
Shawn
________________________________
Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The
Netherlands, Registration No. 33158992, Registered in The Netherlands.
