[
https://issues.apache.org/jira/browse/SOLR-16561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17638364#comment-17638364
]
Shawn Heisey commented on SOLR-16561:
-------------------------------------
I encouraged [~sunh11373] to open this issue on ApacheSolr slack. I think this
is how the replication interval should have been determined from the beginning.
Opening a new searcher is particularly heavy on TLOG/PULL followers, as noted
elsewhere with the report that all the existing SegmentReader instances are not
re-used. The current code is particularly aggressive for most users given that
the autoCommit in our example configs is set to 15 seconds.
Do any of my peers disagree with committing this change? I think that this
should be documented in the section on TLOG and PULL replicas, with some
discussion about general recommendations regarding autoSoftCommit and
autoCommit. I will work on some documentation changes to pair with this.
> Use autoSoftCommmitMaxTime as preferred poll interval of IndexFetcher
> ---------------------------------------------------------------------
>
> Key: SOLR-16561
> URL: https://issues.apache.org/jira/browse/SOLR-16561
> Project: Solr
> Issue Type: Improvement
> Components: replication (java)
> Affects Versions: 8.8.2
> Reporter: Hang Sun
> Assignee: Shawn Heisey
> Priority: Minor
> Labels: replication-performance
> Attachments: SOLR-16561.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> TLOG/PULL replicas use *IndexFetcher* to fetch segment files from leaders.
> Once new segment files are downloaded and merged into existing index, a new
> Searcher is opened so the updated data is made available to the clients. The
> poll interval is determined by following code in *ReplicateFromLeader*
> {code:java}
> if (uinfo.autoCommmitMaxTime != -1) {
> pollIntervalStr = toPollIntervalStr(uinfo.autoCommmitMaxTime/2);
> } else if (uinfo.autoSoftCommmitMaxTime != -1) {
> pollIntervalStr = toPollIntervalStr(uinfo.autoSoftCommmitMaxTime/2);
> }{code}
>
> In a typical config for replication using TLOG/PULL replicas where data
> visibility is less important (a trade-off to avoid NRT replicas), we set a
> short commit time to persist changes and long soft-commit time to make
> changes visible.
>
> {code:java}
> <autoCommit>
> <maxTime>15000</maxTime>
> <openSearcher>false</openSearcher>
> </autoCommit>
> <autoSoftCommit>
> <maxTime>3600000</maxTime>
> </autoSoftCommit>
> {code}
>
> With about config, the poll interval will be 15/2 = 7 sec. This leads to
> frequent opening of new Searchers which causes huge impact on realtime user
> queries, especially if the new Searcher takes long time to warmup. This also
> makes changes visible on followers ahead of leaders.
> Because the polling of new segment files is more about visibility because
> TLOG replicas still get updates to tlog files via UpdateHandler (this is my
> understanding). It seems more appropriate to use *autoSoftCommmitMaxTime* as
> the poll interval.
> I would proposed change below where *autoSoftCommmitMaxTime* is chosen as
> the preferred interval. This will make the poll interval much longer and
> make the visibility order more inline with eventual consistency pattern.
>
> {code:java}
> if (uinfo.autoSoftCommmitMaxTime != -1) {
> pollIntervalStr = toPollIntervalStr(uinfo.autoSoftCommmitMaxTime);
> } else if (uinfo.autoCommmitMaxTime != -1) {
> pollIntervalStr = toPollIntervalStr(uinfo.autoCommmitMaxTime);
> }
> {code}
> The change has been tried and showed much less impact on realtime queries.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
