[
https://issues.apache.org/jira/browse/SOLR-18406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhenyuLi updated SOLR-18406:
----------------------------
Description:
A follower can receive HTTP 500 while downloading an index if the leader
deletes the requested index generation after filelist completes but before a
subsequent filecontent request begins.
Follower index replication consists of separate HTTP requests:
indexversion
-> filelist(generation=G)
-> filecontent(file=X, generation=G)
-> filecontent(file=Y, generation=G)
-> ...
The leader must retain commit G across these requests.
While a filecontent request is active, the commit is protected with
saveCommitPoint()/releaseCommitPoint(). Between requests, however, it is only
protected by a timed reservation. The default commitReserveDuration is 10
seconds.
If the follower takes longer than that interval before requesting the next
file, the reservation expires. A later hard commit on the leader invokes the
index deletion policy and may delete G. The next filecontent request then calls
saveCommitPoint(G), which throws IllegalStateException.
This exception is currently exposed as HTTP 500.
The same stale-generation condition is already handled explicitly by the
filelist path. filelist catches IllegalStateException from
getAndSaveCommitPoint(G) and reports:invalid index generation
The filecontent path does not provide equivalent handling.
The relevant difference is approximately:
ReplicationHandler.getFileList():
try
{ commit = delPol.getAndSaveCommitPoint(gen); }
catch (IllegalStateException ignored)
{ // handled as an invalid generation below }
ReplicationHandler.DirectoryFileStream.initWrite():
if (indexGen != null)
{ delPolicy.saveCommitPoint(indexGen); }
The equivalent behavior is present in the newer replication implementation.
Reproduction
============
The issue does not require a large index. One committed document is sufficient.
1. Create and hard-commit one document on the leader, producing generation G.
2. Submit another update or delete without committing it.
3. Start an explicit fetchindex request on the follower.
4. Allow the follower to complete indexversion and filelist(G).
5. Pause before the first filecontent(G) request.
6. Wait until the reservation for G has expired.
7. Hard-commit the pending leader update, producing a newer generation and
allowing the deletion policy to delete G.
8. Resume the follower so that it requests filecontent(G).
The hard commit must occur after the reservation expires. Expiration alone does
not delete the commit; the subsequent leader commit is what invokes the
deletion policy.
was:
A follower can receive HTTP 500 while downloading an index if the leader deletes
the requested index generation after filelist completes but before a
subsequent
filecontent request begins.
Follower index replication consists of separate HTTP requests:
indexversion
-> filelist(generation=G)
-> filecontent(file=X, generation=G)
-> filecontent(file=Y, generation=G)
-> ...
The leader must retain commit G across these requests.
While a filecontent request is active, the commit is protected with
saveCommitPoint()/releaseCommitPoint(). Between requests, however, it is only
protected by a timed reservation. The default commitReserveDuration is 10
seconds.
If the follower takes longer than that interval before requesting the next
file,
the reservation expires. A later hard commit on the leader invokes the index
deletion policy and may delete G. The next filecontent request then calls
saveCommitPoint(G), which throws IllegalStateException.
This exception is currently exposed as HTTP 500.
The same stale-generation condition is already handled explicitly by the
filelist path. filelist catches IllegalStateException from
getAndSaveCommitPoint(G) and reports:
invalid index generation
The filecontent path does not provide equivalent handling.
In Solr 8.11.4, the relevant difference is approximately:
ReplicationHandler.getFileList():
try {
commit = delPol.getAndSaveCommitPoint(gen);
} catch (IllegalStateException ignored) {
// handled as an invalid generation below
}
ReplicationHandler.DirectoryFileStream.initWrite():
if (indexGen != null) {
delPolicy.saveCommitPoint(indexGen);
}
The equivalent behavior is present in the newer replication implementation.
Reproduction
============
The issue does not require a large index. One committed document is
sufficient.
1. Create and hard-commit one document on the leader, producing generation G.
2. Submit another update or delete without committing it.
3. Start an explicit fetchindex request on the follower.
4. Allow the follower to complete indexversion and filelist(G).
5. Pause before the first filecontent(G) request.
6. Wait until the reservation for G has expired.
7. Hard-commit the pending leader update, producing a newer generation and
allowing the deletion policy to delete G.
8. Resume the follower so that it requests filecontent(G).
The hard commit must occur after the reservation expires. Expiration alone
does
not delete the commit; the subsequent leader commit is what invokes the
deletion
policy.
> Replication filecontent returns HTTP 500 when the commit reservation expires
> ----------------------------------------------------------------------------
>
> Key: SOLR-18406
> URL: https://issues.apache.org/jira/browse/SOLR-18406
> Project: Solr
> Issue Type: Bug
> Components: replication (java)
> Affects Versions: 8.11.4, 9.10.1
> Reporter: ZhenyuLi
> Priority: Major
>
> A follower can receive HTTP 500 while downloading an index if the leader
> deletes the requested index generation after filelist completes but before a
> subsequent filecontent request begins.
> Follower index replication consists of separate HTTP requests:
> indexversion
> -> filelist(generation=G)
> -> filecontent(file=X, generation=G)
> -> filecontent(file=Y, generation=G)
> -> ...
> The leader must retain commit G across these requests.
> While a filecontent request is active, the commit is protected with
> saveCommitPoint()/releaseCommitPoint(). Between requests, however, it is only
> protected by a timed reservation. The default commitReserveDuration is 10
> seconds.
> If the follower takes longer than that interval before requesting the next
> file, the reservation expires. A later hard commit on the leader invokes the
> index deletion policy and may delete G. The next filecontent request then
> calls
> saveCommitPoint(G), which throws IllegalStateException.
> This exception is currently exposed as HTTP 500.
> The same stale-generation condition is already handled explicitly by the
> filelist path. filelist catches IllegalStateException from
> getAndSaveCommitPoint(G) and reports:invalid index generation
> The filecontent path does not provide equivalent handling.
> The relevant difference is approximately:
> ReplicationHandler.getFileList():
> try
> { commit = delPol.getAndSaveCommitPoint(gen); }
> catch (IllegalStateException ignored)
> { // handled as an invalid generation below }
> ReplicationHandler.DirectoryFileStream.initWrite():
> if (indexGen != null)
> { delPolicy.saveCommitPoint(indexGen); }
> The equivalent behavior is present in the newer replication implementation.
> Reproduction
> ============
> The issue does not require a large index. One committed document is
> sufficient.
> 1. Create and hard-commit one document on the leader, producing generation G.
> 2. Submit another update or delete without committing it.
> 3. Start an explicit fetchindex request on the follower.
> 4. Allow the follower to complete indexversion and filelist(G).
> 5. Pause before the first filecontent(G) request.
> 6. Wait until the reservation for G has expired.
> 7. Hard-commit the pending leader update, producing a newer generation and
> allowing the deletion policy to delete G.
> 8. Resume the follower so that it requests filecontent(G).
> The hard commit must occur after the reservation expires. Expiration alone
> does not delete the commit; the subsequent leader commit is what invokes the
> deletion policy.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
