showuon opened a new pull request, #15947:
URL: https://github.com/apache/kafka/pull/15947
We use topicIdPartition as the key for each RLM task. It will cause
highestOffsetInRemoteStorage in log not updated after logDir alter completion.
The reproducing flow is like this:
1. tp-0 locating in dirA is the leader of the partition
2. tp-0 is altering logDir to dirB
3. tp-0 is copying segments to remote storage (note: the log in dirA)
4. The logDir altering for tp-0 is completed
5. `remoteLogManager#onLeadershipChange` is invoked, `copiedOffsetOption`
is reset to Optional.empty()
6. The copying segments to remote storage in step 3 for tp-0 is
completed, updating `copiedOffsetOption` to new offset, as well as the
`log#highestOffsetInRemoteStorage`. (note: the log in dirA)
7. In the next run of RLMTask, the log will be the one in target dir
(dirB), and the `log#highestOffsetInRemoteStorage` (dirB) will be the default
value (-1), which will block the log cleanup operation.
This PR makes sure after log dir changed, the RLMTask state will be reset.
Also adding a test for the above scenario.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
