Caideyipi opened a new pull request, #18399:
URL: https://github.com/apache/iotdb/pull/18399
## Description
### Retain WAL required by uncommitted consumers
Previously, subscription WAL reclamation only considered replication
progress and Topic retention policies. Consumer-group committed progress was
not part of the deletion boundary, so retention could reclaim WAL still
required by a lagging consumer.
This PR derives a file-level retained WAL version for each `(consumerGroup,
topic, region)` queue from its per-writer committed `RegionProgress`.
IoTConsensus combines these boundaries with Topic size/time retention and uses
the most conservative boundary.
### Multi-writer and leader migration handling
Committed progress remains represented by `(physicalTime, writerNodeId,
localSeq)` rather than local `searchIndex`. Rolled WAL V3 metadata is
summarized per writer, allowing follower WAL records and records produced
before leader migration to be protected correctly.
WAL files with incomplete or unsupported writer metadata are conservatively
retained.
### Boundary refresh and caching
The committed WAL boundary is refreshed after ACK, direct commit,
initialization, seek, committed-progress broadcast, or WAL roll. Per-file
writer-progress requirements are cached so repeated safe-delete checks do not
repeatedly scan unchanged WAL metadata.
<hr>
This PR has:
- [x] been self-reviewed.
- [x] concurrent read
- [x] concurrent write
- [x] concurrent read and write
- [x] added Javadocs for non-trivial behavior.
- [x] added comments explaining intent.
- [x] added unit tests covering new code paths.
### Tests
- `mvn spotless:apply -pl iotdb-core/consensus,iotdb-core/datanode`
- `mvn test -pl iotdb-core/consensus
-Dtest=SubscriptionWalRetentionCalculatorTest`
- `mvn test -pl iotdb-core/consensus,iotdb-core/datanode
-Dtest=ConsensusPrefetchingQueueTest#testWalFileCommitRequirementUsesPerWriterMaximum+testWalFileCommitRequirementRejectsUnsupportedWriterMetadata
-Dsurefire.failIfNoSpecifiedTests=false`
- `mvn test-compile -pl iotdb-core/consensus,iotdb-core/datanode -P
with-zh-locale -DskipTests`
<hr>
##### Key changed/added classes
- `SubscriptionWalRetentionCalculator`
- `SubscriptionQueueRegistry`
- `IoTConsensusServerImpl`
- `ConsensusPrefetchingQueue`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
