Caideyipi opened a new pull request, #18718:
URL: https://github.com/apache/iotdb/pull/18718
## Description
Add the incremental topic attribute `retain.progress.after.unsubscribe` to
preserve a consumer group's committed progress and the WAL files required to
resume from that progress after the group unsubscribes from a topic.
### Configuration and validation
- The attribute defaults to `false`, preserving the existing unsubscribe
behavior.
- Attribute names are matched case-insensitively. Values are trimmed and
must be `true` or `false`, also case-insensitively.
- The attribute is supported only by incremental topics, together with the
existing retention attributes.
- The attribute can be updated online. Changing it from `true` to `false`
removes retained state for groups that are no longer subscribed. The existing
`retention.bytes` and `retention.ms` attributes remain immutable after topic
creation.
### Progress and WAL lifecycle
- When the final consumer in a group unsubscribes, ConfigNode retains that
group's committed topic progress when the attribute is enabled.
- DataNode atomically replaces the active prefetching queue with detached
WAL retention. Detached retention does not receive live subscription requests
and only prevents deletion of WAL files needed by the committed progress.
- Resubscribing atomically replaces detached retention with an active queue
and resumes from the retained progress.
- During DataNode recovery, IoTConsensus installs a conservative retention
guard before the recovered peer starts. Subscription metadata recovery then
replaces that guard with the active queue, the final detached retention policy,
or removes it when the retained state is no longer valid.
- Dropping a topic, consumer group, or region always removes its progress
and WAL retention state. Setup rollback restores the startup retention guard.
### Tests
- `TopicConfigTest`: 9 tests
- `CommitProgressKeeperTest`: 14 tests
- `SubscriptionQueueRegistryTest`: 4 tests
- Targeted DataNode subscription tests: 32 tests
- Targeted ConfigNode subscription tests: 43 tests
- `mvn test-compile -DskipTests`: 53-module reactor, `BUILD SUCCESS`
- `mvn test-compile -DskipTests -P with-zh-locale`: 53-module reactor,
`BUILD SUCCESS`
<hr>
This PR has:
- [x] been self-reviewed.
- [x] concurrent read
- [x] concurrent write
- [x] concurrent read and write
- [x] added comments explaining the "why" and the intent of the code
wherever it would not be obvious to an unfamiliar reader.
- [x] added or modified unit tests to cover the new code paths.
<hr>
##### Key changed/added classes (or packages if there are too many classes)
in this PR
- `TopicConfig` and `TopicConstant`: expose the new topic attribute and its
default.
- `SubscriptionInfo` and `CommitProgressKeeper`: validate the attribute and
retain or clean ConfigNode progress.
- `ConsensusSubscriptionSetupHandler` and
`ConsensusSubscriptionWalRetention`: switch between active queues and detached
WAL retention, including recovery and rollback.
- `IoTConsensusServerImpl` and `SubscriptionQueueRegistry`: manage WAL
retention policies and their atomic lifecycle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]