Caideyipi opened a new pull request, #18704:
URL: https://github.com/apache/iotdb/pull/18704
## Description
### Problem
Closing a subscription consumer can deadlock with a consumer-group metadata
push on a DataNode:
1. The metadata push holds the consumer-group metadata write lock while it
closes consensus prefetch queues and waits for in-flight prefetch work to
finish.
2. Event materialization in the prefetch path calls
`SubscriptionAgent.consumer().isTableModel(consumerGroupId)`, which tries to
acquire the same metadata read lock.
3. The metadata push and prefetch task then wait for each other. The
ConfigNode `DropSubscriptionProcedure` times out and remains stuck, while the
client blocks waiting for the close response.
This was identified from the stable subscription regression represented by
internal test case 1379/result 27213.
### Fix
Derive the data model from the converter's immutable tree/table conversion
pattern and cache it when `ConsensusPrefetchingQueue` is constructed. Event
materialization now uses the cached value when obtaining the column filter
matcher, so it no longer reads consumer-group metadata while queue shutdown is
waiting for prefetch completion.
This preserves synchronous queue shutdown and the existing subscription
procedure consistency semantics while removing the lock cycle.
### Tests
Added unit coverage for:
- deriving table/tree model from the immutable converter pattern;
- materializing a prefetched event while the real consumer metadata write
lock is held, proving that prefetch no longer depends on the metadata read lock.
Validated with:
```text
mvn test -pl iotdb-core/datanode -am \
-Dtest=ConsensusLogToTabletConverterTest#testDataModelIsDerivedFromImmutableConversionPattern,ConsensusPrefetchingQueueTest#testPrefetchDoesNotDependOnConsumerMetaReadLock
\
-Dsurefire.failIfNoSpecifiedTests=false \
-DfailIfNoTests=false
```
Result: 2 tests run, 0 failures, 0 errors; full reactor build successful.
<hr>
This PR has:
- [x] been self-reviewed.
- [x] concurrent read and write
- [x] added comments explaining the why and the intent of the code wherever
it would not be obvious for an unfamiliar reader.
- [x] added unit tests to cover the new code path.
<hr>
##### Key changed/added classes
- `ConsensusLogToTabletConverter`
- `ConsensusPrefetchingQueue`
- `ConsensusLogToTabletConverterTest`
- `ConsensusPrefetchingQueueTest`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]