Pengzna opened a new pull request, #17977:
URL: https://github.com/apache/iotdb/pull/17977
## Description
This PR mitigates IoTConsensusV2 table-model wide-table sync stalls observed
when a workload has many columns and 10 pipe devices. In the reported case,
both stream and batch modes could not finish within 30 minutes, and stream mode
still did not finish after 5 hours.
The code-path analysis is documented in
`iotconsensusv2_wide_table_sync_analysis.md`.
## Changes
- Force IoTConsensusV2 consensus-pipe table-model wide or large
`PipeInsertNodeTabletInsertionEvent`s to downgrade from realtime tablet
extraction to TsFile extraction.
- Add two tuning configs:
- `pipe_realtime_iot_consensus_v2_force_tsfile_measurement_count_threshold`
- `pipe_realtime_iot_consensus_v2_force_tsfile_memory_threshold_in_bytes`
- Move table-model table disk-usage calculation for IoTConsensusV2 receiver
TsFile loads out of the ordered receiver apply path.
- Reuse `TableDiskUsageIndex`'s background worker for async table-size
calculation, with non-blocking enqueue for this new statistic task.
- Add an analysis document covering the background, root-cause analysis,
implemented fixes, validation plan, and follow-up work.
## Validation
- `JAVA_HOME=$(/usr/libexec/java_home -v 17) mvn spotless:apply -pl
iotdb-core/datanode,iotdb-core/node-commons`
- `JAVA_HOME=$(/usr/libexec/java_home -v 17) mvn compile -pl
iotdb-core/node-commons,iotdb-core/datanode -DskipTests`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
