Caideyipi opened a new pull request, #18088:
URL: https://github.com/apache/iotdb/pull/18088
## Description
This PR adds a new historical pipe source option:
- `extractor.history.tsfile.order-by-flush-time`
- `source.history.tsfile.order-by-flush-time`
- default: `true`
When enabled, historical TsFile-only extraction sends selected TsFiles in
source-side file creation / flush-time order instead of progressIndex order.
The goal is to preserve overwrite semantics for duplicated timestamps: older
TsFiles are transferred first, and newer TsFiles are transferred later so the
receiver can overwrite older values with newer values.
## Semantic Changes
For historical TsFile extraction where insertions are captured and deletions
are not captured:
1. Historical working TsFile processors are closed synchronously before
extraction, so the source has a stable set of TsFiles to order.
2. Selected TsFiles are sorted by:
- sequence files before unsequence files,
- file-name timestamp / flush time ascending,
- file version ascending,
- compaction version ascending,
- file path as a deterministic tie-breaker.
3. TsFiles filtered out by progress, time, path, delete status,
pipe-generated status, or pin failure are removed before computing delayed
progress. Their progressIndex is not reported by this historical flush-time
path.
4. Because flush-time order is not guaranteed to be compatible with
progressIndex topological order, per-TsFile commit progress reporting is
disabled for reordered historical TsFile events.
5. Tablet events generated from those TsFile events inherit the no-report
behavior, so decomposing a TsFile into tablets does not accidentally advance
progress early.
6. After all selected reordered historical TsFiles are supplied, the source
emits one `ProgressReportEvent` with the max progressIndex of the selected
resources.
When historical deletions are captured together with insertions, the source
keeps the previous progressIndex ordering. Deletion resources only carry
progressIndex ordering information, so this avoids changing insertion/deletion
ordering semantics.
The option can be set to `false` to keep the previous progressIndex-based
ordering.
## Tests / Coverage
Added unit coverage for:
- the new option defaulting to `true`,
- sorting older flush-time TsFiles before newer flush-time TsFiles,
- explicitly disabling the new option and falling back to progressIndex
order,
- delaying progress reporting until all reordered historical resources are
consumed,
- excluding filtered-out TsFiles from the delayed max progressIndex,
- preserving the no-progress-report flag when a TsFile event is
shallow-copied,
- making generated tablet events inherit the source TsFile event's
no-progress-report behavior.
I did not add an integration test because ordinary end-to-end write/flush
scenarios usually produce progressIndex order that matches flush-time order, so
such an IT would not deterministically cover the regression. The deterministic
behavior that matters here is the historical source's resource ordering and
progress reporting contract, which is covered by unit tests with explicit
TsFileResource ordering and progressIndex setup.
Local verification:
- `mvn spotless:apply -pl iotdb-core/datanode`
- `mvn spotless:apply -pl iotdb-core/node-commons`
- `mvn "-Ddevelocity.off=true" -o install -pl iotdb-core/node-commons
"-DskipTests=true" "-Dcheckstyle.skip=true" "-Dspotless.check.skip=true"`
- `git diff --check origin/master..HEAD`
Attempted datanode targeted UT execution locally, but the module currently
fails during main compilation in this checkout before reaching the test phase
due to unrelated generated-source / thrift cache errors such as missing
`TShowRepairDataPartitionTableProgressResp`, `IFill`, and `Accumulator` classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
