Re: [PR] Pipe: order historical TsFiles by flush time [iotdb]
codecov[bot] commented on PR #18088: URL: https://github.com/apache/iotdb/pull/18088#issuecomment-4861947288 ## [Codecov](https://app.codecov.io/gh/apache/iotdb/pull/18088?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report :x: Patch coverage is `75.24752%` with `25 lines` in your changes missing coverage. Please review. :white_check_mark: Project coverage is 41.66%. Comparing base ([`220e7a3`](https://app.codecov.io/gh/apache/iotdb/commit/220e7a319c283deeaa27a25bccdad82a2d2e18db?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`3a32f1e`](https://app.codecov.io/gh/apache/iotdb/commit/3a32f1ea9653af3b9a8b7c3383148d9759492af8?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)). | [Files with missing lines](https://app.codecov.io/gh/apache/iotdb/pull/18088?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines | |---|---|---| | [...peHistoricalDataRegionTsFileAndDeletionSource.java](https://app.codecov.io/gh/apache/iotdb/pull/18088?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fpipe%2Fsource%2Fdataregion%2Fhistorical%2FPipeHistoricalDataRegionTsFileAndDeletionSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcGlwZS9zb3VyY2UvZGF0YXJlZ2lvbi9oaXN0b3JpY2FsL1BpcGVIaXN0b3JpY2FsRGF0YVJlZ2lvblRzRmlsZUFuZERlbGV0aW9uU291cmNlLmphdmE=) | 70.51% | [23 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/18088?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | | [.../event/common/tsfile/PipeTsFileInsertionEvent.java](https://app.codecov.io/gh/apache/iotdb/pull/18088?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fpipe%2Fevent%2Fcommon%2Ftsfile%2FPipeTsFileInsertionEvent.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcGlwZS9ldmVudC9jb21tb24vdHNmaWxlL1BpcGVUc0ZpbGVJbnNlcnRpb25FdmVudC5qYXZh) | 77.77% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/18088?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Additional details and impacted files ```diff @@Coverage Diff@@ ## master #18088 +/- ## = Coverage 41.65% 41.66% Complexity 318 318 = Files 5296 5296 Lines371663 371744 +81 Branches 4808848103 +15 = + Hits 154819 154889 +70 - Misses 216844 216855 +11 ``` [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/iotdb/pull/18088?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache). :rocket: New features to boost your workflow: - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Pipe: order historical TsFiles by flush time [iotdb]
sonarqubecloud[bot] commented on PR #18088: URL: https://github.com/apache/iotdb/pull/18088#issuecomment-4861757029 ## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=18088) **Quality Gate passed** Issues  [5 New issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=18088&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=18088&issueStatuses=ACCEPTED) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb&pullRequest=18088&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=18088&metric=new_coverage&view=list)  [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=18088&metric=new_duplicated_lines_density&view=list) [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=18088) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
[PR] Pipe: order historical TsFiles by flush time [iotdb]
Caideyipi opened a new pull request, #18088: URL: https://github.com/apache/iotdb/pull/18088 ## Description This PR adds a new historical pipe source option: - `extractor.history.tsfile.order-by-flush-time` - `source.history.tsfile.order-by-flush-time` - default: `true` When enabled, historical TsFile-only extraction sends selected TsFiles in source-side file creation / flush-time order instead of progressIndex order. The goal is to preserve overwrite semantics for duplicated timestamps: older TsFiles are transferred first, and newer TsFiles are transferred later so the receiver can overwrite older values with newer values. ## Semantic Changes For historical TsFile extraction where insertions are captured and deletions are not captured: 1. Historical working TsFile processors are closed synchronously before extraction, so the source has a stable set of TsFiles to order. 2. Selected TsFiles are sorted by: - sequence files before unsequence files, - file-name timestamp / flush time ascending, - file version ascending, - compaction version ascending, - file path as a deterministic tie-breaker. 3. TsFiles filtered out by progress, time, path, delete status, pipe-generated status, or pin failure are removed before computing delayed progress. Their progressIndex is not reported by this historical flush-time path. 4. Because flush-time order is not guaranteed to be compatible with progressIndex topological order, per-TsFile commit progress reporting is disabled for reordered historical TsFile events. 5. Tablet events generated from those TsFile events inherit the no-report behavior, so decomposing a TsFile into tablets does not accidentally advance progress early. 6. After all selected reordered historical TsFiles are supplied, the source emits one `ProgressReportEvent` with the max progressIndex of the selected resources. When historical deletions are captured together with insertions, the source keeps the previous progressIndex ordering. Deletion resources only carry progressIndex ordering information, so this avoids changing insertion/deletion ordering semantics. The option can be set to `false` to keep the previous progressIndex-based ordering. ## Tests / Coverage Added unit coverage for: - the new option defaulting to `true`, - sorting older flush-time TsFiles before newer flush-time TsFiles, - explicitly disabling the new option and falling back to progressIndex order, - delaying progress reporting until all reordered historical resources are consumed, - excluding filtered-out TsFiles from the delayed max progressIndex, - preserving the no-progress-report flag when a TsFile event is shallow-copied, - making generated tablet events inherit the source TsFile event's no-progress-report behavior. I did not add an integration test because ordinary end-to-end write/flush scenarios usually produce progressIndex order that matches flush-time order, so such an IT would not deterministically cover the regression. The deterministic behavior that matters here is the historical source's resource ordering and progress reporting contract, which is covered by unit tests with explicit TsFileResource ordering and progressIndex setup. Local verification: - `mvn spotless:apply -pl iotdb-core/datanode` - `mvn spotless:apply -pl iotdb-core/node-commons` - `mvn "-Ddevelocity.off=true" -o install -pl iotdb-core/node-commons "-DskipTests=true" "-Dcheckstyle.skip=true" "-Dspotless.check.skip=true"` - `git diff --check origin/master..HEAD` Attempted datanode targeted UT execution locally, but the module currently fails during main compilation in this checkout before reaching the test phase due to unrelated generated-source / thrift cache errors such as missing `TShowRepairDataPartitionTableProgressResp`, `IFill`, and `Accumulator` classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
