luoluoyuyu opened a new pull request, #19184: URL: https://github.com/apache/iotdb/pull/19184
## Description This PR completes and supersedes [apache/iotdb#19032](https://github.com/apache/iotdb/pull/19032) (by @Caideyipi). #19032's head branch lives inside `apache/iotdb`, so the follow-up work cannot be added to it from a fork; this PR carries the whole change instead: commit 1 is the original fix (content unchanged, authorship preserved), commit 2 fixes the defects that remain in the splitter, commit 3 is the regression integration test. Please review this PR instead of #19032 and close #19032 if the combined change is acceptable. ### Problem LOAD splits a TsFile by time partition whenever a device spans more than one partition slot (`LoadSingleTsFileNode#needDecodeTsFile`), and the splitter then re-encodes the value chunks of a batched aligned chunk group. That is the layout the compaction writer produces: the series batches behind the first one are standalone `VALUE_CHUNK` sections that follow all time chunks of the chunk group, and `BatchedAlignedValueChunkData` rebuilds one chunk per (device, time partition) out of the decoded pages. Rebuilding such a chunk has to preserve the row and page positions of the source chunk, because an aligned value chunk is read page by page in lockstep with its time chunk: a null value still occupies one bit of the value page bitmap, and an all-null page still occupies its page slot. The original code assumed compact data and compact page slots, so LOAD of such a file either returned wrong values, dropped rows, or produced an unreadable page stream. ### Content **1. Original fix (#19032).** Null rows keep their bitmap position instead of shifting later rows left; values are encoded and their statistics computed with the real timestamp instead of a synthetic one; decoded pages are rebuilt into the chunk instead of being dropped. **2. Follow-up fixes.** - Page slots are matched against the partition range by page, not by "the chunk buffered something": `hasPageWithRows(chunkPages, startTime, endTime)` replaces the coarse `hasBufferedData` check, so a partition whose rows all live in some pages of the chunk gets exactly those pages, and a page that is entirely null keeps its slot. - A copied empty page is re-emitted with `PageHeader(0, 0, null)`; with statistics attached, `ValueChunkWriter` writes a compressed-size varint after a zero-length body, so the empty page takes 2 bytes instead of the single `0` varint the format expects and every following page boundary shifts. - The decoded loop tolerates a batch array shorter than the page (`j >= page.valueBatch.length`) and seals a page only when it actually received rows; a chunk that has rows in the partition but no data page emits one empty page slot per time page, matching `AlignedChunkData#hasPageWithData`. A chunk without any row keeps the zero-page layout (`EMPTY_CHUNK_DATA`), because the metadata index aligns measurements by chunk-group position and dropping the chunk entirely would break that alignment. **3. Tests.** - `TsFileSplitterTest#testSplitBatchedAlignedChunkPreservesCopiedPageBeforeDecodedPage` — a copied page followed by a decoded page in one partition. - `TsFileSplitterTest#testSplitBatchedAlignedChunkPreservesEmptyPageAfterDataPages` — an all-null page behind data pages. - Both assert the rebuilt chunk against its header (`getDataSize() == getData().remaining()`) so a mis-sized page cannot pass. - `IoTDBLoadTsFileIT#testLoadSparseBatchedAlignedTsFile` — builds the compaction layout with `TsFileIOWriter` (a time chunk with `f1`, plus `f2` as a standalone `ValueChunkWriter` chunk behind it) whose three pages are: a page inside one partition (copied), a page crossing a partition boundary with one null row, and an entirely null page. The rows span three time partitions of the interval the IT configures, so LOAD decodes and splits the file, writes three new TsFiles, and the test then checks every row's timestamp, `f1`, and `f2` including the null rows. <hr> This PR has: - [x] been self-reviewed. - [x] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. - [x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage. - [x] added integration tests. ### Validation - `mvn spotless:apply -pl iotdb-core/datanode` and `-pl integration-test -P with-integration-tests`: clean (`spotless:check` reports 0 files to change). - `mvn validate -pl integration-test -P with-integration-tests`: 0 Checkstyle violations. - The IT's TsFile fixture was verified outside the cluster: writing it and reading it back with `TsFileReader` returns the 11 expected rows with the expected nulls. This is how the fixture's missing `endFile()` was caught, so the file the IT loads is known to be well formed. - Not executed here: `TsFileSplitterTest` and the IT itself. The local environment has pre-existing compilation errors in unrelated datanode test sources (stale `~/.m2` artifacts) and no `thrift` compiler for `-am` builds, so neither can be started locally; the affected files were verified with `javac` against the module sourcepath instead. CI is expected to run `TsFileSplitterTest` and `IoTDBLoadTsFileIT` for this PR. <hr> ##### Key changed/added classes (or packages if there are too many classes) in this PR - `org.apache.iotdb.db.storageengine.load.splitter.BatchedAlignedValueChunkData` - `org.apache.iotdb.db.storageengine.load.TsFileSplitterTest` - `org.apache.iotdb.relational.it.db.it.IoTDBLoadTsFileIT` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
