luoluoyuyu opened a new pull request, #19184:
URL: https://github.com/apache/iotdb/pull/19184

   ## Description
   
   This PR completes and supersedes 
[apache/iotdb#19032](https://github.com/apache/iotdb/pull/19032) (by 
@Caideyipi). #19032's head branch lives inside `apache/iotdb`, so the follow-up 
work cannot be added to it from a fork; this PR carries the whole change 
instead: commit 1 is the original fix (content unchanged, authorship 
preserved), commit 2 fixes the defects that remain in the splitter, commit 3 is 
the regression integration test. Please review this PR instead of #19032 and 
close #19032 if the combined change is acceptable.
   
   ### Problem
   
   LOAD splits a TsFile by time partition whenever a device spans more than one 
partition slot (`LoadSingleTsFileNode#needDecodeTsFile`), and the splitter then 
re-encodes the value chunks of a batched aligned chunk group. That is the 
layout the compaction writer produces: the series batches behind the first one 
are standalone `VALUE_CHUNK` sections that follow all time chunks of the chunk 
group, and `BatchedAlignedValueChunkData` rebuilds one chunk per (device, time 
partition) out of the decoded pages.
   
   Rebuilding such a chunk has to preserve the row and page positions of the 
source chunk, because an aligned value chunk is read page by page in lockstep 
with its time chunk: a null value still occupies one bit of the value page 
bitmap, and an all-null page still occupies its page slot. The original code 
assumed compact data and compact page slots, so LOAD of such a file either 
returned wrong values, dropped rows, or produced an unreadable page stream.
   
   ### Content
   
   **1. Original fix (#19032).** Null rows keep their bitmap position instead 
of shifting later rows left; values are encoded and their statistics computed 
with the real timestamp instead of a synthetic one; decoded pages are rebuilt 
into the chunk instead of being dropped.
   
   **2. Follow-up fixes.**
   
   - Page slots are matched against the partition range by page, not by "the 
chunk buffered something": `hasPageWithRows(chunkPages, startTime, endTime)` 
replaces the coarse `hasBufferedData` check, so a partition whose rows all live 
in some pages of the chunk gets exactly those pages, and a page that is 
entirely null keeps its slot.
   - A copied empty page is re-emitted with `PageHeader(0, 0, null)`; with 
statistics attached, `ValueChunkWriter` writes a compressed-size varint after a 
zero-length body, so the empty page takes 2 bytes instead of the single `0` 
varint the format expects and every following page boundary shifts.
   - The decoded loop tolerates a batch array shorter than the page (`j >= 
page.valueBatch.length`) and seals a page only when it actually received rows; 
a chunk that has rows in the partition but no data page emits one empty page 
slot per time page, matching `AlignedChunkData#hasPageWithData`. A chunk 
without any row keeps the zero-page layout (`EMPTY_CHUNK_DATA`), because the 
metadata index aligns measurements by chunk-group position and dropping the 
chunk entirely would break that alignment.
   
   **3. Tests.**
   
   - 
`TsFileSplitterTest#testSplitBatchedAlignedChunkPreservesCopiedPageBeforeDecodedPage`
 — a copied page followed by a decoded page in one partition.
   - 
`TsFileSplitterTest#testSplitBatchedAlignedChunkPreservesEmptyPageAfterDataPages`
 — an all-null page behind data pages.
   - Both assert the rebuilt chunk against its header (`getDataSize() == 
getData().remaining()`) so a mis-sized page cannot pass.
   - `IoTDBLoadTsFileIT#testLoadSparseBatchedAlignedTsFile` — builds the 
compaction layout with `TsFileIOWriter` (a time chunk with `f1`, plus `f2` as a 
standalone `ValueChunkWriter` chunk behind it) whose three pages are: a page 
inside one partition (copied), a page crossing a partition boundary with one 
null row, and an entirely null page. The rows span three time partitions of the 
interval the IT configures, so LOAD decodes and splits the file, writes three 
new TsFiles, and the test then checks every row's timestamp, `f1`, and `f2` 
including the null rows.
   
   <hr>
   
   This PR has:
   - [x] been self-reviewed.
   - [x] added comments explaining the "why" and the intent of the code 
wherever would not be obvious for an unfamiliar reader.
   - [x] added unit tests or modified existing tests to cover new code paths, 
ensuring the threshold for code coverage.
   - [x] added integration tests.
   
   ### Validation
   
   - `mvn spotless:apply -pl iotdb-core/datanode` and `-pl integration-test -P 
with-integration-tests`: clean (`spotless:check` reports 0 files to change).
   - `mvn validate -pl integration-test -P with-integration-tests`: 0 
Checkstyle violations.
   - The IT's TsFile fixture was verified outside the cluster: writing it and 
reading it back with `TsFileReader` returns the 11 expected rows with the 
expected nulls. This is how the fixture's missing `endFile()` was caught, so 
the file the IT loads is known to be well formed.
   - Not executed here: `TsFileSplitterTest` and the IT itself. The local 
environment has pre-existing compilation errors in unrelated datanode test 
sources (stale `~/.m2` artifacts) and no `thrift` compiler for `-am` builds, so 
neither can be started locally; the affected files were verified with `javac` 
against the module sourcepath instead. CI is expected to run 
`TsFileSplitterTest` and `IoTDBLoadTsFileIT` for this PR.
   
   <hr>
   
   ##### Key changed/added classes (or packages if there are too many classes) 
in this PR
   
   - 
`org.apache.iotdb.db.storageengine.load.splitter.BatchedAlignedValueChunkData`
   - `org.apache.iotdb.db.storageengine.load.TsFileSplitterTest`
   - `org.apache.iotdb.relational.it.db.it.IoTDBLoadTsFileIT`
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to