Caideyipi opened a new pull request, #18010: URL: https://github.com/apache/iotdb/pull/18010
## Description This PR improves the pipe TsFile scan parser for legal aligned TsFiles whose value chunks are physically written in column batches, such as files produced by batched aligned compaction. The current scan parser emits an aligned tablet when the value chunk occurrence index changes. For batched aligned compaction output, value chunks can be laid out as: - time chunk 0, time chunk 1 - value columns 0-9 for chunk 0 and chunk 1 - value columns 10-19 for chunk 0 and chunk 1 - ... This layout is valid, but the previous parser behavior makes the emitted tablets inherit the physical compaction batch width, commonly 10 columns from `compaction_max_aligned_series_num_in_one_batch`, even when pipe reader memory allows a wider aligned tablet. That increases the number of tablets and hurts pipe performance. This PR changes the scan parser to cache pending aligned value chunk groups by time chunk index and emit them only when memory limits or chunk group boundaries require it. With enough memory, consecutive physical value column batches for the same aligned chunks are merged into wider aligned tablets instead of being split at the compaction batch boundary. It also defines `pipeDataStructureTabletRowSize <= 0` as disabling the row-count cap for pipe tablets. In that mode, tablet row count is calculated only from `pipe_data_structure_tablet_size_in_bytes`, so users can rely on the memory-size limit instead of the fixed row-count limit. ## Changes - Cache aligned value chunks in pending groups keyed by time chunk index in `TsFileInsertionEventScanParser`. - Preserve chunk/page memory protection when merging multiple physical aligned value chunk groups. - Keep cached value chunk replay subject to the same memory threshold checks. - Treat non-positive `pipeDataStructureTabletRowSize` as no row-count cap in `PipeMemoryWeightUtil`. - Add tests for batched aligned value chunk layout and disabling the tablet row-size cap. ## Validation - `mvn spotless:apply -pl iotdb-core/datanode` - `git diff --check` I also tried: - `mvn -Ddevelocity.off=true -pl iotdb-core/datanode -DskipTests compile` - `mvn -Ddevelocity.off=true -Dmaven.main.skip=true -pl iotdb-core/datanode -Dtest=TsFileInsertionEventParserTest#testScanParserMergesBatchedAlignedValueChunkGroups+testPipeTabletRowSizeCanBeDisabledByNonPositiveValue test` Both Maven compile/test attempts are blocked in this workspace by existing datanode-wide compile issues outside this PR, including generated query fill/aggregation classes and `IOUtils.readFully` unresolved symbols in unrelated files. The focused tests did not get executed because `testCompile` fails before Surefire runs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
