Caideyipi opened a new pull request, #19282: URL: https://github.com/apache/iotdb/pull/19282
## Description Pipe's TsFile query parser converts each primitive `Field` to a boxed object before storing it in a Tablet. Raw Tablet batches also encode their payloads through `PublicBAOS` and `DataOutputStream`. This change copies primitive fields directly into Tablet columns and writes both tree-model and merged table-model batches into exactly sized ByteBuffers. The existing null/default conversions, DATE representation compatibility, OBJECT fallback, deletion handling, and request wire format are preserved. For example, if timestamp 0 is fully deleted and timestamps 1 and 2 contain `(11, null)` and `(12, "value")`, the output still contains those two rows with the same null bitmap, even though the deleted row's slot is reused. Aligned and non-aligned tree batches and merged table batches retain byte-for-byte equivalence with stream serialization, including NaNs and signed zero. Dependency: [apache/tsfile#1101](https://github.com/apache/tsfile/pull/1101), specifically `eaaad099d6a29bff60cc374ad0ff30fad6b40e3d`, supplies `Tablet.serialize(ByteBuffer)` and matching NaN encoding. The POM selects that source's `2.5.1-SNAPSHOT` coordinates. This dependency must be integrated and made available before this PR can merge; the local checks used artifacts built from the specified source revision. ## Performance The following local JMH measurements compare the IoTDB implementation before and after these changes. Both variants use the **same TsFile revision** above, so these measurements isolate the IoTDB changes and do not attribute the results to changes between TsFile versions. They were collected on November 13 on the earlier development snapshot, before transplanting the same production-file changes onto public `master`; timings were not rerun on the final publication base. | Operation / input | Baseline ms/op | Optimized ms/op | Mean latency reduction | Allocated bytes/op reduction | | --- | ---: | ---: | ---: | ---: | | QueryParser, 64 INT64 columns | 336.106 | 232.381 | 30.9% | 13.1% | | QueryParser, 256 INT64 columns | 1,439.066 | 1,140.021 | 20.8% | 12.9% | | Tree-model batch, mostly INT64 | 1.435 | 0.417 | 71.0% | 0.3% | | Tree-model batch, mixed types with nulls | 2.239 | 0.699 | 68.8% | 0.4% | | Merged table-model batch, mostly INT64 | 2.117 | 1.670 | 21.1% | 0.1% | | Merged table-model batch, mixed types with nulls | 3.412 | 2.411 | 29.4% | 0.1% | - Windows x86_64, Intel Core i9-13900H, JDK 17.0.15, JMH 1.37; one benchmark thread, `-Xms1g -Xmx1g -XX:ActiveProcessorCount=4`. - Each variant used two independent JVM forks, each with three 1-second warmup iterations and five 1-second measurement iterations. Allocation is JMH GCProfiler's normalized allocated bytes per operation. - QueryParser read one aligned device with 16,384 rows and either 64 or 256 INT64 columns (1,048,576 or 4,194,304 points per operation). Input generation was excluded, reads used a warm OS cache, and returned values and timestamps were consumed. - Batch inputs used 64 columns and 1,024 rows per Tablet. Mostly-INT64 input had one STRING column and 63 INT64 columns. Mixed input covered BOOLEAN, INT32, INT64, FLOAT, DOUBLE, DATE, STRING, and BLOB, with about 14% nulls outside the first column. Tree batches used one Tablet; merged table batches used two. Timing included production batch construction, memory accounting, merge/copy where applicable, final request encoding, and batch reset. Stream-format byte equality was checked outside timed regions. These are preliminary component measurements on a shared workstation. The JMH 99.9% timing confidence intervals overlap for the 256-column QueryParser and both merged table-model cases, so their percentage changes should not be treated as established, stable speedups. End-to-end sender/network/receiver Pipe throughput and distribution packaging were not tested. ## Validation On public `master` base `ea7b4ffc5d68021f18a18f627218ab625eee7f7a`, the required 28-module dependency closure was rebuilt from clean source, and the focused reactor test run completed with **16 tests passed, zero failures, errors, or skips** (Windows, JDK 17.0.15, Maven 3.9.12). This includes all five added regression tests. The deletion fixture uses the public branch's existing PatternTreeMap factory; its deletion records and assertions are unchanged. The original implementation snapshot passed 19 selected local tests. All five newly added tests have been run locally. Verification temporarily disabled build-scan extensions; that configuration change is excluded from the submitted diff. ## Checklist - [x] Independently self-reviewed once with no actionable findings in the original implementation snapshot; publication preserves the reviewed production-file contents. - [x] Added unit tests covering direct Field copy, deleted-row slot reuse, and complete request byte compatibility. - [x] Measured local component performance with the limitations stated above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
