Caideyipi opened a new pull request, #19282:
URL: https://github.com/apache/iotdb/pull/19282

   ## Description
   
   Pipe's TsFile query parser converts each primitive `Field` to a boxed object 
before storing it in a Tablet. Raw Tablet batches also encode their payloads 
through `PublicBAOS` and `DataOutputStream`. This change copies primitive 
fields directly into Tablet columns and writes both tree-model and merged 
table-model batches into exactly sized ByteBuffers.
   
   The existing null/default conversions, DATE representation compatibility, 
OBJECT fallback, deletion handling, and request wire format are preserved. For 
example, if timestamp 0 is fully deleted and timestamps 1 and 2 contain `(11, 
null)` and `(12, "value")`, the output still contains those two rows with the 
same null bitmap, even though the deleted row's slot is reused. Aligned and 
non-aligned tree batches and merged table batches retain byte-for-byte 
equivalence with stream serialization, including NaNs and signed zero.
   
   Dependency: 
[apache/tsfile#1101](https://github.com/apache/tsfile/pull/1101), specifically 
`eaaad099d6a29bff60cc374ad0ff30fad6b40e3d`, supplies 
`Tablet.serialize(ByteBuffer)` and matching NaN encoding. The POM selects that 
source's `2.5.1-SNAPSHOT` coordinates. This dependency must be integrated and 
made available before this PR can merge; the local checks used artifacts built 
from the specified source revision.
   
   ## Performance
   
   The following local JMH measurements compare the IoTDB implementation before 
and after these changes. Both variants use the **same TsFile revision** above, 
so these measurements isolate the IoTDB changes and do not attribute the 
results to changes between TsFile versions. They were collected on November 13 
on the earlier development snapshot, before transplanting the same 
production-file changes onto public `master`; timings were not rerun on the 
final publication base.
   
   | Operation / input | Baseline ms/op | Optimized ms/op | Mean latency 
reduction | Allocated bytes/op reduction |
   | --- | ---: | ---: | ---: | ---: |
   | QueryParser, 64 INT64 columns | 336.106 | 232.381 | 30.9% | 13.1% |
   | QueryParser, 256 INT64 columns | 1,439.066 | 1,140.021 | 20.8% | 12.9% |
   | Tree-model batch, mostly INT64 | 1.435 | 0.417 | 71.0% | 0.3% |
   | Tree-model batch, mixed types with nulls | 2.239 | 0.699 | 68.8% | 0.4% |
   | Merged table-model batch, mostly INT64 | 2.117 | 1.670 | 21.1% | 0.1% |
   | Merged table-model batch, mixed types with nulls | 3.412 | 2.411 | 29.4% | 
0.1% |
   
   - Windows x86_64, Intel Core i9-13900H, JDK 17.0.15, JMH 1.37; one benchmark 
thread, `-Xms1g -Xmx1g -XX:ActiveProcessorCount=4`.
   - Each variant used two independent JVM forks, each with three 1-second 
warmup iterations and five 1-second measurement iterations. Allocation is JMH 
GCProfiler's normalized allocated bytes per operation.
   - QueryParser read one aligned device with 16,384 rows and either 64 or 256 
INT64 columns (1,048,576 or 4,194,304 points per operation). Input generation 
was excluded, reads used a warm OS cache, and returned values and timestamps 
were consumed.
   - Batch inputs used 64 columns and 1,024 rows per Tablet. Mostly-INT64 input 
had one STRING column and 63 INT64 columns. Mixed input covered BOOLEAN, INT32, 
INT64, FLOAT, DOUBLE, DATE, STRING, and BLOB, with about 14% nulls outside the 
first column. Tree batches used one Tablet; merged table batches used two. 
Timing included production batch construction, memory accounting, merge/copy 
where applicable, final request encoding, and batch reset. Stream-format byte 
equality was checked outside timed regions.
   
   These are preliminary component measurements on a shared workstation. The 
JMH 99.9% timing confidence intervals overlap for the 256-column QueryParser 
and both merged table-model cases, so their percentage changes should not be 
treated as established, stable speedups. End-to-end sender/network/receiver 
Pipe throughput and distribution packaging were not tested.
   
   ## Validation
   
   On public `master` base `ea7b4ffc5d68021f18a18f627218ab625eee7f7a`, the 
required 28-module dependency closure was rebuilt from clean source, and the 
focused reactor test run completed with **16 tests passed, zero failures, 
errors, or skips** (Windows, JDK 17.0.15, Maven 3.9.12). This includes all five 
added regression tests. The deletion fixture uses the public branch's existing 
PatternTreeMap factory; its deletion records and assertions are unchanged.
   
   The original implementation snapshot passed 19 selected local tests. All 
five newly added tests have been run locally. Verification temporarily disabled 
build-scan extensions; that configuration change is excluded from the submitted 
diff.
   
   ## Checklist
   
   - [x] Independently self-reviewed once with no actionable findings in the 
original implementation snapshot; publication preserves the reviewed 
production-file contents.
   - [x] Added unit tests covering direct Field copy, deleted-row slot reuse, 
and complete request byte compatibility.
   - [x] Measured local component performance with the limitations stated above.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to