CritasWang opened a new pull request, #16: URL: https://github.com/apache/iotdb-client-nodejs/pull/16
## Problem CPU profiling of an insertTablet-heavy workload (100 devices × 20 sensors × 1000-row batches, 20 clients) showed the event loop 79.5% busy, with `serializeTabletValues` at 26% self time and GC at 24.5% — the write path was dominated by allocation churn, not I/O: 1. `serializeTabletValues` transposed rows→columns (`values.map(row => row[col])` — a fresh array per column), built a `boolean[]` null bitmap per column even when the column had no nulls, serialized each column into its own intermediate buffer, and finally `Buffer.concat`-ed everything (a full payload re-copy). 2. INT64/TIMESTAMP writes allocated a `BigInt` per value for `writeBigInt64BE`. 3. TEXT/STRING columns re-encoded UTF-8 for every row, even constant TAG columns. 4. `BufferPool` was acquire-only (zero `release()` calls in `src/`) — 0% hit rate, pure overhead over `Buffer.allocUnsafe`. 5. `serializeBitMaps` allocated a 1-byte `Buffer.from([flag])` per column. ## Changes - **Single-pass, single-buffer tablet serialization** (`serializeTabletValuesFast`): pre-compute the exact payload size (fixed widths from schema; one `byteLength` scan for variable-width columns), allocate one buffer, write values column-by-column reading `values[row][col]` directly, and pack null bitmaps inline — flag byte stays 0 and no bitmap bytes are emitted when a column has no nulls. No transpose, no intermediate buffers, no `Buffer.concat`. - **BigInt-free int64 writes**: sign-correct hi/lo 32-bit pair (`hi = Math.floor(v / 2^32)`, `lo = v >>> 0`); BigInt inputs and non-safe integers still use `writeBigInt64BE`. - **String encode cache**: single-entry memo reuses the encoded buffer for consecutive identical strings (constant TAG columns). - **BufferPool deprecated**: removed from the write path (kept exported for API compatibility, marked `@deprecated`). The legacy path (`enableFastSerialization: false`) is unchanged. ## Results (IoTDB 2.0.10 standalone, 16-core box, DOUBLE workload, 10^9 points/run, interleaved A/B, 2 rounds) | Run | Throughput | Avg latency | P99 | |---|---|---|---| | baseline #1 | 8.70M pts/s | 26.83 ms | 78.11 ms | | optimized #1 | 13.11M pts/s | 10.60 ms | 23.61 ms | | baseline #2 | 8.53M pts/s | 27.07 ms | 78.90 ms | | optimized #2 | 13.16M pts/s | 10.53 ms | 23.18 ms | **+52% throughput, −61% average latency, −70% P99.** Server-side `count(*)` verified 100,000,000 rows for every run. Serializer self time dropped from 26% to 14.2% in `--cpu-prof`; the profiled process now shows 16% idle where it was previously saturated. ## Wire compatibility - All pre-existing serialization unit tests (which assert exact wire bytes) pass unchanged. - New golden tests assert the fast path equals the legacy path **byte-for-byte** for mixed-type tablets with nulls, all-null columns, bitmap-width edge cases, BLOB/DATE, and negative/`MIN_SAFE_INTEGER`/BigInt int64 values. - New unit tests cover hi/lo int64 correctness (two's complement negatives, i64 min/max BigInt) and string-cache correctness. - e2e `AllDataTypes` + `TableModelDataTypes` suites pass against a live IoTDB 2.0.10. ## Caveats - `BufferPool` is deprecated but still exported; removal would be semver-major. - The remaining GC in profiles is dominated by the benchmark harness's own tablet construction, not the client. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
