CritasWang commented on code in PR #16:
URL:
https://github.com/apache/iotdb-client-nodejs/pull/16#discussion_r3584475804
##########
.specify/memory/constitution.md:
##########
@@ -0,0 +1,50 @@
+# [PROJECT_NAME] Constitution
Review Comment:
Removed — all `.claude/` and `.specify/` files are dropped from the PR (they
were accidental local scaffolding; the diff is now 7 files / +611). Sorry for
the noise. (c5f5522)
##########
src/utils/FastSerializer.ts:
##########
@@ -265,3 +278,245 @@ export function serializeColumnFast(values: any[],
dataType: number): Buffer {
throw new Error(`Unsupported data type: ${dataType}`);
}
}
+
+/**
+ * Serialize a whole tablet's values (all columns + null bitmaps) into ONE
buffer.
+ *
+ * Wire format (identical to the legacy per-column path):
+ * [col0 data][col1 data]...[colN data]
+ * then per column: [hasNull flag byte][bitmap bytes, only when flag=1]
+ * Bitmap: ceil(rowCount/8) bytes, LSB-first, bit=1 means NULL.
+ *
+ * Optimizations over the legacy path:
+ * - No rows→columns transpose (reads values[row][col] directly)
+ * - No per-column intermediate buffers or boolean[] bitmaps
+ * - No trailing Buffer.concat (exact data size pre-computed; bitmap section
+ * allocated worst-case and trimmed with a zero-copy subarray)
+ * - Null bitmaps are packed inline during the value pass; the flag byte
+ * stays 0 and no bitmap bytes are emitted when a column has no nulls
+ *
+ * @param values Row-major tablet values: values[rowIndex][colIndex]
+ * @param dataTypes TSDataType code per column
+ * @param rowCount Number of rows
+ */
+export function serializeTabletValuesFast(
+ values: any[][],
+ dataTypes: number[],
+ rowCount: number,
+): Buffer {
+ const numCols = dataTypes.length;
+ const bitmapBytes = Math.ceil(rowCount / 8);
+
+ // ---- Pass 1: exact data-section size (fixed widths from schema; one
+ // byteLength scan for variable-width columns, memoizing repeated strings)
----
+ let dataSize = 0;
+ for (let c = 0; c < numCols; c++) {
+ switch (dataTypes[c]) {
+ case 0: // BOOLEAN
+ dataSize += rowCount;
+ break;
+ case 1: // INT32
+ case 3: // FLOAT
+ case 9: // DATE
+ dataSize += rowCount * 4;
+ break;
+ case 2: // INT64
+ case 4: // DOUBLE
+ case 8: // TIMESTAMP
+ dataSize += rowCount * 8;
+ break;
+ case 5: // TEXT
+ case 11: { // STRING
+ let lastStr: string | null = null;
+ let lastLen = 0;
+ for (let r = 0; r < rowCount; r++) {
+ const v = values[r][c];
+ if (v === null || v === undefined) {
+ dataSize += 4;
+ continue;
+ }
+ const str = typeof v === "string" ? v : String(v);
+ if (str !== lastStr) {
+ lastLen = Buffer.byteLength(str, "utf8");
+ lastStr = str;
+ }
+ dataSize += 4 + lastLen;
+ }
+ break;
+ }
+ case 10: { // BLOB
+ for (let r = 0; r < rowCount; r++) {
+ const v = values[r][c];
+ dataSize += 4 + (v === null || v === undefined
+ ? 0
+ : Buffer.isBuffer(v) ? v.length : Buffer.from(v).length);
Review Comment:
Fixed in c5f5522: the sizing pass now uses a new `blobByteLength()` helper
(no allocation — `Uint8Array`/`Buffer` use `.length`, strings use
`Buffer.byteLength`, array-likes use `.length`), and the write pass writes
directly into the target buffer (`buffer.set` for `Uint8Array`, `buffer.write`
for strings, byte loop for array-likes) with no intermediate `Buffer.from`.
Re-ran your microbenchmark scenario (1000 × 64-byte `Uint8Array` cells): 0.127
ms/tablet before → 0.051 ms/tablet after (~2.5× faster, and faster than the
pre-PR baseline). Added a golden fast-vs-legacy test covering `Uint8Array`,
byte-array, string, empty, and null BLOB inputs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
