HTHou commented on code in PR #16:
URL:
https://github.com/apache/iotdb-client-nodejs/pull/16#discussion_r3584365879
##########
src/utils/FastSerializer.ts:
##########
@@ -265,3 +278,245 @@ export function serializeColumnFast(values: any[],
dataType: number): Buffer {
throw new Error(`Unsupported data type: ${dataType}`);
}
}
+
+/**
+ * Serialize a whole tablet's values (all columns + null bitmaps) into ONE
buffer.
+ *
+ * Wire format (identical to the legacy per-column path):
+ * [col0 data][col1 data]...[colN data]
+ * then per column: [hasNull flag byte][bitmap bytes, only when flag=1]
+ * Bitmap: ceil(rowCount/8) bytes, LSB-first, bit=1 means NULL.
+ *
+ * Optimizations over the legacy path:
+ * - No rows→columns transpose (reads values[row][col] directly)
+ * - No per-column intermediate buffers or boolean[] bitmaps
+ * - No trailing Buffer.concat (exact data size pre-computed; bitmap section
+ * allocated worst-case and trimmed with a zero-copy subarray)
+ * - Null bitmaps are packed inline during the value pass; the flag byte
+ * stays 0 and no bitmap bytes are emitted when a column has no nulls
+ *
+ * @param values Row-major tablet values: values[rowIndex][colIndex]
+ * @param dataTypes TSDataType code per column
+ * @param rowCount Number of rows
+ */
+export function serializeTabletValuesFast(
+ values: any[][],
+ dataTypes: number[],
+ rowCount: number,
+): Buffer {
+ const numCols = dataTypes.length;
+ const bitmapBytes = Math.ceil(rowCount / 8);
+
+ // ---- Pass 1: exact data-section size (fixed widths from schema; one
+ // byteLength scan for variable-width columns, memoizing repeated strings)
----
+ let dataSize = 0;
+ for (let c = 0; c < numCols; c++) {
+ switch (dataTypes[c]) {
+ case 0: // BOOLEAN
+ dataSize += rowCount;
+ break;
+ case 1: // INT32
+ case 3: // FLOAT
+ case 9: // DATE
+ dataSize += rowCount * 4;
+ break;
+ case 2: // INT64
+ case 4: // DOUBLE
+ case 8: // TIMESTAMP
+ dataSize += rowCount * 8;
+ break;
+ case 5: // TEXT
+ case 11: { // STRING
+ let lastStr: string | null = null;
+ let lastLen = 0;
+ for (let r = 0; r < rowCount; r++) {
+ const v = values[r][c];
+ if (v === null || v === undefined) {
+ dataSize += 4;
+ continue;
+ }
+ const str = typeof v === "string" ? v : String(v);
+ if (str !== lastStr) {
+ lastLen = Buffer.byteLength(str, "utf8");
+ lastStr = str;
+ }
+ dataSize += 4 + lastLen;
+ }
+ break;
+ }
+ case 10: { // BLOB
+ for (let r = 0; r < rowCount; r++) {
+ const v = values[r][c];
+ dataSize += 4 + (v === null || v === undefined
+ ? 0
+ : Buffer.isBuffer(v) ? v.length : Buffer.from(v).length);
Review Comment:
[P2] Avoid materializing non-`Buffer` BLOB values twice.
The sizing pass calls `Buffer.from(v)` here, and the write pass calls it
again for the same value. The previous serializer normalized each value once
and reused that buffer, so existing accepted inputs such as `Uint8Array` and
number arrays now allocate two temporary buffers per cell. In a local 1,000 ×
64-byte `Uint8Array` microbenchmark, the new tablet path was roughly 30–40%
slower. Please determine the length without allocating where possible, or
retain the normalized buffer for the write pass, and add coverage for
non-`Buffer` BLOB inputs.
##########
.specify/memory/constitution.md:
##########
@@ -0,0 +1,50 @@
+# [PROJECT_NAME] Constitution
Review Comment:
[P1] Please remove the unrelated SpecKit/Claude scaffolding from this PR.
This change adds 20 files under `.claude/` and `.specify/`—most of the PR’s
4,026 added lines—including placeholder governance content and executable
scripts, but none of them are related to tablet serialization. Merging them
would unexpectedly expand the repository’s tooling and governance surface, and
these new files also lack ASF license headers. Please drop all `.claude/` and
`.specify/` additions here; if they are wanted, they should be proposed
separately with their own rationale and review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
