Caideyipi opened a new pull request, #18816: URL: https://github.com/apache/iotdb/pull/18816
## Description An aligned tablet that introduces many measurements previously expanded its TVList and copied full-width accounting, value and bitmap arrays once per missing measurement while holding the DataRegion write lock. The repeated scans and copies made expansion quadratic in schema width, delaying queries waiting for the region read lock. Register missing measurements in insertion order, extend the TVList in one batch, and allocate the final-width reordered value and bitmap arrays once. Memory estimates are refreshed once per batch. Existing rows retain implicit nulls, and new value arrays and bitmaps remain lazy. Single-column extension delegates to the batch API. Regression tests cover mixed existing/new/deleted schemas, duplicate measurement names, bitmap alignment with a nonzero tablet offset, historical nulls across array blocks, binary and primitive memory accounting, clone/clear/reuse, WAL recovery, handover, and a 10,000-column tablet followed by another write. ## Validation - `mvn spotless:apply -pl iotdb-core/datanode` - `mvn test -pl iotdb-core/datanode -am -Dtest=AlignedWritableMemChunkTest,AlignedTVListTest,MemChunkDeserializeTest,MemTableFlushTaskTest,PrimitiveMemTableTest -Dsurefire.failIfNoSpecifiedTests=false` — 65 tests passed; the reactor compiled the DataNode and its dependencies from a fresh worktree. - `git diff --check` Local JDK 17 microbenchmark against the original classes from base commit `21d53dd015c43c230914076619cc34ca636a3882`: one existing populated INT64 column, then one tablet row adding new INT64 columns with a bitmap array. Setup is excluded. Sparse writes supply only the first and last new column; dense writes supply all new columns. Each JVM warms up with 30 expansions of 256 columns. The 10,000-column figures are medians of three runs; the 100,000-column figures are single runs. | New columns | Values | Before (ms) | After (ms) | Before allocated (MiB) | After allocated (MiB) | | --- | --- | ---: | ---: | ---: | ---: | | 10,000 | Sparse | 150.9 | 5.4 | 956.2 | 1.9 | | 10,000 | Dense | 156.9 | 8.4 | 957.1 | 2.9 | | 100,000 | Sparse | 18,151.7 | 48.7 | 95,393.7 | 19.8 | | 100,000 | Dense | 18,592.5 | 73.8 | 95,453.3 | 79.6 | Allocation is the current thread's total allocation during `writeAlignedTablet`, measured with `ThreadMXBean`, rather than retained heap size. These measurements validate the expansion path; a distributed workload was not rerun. This PR has: - [x] been self-reviewed, including the existing region locking and query/flush ownership rules. - [x] added Javadocs and comments explaining the batch expansion. - [x] added unit tests covering the new paths and preserved behavior. Key changed classes: `AlignedWritableMemChunk`, `AlignedTVList`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
