Caideyipi opened a new pull request, #18816:
URL: https://github.com/apache/iotdb/pull/18816

   ## Description
   
   An aligned tablet that introduces many measurements previously expanded its 
TVList and copied full-width accounting, value and bitmap arrays once per 
missing measurement while holding the DataRegion write lock. The repeated scans 
and copies made expansion quadratic in schema width, delaying queries waiting 
for the region read lock.
   
   Register missing measurements in insertion order, extend the TVList in one 
batch, and allocate the final-width reordered value and bitmap arrays once. 
Memory estimates are refreshed once per batch. Existing rows retain implicit 
nulls, and new value arrays and bitmaps remain lazy. Single-column extension 
delegates to the batch API.
   
   Regression tests cover mixed existing/new/deleted schemas, duplicate 
measurement names, bitmap alignment with a nonzero tablet offset, historical 
nulls across array blocks, binary and primitive memory accounting, 
clone/clear/reuse, WAL recovery, handover, and a 10,000-column tablet followed 
by another write.
   
   ## Validation
   
   - `mvn spotless:apply -pl iotdb-core/datanode`
   - `mvn test -pl iotdb-core/datanode -am 
-Dtest=AlignedWritableMemChunkTest,AlignedTVListTest,MemChunkDeserializeTest,MemTableFlushTaskTest,PrimitiveMemTableTest
 -Dsurefire.failIfNoSpecifiedTests=false` — 65 tests passed; the reactor 
compiled the DataNode and its dependencies from a fresh worktree.
   - `git diff --check`
   
   Local JDK 17 microbenchmark against the original classes from base commit 
`21d53dd015c43c230914076619cc34ca636a3882`: one existing populated INT64 
column, then one tablet row adding new INT64 columns with a bitmap array. Setup 
is excluded. Sparse writes supply only the first and last new column; dense 
writes supply all new columns. Each JVM warms up with 30 expansions of 256 
columns. The 10,000-column figures are medians of three runs; the 
100,000-column figures are single runs.
   
   | New columns | Values | Before (ms) | After (ms) | Before allocated (MiB) | 
After allocated (MiB) |
   | --- | --- | ---: | ---: | ---: | ---: |
   | 10,000 | Sparse | 150.9 | 5.4 | 956.2 | 1.9 |
   | 10,000 | Dense | 156.9 | 8.4 | 957.1 | 2.9 |
   | 100,000 | Sparse | 18,151.7 | 48.7 | 95,393.7 | 19.8 |
   | 100,000 | Dense | 18,592.5 | 73.8 | 95,453.3 | 79.6 |
   
   Allocation is the current thread's total allocation during 
`writeAlignedTablet`, measured with `ThreadMXBean`, rather than retained heap 
size. These measurements validate the expansion path; a distributed workload 
was not rerun.
   
   This PR has:
   
   - [x] been self-reviewed, including the existing region locking and 
query/flush ownership rules.
   - [x] added Javadocs and comments explaining the batch expansion.
   - [x] added unit tests covering the new paths and preserved behavior.
   
   Key changed classes: `AlignedWritableMemChunk`, `AlignedTVList`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to