Caideyipi commented on code in PR #19310:
URL: https://github.com/apache/iotdb/pull/19310#discussion_r4609962279


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/datastructure/AlignedTVList.java:
##########
@@ -92,6 +94,21 @@ public abstract class AlignedTVList extends TVList {
   private long materializedBitmapMemoryCost;
   private long arrayMemCostWithoutPrimitiveArraysAndIndex;
 
+  // Cached result of calculateContainerRamCost(null). Structural changes are 
rare (one per block or
+  // per new column) compared with writes, so invalidating this cache on 
structural changes keeps
+  // the per-write memory reconciliation O(1) instead of re-estimating every 
column.
+  private long structureRamCost;
+  private boolean structureRamCostDirty = true;
+
+  // Columns that have at least one materialized value array. A sparse write 
must explicitly mark
+  // the rows it does not write as null for every such column that already has 
a value array in the
+  // target block. Tracking the columns avoids scanning all device columns on 
every sparse write.
+  private final List<Integer> materializedColumns = new ArrayList<>();
+  // Marks which columns belong to the sparse write currently in progress, so 
the null-marking pass
+  // can skip the columns whose nulls are handled together with their values.
+  private long[] columnWriteEpoch;

Review Comment:
   [P2] Include the new write metadata in RAM accounting. 
`getStructureRamSize()` delegates to `calculateContainerRamCost()`, but that 
calculation does not include this `long[]` or the new `materializedColumns` 
list. A 10,000-column TVList now carries about 80 KB of epoch entries alone 
that `calculateRamSize()` does not report, so memory reservations can 
undercount wide lists. Please account for these structures and add a test 
against the expected size.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/datastructure/AlignedTVList.java:
##########
@@ -614,12 +685,15 @@ public void extendColumns(List<TSDataType> newDataTypes) {
     if (newDataTypes.isEmpty()) {
       return;
     }
+    int existingBlockCount = timestamps.size();
     for (TSDataType dataType : newDataTypes) {
-      List<Object> columnValue = new ArrayList<>(timestamps.size());
-      for (int i = 0; i < timestamps.size(); i++) {
-        // New columns contain only implicit nulls until their first non-null 
write.
-        columnValue.add(null);
-      }
+      // New columns contain only implicit nulls until their first non-null 
write. Represent the
+      // existing blocks lazily so adding C columns to a device with B blocks 
is O(C) instead of
+      // O(C * B); only columns that are actually written materialize their 
null prefix.
+      List<Object> columnValue =
+          existingBlockCount == 0
+              ? new ArrayList<>()
+              : new NullPrefixedValueList(existingBlockCount);

Review Comment:
   [P2] Account for the lazy prefix as allocated, not logical, slots. 
`calculateContainerRamCost()` measures every value column with 
`listRamCostWithReferences()`, which sizes an `Object[]` from `List.size()`. 
Here the size includes `implicitNullCount` even while `prefix` is null, so 
adding C columns after B existing blocks still reports O(C * B) references 
despite allocating only O(C) lazy wrappers. Since this estimate feeds TVList 
memory accounting, sparse extensions can trigger premature memory pressure and 
lose the intended benefit. Please account for the actual prefix/tail storage 
and invalidate the cached cost when the prefix is materialized.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to