Caideyipi commented on code in PR #19310:
URL: https://github.com/apache/iotdb/pull/19310#discussion_r4609962279
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/datastructure/AlignedTVList.java:
##########
@@ -92,6 +94,21 @@ public abstract class AlignedTVList extends TVList {
private long materializedBitmapMemoryCost;
private long arrayMemCostWithoutPrimitiveArraysAndIndex;
+ // Cached result of calculateContainerRamCost(null). Structural changes are
rare (one per block or
+ // per new column) compared with writes, so invalidating this cache on
structural changes keeps
+ // the per-write memory reconciliation O(1) instead of re-estimating every
column.
+ private long structureRamCost;
+ private boolean structureRamCostDirty = true;
+
+ // Columns that have at least one materialized value array. A sparse write
must explicitly mark
+ // the rows it does not write as null for every such column that already has
a value array in the
+ // target block. Tracking the columns avoids scanning all device columns on
every sparse write.
+ private final List<Integer> materializedColumns = new ArrayList<>();
+ // Marks which columns belong to the sparse write currently in progress, so
the null-marking pass
+ // can skip the columns whose nulls are handled together with their values.
+ private long[] columnWriteEpoch;
Review Comment:
[P2] Include the new write metadata in RAM accounting.
`getStructureRamSize()` delegates to `calculateContainerRamCost()`, but that
calculation does not include this `long[]` or the new `materializedColumns`
list. A 10,000-column TVList now carries about 80 KB of epoch entries alone
that `calculateRamSize()` does not report, so memory reservations can
undercount wide lists. Please account for these structures and add a test
against the expected size.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/datastructure/AlignedTVList.java:
##########
@@ -614,12 +685,15 @@ public void extendColumns(List<TSDataType> newDataTypes) {
if (newDataTypes.isEmpty()) {
return;
}
+ int existingBlockCount = timestamps.size();
for (TSDataType dataType : newDataTypes) {
- List<Object> columnValue = new ArrayList<>(timestamps.size());
- for (int i = 0; i < timestamps.size(); i++) {
- // New columns contain only implicit nulls until their first non-null
write.
- columnValue.add(null);
- }
+ // New columns contain only implicit nulls until their first non-null
write. Represent the
+ // existing blocks lazily so adding C columns to a device with B blocks
is O(C) instead of
+ // O(C * B); only columns that are actually written materialize their
null prefix.
+ List<Object> columnValue =
+ existingBlockCount == 0
+ ? new ArrayList<>()
+ : new NullPrefixedValueList(existingBlockCount);
Review Comment:
[P2] Account for the lazy prefix as allocated, not logical, slots.
`calculateContainerRamCost()` measures every value column with
`listRamCostWithReferences()`, which sizes an `Object[]` from `List.size()`.
Here the size includes `implicitNullCount` even while `prefix` is null, so
adding C columns after B existing blocks still reports O(C * B) references
despite allocating only O(C) lazy wrappers. Since this estimate feeds TVList
memory accounting, sparse extensions can trigger premature memory pressure and
lose the intended benefit. Please account for the actual prefix/tail storage
and invalidate the cached cost when the prefix is materialized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]