jt2594838 commented on code in PR #18276:
URL: https://github.com/apache/iotdb/pull/18276#discussion_r3629717438
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/datastructure/AlignedTVList.java:
##########
Review Comment:
Thanks, fixed in a3351070. AlignedTVList now updates
materializedValueArrayMemCost when getOrCreateValueArray() allocates a
primitive array. calculateRamSize() uses the per-block cost without value
primitive arrays plus the incrementally tracked materialized-array cost, so it
no longer assumes dense value arrays or scans all blocks. The tracked state is
also handled by clone, projection, column extension, and clear paths.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/memtable/TsFileProcessor.java:
##########
@@ -1124,13 +1132,21 @@ private void updateAlignedMemCost(
+ (newPointNum % PrimitiveArrayManager.ARRAY_SIZE > 0 ? 1 : 0);
long acquireArray = newArrayCnt - currentArrayCnt;
+ boolean writesExistingLastBlock =
+ currentPointNum % PrimitiveArrayManager.ARRAY_SIZE != 0 &&
incomingPointNum > 0;
+ for (TSDataType dataType : dataTypesInTVList) {
+ if (writesExistingLastBlock) {
+ memIncrements[0] += AlignedTVList.primitiveArrayMemCost(dataType);
+ }
+ // Reserve a bitmap as well as a value array for new blocks because
the tablet may contain
+ // null or failed rows in those blocks.
+ memIncrements[0] += acquireArray *
AlignedTVList.valueListArrayMemCost(dataType);
+ }
+
if (acquireArray != 0) {
// memory of extending the TVList
memIncrements[0] +=
acquireArray *
alignedMemChunk.getWorkingTVList().alignedTvListArrayMemCost();
- for (TSDataType dataType : dataTypesInTVList) {
- memIncrements[0] += acquireArray *
AlignedTVList.valueListArrayMemCost(dataType);
- }
}
Review Comment:
Thanks, fixed in a3351070. The aligned row, rows, and tablet estimators now
split the charge into: (1) the base aligned-TVList block cost without value
primitive arrays, (2) placeholder/bitmap cost for newly extended columns, and
(3) primitive-array cost only when a successful non-null write first
materializes that column/block. Sparse, null, and failed-write paths are
covered by tests; the tablet path also skips per-row checks when both bitmap
and results are absent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
