Caideyipi commented on PR #17769:
URL: https://github.com/apache/iotdb/pull/17769#issuecomment-4550799680
1. AlignedTVList.java:1430
(https://github.com/apache/iotdb/blob/ff1190cd414115ebf695a9f1ec11913db22c5dcc/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/datastructure/AlignedTVList.java#L1430)
serializedSize() still calls getBinaryByValueIndex() unconditionally
for TEXT/BLOB/STRING/OBJECT columns. After this PR, a whole value array can be
null to
represent an unwritten sparse aligned column. In that case WAL size
calculation can hit an NPE before serializeToWAL() gets to its valueArray ==
null
handling.
2. TsFileProcessor.java:817
(https://github.com/apache/iotdb/blob/ff1190cd414115ebf695a9f1ec11913db22c5dcc/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/memtable/TsFileProcessor.java#L817)
/ AlignedWritableMemChunk.java:905
(https://github.com/apache/iotdb/blob/ff1190cd414115ebf695a9f1ec11913db22c5dcc/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/memtable/AlignedWritableMemChunk.java#L905)
For an existing aligned memchunk, when an insert expands the TVList
into a new chunk, the memory estimate adds only alignedTvListArrayMemCost().
The extra
value-array estimate checks only whether the current last chunk is
unallocated. If the previous last chunk already had a value array, but the new
chunk
will allocate value arrays for this insert, those value arrays are not
counted. This can underestimate memtable memory usage and weaken write memory
control.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
