Copilot commented on code in PR #19339:
URL: https://github.com/apache/iotdb/pull/19339#discussion_r4638606539
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/datastructure/AlignedTVList.java:
##########
@@ -1430,6 +1449,24 @@ public synchronized long getStructureRamSize() {
return calculateContainerRamCost(null);
}
+ public synchronized boolean isRamSizeAccounted() {
+ return ramSizeAccounted;
+ }
+
+ public synchronized long getAccountedArrayRamSize() {
+ return accountedArrayRamSize;
+ }
+
+ public synchronized long getAccountedStructureRamSize() {
+ return accountedStructureRamSize;
+ }
+
+ public synchronized void setAccountedRamSize(long arrayRamSize, long
structureRamSize) {
+ this.accountedArrayRamSize = arrayRamSize;
+ this.accountedStructureRamSize = structureRamSize;
+ this.ramSizeAccounted = true;
+ }
Review Comment:
`setAccountedRamSize` is a new public mutator that can accept negative or
otherwise invalid sizes, which could reintroduce memory-block underflow (or
corrupt accounting) if misused. Consider adding argument validation (e.g.,
`arrayRamSize >= 0` and `structureRamSize >= 0`) and a brief Javadoc describing
the intended semantics (must reflect bytes already reconciled into the MemTable
write blocks).
##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/memtable/TsFileProcessorTest.java:
##########
@@ -989,6 +989,129 @@ public void
alignedBitmapMemoryAccountingMatchesActualAllocations()
assertAlignedTvListRamCostMatchesActual(deviceId);
}
+ @Test
+ public void
alignedStructureMemoryAccountingSurvivesRepeatedSortAndOutOfOrderWrites()
+ throws ExecutionException, WriteProcessException, IOException,
IllegalPathException {
+ processor = newTestProcessor(filePath + ".sort-structure-accounting");
+ insertAlignedRow(processor, deviceId, 100L, false);
+ insertAlignedRow(processor, deviceId, 50L, false);
+ assertAlignedTvListRamCostMatchesActual(deviceId);
+
+ for (int i = 1; i <= 10; i++) {
+ AlignedTVList list = getAlignedMemChunk(deviceId).getWorkingTVList();
+ list.sort();
+ Assert.assertNotNull(list.getSegmentMovedMap());
+
+ insertAlignedRow(processor, deviceId, 50L - i, false);
+ Assert.assertNull(list.getSegmentMovedMap());
+ assertAlignedTvListRamCostMatchesActual(deviceId);
+ }
+
+ AlignedTVList list = getAlignedMemChunk(deviceId).getWorkingTVList();
+ list.sort();
+ Assert.assertNotNull(list.getSegmentMovedMap());
+
+ // In-order write retains segmentMovedMap, charging its bytes into
structureMemoryBlock.
+ insertAlignedRow(processor, deviceId, 200L, false);
+ Assert.assertNotNull(list.getSegmentMovedMap());
+ assertAlignedTvListRamCostMatchesActual(deviceId);
+
+ // Next out-of-order write clears segmentMovedMap and releases only the
accounted bytes.
+ insertAlignedRow(processor, deviceId, 1L, false);
+ Assert.assertNull(list.getSegmentMovedMap());
+ assertAlignedTvListRamCostMatchesActual(deviceId);
+
+ processor.syncClose();
+ assertEquals(0, sgInfo.getMemCost());
+ }
+
+ @Test
+ public void
alignedStructureMemoryAccountingSurvivesMultiDeviceSortAndOutOfOrderBatch()
+ throws ExecutionException, WriteProcessException, IOException,
IllegalPathException {
+ processor = newTestProcessor(filePath + ".multi-device-sort-accounting");
+ String[] devices = {
+ "root.vehicle.sort_d0", "root.vehicle.sort_d1", "root.vehicle.sort_d2",
"root.vehicle.sort_d3"
+ };
+
+ for (String targetDevice : devices) {
+ insertAlignedRow(processor, targetDevice, 100L, false);
+ insertAlignedRow(processor, targetDevice, 50L, false);
+ }
+ for (String targetDevice : devices) {
+ AlignedTVList list = getAlignedMemChunk(targetDevice).getWorkingTVList();
+ list.sort();
+ Assert.assertNotNull(list.getSegmentMovedMap());
+ }
+
+ InsertRowsNode batch = new InsertRowsNode(new
PlanNodeId("multi-device-sort"));
+ for (int i = 0; i < devices.length; i++) {
+ TSRecord record = new TSRecord(devices[i], 10L);
+ record.addTuple(DataPoint.getDataPoint(dataType, measurementId, "10"));
+ InsertRowNode row = buildInsertRowNodeByTSRecord(record);
+ row.setAligned(true);
+ batch.addOneInsertRowNode(row, i);
+ }
+ processor.insertRows(batch, new long[5]);
Review Comment:
The `new long[5]` argument is a hard-coded size while the test already
derives the batch size from `devices.length`. To make the test resilient to
future edits (e.g., changing the device count), size this array based on the
batch/device count instead of a magic constant.
##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/utils/datastructure/AlignedTVListTest.java:
##########
@@ -967,4 +967,65 @@ public void testReleaseNonQueryColumnsWithBitmaps() {
Assert.assertEquals((long) (i * 100), tvList.getLongByValueIndex(i, 2));
}
}
+
+ @Test
+ public void testAccountedRamSizeLifecycleOnDeserializeAndClear() throws
IOException {
+ AlignedTVList tvList =
+ AlignedTVList.newAlignedList(Arrays.asList(TSDataType.INT64,
TSDataType.INT32));
+ Assert.assertFalse(tvList.isRamSizeAccounted());
+ tvList.putAlignedValue(10L, new Object[] {1L, 2});
+ tvList.putAlignedValue(5L, new Object[] {3L, 4});
+
+ ByteBuffer buffer = ByteBuffer.allocate(tvList.serializedSize());
+ tvList.serializeToWAL(new WALByteBufferForTest(buffer));
+ buffer.flip();
+
+ AlignedTVList restored =
+ AlignedTVList.deserialize(new DataInputStream(new
ByteArrayInputStream(buffer.array())));
Review Comment:
This deserialization uses `buffer.array()` (the full backing array), which
may include trailing unused bytes beyond `buffer.limit()`. Using an input
stream limited to `[0, buffer.limit())` makes the test stricter and avoids
silently accepting over-reads or depending on zero-filled padding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]