This is an automated email from the ASF dual-hosted git repository.

jt2594838 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/iotdb.git


The following commit(s) were added to refs/heads/master by this push:
     new 48168f1f858 perf(datanode): encode aligned duplicate timestamps in one 
pass (#19324)
48168f1f858 is described below

commit 48168f1f858dcf25abf9dc43e8b235920395e9c0
Author: Zhenyu Luo <[email protected]>
AuthorDate: Fri Nov 20 11:10:17 2026 +0800

    perf(datanode): encode aligned duplicate timestamps in one pass (#19324)
    
    * perf(datanode): merge duplicate aligned timestamps in one encoding pass
    
    AlignedWritableMemChunk pre-scanned every row, kept a boolean[rowCount]
    timeDuplicateInfo bitmap, and then traversed the same rows again while
    encoding. Replace that with a row-major pending-timestamp group encoder:
    the last non-null value of a duplicate group is emitted when the group
    ends, and a group may carry across page or chunk boundaries. The first
    pass now only builds page/chunk ranges and tracks whether any duplicate
    exists so the dense segment-batch fast path can still be used.
    
    Update the opt-in duplicate-time benchmark to the single-pass shape and
    add regression tests for groups spanning page/chunk boundaries, with and
    without measurement deletion and all-null rows.
    
    * perf(datanode): tighten the aligned duplicate-timestamp encoder
    
    Reuse one page timestamp buffer per encode instead of allocating a
    long[pageCap] for every page, unbox the TVList-column mapping into an
    int[] before the row loop, and decide whether a pending timestamp group
    crosses a page/chunk boundary by peeking at the next page's first row
    instead of walking forward over deleted rows.
    
    Page ranges contain only non-deleted rows, so the next page's first row
    is exactly the next non-deleted row after the current page: the pending
    group continues iff that row repeats the pending timestamp. This also
    removes hasSameTimeAsNextNonDeletedRow and its forward scan.
    
    For a 1M-row encode the per-page time arrays added up to
    rowCount * 8 B (~8 MB) of transient allocation; the encode now allocates
    that buffer once. The row loop also no longer performs one List.get plus
    unbox per column per row.
    
    Benchmark (AlignedFlushDuplicateTimeCheckPerformanceTest, opt-in,
    16,384 rows x 8 columns, 64 scans/round, 5 rounds, Apple Silicon, JDK 24;
    cpu% = single-pass CPU / legacy CPU). The test now sweeps duplicate
    timestamp ratios 0/50/90% and per-value null ratios 0/50/90%, because a
    single synthetic shape must not be extrapolated to other workloads:
    
       dup   null   legacy(ms)  onePass(ms)   cpu%
      0.00   0.00        1.828        0.160    8.7%
      0.00   0.50        2.756        0.360   13.0%
      0.00   0.90        1.389        0.320   23.0%
      0.50   0.00        1.461        0.169   11.5%
      0.50   0.50        2.095        0.311   14.8%
      0.50   0.90        1.142        0.322   28.1%
      0.90   0.00        1.125        0.159   14.2%
      0.90   0.50        1.993        0.306   15.3%
      0.90   0.90        0.932        0.309   33.2%
    
    The merge loop is 3x to 11x faster depending on the shape; null-heavy
    data narrows the gap because the legacy early-return is cheaper there.
    Numbers are from a shared development machine and move with load; the
    spread, not any single ratio, is the point.
    
    Verification:
    - mvn spotless:check -pl iotdb-core/datanode: passed.
    - mvn test-compile -pl iotdb-core/datanode -am -o: BUILD SUCCESS.
    - MemTableFlushTaskTest 19/19, AlignedWritableMemChunkTest 6/6,
      AlignedTVListTest 29/29: passed, including the duplicate groups that
      cross page/chunk boundaries with measurement deletion and
      ignoreAllNullRows.
    - The opt-in benchmark completes and asserts legacy == single-pass
      results for every scenario.
---
 .../memtable/AlignedWritableMemChunk.java          | 379 ++++++++-------------
 ...gnedFlushDuplicateTimeCheckPerformanceTest.java | 152 +++++----
 .../dataregion/memtable/MemTableFlushTaskTest.java |  91 +++++
 3 files changed, 319 insertions(+), 303 deletions(-)

diff --git 
a/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/memtable/AlignedWritableMemChunk.java
 
b/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/memtable/AlignedWritableMemChunk.java
index 106311f80de..af576b92c57 100644
--- 
a/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/memtable/AlignedWritableMemChunk.java
+++ 
b/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/memtable/AlignedWritableMemChunk.java
@@ -57,7 +57,6 @@ import java.util.Arrays;
 import java.util.LinkedHashMap;
 import java.util.List;
 import java.util.Map;
-import java.util.Objects;
 import java.util.Set;
 import java.util.TreeMap;
 import java.util.concurrent.BlockingQueue;
@@ -597,7 +596,7 @@ public class AlignedWritableMemChunk extends 
AbstractWritableMemChunk {
     allValueColDeletedMap =
         ignoreAllNullRows ? 
alignedWorkingListForFlush.getAllValueColDeletedMap() : null;
 
-    boolean[] timeDuplicateInfo = null;
+    boolean hasDuplicateTimestamp = false;
 
     List<List<Integer>> chunkRange = new ArrayList<>();
     // Eg. chunkRange: ((0,9,10,12),(13,15)) means this TVList contains 2 
chunks,
@@ -640,10 +639,8 @@ public class AlignedWritableMemChunk extends 
AbstractWritableMemChunk {
       }
       if (nextRowIndex != alignedWorkingListForFlush.rowCount()
           && time == alignedWorkingListForFlush.getTime(nextRowIndex)) {
-        if (Objects.isNull(timeDuplicateInfo)) {
-          timeDuplicateInfo = new 
boolean[alignedWorkingListForFlush.rowCount()];
-        }
-        timeDuplicateInfo[sortedRowIndex] = true;
+        // This flag only selects the dense fast path; duplicates are merged 
during encoding.
+        hasDuplicateTimestamp = true;
       }
       sortedRowIndex = nextRowIndex - 1;
     }
@@ -658,7 +655,7 @@ public class AlignedWritableMemChunk extends 
AbstractWritableMemChunk {
     handleEncoding(
         ioTaskQueue,
         chunkRange,
-        timeDuplicateInfo,
+        hasDuplicateTimestamp,
         allValueColDeletedMap,
         maxNumberOfPointsInPage,
         activeSchemaList);
@@ -667,142 +664,133 @@ public class AlignedWritableMemChunk extends 
AbstractWritableMemChunk {
   private void handleEncoding(
       BlockingQueue<Object> ioTaskQueue,
       List<List<Integer>> chunkRange,
-      boolean[] timeDuplicateInfo,
+      boolean hasDuplicateTimestamp,
       BitMap allValueColDeletedMap,
       int maxNumberOfPointsInPage,
       List<IMeasurementSchema> activeSchemaList) {
+    AlignedTVList alignedWorkingListForFlush = (AlignedTVList) 
workingListForFlush;
     // Flushing memtables are immutable, so this identity remains stable 
during encoding.
-    if (activeSchemaList == schemaList) {
-      handleEncodingWithoutDeletedMeasurements(
-          ioTaskQueue,
-          chunkRange,
-          timeDuplicateInfo,
-          allValueColDeletedMap,
-          maxNumberOfPointsInPage);
+    boolean hasTimeDeleted = alignedWorkingListForFlush.getTimeColDeletedMap() 
!= null;
+    if (!hasDuplicateTimestamp
+        && allValueColDeletedMap == null
+        && !hasTimeDeleted
+        && activeSchemaList == schemaList) {
+      handleEncodingWithoutDeletedMeasurementsFastPath(ioTaskQueue, 
chunkRange);
       return;
     }
-    handleEncodingWithDeletedMeasurements(
-        ioTaskQueue,
-        chunkRange,
-        timeDuplicateInfo,
-        allValueColDeletedMap,
-        maxNumberOfPointsInPage,
-        activeSchemaList);
+    encodePages(
+        ioTaskQueue, chunkRange, allValueColDeletedMap, 
maxNumberOfPointsInPage, activeSchemaList);
   }
 
-  private void handleEncodingWithoutDeletedMeasurements(
+  /** Encode pages in one row-major pass, merging duplicate timestamps into 
groups. */
+  private void encodePages(
       BlockingQueue<Object> ioTaskQueue,
       List<List<Integer>> chunkRange,
-      boolean[] timeDuplicateInfo,
       BitMap allValueColDeletedMap,
-      int maxNumberOfPointsInPage) {
-    // No duplicate timestamps, deleted rows, or all-null rows need filtering, 
so avoid metadata
-    // checks.
+      int maxNumberOfPointsInPage,
+      List<IMeasurementSchema> activeSchemaList) {
     AlignedTVList alignedWorkingListForFlush = (AlignedTVList) 
workingListForFlush;
-    boolean hasTimeDeleted = alignedWorkingListForFlush.getTimeColDeletedMap() 
!= null;
-    if (timeDuplicateInfo == null && allValueColDeletedMap == null && 
!hasTimeDeleted) {
-      handleEncodingWithoutDeletedMeasurementsFastPath(ioTaskQueue, 
chunkRange);
-      return;
+    int columnCount = activeSchemaList.size();
+    List<Integer> columnIndexList = buildColumnIndexList(activeSchemaList);
+    // Unbox the column mapping once instead of on every row/column access.
+    int[] columnIndexes = new int[columnCount];
+    for (int columnIndex = 0; columnIndex < columnCount; columnIndex++) {
+      columnIndexes[columnIndex] = columnIndexList.get(columnIndex);
     }
-
-    List<TSDataType> dataTypes = alignedWorkingListForFlush.getTsDataTypes();
-    List<TypeServices.AlignedTVListChunkWriter> valueWriters = new 
ArrayList<>(dataTypes.size());
-    for (TSDataType dataType : dataTypes) {
-      valueWriters.add(
+    TypeServices.AlignedTVListChunkWriter[] valueWriters =
+        new TypeServices.AlignedTVListChunkWriter[columnCount];
+    for (int columnIndex = 0; columnIndex < columnCount; columnIndex++) {
+      valueWriters[columnIndex] =
           TypeServices.StorageEngine.ALIGNED_TV_LIST_CHUNK_WRITER_SERVICE.call(
-              Type.fromTsDataType(dataType)));
-    }
-    long[] lastValidPointTimeForTimeDupCheck =
-        timeDuplicateInfo == null ? null : new long[dataTypes.size()];
-    int[] lastValidPointIndexForTimeDupCheck =
-        timeDuplicateInfo == null ? null : new int[dataTypes.size()];
-    if (lastValidPointIndexForTimeDupCheck != null) {
-      Arrays.fill(lastValidPointIndexForTimeDupCheck, -1);
+              
Type.fromTsDataType(activeSchemaList.get(columnIndex).getType()));
     }
-    for (List<Integer> pageRange : chunkRange) {
+    int[] lastValidPointIndex = new int[columnCount];
+    Arrays.fill(lastValidPointIndex, -1);
+    // write() only reads [0, pointsInPage), so one array can back every page 
of this encode.
+    long[] times =
+        new long[Math.min(maxNumberOfPointsInPage, 
alignedWorkingListForFlush.rowCount())];
+
+    boolean hasPendingGroup = false;
+    long pendingTime = 0;
+    int pendingLastRowValueIndex = -1;
+
+    for (int chunkIndex = 0; chunkIndex < chunkRange.size(); chunkIndex++) {
+      List<Integer> pageRange = chunkRange.get(chunkIndex);
       AlignedChunkWriterImpl alignedChunkWriter =
-          new AlignedChunkWriterImpl(schemaList, encryptParameter);
-      for (int pageNum = 0; pageNum < pageRange.size() / 2; pageNum += 1) {
-        for (int columnIndex = 0; columnIndex < dataTypes.size(); 
columnIndex++) {
-          TypeServices.AlignedTVListChunkWriter valueWriter = 
valueWriters.get(columnIndex);
-          ValueChunkWriter valueChunkWriter =
-              alignedChunkWriter.getValueChunkWriterByIndex(columnIndex);
-          for (int sortedRowIndex = pageRange.get(pageNum * 2);
-              sortedRowIndex <= pageRange.get(pageNum * 2 + 1);
-              sortedRowIndex++) {
-            int valueIndex = 
alignedWorkingListForFlush.getValueIndex(sortedRowIndex);
-            // skip empty row
-            if (allValueColDeletedMap != null && 
allValueColDeletedMap.isMarked(valueIndex)) {
-              continue;
-            }
-            // Keep value pages aligned with the time page when an entire 
timestamp is deleted.
-            if (alignedWorkingListForFlush.isTimeDeleted(valueIndex, false)) {
-              continue;
-            }
-            // skip time duplicated rows
-            long time = alignedWorkingListForFlush.getTime(sortedRowIndex);
-            boolean currentValueIsNull = false;
-            if (timeDuplicateInfo != null) {
-              currentValueIsNull = 
alignedWorkingListForFlush.isNullValue(valueIndex, columnIndex);
-              if (!currentValueIsNull) {
-                lastValidPointTimeForTimeDupCheck[columnIndex] = time;
-                lastValidPointIndexForTimeDupCheck[columnIndex] = valueIndex;
-              }
-              if (timeDuplicateInfo[sortedRowIndex]) {
-                continue;
-              }
-            }
+          new AlignedChunkWriterImpl(activeSchemaList, encryptParameter);
+      ValueChunkWriter[] valueChunkWriters = new ValueChunkWriter[columnCount];
+      for (int columnIndex = 0; columnIndex < columnCount; columnIndex++) {
+        valueChunkWriters[columnIndex] = 
alignedChunkWriter.getValueChunkWriterByIndex(columnIndex);
+      }
+      int pageCount = pageRange.size() / 2;
+      for (int pageNum = 0; pageNum < pageCount; pageNum++) {
+        int pageStart = pageRange.get(pageNum * 2);
+        int pageEnd = pageRange.get(pageNum * 2 + 1);
+        int pointsInPage = 0;
 
-            // The part of code solves the following problem:
-            // Time: 1,2,2,3
-            // Value: 1,2,null,null
-            // When rowIndex:1, no valid duplicate value, 
timeDuplicateInfo:false, write(T:1,V:1)
-            // When rowIndex:2, last valid point=(2,2), 
timeDuplicateInfo:true, skip writing value
-            // When rowIndex:3, last valid point=(2,2), T:2 matches, 
write(T:2,V:2)
-            // When rowIndex:4, last valid point=(2,2), T:3 doesn't match,
-            // write(T:3,V:null)
-
-            int originRowIndex;
-            if (timeDuplicateInfo != null
-                && lastValidPointIndexForTimeDupCheck[columnIndex] >= 0
-                && time == lastValidPointTimeForTimeDupCheck[columnIndex]) {
-              originRowIndex = lastValidPointIndexForTimeDupCheck[columnIndex];
-            } else {
-              originRowIndex = valueIndex;
+        for (int sortedRowIndex = pageStart; sortedRowIndex <= pageEnd; 
sortedRowIndex++) {
+          int valueIndex = 
alignedWorkingListForFlush.getValueIndex(sortedRowIndex);
+          // Skip empty and deleted rows.
+          if ((allValueColDeletedMap != null && 
allValueColDeletedMap.isMarked(valueIndex))
+              || alignedWorkingListForFlush.isTimeDeleted(valueIndex, false)) {
+            continue;
+          }
+          long time = alignedWorkingListForFlush.getTime(sortedRowIndex);
+          if (hasPendingGroup && time != pendingTime) {
+            pointsInPage =
+                appendPendingGroup(
+                    alignedWorkingListForFlush,
+                    pendingTime,
+                    pendingLastRowValueIndex,
+                    lastValidPointIndex,
+                    columnIndexes,
+                    valueWriters,
+                    valueChunkWriters,
+                    times,
+                    pointsInPage);
+            hasPendingGroup = false;
+          }
+          if (!hasPendingGroup) {
+            hasPendingGroup = true;
+            pendingTime = time;
+          }
+          pendingLastRowValueIndex = valueIndex;
+          for (int columnIndex = 0; columnIndex < columnCount; columnIndex++) {
+            int tvListColumnIndex = columnIndexes[columnIndex];
+            if (tvListColumnIndex >= 0
+                && !alignedWorkingListForFlush.isNullValue(valueIndex, 
tvListColumnIndex)) {
+              lastValidPointIndex[columnIndex] = valueIndex;
             }
-
-            boolean isNull =
-                timeDuplicateInfo == null
-                    ? alignedWorkingListForFlush.isNullValue(originRowIndex, 
columnIndex)
-                    : originRowIndex == valueIndex && currentValueIsNull;
-            valueWriter.write(
-                valueChunkWriter,
-                time,
-                alignedWorkingListForFlush,
-                originRowIndex,
-                columnIndex,
-                isNull);
           }
-          alignedChunkWriter.nextColumn();
         }
 
-        long[] times =
-            new long[Math.min(maxNumberOfPointsInPage, 
alignedWorkingListForFlush.rowCount())];
-        int pointsInPage = 0;
-        for (int sortedRowIndex = pageRange.get(pageNum * 2);
-            sortedRowIndex <= pageRange.get(pageNum * 2 + 1);
-            sortedRowIndex++) {
-          // skip empty row
-          if (((allValueColDeletedMap != null
-                  && allValueColDeletedMap.isMarked(
-                      
alignedWorkingListForFlush.getValueIndex(sortedRowIndex)))
-              || (alignedWorkingListForFlush.isTimeDeleted(sortedRowIndex)))) {
-            continue;
-          }
-          if (Objects.isNull(timeDuplicateInfo) || 
!timeDuplicateInfo[sortedRowIndex]) {
-            times[pointsInPage++] = 
alignedWorkingListForFlush.getTime(sortedRowIndex);
-          }
+        // Page ranges contain only non-deleted rows, so the next page's first 
row is exactly the
+        // next non-deleted row: the pending group continues iff that row 
repeats pendingTime.
+        int nextPageStart;
+        if (pageNum + 1 < pageCount) {
+          nextPageStart = pageRange.get((pageNum + 1) * 2);
+        } else if (chunkIndex + 1 < chunkRange.size()) {
+          nextPageStart = chunkRange.get(chunkIndex + 1).get(0);
+        } else {
+          nextPageStart = -1;
+        }
+        if (hasPendingGroup
+            && (nextPageStart < 0
+                || alignedWorkingListForFlush.getTime(nextPageStart) != 
pendingTime)) {
+          pointsInPage =
+              appendPendingGroup(
+                  alignedWorkingListForFlush,
+                  pendingTime,
+                  pendingLastRowValueIndex,
+                  lastValidPointIndex,
+                  columnIndexes,
+                  valueWriters,
+                  valueChunkWriters,
+                  times,
+                  pointsInPage);
+          hasPendingGroup = false;
         }
+
         alignedChunkWriter.write(times, pointsInPage, 0);
       }
       alignedChunkWriter.sealCurrentPage();
@@ -815,6 +803,33 @@ public class AlignedWritableMemChunk extends 
AbstractWritableMemChunk {
     }
   }
 
+  private int appendPendingGroup(
+      AlignedTVList list,
+      long pendingTime,
+      int pendingLastRowValueIndex,
+      int[] lastValidPointIndex,
+      int[] columnIndexes,
+      TypeServices.AlignedTVListChunkWriter[] valueWriters,
+      ValueChunkWriter[] valueChunkWriters,
+      long[] times,
+      int pointsInPage) {
+    for (int columnIndex = 0; columnIndex < columnIndexes.length; 
columnIndex++) {
+      int tvListColumnIndex = columnIndexes[columnIndex];
+      boolean isNull = tvListColumnIndex < 0 || 
lastValidPointIndex[columnIndex] < 0;
+      int originRowIndex = isNull ? pendingLastRowValueIndex : 
lastValidPointIndex[columnIndex];
+      valueWriters[columnIndex].write(
+          valueChunkWriters[columnIndex],
+          pendingTime,
+          list,
+          originRowIndex,
+          tvListColumnIndex,
+          isNull);
+    }
+    times[pointsInPage] = pendingTime;
+    Arrays.fill(lastValidPointIndex, -1);
+    return pointsInPage + 1;
+  }
+
   private void handleEncodingWithoutDeletedMeasurementsFastPath(
       BlockingQueue<Object> ioTaskQueue, List<List<Integer>> chunkRange) {
     AlignedTVList alignedWorkingListForFlush = (AlignedTVList) 
workingListForFlush;
@@ -945,128 +960,6 @@ public class AlignedWritableMemChunk extends 
AbstractWritableMemChunk {
         .write(valueChunkWriter, timestamps, valueArray, bitMap, arrayOffset, 
pointsInSegment);
   }
 
-  private void handleEncodingWithDeletedMeasurements(
-      BlockingQueue<Object> ioTaskQueue,
-      List<List<Integer>> chunkRange,
-      boolean[] timeDuplicateInfo,
-      BitMap allValueColDeletedMap,
-      int maxNumberOfPointsInPage,
-      List<IMeasurementSchema> activeSchemaList) {
-    AlignedTVList alignedWorkingListForFlush = (AlignedTVList) 
workingListForFlush;
-    List<Integer> columnIndexList = buildColumnIndexList(activeSchemaList);
-    long[] lastValidPointTimeForTimeDupCheck =
-        timeDuplicateInfo == null ? null : new long[activeSchemaList.size()];
-    int[] lastValidPointIndexForTimeDupCheck =
-        timeDuplicateInfo == null ? null : new int[activeSchemaList.size()];
-    if (lastValidPointIndexForTimeDupCheck != null) {
-      Arrays.fill(lastValidPointIndexForTimeDupCheck, -1);
-    }
-    for (List<Integer> pageRange : chunkRange) {
-      AlignedChunkWriterImpl alignedChunkWriter =
-          new AlignedChunkWriterImpl(activeSchemaList, encryptParameter);
-      for (int pageNum = 0; pageNum < pageRange.size() / 2; pageNum += 1) {
-        for (int columnIndex = 0; columnIndex < activeSchemaList.size(); 
columnIndex++) {
-          int tvListColumnIndex = columnIndexList.get(columnIndex);
-          TSDataType tsDataType = activeSchemaList.get(columnIndex).getType();
-          TypeServices.AlignedTVListColumnWriter columnWriter =
-              
TypeServices.StorageEngine.ALIGNED_TV_LIST_COLUMN_WRITER_SERVICE.call(
-                  Type.fromTsDataType(tsDataType));
-          for (int sortedRowIndex = pageRange.get(pageNum * 2);
-              sortedRowIndex <= pageRange.get(pageNum * 2 + 1);
-              sortedRowIndex++) {
-            int valueIndex = 
alignedWorkingListForFlush.getValueIndex(sortedRowIndex);
-            // skip empty row
-            if (allValueColDeletedMap != null && 
allValueColDeletedMap.isMarked(valueIndex)) {
-              continue;
-            }
-            // Keep value pages aligned with the time page when an entire 
timestamp is deleted.
-            if (alignedWorkingListForFlush.isTimeDeleted(valueIndex, false)) {
-              continue;
-            }
-            // skip time duplicated rows
-            long time = alignedWorkingListForFlush.getTime(sortedRowIndex);
-            boolean currentValueIsNull = true;
-            if (timeDuplicateInfo != null && tvListColumnIndex >= 0) {
-              currentValueIsNull =
-                  alignedWorkingListForFlush.isNullValue(valueIndex, 
tvListColumnIndex);
-              if (!currentValueIsNull) {
-                lastValidPointTimeForTimeDupCheck[columnIndex] = time;
-                lastValidPointIndexForTimeDupCheck[columnIndex] = valueIndex;
-              }
-            }
-            if (timeDuplicateInfo != null) {
-              if (timeDuplicateInfo[sortedRowIndex]) {
-                continue;
-              }
-            }
-
-            // The part of code solves the following problem:
-            // Time: 1,2,2,3
-            // Value: 1,2,null,null
-            // When rowIndex:1, no valid duplicate value, 
timeDuplicateInfo:false, write(T:1,V:1)
-            // When rowIndex:2, last valid point=(2,2), 
timeDuplicateInfo:true, skip writing value
-            // When rowIndex:3, last valid point=(2,2), T:2 matches, 
write(T:2,V:2)
-            // When rowIndex:4, last valid point=(2,2), T:3 doesn't match,
-            // write(T:3,V:null)
-
-            int originRowIndex;
-            if (tvListColumnIndex >= 0
-                && timeDuplicateInfo != null
-                && lastValidPointIndexForTimeDupCheck[columnIndex] >= 0
-                && time == lastValidPointTimeForTimeDupCheck[columnIndex]) {
-              originRowIndex = lastValidPointIndexForTimeDupCheck[columnIndex];
-            } else {
-              originRowIndex = valueIndex;
-            }
-
-            boolean isNull;
-            if (tvListColumnIndex < 0) {
-              isNull = true;
-            } else if (timeDuplicateInfo == null) {
-              isNull = alignedWorkingListForFlush.isNullValue(originRowIndex, 
tvListColumnIndex);
-            } else {
-              isNull = originRowIndex == valueIndex && currentValueIsNull;
-            }
-            columnWriter.write(
-                alignedChunkWriter,
-                time,
-                alignedWorkingListForFlush,
-                originRowIndex,
-                tvListColumnIndex,
-                isNull);
-          }
-          alignedChunkWriter.nextColumn();
-        }
-
-        long[] times =
-            new long[Math.min(maxNumberOfPointsInPage, 
alignedWorkingListForFlush.rowCount())];
-        int pointsInPage = 0;
-        for (int sortedRowIndex = pageRange.get(pageNum * 2);
-            sortedRowIndex <= pageRange.get(pageNum * 2 + 1);
-            sortedRowIndex++) {
-          // skip empty row
-          if (((allValueColDeletedMap != null
-                  && allValueColDeletedMap.isMarked(
-                      
alignedWorkingListForFlush.getValueIndex(sortedRowIndex)))
-              || (alignedWorkingListForFlush.isTimeDeleted(sortedRowIndex)))) {
-            continue;
-          }
-          if (Objects.isNull(timeDuplicateInfo) || 
!timeDuplicateInfo[sortedRowIndex]) {
-            times[pointsInPage++] = 
alignedWorkingListForFlush.getTime(sortedRowIndex);
-          }
-        }
-        alignedChunkWriter.write(times, pointsInPage, 0);
-      }
-      alignedChunkWriter.sealCurrentPage();
-      alignedChunkWriter.clearPageWriter();
-      try {
-        ioTaskQueue.put(alignedChunkWriter);
-      } catch (InterruptedException e) {
-        Thread.currentThread().interrupt();
-      }
-    }
-  }
-
   @Override
   public void encode(
       BlockingQueue<Object> ioTaskQueue,
diff --git 
a/iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/memtable/AlignedFlushDuplicateTimeCheckPerformanceTest.java
 
b/iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/memtable/AlignedFlushDuplicateTimeCheckPerformanceTest.java
index 1f396efb6ff..afbb2d25a49 100644
--- 
a/iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/memtable/AlignedFlushDuplicateTimeCheckPerformanceTest.java
+++ 
b/iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/memtable/AlignedFlushDuplicateTimeCheckPerformanceTest.java
@@ -34,6 +34,15 @@ import java.util.Arrays;
 import java.util.List;
 import java.util.Locale;
 
+/**
+ * Manual benchmark of the aligned duplicate-timestamp merge loop.
+ *
+ * <p>The model compares the legacy per-column traversal against the single 
row-major pass. It
+ * varies the duplicate-timestamp ratio and the per-value null ratio, because 
the ratio reported for
+ * one synthetic shape must not be extrapolated to other workloads. The 
encoder's additional skip
+ * paths (all-null rows with {@code ignoreAllNullRows}, time-deleted rows) are 
not modeled here;
+ * they are covered functionally by {@link MemTableFlushTaskTest}.
+ */
 public class AlignedFlushDuplicateTimeCheckPerformanceTest {
 
   private static final String ENABLED_PROPERTY = 
"iotdb.aligned.flush.duplicate-check.perf.enabled";
@@ -42,6 +51,10 @@ public class AlignedFlushDuplicateTimeCheckPerformanceTest {
   private static final int ROW_COUNT = 16_384;
   private static final int COLUMN_COUNT = 8;
   private static final int WARMUP_ROUNDS = 3;
+  // Fraction of rows that share the previous row's timestamp.
+  private static final double[] DUPLICATE_RATIOS = {0.0, 0.5, 0.9};
+  // Fraction of values that are null.
+  private static final double[] NULL_RATIOS = {0.0, 0.5, 0.9};
 
   private static volatile long benchmarkBlackhole;
 
@@ -64,10 +77,39 @@ public class AlignedFlushDuplicateTimeCheckPerformanceTest {
     Assert.assertTrue(rounds > 0);
     Assert.assertTrue(scans > 0);
 
-    BenchmarkData data = createBenchmarkData();
+    System.out.printf(
+        Locale.ROOT,
+        "Aligned flush duplicate-time merge benchmark: rows=%d, columns=%d, 
value checks/scan=%d,"
+            + " scans/round=%d, rounds=%d%n",
+        ROW_COUNT,
+        COLUMN_COUNT,
+        ROW_COUNT * COLUMN_COUNT,
+        scans,
+        rounds);
+    System.out.printf(
+        Locale.ROOT,
+        "  %8s %10s %13s %15s %8s %8s%n",
+        "dup",
+        "null",
+        "legacy(ms)",
+        "onePass(ms)",
+        "cpu%",
+        "alloc%");
+    for (double duplicateRatio : DUPLICATE_RATIOS) {
+      for (double nullRatio : NULL_RATIOS) {
+        runScenario(duplicateRatio, nullRatio, rounds, scans);
+      }
+    }
+  }
+
+  private static void runScenario(double duplicateRatio, double nullRatio, int 
rounds, int scans) {
+    BenchmarkData data = createBenchmarkData(duplicateRatio, nullRatio);
     long legacyResult = runLegacy(data);
     long optimizedResult = runOptimized(data);
-    Assert.assertEquals(legacyResult, optimizedResult);
+    Assert.assertEquals(
+        String.format(Locale.ROOT, "dup=%.2f null=%.2f", duplicateRatio, 
nullRatio),
+        legacyResult,
+        optimizedResult);
 
     for (int i = 0; i < WARMUP_ROUNDS; i++) {
       runLegacy(data);
@@ -88,7 +130,18 @@ public class AlignedFlushDuplicateTimeCheckPerformanceTest {
 
     Summary legacySummary = 
ManualPerformanceTestUtils.summarize(legacyMeasurements, scans);
     Summary optimizedSummary = 
ManualPerformanceTestUtils.summarize(optimizedMeasurements, scans);
-    printResult(rounds, scans, legacySummary, optimizedSummary);
+    System.out.printf(
+        Locale.ROOT,
+        "  %8.2f %10.2f %13.3f %15.3f %7.1f%% %7.1f%%%n",
+        duplicateRatio,
+        nullRatio,
+        legacySummary.getCpuNanosPerOperation() / 1_000_000.0,
+        optimizedSummary.getCpuNanosPerOperation() / 1_000_000.0,
+        percentage(
+            optimizedSummary.getCpuNanosPerOperation(), 
legacySummary.getCpuNanosPerOperation()),
+        percentage(
+            optimizedSummary.getAllocatedBytesPerOperation(),
+            legacySummary.getAllocatedBytesPerOperation()));
   }
 
   private static Measurement measureLegacy(BenchmarkData data, int scans) {
@@ -133,32 +186,37 @@ public class 
AlignedFlushDuplicateTimeCheckPerformanceTest {
   }
 
   private static long runOptimized(BenchmarkData data) {
+    // Single row-major pass mirroring the encoder's duplicate-timestamp merge.
     long nullCount = 0;
-    long[] lastValidPointTimeForTimeDupCheck = new long[COLUMN_COUNT];
-    int[] lastValidPointIndexForTimeDupCheck = new int[COLUMN_COUNT];
-    Arrays.fill(lastValidPointIndexForTimeDupCheck, -1);
-    for (int columnIndex = 0; columnIndex < COLUMN_COUNT; columnIndex++) {
-      for (int rowIndex = 0; rowIndex < ROW_COUNT; rowIndex++) {
-        int valueIndex = data.list.getValueIndex(rowIndex);
-        long time = data.list.getTime(rowIndex);
-        boolean currentValueIsNull = data.list.isNullValue(valueIndex, 
columnIndex);
-        if (!currentValueIsNull) {
-          lastValidPointTimeForTimeDupCheck[columnIndex] = time;
-          lastValidPointIndexForTimeDupCheck[columnIndex] = valueIndex;
-        }
-        if (data.timeDuplicateInfo[rowIndex]) {
-          continue;
+    int[] lastValidPointIndex = new int[COLUMN_COUNT];
+    Arrays.fill(lastValidPointIndex, -1);
+    boolean hasPendingGroup = false;
+    long pendingTime = 0;
+    for (int rowIndex = 0; rowIndex < ROW_COUNT; rowIndex++) {
+      long time = data.list.getTime(rowIndex);
+      if (hasPendingGroup && time != pendingTime) {
+        for (int columnIndex = 0; columnIndex < COLUMN_COUNT; columnIndex++) {
+          if (lastValidPointIndex[columnIndex] < 0) {
+            nullCount++;
+          }
         }
-
-        int originRowIndex;
-        if (lastValidPointIndexForTimeDupCheck[columnIndex] >= 0
-            && time == lastValidPointTimeForTimeDupCheck[columnIndex]) {
-          originRowIndex = lastValidPointIndexForTimeDupCheck[columnIndex];
-        } else {
-          originRowIndex = valueIndex;
+        Arrays.fill(lastValidPointIndex, -1);
+        hasPendingGroup = false;
+      }
+      if (!hasPendingGroup) {
+        hasPendingGroup = true;
+        pendingTime = time;
+      }
+      int valueIndex = data.list.getValueIndex(rowIndex);
+      for (int columnIndex = 0; columnIndex < COLUMN_COUNT; columnIndex++) {
+        if (!data.list.isNullValue(valueIndex, columnIndex)) {
+          lastValidPointIndex[columnIndex] = valueIndex;
         }
-        boolean isNull = originRowIndex == valueIndex && currentValueIsNull;
-        if (isNull) {
+      }
+    }
+    if (hasPendingGroup) {
+      for (int columnIndex = 0; columnIndex < COLUMN_COUNT; columnIndex++) {
+        if (lastValidPointIndex[columnIndex] < 0) {
           nullCount++;
         }
       }
@@ -167,20 +225,25 @@ public class 
AlignedFlushDuplicateTimeCheckPerformanceTest {
     return nullCount;
   }
 
-  private static BenchmarkData createBenchmarkData() {
+  private static BenchmarkData createBenchmarkData(double duplicateRatio, 
double nullRatio) {
     List<TSDataType> dataTypes = new ArrayList<>(COLUMN_COUNT);
     for (int columnIndex = 0; columnIndex < COLUMN_COUNT; columnIndex++) {
       dataTypes.add(TSDataType.INT64);
     }
     AlignedTVList list = AlignedTVList.newAlignedList(dataTypes);
+    int distinctTimestamps = Math.max(1, (int) Math.round(ROW_COUNT * (1.0 - 
duplicateRatio)));
+    int nullPercent = (int) Math.round(nullRatio * 100);
     for (int rowIndex = 0; rowIndex < ROW_COUNT; rowIndex++) {
+      // Monotonic timestamps: distinct timestamps are shared by 
1/(1-duplicateRatio) rows.
+      long time = (long) rowIndex * distinctTimestamps / ROW_COUNT;
       Object[] values = new Object[COLUMN_COUNT];
       for (int columnIndex = 0; columnIndex < COLUMN_COUNT; columnIndex++) {
-        if ((rowIndex + columnIndex) % 4 != 0) {
+        // Deterministic pattern so the legacy and single-pass models see 
identical data.
+        if (Math.floorMod(rowIndex * 31 + columnIndex * 17, 100) >= 
nullPercent) {
           values[columnIndex] = (long) rowIndex + columnIndex;
         }
       }
-      list.putAlignedValue(rowIndex / 4, values);
+      list.putAlignedValue(time, values);
     }
     list.sort();
 
@@ -191,37 +254,6 @@ public class AlignedFlushDuplicateTimeCheckPerformanceTest 
{
     return new BenchmarkData(list, timeDuplicateInfo);
   }
 
-  private static void printResult(
-      int rounds, int scans, Summary legacySummary, Summary optimizedSummary) {
-    System.out.printf(
-        Locale.ROOT,
-        "Aligned flush duplicate-time check benchmark: rows=%d, columns=%d, 
value checks/scan=%d, scans/round=%d, rounds=%d%n",
-        ROW_COUNT,
-        COLUMN_COUNT,
-        ROW_COUNT * COLUMN_COUNT,
-        scans,
-        rounds);
-    printSummary("legacy", legacySummary);
-    printSummary("optimized", optimizedSummary);
-    System.out.printf(
-        Locale.ROOT,
-        "  optimized/legacy CPU ratio=%.2f%%, allocation ratio=%.2f%%%n",
-        percentage(
-            optimizedSummary.getCpuNanosPerOperation(), 
legacySummary.getCpuNanosPerOperation()),
-        percentage(
-            optimizedSummary.getAllocatedBytesPerOperation(),
-            legacySummary.getAllocatedBytesPerOperation()));
-  }
-
-  private static void printSummary(String label, Summary summary) {
-    System.out.printf(
-        Locale.ROOT,
-        "  %-10s CPU=%.3f ms/scan, allocated=%.2f bytes/scan%n",
-        label,
-        summary.getCpuNanosPerOperation() / 1_000_000.0,
-        summary.getAllocatedBytesPerOperation());
-  }
-
   private static double percentage(double numerator, double denominator) {
     return denominator == 0 ? 0 : numerator * 100.0 / denominator;
   }
diff --git 
a/iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/memtable/MemTableFlushTaskTest.java
 
b/iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/memtable/MemTableFlushTaskTest.java
index fd77297b86e..0064637887d 100644
--- 
a/iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/memtable/MemTableFlushTaskTest.java
+++ 
b/iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/memtable/MemTableFlushTaskTest.java
@@ -409,6 +409,97 @@ public class MemTableFlushTaskTest {
     }
   }
 
+  @Test
+  public void testAlignedFlushMergesDuplicateTimestampsAcrossBoundaries() 
throws Exception {
+    checkAlignedFlushMergesDuplicateTimestampsAcrossBoundaries(false, false);
+  }
+
+  @Test
+  public void 
testAlignedFlushMergesDuplicateTimestampsAcrossBoundariesAfterDeletion()
+      throws Exception {
+    checkAlignedFlushMergesDuplicateTimestampsAcrossBoundaries(true, false);
+  }
+
+  @Test
+  public void testAlignedFlushSkipsAllNullDuplicateRowsAcrossBoundaries() 
throws Exception {
+    checkAlignedFlushMergesDuplicateTimestampsAcrossBoundaries(false, true);
+  }
+
+  private void checkAlignedFlushMergesDuplicateTimestampsAcrossBoundaries(
+      boolean removeSecondMeasurement, boolean ignoreAllNullRows) throws 
Exception {
+    List<IMeasurementSchema> schemas =
+        Arrays.asList(
+            new MeasurementSchema("s0", TSDataType.INT64, TSEncoding.PLAIN),
+            new MeasurementSchema("s1", TSDataType.INT64, TSEncoding.PLAIN));
+    AlignedWritableMemChunk memChunk = new AlignedWritableMemChunk(schemas);
+    String alignedFilePath =
+        TestConstant.OUTPUT_DATA_DIR.concat(
+            "duplicateBoundary-" + removeSecondMeasurement + "-" + 
ignoreAllNullRows + ".tsfile");
+
+    try {
+      // Timestamp 5 has four duplicate rows, so it spans page and chunk caps 
2 and 3.
+      memChunk.putAlignedRow(6, new Object[] {6L, 66L});
+      memChunk.putAlignedRow(5, new Object[] {null, 55L});
+      memChunk.putAlignedRow(1, new Object[] {1L, null});
+      memChunk.putAlignedRow(3, new Object[] {3L, null});
+      memChunk.putAlignedRow(5, new Object[] {5L, null});
+      memChunk.putAlignedRow(1, new Object[] {null, 11L});
+      memChunk.putAlignedRow(5, new Object[] {null, null});
+      memChunk.putAlignedRow(3, new Object[] {null, 33L});
+      memChunk.putAlignedRow(1, new Object[] {null, null});
+      memChunk.putAlignedRow(5, new Object[] {null, null});
+      memChunk.putAlignedRow(2, new Object[] {2L, 22L});
+      if (ignoreAllNullRows) {
+        memChunk.putAlignedRow(4, new Object[] {null, null});
+      } else {
+        memChunk.putAlignedRow(4, new Object[] {4L, 44L});
+      }
+      if (removeSecondMeasurement) {
+        memChunk.removeColumn("s1");
+      }
+      memChunk.sortTvListForFlush();
+
+      BlockingQueue<Object> ioTaskQueue = new LinkedBlockingQueue<>();
+      memChunk.encodeWorkingAlignedTVList(ioTaskQueue, 3, 2, 
ignoreAllNullRows);
+      try (TsFileIOWriter alignedWriter = new TsFileIOWriter(new 
File(alignedFilePath))) {
+        
alignedWriter.startChunkGroup(IDeviceID.Factory.DEFAULT_FACTORY.create("root.d"));
+        Object task;
+        while ((task = ioTaskQueue.poll()) != null) {
+          if (task instanceof IChunkWriter chunkWriter) {
+            chunkWriter.writeToFileWriter(alignedWriter);
+          }
+        }
+        alignedWriter.endChunkGroup();
+        alignedWriter.endFile();
+      }
+
+      try (TsFileSequenceReader sequenceReader = new 
TsFileSequenceReader(alignedFilePath);
+          TsFileReader fileReader = new TsFileReader(sequenceReader)) {
+        List<Path> paths = new ArrayList<>();
+        paths.add(new Path("root.d", "s0", false));
+        if (!removeSecondMeasurement) {
+          paths.add(new Path("root.d", "s1", false));
+        }
+        QueryDataSet dataSet = fileReader.query(QueryExpression.create(paths, 
null));
+
+        int[] expectedTimes =
+            ignoreAllNullRows ? new int[] {1, 2, 3, 5, 6} : new int[] {1, 2, 
3, 4, 5, 6};
+        for (int time : expectedTimes) {
+          assertTrue(dataSet.hasNext());
+          RowRecord row = dataSet.next();
+          assertEquals(time, row.getTimestamp());
+          assertEquals(time, row.getFields().get(0).getLongV());
+          if (!removeSecondMeasurement) {
+            assertEquals(time * 11L, row.getFields().get(1).getLongV());
+          }
+        }
+        assertFalse(dataSet.hasNext());
+      }
+    } finally {
+      memChunk.release();
+    }
+  }
+
   @Test
   public void 
testAlignedFastPathKeepsPagesAndValuesAlignedAfterPartialSegmentSort()
       throws IOException, InterruptedException {

Reply via email to