JackieTien97 commented on code in PR #18647:
URL: https://github.com/apache/iotdb/pull/18647#discussion_r4023182998
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/datastructure/TVList.java:
##########
@@ -966,9 +966,10 @@ protected void skipToCurrentTimeRangeStartPosition() {
this.getQueryContext().getQueryStatistics().addFilteredRowsOfRowLevel(newIndex
- index);
}
index = newIndex;
+ // If the cursor does not move, a duplicate-timestamp group prepared
for the current
+ // position remains valid. Invalidate it only after the cursor
actually advances.
+ probeNext = false;
Review Comment:
**[P1] Invalidate non-aligned batch state before preserving `probeNext`**
`newIndex == index` does not guarantee that `probeNext` still describes the
current record. Non-aligned `TVListIterator.nextBatch()` advances `index`
directly but does not clear `probeNext` before returning. With this change,
switching through an empty fake page preserves that stale `true`, so subsequent
point reads skip `prepareNext()` and can return duplicate timestamps or deleted
points. This matters when an earlier non-overlapping page is read in batches
and a later overlapping page is consumed through the point-reader path.
For example, consider a single non-aligned `LongTVList` with these physical
records, in write order:
```text
index time value
0 1 1
1 100 2
2 100 3
```
Read `[1,33]` in batches, visit the empty page `[34,66]`, then read
`[67,100]` as points:
1. The first `hasNextBatch()` prepares index 0 and sets `probeNext = true`.
2. `nextBatch()` emits `(1,1)`, advances to index 1, and stops because time
100 exceeds the first page boundary. It leaves `probeNext = true`, although
index 1 has not been prepared for point reading.
3. For the empty page, binary search returns index 1 again. Previously, this
branch cleared `probeNext`, allowing `hasNextBatch()` to prepare the time-100
group and advance to its latest record, index 2. After this change, preparation
is skipped and index 1 remains selected.
4. When the last page is selected, time 100 is already inside its range, so
the early return leaves the stale flag untouched. The first point read emits
`(100,2)`; the next read also emits `(100,3)`.
```text
Expected last-page output: [(100,3)]
Parent implementation: [(100,3)]
This PR: [(100,2), (100,3)]
```
The same stale flag bypasses deletion checks. My ASC and DESC deletion
variants both passed with the parent implementation and returned an extra
deleted point with this PR.
Please clear `probeNext` after non-aligned `nextBatch()` advances the
cursor, as the aligned batch implementation already does, and cover the
batch-to-point transition while retaining these aligned regressions.
<details>
<summary>Minimal JUnit reproducer (imports omitted)</summary>
```java
@Test
public void testNonAlignedBatchToPointAfterEmptyPage() throws Exception {
LongTVList list = LongTVList.newList();
list.putLong(1, 1);
list.putLong(100, 2);
list.putLong(100, 3);
MemPointIterator iterator =
list.iterator(
Ordering.ASC,
list.rowCount(),
null,
Collections.emptyList(),
0,
TSEncoding.PLAIN,
1024,
null);
iterator.setCurrentPageTimeRange(new TimeRange(1, 33));
int firstPageRows = 0;
while (iterator.hasNextBatch()) {
firstPageRows += iterator.nextBatch().getPositionCount();
}
Assert.assertEquals(1, firstPageRows);
iterator.setCurrentPageTimeRange(new TimeRange(34, 66));
Assert.assertFalse(iterator.hasNextBatch());
iterator.setCurrentPageTimeRange(new TimeRange(67, 100));
List<String> result = new ArrayList<>();
while (iterator.hasNextTimeValuePair()) {
TimeValuePair pair = iterator.nextTimeValuePair();
result.add(pair.getTimestamp() + ":" + pair.getValue().getLong());
}
Assert.assertEquals(Collections.singletonList("100:3"), result);
}
```
This minimal test supplies the page ranges explicitly. I also reproduced it
with 300,000 physical rows: 299,998 records at time 1 followed by `(100,2)` and
`(100,3)`. `ReadOnlyMemChunk` naturally generated `[1,33]`, `[34,66]`, and
`[67,100]`; the same batch/batch/point sequence through
`LazyMemVersionPageReader` and `PriorityMergeReader` returned `[100:2, 100:3]`
on this PR and `[100:3]` with the parent implementation.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]