ColinLeeo commented on code in PR #19035:
URL: https://github.com/apache/iotdb/pull/19035#discussion_r4478126724
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/tablet/parser/TabletInsertionEventParser.java:
##########
@@ -443,9 +443,16 @@ private List<Integer> generateRowIndexList(final long[]
originTimestampColumn) {
}
final List<Integer> rowIndexList = new ArrayList<>();
- // We assume that `originTimestampColumn` is ordered.
- if (originTimestampColumn[originTimestampColumn.length - 1] <
sourceEvent.getStartTime()
- || originTimestampColumn[0] > sourceEvent.getEndTime()) {
+ // The timestamps are not necessarily ordered: the rows of a table-model
tablet are grouped by
+ // device, so its first and last timestamp are not the minimum and the
maximum. The event can
+ // only be skipped without looking at the rows when all of them are
outside the time range.
+ long minTimestamp = Long.MAX_VALUE;
+ long maxTimestamp = Long.MIN_VALUE;
+ for (final long timestamp : originTimestampColumn) {
+ minTimestamp = Math.min(minTimestamp, timestamp);
+ maxTimestamp = Math.max(maxTimestamp, timestamp);
+ }
+ if (maxTimestamp < sourceEvent.getStartTime() || minTimestamp >
sourceEvent.getEndTime()) {
Review Comment:
That comment belonged to an earlier revision of this PR, where the
coordinator reordered every tablet statement by device and then by time — the
pipe then received device-grouped rows, and its "first and last timestamp are
the extremes" shortcut dropped whole events that did contain in-range rows
(`IoTDBPipeNullValueIT` went from 100 received rows to 78). That revision was
reworked: the sort now lives in `TableInsertTabletStatementGenerator` and
orders the table-model INTO tablet by time only, so the pipe receives
time-ordered tablets again and the parser change in this thread is reverted.
The statement was about the order that revision produced, not a claim that
table-model tablets are grouped by device in general.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]