ColinLeeo commented on code in PR #19035:
URL: https://github.com/apache/iotdb/pull/19035#discussion_r4478126724
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/tablet/parser/TabletInsertionEventParser.java:
##########
@@ -443,9 +443,16 @@ private List<Integer> generateRowIndexList(final long[]
originTimestampColumn) {
}
final List<Integer> rowIndexList = new ArrayList<>();
- // We assume that `originTimestampColumn` is ordered.
- if (originTimestampColumn[originTimestampColumn.length - 1] <
sourceEvent.getStartTime()
- || originTimestampColumn[0] > sourceEvent.getEndTime()) {
+ // The timestamps are not necessarily ordered: the rows of a table-model
tablet are grouped by
+ // device, so its first and last timestamp are not the minimum and the
maximum. The event can
+ // only be skipped without looking at the rows when all of them are
outside the time range.
+ long minTimestamp = Long.MAX_VALUE;
+ long maxTimestamp = Long.MIN_VALUE;
+ for (final long timestamp : originTimestampColumn) {
+ minTimestamp = Math.min(minTimestamp, timestamp);
+ maxTimestamp = Math.max(maxTimestamp, timestamp);
+ }
+ if (maxTimestamp < sourceEvent.getStartTime() || minTimestamp >
sourceEvent.getEndTime()) {
Review Comment:
That comment belonged to an earlier revision of this PR, where the
coordinator reordered every tablet statement by device and then by time — the
pipe then received device-grouped rows, and its "first and last timestamp are
the extremes" shortcut dropped whole events that did contain in-range rows
(`IoTDBPipeNullValueIT` went from 100 received rows to 78). That revision was
reworked: the sort now lives in `TableInsertTabletStatementGenerator` and
orders the table-model INTO tablet by time only, so the pipe receives
time-ordered tablets again and the parser change in this thread is reverted.
The statement was about the order that revision produced, not a claim that
table-model tablets are grouped by device in general.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/sorter/InsertEventSorter.java:
##########
@@ -166,4 +168,57 @@ private int getLastNonnullIndex(
}
return index[lastNonnullIndex];
}
+
+ /** Sort by time only, keeping the relative order of rows that share a
timestamp. */
+ public void sortByTimestampIfNecessary() {
+ if (dataAdapter == null || dataAdapter.getRowSize() == 0) {
+ return;
+ }
+
+ final long[] timestamps = dataAdapter.getTimestamps();
+ final int rowSize = dataAdapter.getRowSize();
+ for (int i = 1; i < rowSize; ++i) {
+ if (timestamps[i] < timestamps[i - 1]) {
+ isSorted = false;
+ break;
+ }
+ }
+
+ if (isSorted) {
+ return;
+ }
+
+ index = new Integer[rowSize];
+ for (int i = 0; i < rowSize; i++) {
+ index[i] = i;
+ }
+ sortTimestamps();
+
+ deDuplicatedIndex = null;
+ deDuplicatedSize = rowSize;
+ sortAndMayDeduplicateValuesAndBitMaps();
+ }
+
+ /**
+ * The permutation the last sort applied: the row at position {@code i} came
from row {@code
+ * permutation[i]} of the input. {@code null} when the rows were already in
order and nothing was
+ * moved.
+ */
+ public int[] getRowPermutation() {
+ if (index == null) {
+ return null;
+ }
+ final int[] permutation = new int[index.length];
+ for (int i = 0; i < index.length; i++) {
+ permutation[i] = index[i];
+ }
+ return permutation;
Review Comment:
`index` is an `Integer[]` (the stable sort needs a comparator over boxed
values) while the permutation is an `int[]`, and
`System.arraycopy`/`Arrays.copyOf` cannot unbox — `System.arraycopy` would
throw `ArrayStoreException`. The loop performs the unboxing and only runs when
a sort actually moved rows.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/sorter/InsertEventSorter.java:
##########
@@ -166,4 +168,57 @@ private int getLastNonnullIndex(
}
return index[lastNonnullIndex];
}
+
+ /** Sort by time only, keeping the relative order of rows that share a
timestamp. */
+ public void sortByTimestampIfNecessary() {
+ if (dataAdapter == null || dataAdapter.getRowSize() == 0) {
+ return;
+ }
+
+ final long[] timestamps = dataAdapter.getTimestamps();
+ final int rowSize = dataAdapter.getRowSize();
+ for (int i = 1; i < rowSize; ++i) {
+ if (timestamps[i] < timestamps[i - 1]) {
+ isSorted = false;
+ break;
+ }
+ }
+
+ if (isSorted) {
+ return;
+ }
+
+ index = new Integer[rowSize];
+ for (int i = 0; i < rowSize; i++) {
+ index[i] = i;
+ }
+ sortTimestamps();
+
+ deDuplicatedIndex = null;
+ deDuplicatedSize = rowSize;
+ sortAndMayDeduplicateValuesAndBitMaps();
+ }
+
+ /**
+ * The permutation the last sort applied: the row at position {@code i} came
from row {@code
+ * permutation[i]} of the input. {@code null} when the rows were already in
order and nothing was
+ * moved.
+ */
+ public int[] getRowPermutation() {
+ if (index == null) {
+ return null;
+ }
+ final int[] permutation = new int[index.length];
+ for (int i = 0; i < index.length; i++) {
+ permutation[i] = index[i];
+ }
+ return permutation;
+ }
+
+ protected void sortTimestamps() {
+ // Index is sorted stably because it is Integer[]
+ Arrays.sort(this.index,
Comparator.comparingLong(dataAdapter::getTimestamp));
+ final long[] timestamps = dataAdapter.getTimestamps();
+ Arrays.sort(timestamps, 0, dataAdapter.getRowSize());
Review Comment:
Done in b4b58c0ca5: `sortTimestamps()` now rearranges the timestamps with
the index it just computed (O(N)) instead of sorting them again (O(N log N)).
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]