Copilot commented on code in PR #15600:
URL: https://github.com/apache/iotdb/pull/15600#discussion_r2113055391
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/extractor/dataregion/historical/PipeHistoricalDataRegionTsFileAndDeletionExtractor.java:
##########
@@ -617,8 +678,19 @@ private boolean mayTsFileContainUnprocessedData(final
TsFileResource resource) {
if (startIndex instanceof StateProgressIndex) {
startIndex = ((StateProgressIndex) startIndex).getInnerProgressIndex();
}
- return !startIndex.isAfter(resource.getMaxProgressIndexAfterClose())
- && !startIndex.equals(resource.getMaxProgressIndexAfterClose());
+
+ if (pipeName.startsWith(PipeStaticMeta.CONSENSUS_PIPE_PREFIX)) {
Review Comment:
You unwrap `startIndex` for StateProgressIndex but never unwrap the
resource's progress index. To ensure apples-to-apples comparison, you should
also unwrap `resource.getMaxProgressIndexAfterClose()` (and similar) before
calling `greaterThanStartIndex`.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/extractor/dataregion/historical/PipeHistoricalDataRegionTsFileAndDeletionExtractor.java:
##########
@@ -426,6 +435,58 @@ private void flushDataRegionAllTsFiles() {
}
}
+ /**
+ * IoTV2 will only resend event that contains un-replicated local write
data. So we only extract
+ * ProgressIndex containing local writes for comparison to prevent
misjudgment on whether
+ * high-level tsFiles with mixed progressIndexes need to be retransmitted
+ *
+ * @return recoverProgressIndex dedicated in local DataNodeId or origin for
fallback.
+ */
+ private ProgressIndex tryToExtractLocalProgressIndexForIoTV2(ProgressIndex
origin) {
+ // There are only 2 cases:
+ // 1. origin is RecoverProgressIndex
+ if (origin instanceof RecoverProgressIndex) {
+ RecoverProgressIndex toBeTransformed = (RecoverProgressIndex) origin;
+ return new RecoverProgressIndex(
+ toBeTransformed.getDataNodeId2LocalIndex().entrySet().stream()
+ .filter(
+ entry ->
+ entry
+ .getKey()
+
.equals(IoTDBDescriptor.getInstance().getConfig().getDataNodeId()))
+ .collect(Collectors.toMap(Map.Entry::getKey,
Map.Entry::getValue)));
Review Comment:
If the filtered map is empty (no local entry), this will create an empty
RecoverProgressIndex, which may misrepresent progress. Consider detecting when
the map is empty and falling back to the original `origin` index.
```suggestion
Map<Integer, ProgressIndex> filteredMap =
toBeTransformed.getDataNodeId2LocalIndex().entrySet().stream()
.filter(
entry ->
entry
.getKey()
.equals(IoTDBDescriptor.getInstance().getConfig().getDataNodeId()))
.collect(Collectors.toMap(Map.Entry::getKey,
Map.Entry::getValue));
if (filteredMap.isEmpty()) {
return origin; // fallback to origin if no local entry is found
}
return new RecoverProgressIndex(filteredMap);
```
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/extractor/dataregion/historical/PipeHistoricalDataRegionTsFileAndDeletionExtractor.java:
##########
@@ -426,6 +435,58 @@ private void flushDataRegionAllTsFiles() {
}
}
+ /**
+ * IoTV2 will only resend event that contains un-replicated local write
data. So we only extract
+ * ProgressIndex containing local writes for comparison to prevent
misjudgment on whether
+ * high-level tsFiles with mixed progressIndexes need to be retransmitted
+ *
+ * @return recoverProgressIndex dedicated in local DataNodeId or origin for
fallback.
+ */
+ private ProgressIndex tryToExtractLocalProgressIndexForIoTV2(ProgressIndex
origin) {
+ // There are only 2 cases:
+ // 1. origin is RecoverProgressIndex
+ if (origin instanceof RecoverProgressIndex) {
+ RecoverProgressIndex toBeTransformed = (RecoverProgressIndex) origin;
+ return new RecoverProgressIndex(
+ toBeTransformed.getDataNodeId2LocalIndex().entrySet().stream()
Review Comment:
[nitpick] The stream-and-filter logic to extract local entries is duplicated
for both `RecoverProgressIndex` and `HybridProgressIndex`. Consider extracting
a helper method to avoid duplication and improve readability.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
