deepthi912 commented on code in PR #19084:
URL: https://github.com/apache/pinot/pull/19084#discussion_r4407881149
##########
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/upsertcompaction/UpsertCompactionTaskExecutor.java:
##########
@@ -137,6 +144,104 @@ protected SegmentConversionResult convert(PinotTaskConfig
pinotTaskConfig, File
return result;
}
+ /// Check B: task-generation (ZK) segment CRC must match the downloaded
deepstore/on-disk copy, with the same
+ /// data-CRC fallback used by server matching ([MinionTaskUtils#crcMatches]).
+ @VisibleForTesting
+ void validateDeepStoreCrc(String tableNameWithType, String segmentName, long
expectedSegmentCrc,
+ long deepstoreSegmentCrc, long deepstoreDataCrc, boolean
ignoreCrcMismatch) {
+ if (ignoreCrcMismatch) {
+ return;
+ }
+ long zkDataCrc = getZkDataCrc(tableNameWithType, segmentName);
+ if (MinionTaskUtils.crcMatches(expectedSegmentCrc, zkDataCrc,
deepstoreSegmentCrc, deepstoreDataCrc)) {
+ return;
+ }
+ String message = "Crc mismatched between ZK and deepstore copy of segment:
" + segmentName
+ + ". Expected crc from ZK: " + expectedSegmentCrc + ", crc from
deepstore: " + deepstoreSegmentCrc
+ + ", zkDataCrc: " + zkDataCrc + ", deepstoreDataCrc: " +
deepstoreDataCrc;
+ LOGGER.error(message);
+ _minionMetrics.addMeteredTableValue(tableNameWithType,
MinionMeter.CRC_MISMATCH_DEEPSTORE, 1L);
+ throw new IllegalStateException(message);
+ }
+
+ /// Check C with a short bounded retry so transient server reload races
(segment uploaded while a replica is still
+ /// rebuilding upsert metadata) do not fail the task on the first attempt.
Does not re-download the segment.
+ @VisibleForTesting
+ RoaringBitmap fetchValidDocIdsWithRetry(PinotTaskConfig pinotTaskConfig,
String tableNameWithType, String segmentName,
Review Comment:
how will retry of fetching valid docIds help? Usually valid docIds will be
missing if the segment is a latest one!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
