9aman commented on code in PR #14741:
URL: https://github.com/apache/pinot/pull/14741#discussion_r1906671563
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java:
##########
@@ -544,6 +557,55 @@ private void doAddConsumingSegment(String segmentName)
_logger.info("Added new CONSUMING segment: {}", segmentName);
}
+ @Override
+ public File downloadSegment(SegmentZKMetadata zkMetadata)
+ throws Exception {
+ if (!PauselessConsumptionUtils.isPauselessEnabled(_tableConfig)) {
+ return super.downloadSegment(zkMetadata);
+ }
+
+ final long startTime = System.currentTimeMillis();
+
+ while (System.currentTimeMillis() - startTime < TIMEOUT_MS) {
+ // ZK Metadata may change during segment download process; fetch it on
every retry.
+ zkMetadata = fetchZKMetadata(zkMetadata.getSegmentName());
+
+ if (zkMetadata.getDownloadUrl() != null) {
+ // The downloadSegment() will throw an exception in case there are
some genuine issues.
+ // We don't want to retry in those scenarios and will throw an
exception
+ return downloadSegmentFromDeepStore(zkMetadata);
+ }
+
+ if (_peerDownloadScheme != null) {
+ _logger.info("Peer download is enabled for the segment: {}",
zkMetadata.getSegmentName());
+ try {
+ return downloadSegmentFromPeers(zkMetadata);
Review Comment:
Have added the check where presence of Online server is a pre-requisite for
us to attempt peer download.
I didn't realize that directly going for peer-download would introduce yet
another retry mechanism. We already are retrying.
Thanks for pointing it out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
