jackjlli commented on a change in pull request #7585:
URL: https://github.com/apache/pinot/pull/7585#discussion_r732109563
##########
File path:
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/SegmentConversionUtils.java
##########
@@ -72,19 +76,33 @@ public static void uploadSegment(Map<String, String>
configs, List<Header> httpH
// Upload the segment with retry policy
SSLContext sslContext = MinionContext.getInstance().getSSLContext();
+ // Create a RoundRobinURIProvider to round robin IP addresses when retry
uploading. Otherwise may always try to
+ // upload to a same broken host as: 1) DNS may not RR the IP addresses 2)
OS cache the DNS resolution result.
+ RoundRobinURIProvider uriProvider = new RoundRobinURIProvider(new
URI(uploadURL));
Review comment:
Same here. Should we use the max count of `maxNumAttempts` or
`uriProvider.numAddresses()`?
##########
File path:
pinot-common/src/main/java/org/apache/pinot/common/utils/fetcher/HttpSegmentFetcher.java
##########
@@ -35,19 +43,36 @@ protected void doInit(PinotConfiguration config) {
}
@Override
- public void fetchSegmentToLocal(URI uri, File dest)
+ public void fetchSegmentToLocal(URI downloadURI, File dest)
throws Exception {
+ // Create a RoundRobinURIProvider to round robin IP addresses when retry
uploading. Otherwise may always try to
+ // download from a same broken host as: 1) DNS may not RR the IP addresses
2) OS cache the DNS resolution result.
+ RoundRobinURIProvider uriProvider = new RoundRobinURIProvider(downloadURI);
+ _retryCount = Math.max(_retryCount, uriProvider.numAddresses());
Review comment:
Can we use a local variable to store the max retry count? The one from
uriProvider is from an external downloadURI and it shouldn't override the
config of local pinot server. Plus, even if the `_retryCount` needs to be
overriden, it should be marked as `volatile`.
##########
File path:
pinot-common/src/main/java/org/apache/pinot/common/utils/fetcher/HttpSegmentFetcher.java
##########
@@ -35,19 +43,36 @@ protected void doInit(PinotConfiguration config) {
}
@Override
- public void fetchSegmentToLocal(URI uri, File dest)
+ public void fetchSegmentToLocal(URI downloadURI, File dest)
throws Exception {
+ // Create a RoundRobinURIProvider to round robin IP addresses when retry
uploading. Otherwise may always try to
+ // download from a same broken host as: 1) DNS may not RR the IP addresses
2) OS cache the DNS resolution result.
+ RoundRobinURIProvider uriProvider = new RoundRobinURIProvider(downloadURI);
+ _retryCount = Math.max(_retryCount, uriProvider.numAddresses());
+ _logger.info("set retryCount as: {}", _retryCount);
Review comment:
Please provide more detailed information in the log.
##########
File path: pinot-common/pom.xml
##########
@@ -245,7 +245,7 @@
</dependency>
<dependency>
<groupId>org.mockito</groupId>
- <artifactId>mockito-core</artifactId>
+ <artifactId>mockito-inline</artifactId>
Review comment:
Any reason to replace the dependency?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
