ccaominh commented on a change in pull request #8903: S3 input source
URL: https://github.com/apache/incubator-druid/pull/8903#discussion_r349761023
##########
File path:
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3TimestampVersionedDataFinder.java
##########
@@ -57,39 +57,32 @@ public
S3TimestampVersionedDataFinder(ServerSideEncryptingAmazonS3 s3Client)
public URI getLatestVersion(final URI uri, final @Nullable Pattern pattern)
{
try {
- return RetryUtils.retry(
- () -> {
- final S3Coords coords = new S3Coords(checkURI(uri));
- long mostRecent = Long.MIN_VALUE;
- URI latest = null;
- final Iterator<S3ObjectSummary> objectSummaryIterator =
S3Utils.objectSummaryIterator(
- s3Client,
- coords.bucket,
- coords.path,
- MAX_LISTING_KEYS
- );
- while (objectSummaryIterator.hasNext()) {
- final S3ObjectSummary objectSummary =
objectSummaryIterator.next();
- String keyString =
objectSummary.getKey().substring(coords.path.length());
- if (keyString.startsWith("/")) {
- keyString = keyString.substring(1);
- }
- if (pattern != null && !pattern.matcher(keyString).matches()) {
- continue;
- }
- final long latestModified =
objectSummary.getLastModified().getTime();
- if (latestModified >= mostRecent) {
- mostRecent = latestModified;
- latest = new URI(
- StringUtils.format("s3://%s/%s",
objectSummary.getBucketName(), objectSummary.getKey())
- );
- }
- }
- return latest;
- },
- shouldRetryPredicate(),
- DEFAULT_RETRY_COUNT
+ final CloudObjectLocation coords = new
CloudObjectLocation(S3Utils.checkURI(uri));
+ long mostRecent = Long.MIN_VALUE;
+ URI latest = null;
+ final Iterator<S3ObjectSummary> objectSummaryIterator =
S3Utils.objectSummaryIterator(
+ s3Client,
+ uri,
+ MAX_LISTING_KEYS
);
+ while (objectSummaryIterator.hasNext()) {
+ final S3ObjectSummary objectSummary = objectSummaryIterator.next();
+ String keyString =
objectSummary.getKey().substring(coords.getPath().length());
+ if (keyString.startsWith("/")) {
+ keyString = keyString.substring(1);
+ }
Review comment:
This logic is used in a few places (e.g., `CloudObjectLocation(URI uri`).
May be useful to add a helper function.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
