>From Ali Alsuliman <[email protected]>:
Ali Alsuliman has uploaded this change for review. (
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21315?usp=email )
Change subject: [NO ISSUE][OTH] Ignore directory objects returned for listing
objects from S3
......................................................................
[NO ISSUE][OTH] Ignore directory objects returned for listing objects from S3
- user model changes: no
- storage format changes: no
- interface changes: no
Details:
Some S3-storage implementations return directory objects
in addition to the real objects when listing objects from a bucket.
This causes a failure since we don’t expect such objects to exist.
Directory objects are the ones created with a training
slash / and 0-byte size. We should filter out the objects
ending with / since we don’t create such objects.
Ext-ref: MB-72272
Change-Id: I1a34f22e250403f3b93e7fe1291bd50b97c00ed8
---
M
asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/aws/s3/S3ClientUtils.java
1 file changed, 3 insertions(+), 1 deletion(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb
refs/changes/15/21315/1
diff --git
a/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/aws/s3/S3ClientUtils.java
b/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/aws/s3/S3ClientUtils.java
index 1855110..334ac75 100644
---
a/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/aws/s3/S3ClientUtils.java
+++
b/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/aws/s3/S3ClientUtils.java
@@ -51,7 +51,9 @@
listObjectsResponse =
s3Client.listObjectsV2(listObjectsBuilder.continuationToken(newMarker).build());
}
- files.addAll(listObjectsResponse.contents());
+ // ignore objects ending with "/" since we don't create such
objects.
+ // S3 can return folders as objects with size 0 and key ending
with "/"
+ listObjectsResponse.contents().stream().filter(object ->
!object.key().endsWith("/")).forEach(files::add);
// Mark the flag as done if done, otherwise, get the marker of the
previous response for the next request
if (Boolean.FALSE.equals(listObjectsResponse.isTruncated())) {
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21315?usp=email
To unsubscribe, or for help writing mail filters, visit
https://asterix-gerrit.ics.uci.edu/settings?usp=email
Gerrit-MessageType: newchange
Gerrit-Project: asterixdb
Gerrit-Branch: lumina
Gerrit-Change-Id: I1a34f22e250403f3b93e7fe1291bd50b97c00ed8
Gerrit-Change-Number: 21315
Gerrit-PatchSet: 1
Gerrit-Owner: Ali Alsuliman <[email protected]>
