>From Hussain Towaileb <[email protected]>: Attention is currently required from: Hussain Towaileb, Utsav Singh. Hussain Towaileb has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18206 )
Change subject: [NO ISSUE]: Support Reading Single Depth Files/Folder from S3 Container ...................................................................... Patch Set 2: (5 comments) File asterixdb/asterix-app/src/test/java/org/apache/asterix/test/external_dataset/ExternalDatasetTestUtils.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18206/comment/d7dc49f8_cbf9656f PS2, Line 200: file hierarchy inside browse container Let's also have a test that adds 1500 folders and 1500 files, we need to make sure that we are returning everything since AWS SDK returns 1000 objects at a time. File asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/aws/s3/S3Utils.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18206/comment/dd05a4bb_f490014c PS2, Line 522: Map<String, List<String>> allObjects = listS3ObjectsOfSingleDepth(s3Client, container, prefix); : return allObjects; You can just return the result directly: return listS3ObjectsOfSingleDepth(s3Client, container, prefix); https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18206/comment/36524677_feace699 PS2, Line 543: Boolean boolean https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18206/comment/047a2085_54006773 PS2, Line 546: for (S3Object object : listObjectsInterable.contents()) { I'm not sure if this is handling the 1000+ folders/files correctly, check the comment I added for adding a test for this case to confirm it is working as expected. If this returns the result correctly, then ignore this comment, all good. https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18206/comment/d2d6a99e_00ac76ae PS2, Line 560: for (CommonPrefix object : listObjectsInterable.commonPrefixes()) { Same as above. -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18206 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: Iae9d85eb9899419e63c86322cd8da2273adf89c6 Gerrit-Change-Number: 18206 Gerrit-PatchSet: 2 Gerrit-Owner: Utsav Singh <[email protected]> Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Hussain Towaileb <[email protected]> Gerrit-Reviewer: Jenkins <[email protected]> Gerrit-Reviewer: Utsav Singh <[email protected]> Gerrit-CC: Hussain Towaileb <[email protected]> Gerrit-Attention: Hussain Towaileb <[email protected]> Gerrit-Attention: Utsav Singh <[email protected]> Gerrit-Comment-Date: Mon, 25 Mar 2024 00:18:58 +0000 Gerrit-HasComments: Yes Gerrit-Has-Labels: No Gerrit-MessageType: comment
