walterddr commented on code in PR #9466:
URL: https://github.com/apache/pinot/pull/9466#discussion_r1028741045
##########
pinot-plugins/pinot-file-system/pinot-s3/src/main/java/org/apache/pinot/plugin/filesystem/S3PinotFS.java:
##########
@@ -435,8 +437,48 @@ public long length(URI fileUri)
@Override
public String[] listFiles(URI fileUri, boolean recursive)
throws IOException {
+ ImmutableList.Builder<String> builder = ImmutableList.builder();
+ visitFiles(fileUri, recursive, s3Object -> {
+ // TODO: Looks like S3PinotFS filters out directories, inconsistent with
the other implementations.
+ // Only add files and not directories
Review Comment:
yeah. even though S3 is an object store it does provide the file-system
based APIs. Users might not know the differences since they are all PinotFS
impl to users and thus it is reasonable to expect same behavior
i think for here, either we dont allow recursive on S3 (explicitly throw
exception if that config is given); or we can do the recursive properly (i dont
know how or even if that's possible though)
##########
pinot-plugins/pinot-file-system/pinot-s3/src/main/java/org/apache/pinot/plugin/filesystem/S3PinotFS.java:
##########
@@ -435,8 +437,48 @@ public long length(URI fileUri)
@Override
public String[] listFiles(URI fileUri, boolean recursive)
throws IOException {
+ ImmutableList.Builder<String> builder = ImmutableList.builder();
+ visitFiles(fileUri, recursive, s3Object -> {
+ // TODO: Looks like S3PinotFS filters out directories, inconsistent with
the other implementations.
+ // Only add files and not directories
Review Comment:
yeah. even though S3 is an object store it does provide the file-system like
APIs. Users might not know the differences since they are all PinotFS impl to
users and thus it is reasonable to expect same behavior
i think for here, either we dont allow recursive on S3 (explicitly throw
exception if that config is given); or we can do the recursive properly (i dont
know how or even if that's possible though)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
