danny0405 commented on code in PR #13007:
URL: https://github.com/apache/hudi/pull/13007#discussion_r2017974085
##########
hudi-io/src/main/java/org/apache/hudi/storage/StorageSchemes.java:
##########
@@ -86,6 +88,11 @@ public enum StorageSchemes {
// Hopsworks File System
HOPSFS("hopsfs", false, true);
+ // list files may bring pressure to storage with centralized meta service
like HDFS.
+ // when we want to get only part of files under a directory rather than all
files, use getStatus may be more friendly than listStatus.
+ // here is a trade-off between rpc times and throughput of storage meta
service
+ private static final Set<String> LIST_STATUS_FRIENDLY_SCHEMES = new
HashSet<>(Arrays.asList(FILE.scheme, S3.scheme, S3A.scheme, GCS.scheme));
Review Comment:
it looks like you said for the early commits, but I think the current change
is better because only one invoker uses it, we can generalize it in the future
if there are more use cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
