steveloughran opened a new pull request #2125: URL: https://github.com/apache/iceberg/pull/2125
Move to RemoteIterator for scanning directories. It's not as elegant as using the java8 streaming, but it works with the prefetching that the s3a and (soon) abfs connectors do, as well as bailing out more efficiently. Because each directory is probed with its own getFileStatus and list calls, the overhead of the outer list could be entirely swallowed by those inner probes -at least if there is >1 page of results in the listing *and* the implementation is prefetching. Also added check for access errors to also look for AccessDeniedException; that's to support other filesystems and to prepare for HADOOP-15710 ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
