smdsbz commented on code in PR #4834:
URL: https://github.com/apache/paimon/pull/4834#discussion_r1915878011
##########
paimon-common/src/main/java/org/apache/paimon/fs/FileIO.java:
##########
@@ -104,6 +107,59 @@ public interface FileIO extends Serializable {
*/
FileStatus[] listStatus(Path path) throws IOException;
+ /**
+ * List the statuses of the files in the given path if the path is a
directory.
+ *
+ * @param path given path
+ * @param recursive if set to <code>true</code> will recursively list
files in subdirectories,
+ * otherwise only files in the current directory will be listed
+ * @return the statuses of the files in the given path
+ */
+ default FileStatus[] listFiles(Path path, boolean recursive) throws
IOException {
+ List<FileStatus> files = new ArrayList<>();
+ Iterator<FileStatus> iter = listFilesIterative(path, recursive);
+ iter.forEachRemaining(files::add);
+ return files.toArray(new FileStatus[0]);
+ }
+
+ /**
+ * List the statuses of the files iteratively in the given path if the
path is a directory.
+ *
+ * @param path given path
+ * @param recursive if set to <code>true</code> will recursively list
files in subdirectories,
+ * otherwise only files in the current directory will be listed
+ * @return an {@link Iterator} over the statuses of the files in the given
path
+ */
+ default Iterator<FileStatus> listFilesIterative(Path path, boolean
recursive)
+ throws IOException {
+ Queue<FileStatus> files = new LinkedList<>();
+ for (Queue<Path> toUnpack = new
LinkedList<>(Collections.singletonList(path));
+ !toUnpack.isEmpty(); ) {
+ FileStatus[] statuses = listStatus(toUnpack.remove());
+ for (FileStatus f : statuses) {
+ if (!f.isDir()) {
+ files.add(f);
+ continue;
+ }
+ if (!recursive) {
+ continue;
+ }
+ toUnpack.add(f.getPath());
+ }
+ }
+ return new Iterator<FileStatus>() {
+ @Override
+ public boolean hasNext() {
+ return !files.isEmpty();
+ }
+
+ @Override
+ public FileStatus next() {
+ return files.remove();
Review Comment:
Adding a `FileStatusIterator` interface to allow `hasNext` and `next`
throwing `IOException` while lazy listing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@paimon.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
