tonymtu opened a new pull request, #11247:
URL: https://github.com/apache/paimon/pull/11247

   ### Purpose
   
   Directory discovery currently uses `FileIO.listStatus`, which materializes 
all entries. The existing `listFilesIterative` returns files only. Add 
`FileIO.listStatusIterative(Path)` to iterate over the immediate files and 
subdirectories, with a default implementation backed by `listStatus`. The Jindo 
implementation pages through the existing SDK iterator.
   
   Add an optional `FileStatus.getStorageClass()` for callers choosing which 
objects to process. Both Jindo directory listing methods expose the class 
already present in the listing. A directory, a single-file lookup, or an 
unreported class returns `null`. Reading the attribute performs no I/O.
   
   ### Tests
   
   - Built the Jindo plugin and its dependencies. All 115 selected FileIO and 
Jindo tests passed, including wrapper delegation, page errors and the 
single-file case.
   - Tested on OSS with Jindo 6.9.1, 100 keys per page, 205 immediate files 
across five storage classes, and one subdirectory. All three methods returned 
the same files over six traversals each. Both directory listing methods 
returned the subdirectory and the correct class for every file. Also checked a 
single Archive object and an empty directory.
   
   OSS access logs gave the following request counts per traversal. All 
metadata HEAD requests targeted the directory.
   
   | Method | Data page LIST | Directory probe LIST | Directory HEAD | Total |
   | --- | ---: | ---: | ---: | ---: |
   | `listFilesIterative(path, false)` | 3 | 3 | 6 | 12 |
   | `listStatusIterative(path)` | 3 | 3 | 6 | 12 |
   | `listStatus(path)` | 3 | 1 | 2 | 6 |
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to