tustvold commented on issue #7237: URL: https://github.com/apache/arrow-rs/issues/7237#issuecomment-2700567681
I wonder if you could use list_with_offset in order to achieve pagination on top of the current APIs? Unless you need to support Azure (#6173) this should function similarly. > Not sure what you mean but it should be possible to explore the file system in a similar way. The underlying listing API is rather specific to the way that object store listing APIs look, and would be a bit of an awkward fit for LocalFileSystems which one could view as cursor based, instead of paginated. > Alternative could be to expose the ListClientExt trait so that we can use list_paginated but only on AWS/GCP/Azure. This is more of an implementation detail and not something I would want to commit to in a public API. There could be a possibility of exposing a dedicated trait for this, in a similar vein to MultipartStore though. This would have the advantage of not needing to wait for a breaking release, as would be required for changes to the main trait. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
