rdblue edited a comment on pull request #3691: URL: https://github.com/apache/iceberg/pull/3691#issuecomment-1016999181
@emkornfield, yes. The FileIO abstraction is a weaker abstraction. Iceberg doesn't require file listing, directories, or many of the features of a file system. It only cares that it can read a file as a stream with seek, or write a file and get the byte position from time to time. That's why we maintain a light-weight way to plug in. Having a fsspec FileIO makes a lot of sense for generic use cases, but we don't want to make strict requirements about large dependencies. This has really helped on the Java side, where we are using this to lower dependence on Hadoop's FileSystem API. And it also allows us to do some custom operations, like hook into S3 setup for non-standard credentials chains. fsspec is definitely one of the base implementations that we're talking about. But if you want to avoid that dependency and use boto3 directly, you should be able to. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
