emkornfield commented on pull request #3691: URL: https://github.com/apache/iceberg/pull/3691#issuecomment-1017022508
> This still leaves an open question on how we manage dependencies for all of the implementations. For example, if a user does not plan on using S3FileIO or has their own s3 file io implementation that does not depend on boto3, it should not be forced as a hard dependency. I think as a generic answer looking at what [fsspec](https://filesystem-spec.readthedocs.io/en/latest/developer.html#implementing-a-backend) has done (and having these as separate packages) that the use can install in there environment probably makes sense. Specifically for S3, if pyarrow is a hard dependency for parquet reading providing reference implementations based off of its file systems (it comes prepackaged with S3) could make sense. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
