I believe the Boto3 lib should be helpful with right credential configuration when creating a client: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/quickstart.html#configuration
On Thu, Feb 21, 2019 at 6:15 PM Suneel Marthi <suneel.mar...@gmail.com> wrote: > Couldn't u just use Boto python package for doing that ? > > I am writing one now to read from S3 via the Python api > > On Thu, Feb 21, 2019 at 6:19 PM Pablo Estrada <pabl...@google.com> wrote: > >> Hello all, >> I was thinking that a filesystem with support for s3 would be great to >> have in the Python SDK. If I am not wrong, it would simply involve >> implementing the filesystem classes >> <https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/filesystem.py> >> with >> s3, right? >> >> I am not familiar enough with s3, nor with filesystems, nor with AWS in >> general - but I have some outstanding questions: >> >> - Does this mean that we probably would need an extra [s3] target for >> installing apache_beam, like we do with [gcp]? >> - Not strictly necessary, but probably desirable... >> - How do we handle KMS in GCS filesystem? >> - Would the filesystem encapsulation make KMS support in an s3 >> filesystem difficult? >> - Or even more... is the KMS support in AWS very different than in >> GCP? >> - I'd love comments from anyone informed around this : ) >> - Is this project of an appropriate size for a GSoC student? >> >> Thoughts? >> Best >> -P. >> >
