This is a random idea, but the whole file IO system inside Beam would actually be awesome to extract into its own project. IIRC, it’s not particularly tied to Beam.
I’m not saying this should be done now, but it’s be nice to keep it mind for a future goal. -chad On Wed, May 19, 2021 at 10:23 AM Pablo Estrada <pabl...@google.com> wrote: > That would be great to add, Matt. Of course it's important to make this > backwards compatible, but other than that, the addition would be very > welcome. > > On Wed, May 19, 2021 at 9:41 AM Matt Rudary <matt.rud...@twosigma.com> > wrote: > >> Hi, >> >> >> >> This is a quick sketch of a proposal – I wanted to get a sense of whether >> there’s general support for this idea before fleshing it out further, >> getting internal approvals, etc. >> >> >> >> I’m working with multiple storage systems that speak the S3 api. I would >> like to support FileIO operations for these storage systems, but >> S3FileSystem hardcodes the s3 scheme (the various systems use different URI >> schemes) and it is in any case impossible to instantiate more than one in >> the current design. >> >> >> >> I’d like to refactor the code in org.apache.beam.sdk.io.aws.s3 (and maybe >> …aws.options) somewhat to enable this use-case. I haven’t worked out the >> details yet, but it will take some thought to make this work in a non-hacky >> way. >> >> >> >> Thanks >> >> Matt Rudary >> >
