Yes, thanks, I saw this commit. I think we need to add the similar things for S3FileSystem as well to support multiple S3 configs.
> On 20 Aug 2019, at 17:45, Lukasz Cwik <[email protected]> wrote: > > HDFS support in Beam was recently[1] improved to support more than one > cluster. > > 1: > https://github.com/apache/beam/commit/f1dc92f8ec2d4d78b9b60440f821df43dc374e21 > > <https://github.com/apache/beam/commit/f1dc92f8ec2d4d78b9b60440f821df43dc374e21> > On Tue, Aug 20, 2019 at 7:56 AM Alexey Romanenko <[email protected] > <mailto:[email protected]>> wrote: > Hi all, > > I’m looking for a working solution for cases where it’s needed (or even > required) to use different file system configuration (HDFS, S3, GC) in the > same pipeline and where IO is Beam FileSystems based (FileIO, TextIO, etc). > For example: > - reading data from one HDFS cluster and writing results into another one > which requires different configuration; > - reading objects from one S3 bucket, writing into another one and we need to > use different credentials and/or regions for that; > - we even can have heterogeneous case, where we need to read data from HDFS > and write results into S3 or vice versa. > > Usually, in other IOs, we can do this easily by having specific methods, like > “withConfiguration()”, “withCredentialsProvider()”, etc. for Read and Write, > but FileSystems based IO could be configured only with PipelineOptions afaik. > There was a thread about that a while ago [1] where Lukasz Cwik said that > it’s feasible by using different schemes but, unfortunately, I haven’t > managed to make it working on my side (neither for HDFS nor for S3). > > So, any additional inputs or working solutions would be very welcomed if > someone has any. In the long term, I’d like to document this in details > since, I guess, this case can be quite demanded. > > [1] > https://lists.apache.org/thread.html/bb5f98c4154cc72d097ce5b404ff0b3bcb52b7360b0834af7116883b@%3Cdev.beam.apache.org%3E > > <https://lists.apache.org/thread.html/bb5f98c4154cc72d097ce5b404ff0b3bcb52b7360b0834af7116883b@%3Cdev.beam.apache.org%3E> > >
