Re: Writing to multiple S3 buckets in multiple regions

2021-04-28 Thread Alexey Romanenko
Hi Valeri, For now it’s not possible to write to the different AWS regions from the same write transform instance. There is an open Jira about this [1]. As a workaround (not very effective maybe, I didn’t try) but I guess you can branch your input PCollection into several branches, depending

Writing to multiple S3 buckets in multiple regions

2021-04-27 Thread Valeri Tsolov
Hello all, we are trying to run an Apache Beam pipeline which reads from Pub/Sub and writes to multiple S3 buckets. The problem comes from the fact that we do not know the region of destination buckets in Amazon. The items are mixed on the Pub/Sub side for multiple s3 buckets in multiple regions.