In that case, why can't you put them all in the same pipeline? On Mon, Jan 4, 2021 at 12:12 AM Arif Alili <[email protected]> wrote:
> Yes, the set is known. > > Each PubSub topic should indexed in different indices in Elasticsearch. > > For example, I have: > PubSub topics: topicA, topicB, topicC - that should be indexed as: IndexA, > indexB, indexC in Elasthcsearch. > > On Sun, Jan 3, 2021 at 4:44 AM Reuven Lax <[email protected]> wrote: > >> Do you know the set of PubSub topics when you launch your pipeline? >> >> On Mon, Dec 28, 2020 at 5:14 AM Arif Alili <[email protected]> wrote: >> >>> Hi all, >>> >>> I am writing to Elasticsearch using Beam (Google Dataflow). The pipeline >>> is ingesting data from PubSub subscription and writing them to >>> Elasticsearch index (using this Dataflow Template >>> <https://github.com/GoogleCloudPlatform/DataflowTemplates/tree/master/v2/pubsub-to-elasticsearch>). >>> This works fine. >>> >>> What I am trying to do now is listen to multiple pubsub topics and write >>> to multiple Elasticsearch indices from one DataFLow job, I want to skip >>> creating multiple DataFlow jobs because the number of >>> pubsub-to-elasticsearch topics can grow fast in near future. >>> >>> I am using Beam's ElasticsearchIO to write data from a single PubSub >>> topic to Elasticsearch, what I need is to change ElasticsearchIO to write >>> to multiple indices. >>> >>> Is anyone familiar with similar architecture? What's the best approach >>> for this scenario? >>> >>> Best, >>> -- >>> *Arif Alili* >>> >>> A: >>> >>> T: >>> E: >>> I: >>> >>> >>> Pilotenstraat 43 bg >>> 1059 CH Amsterdam >>> 020 - 6 71 71 71 <+31+20+6+71+71+71> >>> [email protected] <[name]@propellor.eu> >>> www.propellor.eu >>> >>> >>> <https://www.facebook.com/PropellorEU/> >>> <https://www.linkedin.com/company/10870471> >>> <https://twitter.com/PropellorEU/> >>> >> > > -- > *Arif Alili* > > A: > > T: > E: > I: > > > Pilotenstraat 43 bg > 1059 CH Amsterdam > 020 - 6 71 71 71 <+31+20+6+71+71+71> > [email protected] <[name]@propellor.eu> > www.propellor.eu > > > <https://www.facebook.com/PropellorEU/> > <https://www.linkedin.com/company/10870471> > <https://twitter.com/PropellorEU/> >
