Re: ElasticsearchIO - Read multiple PubSub topics and write to different Indices

In that case, why can't you put them all in the same pipeline? On Mon, Jan 4, 2021 at 12:12 AM Arif Alili wrote: > Yes, the set is known. > > Each PubSub topic should indexed in different indices in Elasticsearch. > > For example, I have: > PubSub topics: topicA, topicB, topicC - that should be

Re: ElasticsearchIO - Read multiple PubSub topics and write to different Indices

Yes, the set is known. Each PubSub topic should indexed in different indices in Elasticsearch. For example, I have: PubSub topics: topicA, topicB, topicC - that should be indexed as: IndexA, indexB, indexC in Elasthcsearch. On Sun, Jan 3, 2021 at 4:44 AM Reuven Lax wrote: > Do you know the

Re: ElasticsearchIO - Read multiple PubSub topics and write to different Indices

Do you know the set of PubSub topics when you launch your pipeline? On Mon, Dec 28, 2020 at 5:14 AM Arif Alili wrote: > Hi all, > > I am writing to Elasticsearch using Beam (Google Dataflow). The pipeline > is ingesting data from PubSub subscription and writing them to > Elasticsearch index

ElasticsearchIO - Read multiple PubSub topics and write to different Indices

Hi all, I am writing to Elasticsearch using Beam (Google Dataflow). The pipeline is ingesting data from PubSub subscription and writing them to Elasticsearch index (using this Dataflow Template ).