Great, thanks for the feedback! Best, Fabian
2018-02-03 9:37 GMT+01:00 Teena Kappen // BPRISE <teena.kap...@bprise.com>: > Hi Fabian, > > > > We tried the fix that was merged and the sinks are working correctly now. > Thank you for resolving the issue. > > > > Regards, > > Teena > > > > *From:* Teena Kappen // BPRISE > *Sent:* 01 February 2018 19:12 > *To:* Fabian Hueske <fhue...@gmail.com>; Stephan Ewen <se...@apache.org> > *Cc:* Timo Walther <twal...@apache.org>; user@flink.apache.org > *Subject:* RE: Multiple Elasticsearch sinks not working in Flink > > > > @Fabian: I will run the code with the Git repo source and let you know the > results. > > > > @Stephan: Sorry I missed the email from you somehow. I understand from the > JIRA link that you already have the answer for this. Yet I tried using two > separate config map objects in my code and that resolved the issue. Both > the sinks wrote into Elasticsearch as expected. > > > > Thank you for taking this up. I will report back on the test results soon. > > > > Regards, > > Teena > > > > *From:* Fabian Hueske [mailto:fhue...@gmail.com <fhue...@gmail.com>] > *Sent:* 31 January 2018 19:41 > *To:* Stephan Ewen <se...@apache.org> > *Cc:* Teena Kappen // BPRISE <teena.kap...@bprise.com>; Timo Walther < > twal...@apache.org>; user@flink.apache.org > *Subject:* Re: Multiple Elasticsearch sinks not working in Flink > > > > Hi Teena, > > a potential fix for the issue has been merged: https://issues.apache.org/ > jira/browse/FLINK-8489 > <https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-8489&data=02%7C01%7Cteena.kappen%40bprise.com%7C28211d80416345c8fecf08d568b49a35%7C9a9bc2522d894aabb48c9884315adfa9%7C0%7C0%7C636530047252331606&sdata=4VLZXSwvjt9dJtEQi8VvyhtrZ51JMDFJse5NRs5bKcQ%3D&reserved=0> > > It would be great if you could check if that fixes the problem and report > back. > > Thank you, > > Fabian > > > > 2018-01-23 20:04 GMT+01:00 Stephan Ewen <se...@apache.org>: > > As mentioned in the issue, please check if using two different config map > objects solves the issue. > > > > On Tue, Jan 23, 2018 at 1:32 PM, Teena Kappen // BPRISE < > teena.kap...@bprise.com> wrote: > > Thanks Fabian. I will go through it and add info if required. > > > > *From:* Fabian Hueske [mailto:fhue...@gmail.com] > *Sent:* 23 January 2018 15:20 > *To:* Teena Kappen // BPRISE <teena.kap...@bprise.com> > *Cc:* Timo Walther <twal...@apache.org>; user@flink.apache.org > > > *Subject:* Re: Multiple Elasticsearch sinks not working in Flink > > > > Hi Teena, > > I created FLINK-8489 [1] to track the issue. > > Please have a look and add information that might be relevant. > > > > Best, Fabian > > > [1] https://issues.apache.org/jira/browse/FLINK-8489 > <https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-8489&data=02%7C01%7Cteena.kappen%40bprise.com%7C13d8e8f5a69346af547c08d56246ca5d%7Cb7cfea21c1ff4c0a8e2ee7ccd5d8f54b%7C0%7C0%7C636522978545765383&sdata=ZSiaFbWWlnKEgAPZMdqE%2F%2FmdiFixS2T8jDqjAtt%2Btzs%3D&reserved=0> > > > > 2018-01-18 14:16 GMT+01:00 Teena Kappen // BPRISE <teena.kap...@bprise.com > >: > > Hi Timo, > > > > It works fine when the second sink is a Cassandra Sink. The data gets read > from KafkaTopic2 and it gets written to Cassandra as expected. > > > > Regards, > > Teena > > > > *From:* Timo Walther [mailto:twal...@apache.org] > *Sent:* 18 January 2018 18:41 > *To:* user@flink.apache.org > *Subject:* Re: Multiple Elasticsearch sinks not working in Flink > > > > Hi Teena, > > what happens if you replace the second sink with a non-ElasticSearchSink? > Is there the same result? Is the data read from the KafkaTopic2? > > We should determine which system is the bottleneck. > > Regards, > Timo > > > Am 1/18/18 um 9:53 AM schrieb Teena Kappen // BPRISE: > > Hi, > > > > I am running flink 1.4 in single node. My job has two Kafka consumers > reading from separate topics. After fetching the data, the job writes it to > two separate Elasticsearch sinks. So the process is like this > > > > KafkaTopic1 -> Kafkaconsumer1 -> create output record -> > Elasticsearchsink1 > > KafkaTopic2 -> Kafkaconsumer2 -> create output record -> Elasticsearchsink2 > > > > Both the streams and their processing are completely unrelated. The first > sink works as expected and it writes the output for all input records. The > second sink writes to Elasticsearch only once and after that it stops > writing to Elasticsearch even if there is more data that gets fed into > Kafka. Sometimes, it does not even write once. We tested this in two other > jobs and the same issue is there in all of them. > > > > I have attached a sample code I had created to illustrate the issue. We > are using Elasticsearch version 5.6.4 and hence the dependency used is > ‘flink-connector-elasticsearch5_2.11’. > > > > Regards, > > Teena > > > > > > > > > > > > > > > > >
