Yeah, this seems like a problem with flink check-pointing. The fact that flink thinks that a checkpoint was successful, but in fact it wasn't.
On Jun 4, 2017 7:37 AM, "Tzu-Li (Gordon) Tai [via Apache Flink User Mailing List archive.]" <ml+s2336050n1347...@n4.nabble.com> wrote: > Thanks for the updates and testing efforts on this! > > I’m sorry that I currently haven’t found the change to look closely into > the testing scenarios you’ve listed, yet. > But please keep us updated on this thread after testing it out also with > the Cloudera build. > > One other suggestion for your test to make sure that some failed record is > actually retried: you can add a dummy verifying operator right before the > Kafka sink. > At least that way you should be able to eliminate the possibility that the > Kafka sink is incorrectly ignoring failed records when checkpointing. From > another look at the Kafka sink code, I’m pretty sure this shouldn’t be the > case. > > Many thanks, > Gordon > > On 4 June 2017 at 2:14:40 PM, ninad ([hidden email] > <http:///user/SendEmail.jtp?type=node&node=13479&i=0>) wrote: > > I tested this with the standalone cluster, and I don't see this problem. > So, > the problem could be that we haven't built Flink against cloudera Hadoop? > I > will test it out. > > > > -- > View this message in context: http://apache-flink-user- > mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss- > tp11413p13477.html > Sent from the Apache Flink User Mailing List archive. mailing list archive > at Nabble.com. > > > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink- > KafkaProducer-Data-Loss-tp11413p13479.html > To unsubscribe from Fink: KafkaProducer Data Loss, click here > <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=11413&code=bm5pbmFkQGdtYWlsLmNvbXwxMTQxM3wtNTE2ODM5Mzg5> > . > NAML > <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p13480.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
