GitHub user tzulitai opened a pull request:
https://github.com/apache/flink/pull/3278
[FLINK-5701] [kafka] FlinkKafkaProducer should check asyncException on
checkpoints
Prior to this change, at-least-once is violated for the FlinkKafkaProducer
because we were not failing checkpoints if there were async exceptions from the
Kafka producer.
With this PR, on `snapshotState()`, we fail if there previously were async
exceptions. We also fail if the flushed records on checkpoint resulted in
exceptions.
This PR also improves the tests in `FlinkKafkaProducerBaseTest` to use
one-shot latches instead of sleeping for more stable tests. It also removes the
test `testAtLeastOnceProducerFailsIfFlushingDisabled()`. My reasoning is that
essentially, you _might_ still have at-least-once even if flushing is disabled
(i.e. always no pending records to flush on checkpoint), so I don't see the
necessity in having that test. I'm open to discussing the removal of that test
and adding it back if others think it's necessary.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tzulitai/flink FLINK-5701
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/3278.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3278
----
commit c3eb0a905b86c6265af91c4bda7d2c9da2dc6ce2
Author: Tzu-Li (Gordon) Tai <[email protected]>
Date: 2017-02-06T16:37:13Z
[FLINK-5701] [kafka] FlinkKafkaProducer should check asyncException on
checkpoints
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
