[
https://issues.apache.org/jira/browse/FLINK-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16224668#comment-16224668
]
ASF GitHub Bot commented on FLINK-7784:
---------------------------------------
Github user pnowojski commented on a diff in the pull request:
https://github.com/apache/flink/pull/4910#discussion_r147655000
--- Diff:
flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer011.java
---
@@ -922,6 +945,22 @@ private void readObject(java.io.ObjectInputStream in)
throws IOException, ClassN
producersPool = new ProducersPool();
}
+ /**
+ * Disables the propagation of exceptions thrown when committing
presumably timed out Kafka
+ * transactions during recovery of the job. If a Kafka transaction is
timed out, a commit will
+ * never be successful. Hence, use this feature to avoid recovery loops
of the Job. Exceptions
+ * will still be logged to inform the user that data loss might have
occurred.
+ *
+ * <p>Note that we use {@link System#currentTimeMillis()} to track the
age of a transaction.
+ * Moreover, only exceptions thrown during the recovery are caught,
i.e., the producer will
+ * attempt at least one commit of the transaction before giving up.</p>
+ */
+ @Override
+ public FlinkKafkaProducer011<IN>
disableFailurePropagationAfterTransactionTimeout() {
--- End diff --
nit: Please move this method to the top, somewhere around
`setLogFailuresOnly`
> Don't fail TwoPhaseCommitSinkFunction when failing to commit
> ------------------------------------------------------------
>
> Key: FLINK-7784
> URL: https://issues.apache.org/jira/browse/FLINK-7784
> Project: Flink
> Issue Type: Bug
> Components: DataStream API
> Affects Versions: 1.4.0
> Reporter: Aljoscha Krettek
> Assignee: Gary Yao
> Priority: Blocker
> Fix For: 1.4.0
>
>
> Currently, {{TwoPhaseCommitSinkFunction}} will fail if committing fails
> (either when doing it via the completed checkpoint notification or when
> trying to commit after restoring after failure). This means that the job will
> go into an infinite recovery loop because we will always keep failing.
> In some cases it might be better to ignore those failures and keep on
> processing and this should be the default. We can provide an option that
> allows failing the sink on failing commits.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
