[
https://issues.apache.org/jira/browse/FLINK-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16194344#comment-16194344
]
ASF GitHub Bot commented on FLINK-6988:
---------------------------------------
GitHub user pnowojski opened a pull request:
https://github.com/apache/flink/pull/4779
[FLINK-6988] Add additional tests coverage for Kafka 0.11 connector
This PR only adds additional tests coverage for Kafka 0.11 connector.
Please check individual commit messages for change log.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pnowojski/flink kafka011-pool
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4779.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4779
----
commit f3e6ef67be4e54471c0143d849e7b905f6e036ae
Author: Aljoscha Krettek <aljoscha.kret...@gmail.com>
Date: 2017-09-25T14:16:34Z
[FLINK-6988] Make TwoPhaseCommitSinkFunction work with Context
commit 7bc6154ef8bdfe1181e404e1f7801f9fbd93543d
Author: Piotr Nowojski <piotr.nowoj...@gmail.com>
Date: 2017-09-06T14:42:59Z
[FLINK-6988] Add Kafka 0.11 connector maven module
commit 3ee004710bf3e875dacb4dd5ebbf0820f9453739
Author: Piotr Nowojski <piotr.nowoj...@gmail.com>
Date: 2017-07-12T13:14:13Z
[FLINK-6988][kafka] Implement our own KafkaProducer class with transactions
recovery
commit 262eb5bfa998083eb28b81cfcb2ae1cc1188c11c
Author: Piotr Nowojski <piotr.nowoj...@gmail.com>
Date: 2017-06-23T07:14:28Z
[FLINK-6988][kafka] Add flink-connector-kafka-0.11 with exactly-once
semantic
commit 489ad56b9750acb1a76b3315cf7723cf0da012ef
Author: Aljoscha Krettek <aljoscha.kret...@gmail.com>
Date: 2017-09-28T12:53:24Z
[hotfix] Don't use deprecated writeWithTimestamps in Kafka 0.10 tests
commit 62a5848ba5ff4f1cbfaa6d2671835cd16e726896
Author: Piotr Nowojski <piotr.nowoj...@gmail.com>
Date: 2017-08-24T11:16:14Z
[hotfix][streaming] Fix typo in parameter and unify naming with parent class
commit 32bfd6300e6fda5f05b495439a581b79abbdae07
Author: Piotr Nowojski <piotr.nowoj...@gmail.com>
Date: 2017-08-24T12:16:55Z
[FLINK-6988][kafka] Add test for failure before before checkpoint and
scaling down
commit 3f8afba1fa49a3c69e3f9b36d7bea3c6316205dd
Author: Piotr Nowojski <piotr.nowoj...@gmail.com>
Date: 2017-08-25T07:47:12Z
[FLINK-6988][kafka] Add Kafka 0.11 tests for scaling down and up again
----
> Add Apache Kafka 0.11 connector
> -------------------------------
>
> Key: FLINK-6988
> URL: https://issues.apache.org/jira/browse/FLINK-6988
> Project: Flink
> Issue Type: Improvement
> Components: Kafka Connector
> Affects Versions: 1.3.1
> Reporter: Piotr Nowojski
> Assignee: Piotr Nowojski
> Fix For: 1.4.0
>
>
> Kafka 0.11 (it will be released very soon) add supports for transactions.
> Thanks to that, Flink might be able to implement Kafka sink supporting
> "exactly-once" semantic. API changes and whole transactions support is
> described in
> [KIP-98|https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging].
> The goal is to mimic implementation of existing BucketingSink. New
> FlinkKafkaProducer011 would
> * upon creation begin transaction, store transaction identifiers into the
> state and would write all incoming data to an output Kafka topic using that
> transaction
> * on `snapshotState` call, it would flush the data and write in state
> information that current transaction is pending to be committed
> * on `notifyCheckpointComplete` we would commit this pending transaction
> * in case of crash between `snapshotState` and `notifyCheckpointComplete` we
> either abort this pending transaction (if not every participant successfully
> saved the snapshot) or restore and commit it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
