Piotr Nowojski created FLINK-6988:
-------------------------------------
Summary: Add Apache Kafka 0.11 connector
Key: FLINK-6988
URL: https://issues.apache.org/jira/browse/FLINK-6988
Project: Flink
Issue Type: Improvement
Components: Kafka Connector
Affects Versions: 1.3.1
Reporter: Piotr Nowojski
Assignee: Piotr Nowojski
Kafka 0.11 (it will be released very soon) add supports for transactions.
Thanks to that, Flink might be able to implement Kafka sink supporting
"exactly-once" semantic. API changes and whole transactions support is
described in
[KIP-98|https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging].
The goal is to mimic implementation of existing BucketingSink. New
KafkaProducer011 would
* upon creation begin transaction, store transaction identifiers into the state
and would write all incoming data to output topic using that transaction
* on `snapshotState` call, it would flush the data and write in state
information that current transaction is pending to be committed
* on `notifyCheckpointComplete` we would commit this pending transaction
* in case of crash between `snapshotState` and `notifyCheckpointComplete` we
either abort this pending transaction (if not every participant successfully
saved the snapshot) or restore and commit it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
