[
https://issues.apache.org/jira/browse/FLINK-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16092701#comment-16092701
]
ASF GitHub Bot commented on FLINK-6988:
---------------------------------------
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/4239
Thanks a lot for opening a pull request for this very important feature
@pnowojski.
I did a rough first pass and had some comments I would like to clear out
first (this is a big chunk of code, we would probably need to go through this
quite a few times before it can be mergeable.)
Most notably, some comments so far:
1. I think we need UTs for the `TwoPhaseCommitSinkFunction`. It alone is a
very important addition (I would even prefer a separate PR for it and try to
merge that first.)
2. Serialization of the transaction state in `TwoPhaseCommitSinkFunction`
needs to be changed
2. Is the `FlinkKafkaProducer011` actually supporting hybrid (normal sink
function and `writeToKafkaWithTimestamps` as a custom sink operator)? From the
looks of it, it doesn't seem like it.
> Add Apache Kafka 0.11 connector
> -------------------------------
>
> Key: FLINK-6988
> URL: https://issues.apache.org/jira/browse/FLINK-6988
> Project: Flink
> Issue Type: Improvement
> Components: Kafka Connector
> Affects Versions: 1.3.1
> Reporter: Piotr Nowojski
> Assignee: Piotr Nowojski
>
> Kafka 0.11 (it will be released very soon) add supports for transactions.
> Thanks to that, Flink might be able to implement Kafka sink supporting
> "exactly-once" semantic. API changes and whole transactions support is
> described in
> [KIP-98|https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging].
> The goal is to mimic implementation of existing BucketingSink. New
> FlinkKafkaProducer011 would
> * upon creation begin transaction, store transaction identifiers into the
> state and would write all incoming data to an output Kafka topic using that
> transaction
> * on `snapshotState` call, it would flush the data and write in state
> information that current transaction is pending to be committed
> * on `notifyCheckpointComplete` we would commit this pending transaction
> * in case of crash between `snapshotState` and `notifyCheckpointComplete` we
> either abort this pending transaction (if not every participant successfully
> saved the snapshot) or restore and commit it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
