AHeise opened a new pull request, #152:
URL: https://github.com/apache/flink-connector-kafka/pull/152
Backchannel provides a way for the committer to communicate with the writer,
even in simple non-chained cases, thanks to colocation constraints. This is the
same trick employed in StateFun.
A backchannel is stateless because its state can be entirely derived from
the committer state. This makes it much easier to handle than a StateFun
backchannel.
Backchannel will be used to communicate committed transactions to the writer
in future commits.
- [FLINK-37282] Introduce internal package to sink
Move FlinkKafkaInternalProducer to internal.
- [FLINK-37282] Split KafkaWriter into EOS/non-EOS
Split the easy case of non-transactional writer from the transactional
writer to simplify reasoning about the state (e.g. which fields are used when).
- [FLINK-37282] Add ProducerPool
Add first class producer pool that self-manages all resources and allows to
recycle producers by transactional ids.
- [FLINK-37282] Add Backchannel
- [FLINK-37282] Incorporate Backchannel and ProducerPool
Note that separating backchannel from producer pool into two PRs would
require sending a different message from committer to writer. I wanted to avoid
adding code that lives only for a moment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
