[
https://issues.apache.org/jira/browse/FLINK-24041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Cranmer updated FLINK-24041:
----------------------------------
Fix Version/s: 1.15.0
> [FLIP-171] Generic AsyncSinkBase
> --------------------------------
>
> Key: FLINK-24041
> URL: https://issues.apache.org/jira/browse/FLINK-24041
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / Common
> Reporter: Zichen Liu
> Assignee: Zichen Liu
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> h2. Motivation
> Apache Flink has a rich connector ecosystem that can persist data in various
> destinations. Flink natively supports Apache Kafka, Amazon Kinesis Data
> Streams, Elasticsearch, HBase, and many more destinations. Additional
> connectors are maintained in Apache Bahir or directly on GitHub. The basic
> functionality of these sinks is quite similar. They batch events according to
> user defined buffering hints, sign requests and send them to the respective
> endpoint, retry unsuccessful or throttled requests, and participate in
> checkpointing. They primarily just differ in the way they interface with the
> destination. Yet, all the above-mentioned sinks are developed and maintained
> independently.
> We hence propose to create a sink that abstracts away this common
> functionality into a generic sink. Adding support for a new destination then
> just means creating a lightweight shim that only implements the specific
> interfaces of the destination using a client that supports async requests.
> Having a common abstraction will reduce the effort required to maintain all
> these individual sinks. It will also make it much easier and faster to create
> integrations with additional destinations. Moreover, improvements or bug
> fixes to the core of the sink will benefit all implementations that are based
> on it.
> The design of the sink focusses on extensibility and a broad support of
> destinations. The core of the sink is kept generic and free of any connector
> specific dependencies. The sink is designed to participate in checkpointing
> to provide at-least once semantics, but it is limited to destinations that
> provide a client that supports async requests.
> h2. References
> More details to be found
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
