infoverload commented on a change in pull request #517: URL: https://github.com/apache/flink-web/pull/517#discussion_r838146254
########## File path: _posts/2022-03-16-async-sink-base.md ########## @@ -0,0 +1,138 @@ +--- +layout: post +title: "Asynchronous Base Sink" +date: 2022-03-17 16:00:00 +authors: +- CrynetLogistics: + name: "Zichen Liu" + twitter: "CrynetLogistics" +excerpt: An overview of the new features of the new Async Base Sink and pointers for building your own concrete sink atop +--- + +The basic functionalities of sinks in general are quite similar. They batch records according to user defined buffering hints, sign requests, write them to the destination, retry unsuccessful or throttled requests, and participate in checkpointing. + +New for [Flink 1.15](https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink) is the Async Base Sink - an abstract sink with a number of common functionalities extracted. Adding support for a new destination now only requires a lightweight shim that implements the specific interfaces of the destination using a client that supports async requests. + +This common abstraction will reduce the effort required to maintain all these individual sinks, with bugfixes and improvements to the sink core benefiting all implementations that extend it. + +**Attention** The sink is designed to participate in checkpointing to provide at-least once semantics, but it is limited to destinations that provide a client that supports async requests. + +The design of the sink focuses on extensibility and a broad support of destinations. The core of the sink is kept generic and free of any connector specific dependencies. + + +{% toc %} + +# Dependency +To use this base sink, add the following dependency to your project: + +```xml +<dependency> + <groupId>org.apache.flink</groupId> + <artifactId>flink-connector-base</artifactId> + <version>${flink.version}</version> +</dependency> +``` + + + +# Public Interfaces + +## Generic Types + +`<InputT>` – type of elements in a DataStream that should be passed to the sink + +`<RequestEntryT>` – type of a payload containing the element and additional metadata that is required to submit a single element to the destination + + +## Element Converter Interface +```java +public interface ElementConverter<InputT, RequestEntryT> extends Serializable { + RequestEntryT apply(InputT element, SinkWriter.Context context); +} +``` +Concrete sink implementers should provide a way to convert from an element in the DataStream to the payload type that contains all the additional metadata required to submit that element to the destination by the sink. +Ideally this would be hidden from the end user as it allows concrete sink implementers to adapt to changes in the destination api without breaking end user code. Review comment: ```suggestion ## ElementConverter Interface ```java public interface ElementConverter<InputT, RequestEntryT> extends Serializable { RequestEntryT apply(InputT element, SinkWriter.Context context); } ``` Classes that implement the concrete sink should provide a way to convert from an element in the DataStream to the payload type that contains all the additional metadata required to submit that element to the destination by the sink. Ideally, this would be hidden from the end user since it allows concrete sink implementers to adapt to changes in the destination API without breaking end user code. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
