dannycranmer commented on a change in pull request #517:
URL: https://github.com/apache/flink-web/pull/517#discussion_r832979017
##########
File path: _posts/2022-03-16-async-sink-base.md
##########
@@ -0,0 +1,138 @@
+---
+layout: post
+title: "Asynchronous Base Sink"
+date: 2022-03-17 16:00:00
+authors:
+- CrynetLogistics:
+ name: "Zichen Liu"
+ twitter: "CrynetLogistics"
+excerpt: An overview of the new features of the new Async Base Sink and
pointers for building your own concrete sink atop
+---
+
+The basic functionalities of sinks in general are quite similar. They batch
records according to user defined buffering hints, sign requests, write them to
the destination, retry unsuccessful or throttled requests, and participate in
checkpointing.
Review comment:
> They batch records according to user defined buffering hints
I would say this is not a "basic functionalities" of sink, this is a more
advanced feature. A basic sink would just send records one by one
##########
File path: _posts/2022-03-16-async-sink-base.md
##########
@@ -0,0 +1,138 @@
+---
+layout: post
+title: "Asynchronous Base Sink"
+date: 2022-03-17 16:00:00
+authors:
+- CrynetLogistics:
+ name: "Zichen Liu"
+ twitter: "CrynetLogistics"
+excerpt: An overview of the new features of the new Async Base Sink and
pointers for building your own concrete sink atop
+---
+
+The basic functionalities of sinks in general are quite similar. They batch
records according to user defined buffering hints, sign requests, write them to
the destination, retry unsuccessful or throttled requests, and participate in
checkpointing.
+
+New for [Flink
1.15](https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink)
is the Async Base Sink - an abstract sink with a number of common
functionalities extracted. Adding support for a new destination now only
requires a lightweight shim that implements the specific interfaces of the
destination using a client that supports async requests.
Review comment:
> is the Async Base Sink - an ab
nit:
> is the Async Base Sink; an ab
##########
File path: _posts/2022-03-16-async-sink-base.md
##########
@@ -0,0 +1,138 @@
+---
Review comment:
General comment, there is some good overview text in the FLIP that you
could reuse to give context here
##########
File path: _posts/2022-03-16-async-sink-base.md
##########
@@ -0,0 +1,138 @@
+---
+layout: post
+title: "Asynchronous Base Sink"
+date: 2022-03-17 16:00:00
+authors:
+- CrynetLogistics:
+ name: "Zichen Liu"
+ twitter: "CrynetLogistics"
+excerpt: An overview of the new features of the new Async Base Sink and
pointers for building your own concrete sink atop
+---
+
+The basic functionalities of sinks in general are quite similar. They batch
records according to user defined buffering hints, sign requests, write them to
the destination, retry unsuccessful or throttled requests, and participate in
checkpointing.
+
+New for [Flink
1.15](https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink)
is the Async Base Sink - an abstract sink with a number of common
functionalities extracted. Adding support for a new destination now only
requires a lightweight shim that implements the specific interfaces of the
destination using a client that supports async requests.
+
+This common abstraction will reduce the effort required to maintain all these
individual sinks, with bugfixes and improvements to the sink core benefiting
all implementations that extend it.
+
+**Attention** The sink is designed to participate in checkpointing to provide
at-least once semantics, but it is limited to destinations that provide a
client that supports async requests.
+
+The design of the sink focuses on extensibility and a broad support of
destinations. The core of the sink is kept generic and free of any connector
specific dependencies.
+
+
+{% toc %}
+
+# Dependency
+To use this base sink, add the following dependency to your project:
+
+```xml
+<dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-connector-base</artifactId>
+ <version>${flink.version}</version>
+</dependency>
+```
+
+
+
+# Public Interfaces
+
+## Generic Types
+
+`<InputT>` – type of elements in a DataStream that should be passed to the sink
+
+`<RequestEntryT>` – type of a payload containing the element and additional
metadata that is required to submit a single element to the destination
+
+
+## Element Converter Interface
+```java
+public interface ElementConverter<InputT, RequestEntryT> extends Serializable {
+ RequestEntryT apply(InputT element, SinkWriter.Context context);
+}
+```
+Concrete sink implementers should provide a way to convert from an element in
the DataStream to the payload type that contains all the additional metadata
required to submit that element to the destination by the sink.
+Ideally this would be hidden from the end user as it allows concrete sink
implementers to adapt to changes in the destination api without breaking end
user code.
+
+
+## Sink Writer Interface
+
+```java
+public abstract class AsyncSinkWriter<InputT, RequestEntryT extends
Serializable>
+ implements SinkWriter<InputT, Void, Collection<RequestEntryT>> {
+ // ...
+ protected abstract void submitRequestEntries
+ (List<RequestEntryT> requestEntries, Consumer<List<RequestEntryT>>
requestResult);
+ // ...
+}
+```
+
+This method should specify how a list of elements from the datastream may be
persisted into the destination. Sink implementers of various datastore and data
processing vendors may use their own clients in connecting to and persisting
the requestEntries received by this method.
Review comment:
> This method should specify how a list of elements from the datastream
may be persisted into the destination
I thought this method writes elements to the destination? This is worded in
a confusing way, sounds like it should return an Object to tell something else
how to write.
##########
File path: _posts/2022-03-16-async-sink-base.md
##########
@@ -0,0 +1,138 @@
+---
+layout: post
+title: "Asynchronous Base Sink"
+date: 2022-03-17 16:00:00
+authors:
+- CrynetLogistics:
+ name: "Zichen Liu"
+ twitter: "CrynetLogistics"
+excerpt: An overview of the new features of the new Async Base Sink and
pointers for building your own concrete sink atop
+---
+
+The basic functionalities of sinks in general are quite similar. They batch
records according to user defined buffering hints, sign requests, write them to
the destination, retry unsuccessful or throttled requests, and participate in
checkpointing.
+
+New for [Flink
1.15](https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink)
is the Async Base Sink - an abstract sink with a number of common
functionalities extracted. Adding support for a new destination now only
requires a lightweight shim that implements the specific interfaces of the
destination using a client that supports async requests.
+
+This common abstraction will reduce the effort required to maintain all these
individual sinks, with bugfixes and improvements to the sink core benefiting
all implementations that extend it.
+
+**Attention** The sink is designed to participate in checkpointing to provide
at-least once semantics, but it is limited to destinations that provide a
client that supports async requests.
Review comment:
> but it is limited to destinations that provide a client that supports
async requests.
nit: Technically it is not, you could manage a thread pool in the concrete
implementation and use a sync client
##########
File path: _posts/2022-03-16-async-sink-base.md
##########
@@ -0,0 +1,138 @@
+---
+layout: post
+title: "Asynchronous Base Sink"
+date: 2022-03-17 16:00:00
+authors:
+- CrynetLogistics:
+ name: "Zichen Liu"
+ twitter: "CrynetLogistics"
+excerpt: An overview of the new features of the new Async Base Sink and
pointers for building your own concrete sink atop
+---
+
+The basic functionalities of sinks in general are quite similar. They batch
records according to user defined buffering hints, sign requests, write them to
the destination, retry unsuccessful or throttled requests, and participate in
checkpointing.
+
+New for [Flink
1.15](https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink)
is the Async Base Sink - an abstract sink with a number of common
functionalities extracted. Adding support for a new destination now only
requires a lightweight shim that implements the specific interfaces of the
destination using a client that supports async requests.
+
+This common abstraction will reduce the effort required to maintain all these
individual sinks, with bugfixes and improvements to the sink core benefiting
all implementations that extend it.
Review comment:
> maintain all these individual sinks
Which individual sinks?
##########
File path: _posts/2022-03-16-async-sink-base.md
##########
@@ -0,0 +1,138 @@
+---
+layout: post
+title: "Asynchronous Base Sink"
+date: 2022-03-17 16:00:00
+authors:
+- CrynetLogistics:
+ name: "Zichen Liu"
+ twitter: "CrynetLogistics"
+excerpt: An overview of the new features of the new Async Base Sink and
pointers for building your own concrete sink atop
+---
+
+The basic functionalities of sinks in general are quite similar. They batch
records according to user defined buffering hints, sign requests, write them to
the destination, retry unsuccessful or throttled requests, and participate in
checkpointing.
+
+New for [Flink
1.15](https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink)
is the Async Base Sink - an abstract sink with a number of common
functionalities extracted. Adding support for a new destination now only
requires a lightweight shim that implements the specific interfaces of the
destination using a client that supports async requests.
+
+This common abstraction will reduce the effort required to maintain all these
individual sinks, with bugfixes and improvements to the sink core benefiting
all implementations that extend it.
+
+**Attention** The sink is designed to participate in checkpointing to provide
at-least once semantics, but it is limited to destinations that provide a
client that supports async requests.
+
+The design of the sink focuses on extensibility and a broad support of
destinations. The core of the sink is kept generic and free of any connector
specific dependencies.
+
+
+{% toc %}
+
+# Dependency
+To use this base sink, add the following dependency to your project:
+
+```xml
+<dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-connector-base</artifactId>
+ <version>${flink.version}</version>
+</dependency>
+```
+
+
+
+# Public Interfaces
+
+## Generic Types
+
+`<InputT>` – type of elements in a DataStream that should be passed to the sink
+
+`<RequestEntryT>` – type of a payload containing the element and additional
metadata that is required to submit a single element to the destination
+
+
+## Element Converter Interface
+```java
+public interface ElementConverter<InputT, RequestEntryT> extends Serializable {
+ RequestEntryT apply(InputT element, SinkWriter.Context context);
+}
+```
+Concrete sink implementers should provide a way to convert from an element in
the DataStream to the payload type that contains all the additional metadata
required to submit that element to the destination by the sink.
+Ideally this would be hidden from the end user as it allows concrete sink
implementers to adapt to changes in the destination api without breaking end
user code.
+
+
+## Sink Writer Interface
+
+```java
+public abstract class AsyncSinkWriter<InputT, RequestEntryT extends
Serializable>
+ implements SinkWriter<InputT, Void, Collection<RequestEntryT>> {
Review comment:
This is out of date:
https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/sink/writer/AsyncSinkWriter.java#L54
##########
File path: _posts/2022-03-16-async-sink-base.md
##########
@@ -0,0 +1,138 @@
+---
+layout: post
+title: "Asynchronous Base Sink"
+date: 2022-03-17 16:00:00
+authors:
+- CrynetLogistics:
+ name: "Zichen Liu"
+ twitter: "CrynetLogistics"
+excerpt: An overview of the new features of the new Async Base Sink and
pointers for building your own concrete sink atop
+---
+
+The basic functionalities of sinks in general are quite similar. They batch
records according to user defined buffering hints, sign requests, write them to
the destination, retry unsuccessful or throttled requests, and participate in
checkpointing.
+
+New for [Flink
1.15](https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink)
is the Async Base Sink - an abstract sink with a number of common
functionalities extracted. Adding support for a new destination now only
requires a lightweight shim that implements the specific interfaces of the
destination using a client that supports async requests.
+
+This common abstraction will reduce the effort required to maintain all these
individual sinks, with bugfixes and improvements to the sink core benefiting
all implementations that extend it.
+
+**Attention** The sink is designed to participate in checkpointing to provide
at-least once semantics, but it is limited to destinations that provide a
client that supports async requests.
+
+The design of the sink focuses on extensibility and a broad support of
destinations. The core of the sink is kept generic and free of any connector
specific dependencies.
+
+
+{% toc %}
+
+# Dependency
+To use this base sink, add the following dependency to your project:
+
+```xml
+<dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-connector-base</artifactId>
+ <version>${flink.version}</version>
+</dependency>
+```
+
+
+
+# Public Interfaces
+
+## Generic Types
+
+`<InputT>` – type of elements in a DataStream that should be passed to the sink
+
+`<RequestEntryT>` – type of a payload containing the element and additional
metadata that is required to submit a single element to the destination
+
+
+## Element Converter Interface
+```java
+public interface ElementConverter<InputT, RequestEntryT> extends Serializable {
+ RequestEntryT apply(InputT element, SinkWriter.Context context);
+}
+```
+Concrete sink implementers should provide a way to convert from an element in
the DataStream to the payload type that contains all the additional metadata
required to submit that element to the destination by the sink.
+Ideally this would be hidden from the end user as it allows concrete sink
implementers to adapt to changes in the destination api without breaking end
user code.
+
+
+## Sink Writer Interface
Review comment:
You did not cover how to snapshot state and handle errors
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
