[
https://issues.apache.org/jira/browse/FLINK-29854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17653981#comment-17653981
]
Ahmed Hamdy commented on FLINK-29854:
-------------------------------------
Proposed Improvement under
https://cwiki.apache.org/confluence/display/FLINK/FLIP-284+%3A+Making+AsyncSinkWriter+Flush+triggers+adjustable
will update issue accordingly
> Make Record Size Flush Strategy Optional for Async Sink
> -------------------------------------------------------
>
> Key: FLINK-29854
> URL: https://issues.apache.org/jira/browse/FLINK-29854
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Common
> Reporter: Danny Cranmer
> Assignee: Ahmed Hamdy
> Priority: Major
>
> h3. Background
> Currently AsyncSinkWriter supports three mechanisms that trigger a flush to
> the destination:
> * TIme based
> * Batch size in bytes
> * Number of records in the batch
> For "batch size in bytes" one must implement
> [getSizeInBytes|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/sink/writer/AsyncSinkWriter.java#L202]
> in order for the base to calculate the total batch size. In some cases
> computing the batch size within the AsyncSinkWriter is an expensive
> operation, or not possible. For example, the DynamoDB connector needs to
> determine the serialized size of {{DynamoDbWriteRequest}}.
> (https://github.com/apache/flink-connector-dynamodb/pull/1/files#r1012223894)
> h3. Scope
> Add a feature to make "size in bytes" support optional, this includes:
> - Connectors will not be required to implement {{getSizeInBytes}}
> - Batches will not be validated for max size
> - Records will not be validated for size
> - Batches are not flushed when max size is exceeded
> The sink implementer can decide if it is appropriate to enable this feature.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
