[
https://issues.apache.org/jira/browse/FLINK-33260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17776078#comment-17776078
]
Emre Kartoglu edited comment on FLINK-33260 at 10/17/23 9:00 AM:
-----------------------------------------------------------------
The current connector offers configurations like below:
{code:java}
flink.shard.getrecords.maxretries
flink.list.shards.maxretries
{code}
In a similar fashion, I am thinking about a new set of config keys:
{code:java}
flink.shard.getrecords.customexceptions=java.lang.Exception;com.mycompany.CustomException
flink.list.shards.customexceptions=java.lang.Exception;com.mycompany.CustomException
{code}
This would allow the users to re-use the existing retry mechanism with a custom
list of exceptions to retry. Note that the user would need to specify the
exceptions per operation.
If we are going for a custom retry mechanism per exception type, I believe a
configuration like below could be a nice user experience:
{code:java}
flink.shard.getrecords.customexceptions.0.exception=java.lang.Exception
flink.shard.getrecords.customexceptions.0.maxretries=5
flink.shard.getrecords.customexceptions.0.backoff.max=20000
flink.shard.getrecords.customexceptions.1.exception=java.lang.Exception
flink.shard.getrecords.customexceptions.1.maxretries=5
flink.shard.getrecords.customexceptions.1.backoff.max=20000
{code}
was (Author: iemre):
The current connector offers configurations like below:
```
flink.shard.getrecords.maxretries
flink.list.shards.maxretries
```
In a similar fashion, I am thinking about a new set of config keys:
```
flink.shard.getrecords.customexceptions=java.lang.Exception;com.mycompany.CustomException
flink.list.shards.customexceptions=java.lang.Exception;com.mycompany.CustomException
```
This would allow the users to re-use the existing retry mechanism with a custom
list of exceptions to retry. Note that the user would need to specify the
exceptions per operation.
If we are going for a custom retry mechanism per exception type, I believe a
configuration like below could be a nice user experience:
```
flink.shard.getrecords.customexceptions.0.exception=java.lang.Exception
flink.shard.getrecords.customexceptions.0.maxretries=5
flink.shard.getrecords.customexceptions.0.backoff.max=20000
flink.shard.getrecords.customexceptions.1.exception=java.lang.Exception
flink.shard.getrecords.customexceptions.1.maxretries=5
flink.shard.getrecords.customexceptions.1.backoff.max=20000
...
```
> Custom Error Handling for Kinesis Consumer
> ------------------------------------------
>
> Key: FLINK-33260
> URL: https://issues.apache.org/jira/browse/FLINK-33260
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kinesis
> Reporter: Danny Cranmer
> Assignee: Emre Kartoglu
> Priority: Major
> Fix For: aws-connector-4.2.0
>
>
> Background
> The Kinesis Consumer exposes various configuration that allows the user to
> define retry and backoff strategies when dealing with errors. However, the
> configuration does not allow the user to configure which errors are
> retryable, or different strategies for different errors. The error handling
> logic is hard coded within the connector. Over time we discover errors that
> should be retryable that are not, for example KDS throwing 500 on
> SubscribeToShare or transient DNS issues. When these arise the user can
> either fork-fix the connector or log an issue and wait for the next version.
> h3. Scope
> Add the ability for the user to define retry/backoff strategy per error. This
> could be achieved using flexible configuration keys, or allowing the user to
> register their own retry strategies on the connector
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
