[
https://issues.apache.org/jira/browse/FLINK-31772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713067#comment-17713067
]
Ahmed Hamdy commented on FLINK-31772:
-------------------------------------
Thanks [~dannycranmer]
I have published a fix for the issue
[https://github.com/apache/flink-connector-aws/pull/70]
Attaching Results of performance benchmark and KDS performance after applying
fix in regression cases.
h2. Performace Benchmark Results
|Parallelism/Shards/Payload|paralellism|shards|payload|records/sec|Async
Sink|Async Sink With Configured Ratelimiting Strategy Thourouput (MB/s)|KPL
Thourouput (MB/s)|Percentage of Maximum Thourouput|Improvement Percentage
against KPL|Improvement Percentage against Async Sink|
|Low/Low/Low|1|1|1024|10000|0.991|1|0.958|1|4.2|0.9|
|Low/Low/High|1|1|102400|100|0.9943|1|0.975|1|2.5|0.57|
|Low/Med/Low|1|8|1024|80000|4.12|4.57|6.45|0.57125|-23.5|5.625|
|Low/Med/High|1|8|102400|800|4.35|7.65|7.45|0.95625|2.5|41.25|
|Med/Low/Low|8|1|1024|20000|0.852|0.846|0.545|0.846|30.1|-0.6|
|Med/Low/High|8|1|102400|200|0.921|0.867|0.975|0.867|-10.8|-5.4|
|Med/Med/Low|8|8|1024|80000|5.37|4.76|5.87|0.595|-13.875|-7.625|
|Med/Med/High|8|8|102400|800|7.53|7.69|5.95|0.96125|21.75|2|
|Med/High/Low|8|64|1024|80000|32.5|37.4|40.7|0.58438|-5.15625|7.65625|
|Med/High/High|8|64|102400|800|47.27|60.4|56.27|0.94375|6.45312|20.51562|
|High/High/Low|256|256|1024|300000|127|127|131|0.49609|-1.5625|0|
|High/High/High|256|256|102400|3000|225|246|215|0.96094|12.10938|8.20313|
h2. KDS Sink performance after applying fix
!Screenshot 2023-04-17 at 13.03.34.png!
!Screenshot 2023-04-17 at 13.02.31.png!
> AsyncSinkWriter Performance regression due to AIMD rate limiting strategy
> -------------------------------------------------------------------------
>
> Key: FLINK-31772
> URL: https://issues.apache.org/jira/browse/FLINK-31772
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common, Connectors / Firehose, Connectors /
> Kinesis
> Affects Versions: 1.16.0, aws-connector-3.0.0, 1.16.1,
> aws-connector-4.0.0, aws-connector-4.1.0
> Reporter: Ahmed Hamdy
> Assignee: Ahmed Hamdy
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.16.2, 1.18.0, 1.17.1, aws-connector-4.2.0
>
> Attachments: Screenshot 2023-04-11 at 12.56.10.png, Screenshot
> 2023-04-11 at 12.58.09.png, Screenshot 2023-04-11 at 13.01.47.png, Screenshot
> 2023-04-17 at 13.02.31.png, Screenshot 2023-04-17 at 13.03.24.png, Screenshot
> 2023-04-17 at 13.03.34-1.png, Screenshot 2023-04-17 at 13.03.34.png
>
>
> h1. Issue
> While benchmarking the {{KinesisStreamSink}} for 1.15 against the legacy
> {{FlinkKinesisProduced}} , it is observed that the new sink has a performance
> regression against the deprecated sink for same environment setting.
> Further investigation identified that the AIMD Ratelimiting strategy is the
> bottleneck for the regression.
> Attached results for {{KinesisStreamSink}} against {FlinkKinesisProducer}
> and {KinesisStreamSink} after disabling {{AIMDRatelimitingStrategy}}
> h2. Environment Settings
> - Benchmarking was performed on AWS KDA.
> - Application logic is just sending records downstream
> - Application parallelism was tested to be 1.
> - Kinesis stream number of shards was tested with 8 and 12.
> - payload size was 1Kb and 100Kb.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
