[jira] [Updated] (FLINK-31872) Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink users for KinesisStreamsSink

2024-04-16 Thread Danny Cranmer (Jira) 


 [ 
https://issues.apache.org/jira/browse/FLINK-31872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Danny Cranmer updated FLINK-31872:
--
Affects Version/s: (was: aws-connector-4.2.0)

> Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink 
> users for KinesisStreamsSink
> --
>
> Key: FLINK-31872
> URL: https://issues.apache.org/jira/browse/FLINK-31872
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / Kinesis
>Reporter: Ahmed Hamdy
>Assignee: Ahmed Hamdy
>Priority: Major
> Fix For: aws-connector-4.4.0
>
>
> h1. Issue
> As part of FLINK-31772
> I performed a complete benchmark for {{KinesisStreamsSink}} after configuring 
> rate limiting strategy.
> It appears that optimum values for rate limiting strategy parameters are 
> dependent on use case (shard number/ parallellism/ record thouroughput)
> We initially implemeted the {{AIMDRateLimitingStrategy}} in accordance with 
> one used for TCP congestion control but since parameters are use case 
> dependent we would like to allow sink users to adjust parameters as suitable.
> h2. Requirements
>  - we *must* allow users to configure increment rate and decrease factor of 
> AIMDRateLimitingStrategy for {{KinesisStreamsSink}}
>  - we *must* provide backward compatible default values identical to current 
> values to introduce no further regressions.
> h2. Appendix
> h3. Performace Benchmark Results
> |Parallelism/Shards/Payload|paralellism|shards|payload|records/sec|Async 
> Sink|Async Sink With Configured Ratelimiting Strategy Thourouput (MB/s)|Async 
> sink/ Maximum Thourouput|% of Improvement|
> |Low/Low/Low|1|1|1024|1|0.991|1|1|0.9|
> |Low/Low/High|1|1|102400|100|0.9943|1|1|0.57|
> |Low/Med/Low|1|8|1024|8|4.12|4.57|0.57125|5.625|
> |Low/Med/High|1|8|102400|800|4.35|7.65|0.95625|41.25|
> |Med/Low/Low|8|1|1024|2|0.852|0.846|0.846|-0.6|
> |Med/Low/High|8|1|102400|200|0.921|0.867|0.867|-5.4|
> |Med/Med/Low|8|8|1024|8|5.37|4.76|0.595|-7.625|
> |Med/Med/High|8|8|102400|800|7.53|7.69|0.96125|2|
> |Med/High/Low|8|64|1024|8|32.5|37.4|0.58438|7.65625|
> |Med/High/High|8|64|102400|800|47.27|60.4|0.94375|20.51562|
> |High/High/Low|256|256|1024|30|127|127|0.49609|0|
> |High/High/High|256|256|102400|3000|225|246|0.96094|8.20313|



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (FLINK-31872) Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink users for KinesisStreamsSink

2024-04-16 Thread Danny Cranmer (Jira) 


 [ 
https://issues.apache.org/jira/browse/FLINK-31872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Danny Cranmer updated FLINK-31872:
--
Fix Version/s: aws-connector-4.4.0
   (was: aws-connector-4.3.0)

> Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink 
> users for KinesisStreamsSink
> --
>
> Key: FLINK-31872
> URL: https://issues.apache.org/jira/browse/FLINK-31872
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / Kinesis
>Affects Versions: aws-connector-4.2.0
>Reporter: Ahmed Hamdy
>Assignee: Ahmed Hamdy
>Priority: Major
> Fix For: aws-connector-4.4.0
>
>
> h1. Issue
> As part of FLINK-31772
> I performed a complete benchmark for {{KinesisStreamsSink}} after configuring 
> rate limiting strategy.
> It appears that optimum values for rate limiting strategy parameters are 
> dependent on use case (shard number/ parallellism/ record thouroughput)
> We initially implemeted the {{AIMDRateLimitingStrategy}} in accordance with 
> one used for TCP congestion control but since parameters are use case 
> dependent we would like to allow sink users to adjust parameters as suitable.
> h2. Requirements
>  - we *must* allow users to configure increment rate and decrease factor of 
> AIMDRateLimitingStrategy for {{KinesisStreamsSink}}
>  - we *must* provide backward compatible default values identical to current 
> values to introduce no further regressions.
> h2. Appendix
> h3. Performace Benchmark Results
> |Parallelism/Shards/Payload|paralellism|shards|payload|records/sec|Async 
> Sink|Async Sink With Configured Ratelimiting Strategy Thourouput (MB/s)|Async 
> sink/ Maximum Thourouput|% of Improvement|
> |Low/Low/Low|1|1|1024|1|0.991|1|1|0.9|
> |Low/Low/High|1|1|102400|100|0.9943|1|1|0.57|
> |Low/Med/Low|1|8|1024|8|4.12|4.57|0.57125|5.625|
> |Low/Med/High|1|8|102400|800|4.35|7.65|0.95625|41.25|
> |Med/Low/Low|8|1|1024|2|0.852|0.846|0.846|-0.6|
> |Med/Low/High|8|1|102400|200|0.921|0.867|0.867|-5.4|
> |Med/Med/Low|8|8|1024|8|5.37|4.76|0.595|-7.625|
> |Med/Med/High|8|8|102400|800|7.53|7.69|0.96125|2|
> |Med/High/Low|8|64|1024|8|32.5|37.4|0.58438|7.65625|
> |Med/High/High|8|64|102400|800|47.27|60.4|0.94375|20.51562|
> |High/High/Low|256|256|1024|30|127|127|0.49609|0|
> |High/High/High|256|256|102400|3000|225|246|0.96094|8.20313|



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (FLINK-31872) Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink users for KinesisStreamsSink

2023-10-31 Thread Danny Cranmer (Jira) 


 [ 
https://issues.apache.org/jira/browse/FLINK-31872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Danny Cranmer updated FLINK-31872:
--
Fix Version/s: aws-connector-4.3.0
   (was: aws-connector-4.2.0)

> Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink 
> users for KinesisStreamsSink
> --
>
> Key: FLINK-31872
> URL: https://issues.apache.org/jira/browse/FLINK-31872
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / Kinesis
>Reporter: Ahmed Hamdy
>Assignee: Ahmed Hamdy
>Priority: Major
> Fix For: aws-connector-4.3.0
>
>
> h1. Issue
> As part of FLINK-31772
> I performed a complete benchmark for {{KinesisStreamsSink}} after configuring 
> rate limiting strategy.
> It appears that optimum values for rate limiting strategy parameters are 
> dependent on use case (shard number/ parallellism/ record thouroughput)
> We initially implemeted the {{AIMDRateLimitingStrategy}} in accordance with 
> one used for TCP congestion control but since parameters are use case 
> dependent we would like to allow sink users to adjust parameters as suitable.
> h2. Requirements
>  - we *must* allow users to configure increment rate and decrease factor of 
> AIMDRateLimitingStrategy for {{KinesisStreamsSink}}
>  - we *must* provide backward compatible default values identical to current 
> values to introduce no further regressions.
> h2. Appendix
> h3. Performace Benchmark Results
> |Parallelism/Shards/Payload|paralellism|shards|payload|records/sec|Async 
> Sink|Async Sink With Configured Ratelimiting Strategy Thourouput (MB/s)|Async 
> sink/ Maximum Thourouput|% of Improvement|
> |Low/Low/Low|1|1|1024|1|0.991|1|1|0.9|
> |Low/Low/High|1|1|102400|100|0.9943|1|1|0.57|
> |Low/Med/Low|1|8|1024|8|4.12|4.57|0.57125|5.625|
> |Low/Med/High|1|8|102400|800|4.35|7.65|0.95625|41.25|
> |Med/Low/Low|8|1|1024|2|0.852|0.846|0.846|-0.6|
> |Med/Low/High|8|1|102400|200|0.921|0.867|0.867|-5.4|
> |Med/Med/Low|8|8|1024|8|5.37|4.76|0.595|-7.625|
> |Med/Med/High|8|8|102400|800|7.53|7.69|0.96125|2|
> |Med/High/Low|8|64|1024|8|32.5|37.4|0.58438|7.65625|
> |Med/High/High|8|64|102400|800|47.27|60.4|0.94375|20.51562|
> |High/High/Low|256|256|1024|30|127|127|0.49609|0|
> |High/High/High|256|256|102400|3000|225|246|0.96094|8.20313|



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (FLINK-31872) Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink users for KinesisStreamsSink

2023-10-31 Thread Danny Cranmer (Jira) 


 [ 
https://issues.apache.org/jira/browse/FLINK-31872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Danny Cranmer updated FLINK-31872:
--
Affects Version/s: aws-connector-4.2.0

> Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink 
> users for KinesisStreamsSink
> --
>
> Key: FLINK-31872
> URL: https://issues.apache.org/jira/browse/FLINK-31872
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / Kinesis
>Affects Versions: aws-connector-4.2.0
>Reporter: Ahmed Hamdy
>Assignee: Ahmed Hamdy
>Priority: Major
> Fix For: aws-connector-4.3.0
>
>
> h1. Issue
> As part of FLINK-31772
> I performed a complete benchmark for {{KinesisStreamsSink}} after configuring 
> rate limiting strategy.
> It appears that optimum values for rate limiting strategy parameters are 
> dependent on use case (shard number/ parallellism/ record thouroughput)
> We initially implemeted the {{AIMDRateLimitingStrategy}} in accordance with 
> one used for TCP congestion control but since parameters are use case 
> dependent we would like to allow sink users to adjust parameters as suitable.
> h2. Requirements
>  - we *must* allow users to configure increment rate and decrease factor of 
> AIMDRateLimitingStrategy for {{KinesisStreamsSink}}
>  - we *must* provide backward compatible default values identical to current 
> values to introduce no further regressions.
> h2. Appendix
> h3. Performace Benchmark Results
> |Parallelism/Shards/Payload|paralellism|shards|payload|records/sec|Async 
> Sink|Async Sink With Configured Ratelimiting Strategy Thourouput (MB/s)|Async 
> sink/ Maximum Thourouput|% of Improvement|
> |Low/Low/Low|1|1|1024|1|0.991|1|1|0.9|
> |Low/Low/High|1|1|102400|100|0.9943|1|1|0.57|
> |Low/Med/Low|1|8|1024|8|4.12|4.57|0.57125|5.625|
> |Low/Med/High|1|8|102400|800|4.35|7.65|0.95625|41.25|
> |Med/Low/Low|8|1|1024|2|0.852|0.846|0.846|-0.6|
> |Med/Low/High|8|1|102400|200|0.921|0.867|0.867|-5.4|
> |Med/Med/Low|8|8|1024|8|5.37|4.76|0.595|-7.625|
> |Med/Med/High|8|8|102400|800|7.53|7.69|0.96125|2|
> |Med/High/Low|8|64|1024|8|32.5|37.4|0.58438|7.65625|
> |Med/High/High|8|64|102400|800|47.27|60.4|0.94375|20.51562|
> |High/High/Low|256|256|1024|30|127|127|0.49609|0|
> |High/High/High|256|256|102400|3000|225|246|0.96094|8.20313|



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (FLINK-31872) Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink users for KinesisStreamsSink

2023-04-20 Thread Ahmed Hamdy (Jira) 


 [ 
https://issues.apache.org/jira/browse/FLINK-31872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed Hamdy updated FLINK-31872:

Description: 
h1. Issue

As part of FLINK-31772

I performed a complete benchmark for {{KinesisStreamsSink}} after configuring 
rate limiting strategy.
It appears that optimum values for rate limiting strategy parameters are 
dependent on use case (shard number/ parallellism/ record thouroughput)
We initially implemeted the {{AIMDRateLimitingStrategy}} in accordance with one 
used for TCP congestion control but since parameters are use case dependent we 
would like to allow sink users to adjust parameters as suitable.
h2. Requirements
 - we *must* allow users to configure increment rate and decrease factor of 
AIMDRateLimitingStrategy for {{KinesisStreamsSink}}
 - we *must* provide backward compatible default values identical to current 
values to introduce no further regressions.

h2. Appendix
h3. Performace Benchmark Results
|Parallelism/Shards/Payload|paralellism|shards|payload|records/sec|Async 
Sink|Async Sink With Configured Ratelimiting Strategy Thourouput (MB/s)|Async 
sink/ Maximum Thourouput|% of Improvement|
|Low/Low/Low|1|1|1024|1|0.991|1|1|0.9|
|Low/Low/High|1|1|102400|100|0.9943|1|1|0.57|
|Low/Med/Low|1|8|1024|8|4.12|4.57|0.57125|5.625|
|Low/Med/High|1|8|102400|800|4.35|7.65|0.95625|41.25|
|Med/Low/Low|8|1|1024|2|0.852|0.846|0.846|-0.6|
|Med/Low/High|8|1|102400|200|0.921|0.867|0.867|-5.4|
|Med/Med/Low|8|8|1024|8|5.37|4.76|0.595|-7.625|
|Med/Med/High|8|8|102400|800|7.53|7.69|0.96125|2|
|Med/High/Low|8|64|1024|8|32.5|37.4|0.58438|7.65625|
|Med/High/High|8|64|102400|800|47.27|60.4|0.94375|20.51562|
|High/High/Low|256|256|1024|30|127|127|0.49609|0|
|High/High/High|256|256|102400|3000|225|246|0.96094|8.20313|

  was:
h1. Issue

As part of FLINK-31772

I performed a complete benchmark for {{KinesisStreamsSink}} after configuring 
rate limiting strategy.
It appears that optimum values for rate limiting strategy parameters are 
dependent on use case (shard number/ parallellism/ record thouroughput)
We initially implemeted the {{AIMDRateLimitingStrategy}} in accordance with one 
used for TCP congestion control but since parameters are use case dependent we 
would like to allow sink users to adjust parameters as suitable. 

h2. Requirements

- we *must* allow users to configure increment rate and decrease factor of 
AIMDRateLimitingStrategy for {{KinesisStreamsSink}}
- we *must* provide backward compatible default values identical to current 
values to introduce no further regressions.


h2. Appendix

h3. Performace Benchmark Results
|Parallelism/Shards/Payload|paralellism|shards|payload|records/sec|Async 
Sink|Async Sink With Configured Ratelimiting Strategy Thourouput (MB/s)|KPL 
Thourouput (MB/s)|Async sink/ Maximum Thourouput|Async sink /KPL|% of 
Improvement|
|Low/Low/Low|1|1|1024|1|0.991|1|0.958|1|4.2|0.9|
|Low/Low/High|1|1|102400|100|0.9943|1|0.975|1|2.5|0.57|
|Low/Med/Low|1|8|1024|8|4.12|4.57|6.45|0.57125|-23.5|5.625|
|Low/Med/High|1|8|102400|800|4.35|7.65|7.45|0.95625|2.5|41.25|
|Med/Low/Low|8|1|1024|2|0.852|0.846|0.545|0.846|30.1|-0.6|
|Med/Low/High|8|1|102400|200|0.921|0.867|0.975|0.867|-10.8|-5.4|
|Med/Med/Low|8|8|1024|8|5.37|4.76|5.87|0.595|-13.875|-7.625|
|Med/Med/High|8|8|102400|800|7.53|7.69|5.95|0.96125|21.75|2|
|Med/High/Low|8|64|1024|8|32.5|37.4|40.7|0.58438|-5.15625|7.65625|
|Med/High/High|8|64|102400|800|47.27|60.4|56.27|0.94375|6.45312|20.51562|
|High/High/Low|256|256|1024|30|127|127|131|0.49609|-1.5625|0|
|High/High/High|256|256|102400|3000|225|246|215|0.96094|12.10938|8.20313|


> Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink 
> users for KinesisStreamsSink
> --
>
> Key: FLINK-31872
> URL: https://issues.apache.org/jira/browse/FLINK-31872
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / Kinesis
>Reporter: Ahmed Hamdy
>Assignee: Ahmed Hamdy
>Priority: Major
> Fix For: aws-connector-4.2.0
>
>
> h1. Issue
> As part of FLINK-31772
> I performed a complete benchmark for {{KinesisStreamsSink}} after configuring 
> rate limiting strategy.
> It appears that optimum values for rate limiting strategy parameters are 
> dependent on use case (shard number/ parallellism/ record thouroughput)
> We initially implemeted the {{AIMDRateLimitingStrategy}} in accordance with 
> one used for TCP congestion control but since parameters are use case 
> dependent we would like to allow sink users to adjust parameters as suitable.
> h2. Requirements
>  - we *must* allow users to configure increment rate and decrease factor of 
> AIMDRateLimitingStrategy for {{KinesisStreamsSink}}
>  - we *must* provide

[jira] [Updated] (FLINK-31872) Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink users for KinesisStreamsSink

2023-04-20 Thread Ahmed Hamdy (Jira) 


 [ 
https://issues.apache.org/jira/browse/FLINK-31872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed Hamdy updated FLINK-31872:

Description: 
h1. Issue

As part of FLINK-31772

I performed a complete benchmark for {{KinesisStreamsSink}} after configuring 
rate limiting strategy.
It appears that optimum values for rate limiting strategy parameters are 
dependent on use case (shard number/ parallellism/ record thouroughput)
We initially implemeted the {{AIMDRateLimitingStrategy}} in accordance with one 
used for TCP congestion control but since parameters are use case dependent we 
would like to allow sink users to adjust parameters as suitable. 

h2. Requirements

- we *must* allow users to configure increment rate and decrease factor of 
AIMDRateLimitingStrategy for {{KinesisStreamsSink}}
- we *must* provide backward compatible default values identical to current 
values to introduce no further regressions.


h2. Appendix

h3. Performace Benchmark Results
|Parallelism/Shards/Payload|paralellism|shards|payload|records/sec|Async 
Sink|Async Sink With Configured Ratelimiting Strategy Thourouput (MB/s)|KPL 
Thourouput (MB/s)|Async sink/ Maximum Thourouput|Async sink /KPL|% of 
Improvement|
|Low/Low/Low|1|1|1024|1|0.991|1|0.958|1|4.2|0.9|
|Low/Low/High|1|1|102400|100|0.9943|1|0.975|1|2.5|0.57|
|Low/Med/Low|1|8|1024|8|4.12|4.57|6.45|0.57125|-23.5|5.625|
|Low/Med/High|1|8|102400|800|4.35|7.65|7.45|0.95625|2.5|41.25|
|Med/Low/Low|8|1|1024|2|0.852|0.846|0.545|0.846|30.1|-0.6|
|Med/Low/High|8|1|102400|200|0.921|0.867|0.975|0.867|-10.8|-5.4|
|Med/Med/Low|8|8|1024|8|5.37|4.76|5.87|0.595|-13.875|-7.625|
|Med/Med/High|8|8|102400|800|7.53|7.69|5.95|0.96125|21.75|2|
|Med/High/Low|8|64|1024|8|32.5|37.4|40.7|0.58438|-5.15625|7.65625|
|Med/High/High|8|64|102400|800|47.27|60.4|56.27|0.94375|6.45312|20.51562|
|High/High/Low|256|256|1024|30|127|127|131|0.49609|-1.5625|0|
|High/High/High|256|256|102400|3000|225|246|215|0.96094|12.10938|8.20313|

  was:
h1. Issue

As part of FLINK-31772

I performed a complete benchmark for


> Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink 
> users for KinesisStreamsSink
> --
>
> Key: FLINK-31872
> URL: https://issues.apache.org/jira/browse/FLINK-31872
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / Kinesis
>Reporter: Ahmed Hamdy
>Assignee: Ahmed Hamdy
>Priority: Major
> Fix For: aws-connector-4.2.0
>
>
> h1. Issue
> As part of FLINK-31772
> I performed a complete benchmark for {{KinesisStreamsSink}} after configuring 
> rate limiting strategy.
> It appears that optimum values for rate limiting strategy parameters are 
> dependent on use case (shard number/ parallellism/ record thouroughput)
> We initially implemeted the {{AIMDRateLimitingStrategy}} in accordance with 
> one used for TCP congestion control but since parameters are use case 
> dependent we would like to allow sink users to adjust parameters as suitable. 
> h2. Requirements
> - we *must* allow users to configure increment rate and decrease factor of 
> AIMDRateLimitingStrategy for {{KinesisStreamsSink}}
> - we *must* provide backward compatible default values identical to current 
> values to introduce no further regressions.
> h2. Appendix
> h3. Performace Benchmark Results
> |Parallelism/Shards/Payload|paralellism|shards|payload|records/sec|Async 
> Sink|Async Sink With Configured Ratelimiting Strategy Thourouput (MB/s)|KPL 
> Thourouput (MB/s)|Async sink/ Maximum Thourouput|Async sink /KPL|% of 
> Improvement|
> |Low/Low/Low|1|1|1024|1|0.991|1|0.958|1|4.2|0.9|
> |Low/Low/High|1|1|102400|100|0.9943|1|0.975|1|2.5|0.57|
> |Low/Med/Low|1|8|1024|8|4.12|4.57|6.45|0.57125|-23.5|5.625|
> |Low/Med/High|1|8|102400|800|4.35|7.65|7.45|0.95625|2.5|41.25|
> |Med/Low/Low|8|1|1024|2|0.852|0.846|0.545|0.846|30.1|-0.6|
> |Med/Low/High|8|1|102400|200|0.921|0.867|0.975|0.867|-10.8|-5.4|
> |Med/Med/Low|8|8|1024|8|5.37|4.76|5.87|0.595|-13.875|-7.625|
> |Med/Med/High|8|8|102400|800|7.53|7.69|5.95|0.96125|21.75|2|
> |Med/High/Low|8|64|1024|8|32.5|37.4|40.7|0.58438|-5.15625|7.65625|
> |Med/High/High|8|64|102400|800|47.27|60.4|56.27|0.94375|6.45312|20.51562|
> |High/High/Low|256|256|1024|30|127|127|131|0.49609|-1.5625|0|
> |High/High/High|256|256|102400|3000|225|246|215|0.96094|12.10938|8.20313|



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (FLINK-31872) Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink users for KinesisStreamsSink

2023-04-20 Thread Ahmed Hamdy (Jira) 


 [ 
https://issues.apache.org/jira/browse/FLINK-31872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed Hamdy updated FLINK-31872:

Description: 
h1. Issue

As part of FLINK-31772

I performed a complete benchmark for

  was:
h1. Issue

While benchmarking the {{KinesisStreamSink}} for 1.15 against the legacy 
{{FlinkKinesisProduced}} , it is observed that the new sink has a performance 
regression against the deprecated sink for same environment setting.

Further investigation identified that the AIMD Ratelimiting strategy is the 
bottleneck for the regression. 

Attached results for {{KinesisStreamSink}}  against {FlinkKinesisProducer} and 
{KinesisStreamSink} after disabling {{AIMDRatelimitingStrategy}}



h2. Environment Settings
- Benchmarking was performed on AWS KDA.
- Application logic is just sending records downstream
- Application parallelism was tested to be 1.
- Kinesis stream number of shards was tested with 8 and 12.
- payload size was 1Kb and 100Kb.

 


> Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink 
> users for KinesisStreamsSink
> --
>
> Key: FLINK-31872
> URL: https://issues.apache.org/jira/browse/FLINK-31872
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / Kinesis
>Reporter: Ahmed Hamdy
>Assignee: Ahmed Hamdy
>Priority: Major
> Fix For: aws-connector-4.2.0
>
>
> h1. Issue
> As part of FLINK-31772
> I performed a complete benchmark for



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (FLINK-31872) Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink users for KinesisStreamsSink

2023-04-20 Thread Ahmed Hamdy (Jira) 


 [ 
https://issues.apache.org/jira/browse/FLINK-31872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed Hamdy updated FLINK-31872:

Affects Version/s: (was: 1.16.0)
   (was: aws-connector-3.0.0)
   (was: 1.16.1)
   (was: aws-connector-4.0.0)
   (was: aws-connector-4.1.0)

> Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink 
> users for KinesisStreamsSink
> --
>
> Key: FLINK-31872
> URL: https://issues.apache.org/jira/browse/FLINK-31872
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / Common, Connectors / Firehose, Connectors / 
> Kinesis
>Reporter: Ahmed Hamdy
>Assignee: Ahmed Hamdy
>Priority: Major
>  Labels: pull-request-available
> Fix For: aws-connector-4.2.0
>
>
> h1. Issue
> While benchmarking the {{KinesisStreamSink}} for 1.15 against the legacy 
> {{FlinkKinesisProduced}} , it is observed that the new sink has a performance 
> regression against the deprecated sink for same environment setting.
> Further investigation identified that the AIMD Ratelimiting strategy is the 
> bottleneck for the regression. 
> Attached results for {{KinesisStreamSink}}  against {FlinkKinesisProducer} 
> and {KinesisStreamSink} after disabling {{AIMDRatelimitingStrategy}}
> h2. Environment Settings
> - Benchmarking was performed on AWS KDA.
> - Application logic is just sending records downstream
> - Application parallelism was tested to be 1.
> - Kinesis stream number of shards was tested with 8 and 12.
> - payload size was 1Kb and 100Kb.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (FLINK-31872) Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink users for KinesisStreamsSink

2023-04-20 Thread Ahmed Hamdy (Jira) 


 [ 
https://issues.apache.org/jira/browse/FLINK-31872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed Hamdy updated FLINK-31872:

Labels:   (was: pull-request-available)

> Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink 
> users for KinesisStreamsSink
> --
>
> Key: FLINK-31872
> URL: https://issues.apache.org/jira/browse/FLINK-31872
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / Kinesis
>Reporter: Ahmed Hamdy
>Assignee: Ahmed Hamdy
>Priority: Major
> Fix For: aws-connector-4.2.0
>
>
> h1. Issue
> While benchmarking the {{KinesisStreamSink}} for 1.15 against the legacy 
> {{FlinkKinesisProduced}} , it is observed that the new sink has a performance 
> regression against the deprecated sink for same environment setting.
> Further investigation identified that the AIMD Ratelimiting strategy is the 
> bottleneck for the regression. 
> Attached results for {{KinesisStreamSink}}  against {FlinkKinesisProducer} 
> and {KinesisStreamSink} after disabling {{AIMDRatelimitingStrategy}}
> h2. Environment Settings
> - Benchmarking was performed on AWS KDA.
> - Application logic is just sending records downstream
> - Application parallelism was tested to be 1.
> - Kinesis stream number of shards was tested with 8 and 12.
> - payload size was 1Kb and 100Kb.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (FLINK-31872) Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink users for KinesisStreamsSink

2023-04-20 Thread Ahmed Hamdy (Jira) 


 [ 
https://issues.apache.org/jira/browse/FLINK-31872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed Hamdy updated FLINK-31872:

Component/s: (was: Connectors / Common)
 (was: Connectors / Firehose)

> Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink 
> users for KinesisStreamsSink
> --
>
> Key: FLINK-31872
> URL: https://issues.apache.org/jira/browse/FLINK-31872
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / Kinesis
>Reporter: Ahmed Hamdy
>Assignee: Ahmed Hamdy
>Priority: Major
>  Labels: pull-request-available
> Fix For: aws-connector-4.2.0
>
>
> h1. Issue
> While benchmarking the {{KinesisStreamSink}} for 1.15 against the legacy 
> {{FlinkKinesisProduced}} , it is observed that the new sink has a performance 
> regression against the deprecated sink for same environment setting.
> Further investigation identified that the AIMD Ratelimiting strategy is the 
> bottleneck for the regression. 
> Attached results for {{KinesisStreamSink}}  against {FlinkKinesisProducer} 
> and {KinesisStreamSink} after disabling {{AIMDRatelimitingStrategy}}
> h2. Environment Settings
> - Benchmarking was performed on AWS KDA.
> - Application logic is just sending records downstream
> - Application parallelism was tested to be 1.
> - Kinesis stream number of shards was tested with 8 and 12.
> - payload size was 1Kb and 100Kb.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (FLINK-31872) Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink users for KinesisStreamsSink

2023-04-20 Thread Ahmed Hamdy (Jira) 


 [ 
https://issues.apache.org/jira/browse/FLINK-31872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed Hamdy updated FLINK-31872:

Fix Version/s: (was: 1.16.2)
   (was: 1.18.0)
   (was: 1.17.1)

> Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink 
> users for KinesisStreamsSink
> --
>
> Key: FLINK-31872
> URL: https://issues.apache.org/jira/browse/FLINK-31872
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / Common, Connectors / Firehose, Connectors / 
> Kinesis
>Affects Versions: 1.16.0, aws-connector-3.0.0, 1.16.1, 
> aws-connector-4.0.0, aws-connector-4.1.0
>Reporter: Ahmed Hamdy
>Assignee: Ahmed Hamdy
>Priority: Major
>  Labels: pull-request-available
> Fix For: aws-connector-4.2.0
>
>
> h1. Issue
> While benchmarking the {{KinesisStreamSink}} for 1.15 against the legacy 
> {{FlinkKinesisProduced}} , it is observed that the new sink has a performance 
> regression against the deprecated sink for same environment setting.
> Further investigation identified that the AIMD Ratelimiting strategy is the 
> bottleneck for the regression. 
> Attached results for {{KinesisStreamSink}}  against {FlinkKinesisProducer} 
> and {KinesisStreamSink} after disabling {{AIMDRatelimitingStrategy}}
> h2. Environment Settings
> - Benchmarking was performed on AWS KDA.
> - Application logic is just sending records downstream
> - Application parallelism was tested to be 1.
> - Kinesis stream number of shards was tested with 8 and 12.
> - payload size was 1Kb and 100Kb.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (FLINK-31872) Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink users for KinesisStreamsSink

2023-04-20 Thread Ahmed Hamdy (Jira) 


 [ 
https://issues.apache.org/jira/browse/FLINK-31872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed Hamdy updated FLINK-31872:

Issue Type: Improvement  (was: Bug)

> Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink 
> users for KinesisStreamsSink
> --
>
> Key: FLINK-31872
> URL: https://issues.apache.org/jira/browse/FLINK-31872
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / Common, Connectors / Firehose, Connectors / 
> Kinesis
>Affects Versions: 1.16.0, aws-connector-3.0.0, 1.16.1, 
> aws-connector-4.0.0, aws-connector-4.1.0
>Reporter: Ahmed Hamdy
>Assignee: Ahmed Hamdy
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.16.2, 1.18.0, 1.17.1, aws-connector-4.2.0
>
>
> h1. Issue
> While benchmarking the {{KinesisStreamSink}} for 1.15 against the legacy 
> {{FlinkKinesisProduced}} , it is observed that the new sink has a performance 
> regression against the deprecated sink for same environment setting.
> Further investigation identified that the AIMD Ratelimiting strategy is the 
> bottleneck for the regression. 
> Attached results for {{KinesisStreamSink}}  against {FlinkKinesisProducer} 
> and {KinesisStreamSink} after disabling {{AIMDRatelimitingStrategy}}
> h2. Environment Settings
> - Benchmarking was performed on AWS KDA.
> - Application logic is just sending records downstream
> - Application parallelism was tested to be 1.
> - Kinesis stream number of shards was tested with 8 and 12.
> - payload size was 1Kb and 100Kb.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (FLINK-31872) Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink users for KinesisStreamsSink

2023-04-20 Thread Ahmed Hamdy (Jira) 


 [ 
https://issues.apache.org/jira/browse/FLINK-31872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed Hamdy updated FLINK-31872:

Summary: Add Support for Configuring AIMD Ratelimiting strategy parameters 
by Sink users for KinesisStreamsSink  (was: CLONE - AsyncSinkWriter Performance 
regression due to AIMD rate limiting strategy)

> Add Support for Configuring AIMD Ratelimiting strategy parameters by Sink 
> users for KinesisStreamsSink
> --
>
> Key: FLINK-31872
> URL: https://issues.apache.org/jira/browse/FLINK-31872
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / Common, Connectors / Firehose, Connectors / 
> Kinesis
>Affects Versions: 1.16.0, aws-connector-3.0.0, 1.16.1, 
> aws-connector-4.0.0, aws-connector-4.1.0
>Reporter: Ahmed Hamdy
>Assignee: Ahmed Hamdy
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.16.2, 1.18.0, 1.17.1, aws-connector-4.2.0
>
>
> h1. Issue
> While benchmarking the {{KinesisStreamSink}} for 1.15 against the legacy 
> {{FlinkKinesisProduced}} , it is observed that the new sink has a performance 
> regression against the deprecated sink for same environment setting.
> Further investigation identified that the AIMD Ratelimiting strategy is the 
> bottleneck for the regression. 
> Attached results for {{KinesisStreamSink}}  against {FlinkKinesisProducer} 
> and {KinesisStreamSink} after disabling {{AIMDRatelimitingStrategy}}
> h2. Environment Settings
> - Benchmarking was performed on AWS KDA.
> - Application logic is just sending records downstream
> - Application parallelism was tested to be 1.
> - Kinesis stream number of shards was tested with 8 and 12.
> - payload size was 1Kb and 100Kb.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)