mxm commented on code in PR #613:
URL: https://github.com/apache/flink-web/pull/613#discussion_r1117121658
##########
docs/content/posts/2023-02-27-release-kubernetes-operator-1.4.0.md:
##########
@@ -0,0 +1,77 @@
+---
+title: "Apache Flink Kubernetes Operator 1.4.0 Release Announcement"
+date: "2023-02-27T08:00:00.000Z"
+authors:
+- gyfora:
+ name: "Gyula Fora"
+ twitter: "GyulaFora"
+- mxm:
+ name: "Max Michels"
+ twitter: "stadtlegende"
+aliases:
+- /news/2023/02/27/release-kubernetes-operator-1.4.0.html
+---
+
+We are proud to announce the latest stable release of the operator. In
addition to the expected stability improvements and fixes, the 1.4.0 release
introduces the first version of the long-awaited autoscaler module.
+
+## Flink Streaming Job Autoscaler
+
+A highly requested feature for Flink applications is the ability to scale the
pipeline based on incoming data load and other performance metrics. While Flink
has already provided some of the required building blocks, this feature has not
yet been realised in the open source ecosystem.
Review Comment:
```suggestion
A highly requested feature for Flink applications is the ability to scale
the pipeline based on incoming data load and the utilization of the dataflow.
While Flink has already provided some of the required building blocks, this
feature has not yet been realized in the open source ecosystem.
```
##########
docs/content/posts/2023-02-27-release-kubernetes-operator-1.4.0.md:
##########
@@ -0,0 +1,77 @@
+---
+title: "Apache Flink Kubernetes Operator 1.4.0 Release Announcement"
+date: "2023-02-27T08:00:00.000Z"
+authors:
+- gyfora:
+ name: "Gyula Fora"
+ twitter: "GyulaFora"
+- mxm:
+ name: "Max Michels"
Review Comment:
NIT
```suggestion
name: "Maximilian Michels"
```
##########
docs/content/posts/2023-02-27-release-kubernetes-operator-1.4.0.md:
##########
@@ -0,0 +1,77 @@
+---
+title: "Apache Flink Kubernetes Operator 1.4.0 Release Announcement"
+date: "2023-02-27T08:00:00.000Z"
+authors:
+- gyfora:
+ name: "Gyula Fora"
+ twitter: "GyulaFora"
+- mxm:
+ name: "Max Michels"
+ twitter: "stadtlegende"
+aliases:
+- /news/2023/02/27/release-kubernetes-operator-1.4.0.html
+---
+
+We are proud to announce the latest stable release of the operator. In
addition to the expected stability improvements and fixes, the 1.4.0 release
introduces the first version of the long-awaited autoscaler module.
+
+## Flink Streaming Job Autoscaler
+
+A highly requested feature for Flink applications is the ability to scale the
pipeline based on incoming data load and other performance metrics. While Flink
has already provided some of the required building blocks, this feature has not
yet been realised in the open source ecosystem.
+
+With
[FLIP-271](https://cwiki.apache.org/confluence/display/FLINK/FLIP-271%3A+Autoscaling)
the community set out to build such autoscaler component as part of the
Kubernetes Operator subproject. The Kubernetes Operator proved to be a great
place for the autoscaler module as it already contains all the necessary bits
for managing and upgrading production streaming applications.
+
+Fast-forward to the 1.4.0 release and we now have the first fully functional
autoscaler implementation in the operator, ready to be tested and used in
production applications. For more, detailed information, please refer to the
[Autoscaler
Documentation](https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-release-1.4/docs/custom-resource/autoscaler/).
+
+### Overview
+
+The autoscaler uses Flink source and operator vertex metrics to efficiently
and independently scale the job vertexes of the streaming pipeline.
+
Review Comment:
```suggestion
The used metrics include:
Source metrics:
- number of pending records (source only)
- number of partitions (source only)
- ingestion rate (source only)
- processing rate
- time spent processing (utilization)
```
##########
docs/content/posts/2023-02-27-release-kubernetes-operator-1.4.0.md:
##########
@@ -0,0 +1,77 @@
+---
+title: "Apache Flink Kubernetes Operator 1.4.0 Release Announcement"
+date: "2023-02-27T08:00:00.000Z"
+authors:
+- gyfora:
+ name: "Gyula Fora"
+ twitter: "GyulaFora"
+- mxm:
+ name: "Max Michels"
+ twitter: "stadtlegende"
+aliases:
+- /news/2023/02/27/release-kubernetes-operator-1.4.0.html
+---
+
+We are proud to announce the latest stable release of the operator. In
addition to the expected stability improvements and fixes, the 1.4.0 release
introduces the first version of the long-awaited autoscaler module.
+
+## Flink Streaming Job Autoscaler
+
+A highly requested feature for Flink applications is the ability to scale the
pipeline based on incoming data load and other performance metrics. While Flink
has already provided some of the required building blocks, this feature has not
yet been realised in the open source ecosystem.
+
+With
[FLIP-271](https://cwiki.apache.org/confluence/display/FLINK/FLIP-271%3A+Autoscaling)
the community set out to build such autoscaler component as part of the
Kubernetes Operator subproject. The Kubernetes Operator proved to be a great
place for the autoscaler module as it already contains all the necessary bits
for managing and upgrading production streaming applications.
+
+Fast-forward to the 1.4.0 release and we now have the first fully functional
autoscaler implementation in the operator, ready to be tested and used in
production applications. For more, detailed information, please refer to the
[Autoscaler
Documentation](https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-release-1.4/docs/custom-resource/autoscaler/).
+
+### Overview
+
+The autoscaler uses Flink source and operator vertex metrics to efficiently
and independently scale the job vertexes of the streaming pipeline.
+
+The algorithm starts from the sources and recursively computes the required
processing capacity for each operator in the pipeline. At the source vertices,
target data rate (processing capacity) is equal to the data rate in Kafka.
+
+For other operators we compute it as the sum of the input (upstream) operators
output data rate.
+
+<p align="center">
+<img
src="/img/blog/2023-02-27-release-kubernetes-operator-1.4.0/autoscaler_fig1.png"
width="70%" height="70%">
+</p>
+
+Users can configure target utilization percentages for the operators in the
pipeline, and if the increased input rate (or slowed down processing) requires
operators will be scaled up to match the target:
Review Comment:
```suggestion
Users configure the target utilization percentage of the operators in the
pipeline, e.g. keep the all operators between 60% - 80% busy. The autoscaler
then finds a configuration such that the output rates of all operators match
the input rates of all their downstream operators at the targeted utilization.
In this example we see an upscale operation:
```
##########
docs/content/posts/2023-02-27-release-kubernetes-operator-1.4.0.md:
##########
@@ -0,0 +1,77 @@
+---
+title: "Apache Flink Kubernetes Operator 1.4.0 Release Announcement"
+date: "2023-02-27T08:00:00.000Z"
+authors:
+- gyfora:
+ name: "Gyula Fora"
+ twitter: "GyulaFora"
+- mxm:
+ name: "Max Michels"
+ twitter: "stadtlegende"
+aliases:
+- /news/2023/02/27/release-kubernetes-operator-1.4.0.html
+---
+
+We are proud to announce the latest stable release of the operator. In
addition to the expected stability improvements and fixes, the 1.4.0 release
introduces the first version of the long-awaited autoscaler module.
+
+## Flink Streaming Job Autoscaler
+
+A highly requested feature for Flink applications is the ability to scale the
pipeline based on incoming data load and other performance metrics. While Flink
has already provided some of the required building blocks, this feature has not
yet been realised in the open source ecosystem.
+
+With
[FLIP-271](https://cwiki.apache.org/confluence/display/FLINK/FLIP-271%3A+Autoscaling)
the community set out to build such autoscaler component as part of the
Kubernetes Operator subproject. The Kubernetes Operator proved to be a great
place for the autoscaler module as it already contains all the necessary bits
for managing and upgrading production streaming applications.
+
+Fast-forward to the 1.4.0 release and we now have the first fully functional
autoscaler implementation in the operator, ready to be tested and used in
production applications. For more, detailed information, please refer to the
[Autoscaler
Documentation](https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-release-1.4/docs/custom-resource/autoscaler/).
+
+### Overview
+
+The autoscaler uses Flink source and operator vertex metrics to efficiently
and independently scale the job vertexes of the streaming pipeline.
Review Comment:
```suggestion
The autoscaler uses Flink task metrics to effectively and independently
scale the job vertices of the streaming pipeline. This removes backpressure
from the job to ensure an optimal flow of data at the lowest possible resource
usage. All kind of jobs, including SQL jobs, can be scaled with this method.
The approach is based on [Three steps is all you need: fast, accurate,
automatic scaling decisions for distributed streaming
dataflows](https://www.usenix.org/system/files/osdi18-kalavri.pdf) by Kalavri
et al. Shoutout to our fellow Flink community member and committer Vasiliki
Kalavri!
```
I think we refer to them in the docs as task metrics.
##########
docs/content/posts/2023-02-27-release-kubernetes-operator-1.4.0.md:
##########
@@ -0,0 +1,77 @@
+---
+title: "Apache Flink Kubernetes Operator 1.4.0 Release Announcement"
+date: "2023-02-27T08:00:00.000Z"
+authors:
+- gyfora:
+ name: "Gyula Fora"
+ twitter: "GyulaFora"
+- mxm:
+ name: "Max Michels"
+ twitter: "stadtlegende"
+aliases:
+- /news/2023/02/27/release-kubernetes-operator-1.4.0.html
+---
+
+We are proud to announce the latest stable release of the operator. In
addition to the expected stability improvements and fixes, the 1.4.0 release
introduces the first version of the long-awaited autoscaler module.
+
+## Flink Streaming Job Autoscaler
+
+A highly requested feature for Flink applications is the ability to scale the
pipeline based on incoming data load and other performance metrics. While Flink
has already provided some of the required building blocks, this feature has not
yet been realised in the open source ecosystem.
+
+With
[FLIP-271](https://cwiki.apache.org/confluence/display/FLINK/FLIP-271%3A+Autoscaling)
the community set out to build such autoscaler component as part of the
Kubernetes Operator subproject. The Kubernetes Operator proved to be a great
place for the autoscaler module as it already contains all the necessary bits
for managing and upgrading production streaming applications.
+
+Fast-forward to the 1.4.0 release and we now have the first fully functional
autoscaler implementation in the operator, ready to be tested and used in
production applications. For more, detailed information, please refer to the
[Autoscaler
Documentation](https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-release-1.4/docs/custom-resource/autoscaler/).
Review Comment:
```suggestion
Fast-forward to the 1.4.0 release, we now have the first fully functional
autoscaler implementation in the operator, ready to be tested and used in
production applications. For more, detailed information, please refer to the
[Autoscaler
Documentation](https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-release-1.4/docs/custom-resource/autoscaler/).
```
##########
docs/content/posts/2023-02-27-release-kubernetes-operator-1.4.0.md:
##########
@@ -0,0 +1,77 @@
+---
+title: "Apache Flink Kubernetes Operator 1.4.0 Release Announcement"
+date: "2023-02-27T08:00:00.000Z"
+authors:
+- gyfora:
+ name: "Gyula Fora"
+ twitter: "GyulaFora"
+- mxm:
+ name: "Max Michels"
+ twitter: "stadtlegende"
+aliases:
+- /news/2023/02/27/release-kubernetes-operator-1.4.0.html
+---
+
+We are proud to announce the latest stable release of the operator. In
addition to the expected stability improvements and fixes, the 1.4.0 release
introduces the first version of the long-awaited autoscaler module.
+
+## Flink Streaming Job Autoscaler
+
+A highly requested feature for Flink applications is the ability to scale the
pipeline based on incoming data load and other performance metrics. While Flink
has already provided some of the required building blocks, this feature has not
yet been realised in the open source ecosystem.
+
+With
[FLIP-271](https://cwiki.apache.org/confluence/display/FLINK/FLIP-271%3A+Autoscaling)
the community set out to build such autoscaler component as part of the
Kubernetes Operator subproject. The Kubernetes Operator proved to be a great
place for the autoscaler module as it already contains all the necessary bits
for managing and upgrading production streaming applications.
Review Comment:
```suggestion
With
[FLIP-271](https://cwiki.apache.org/confluence/display/FLINK/FLIP-271%3A+Autoscaling)
the community set out to build such an autoscaler component as part of the
Kubernetes Operator subproject. The Kubernetes Operator proved to be a great
place for the autoscaler module as it already contains all the necessary bits
for managing and upgrading production streaming applications.
```
##########
docs/content/posts/2023-02-27-release-kubernetes-operator-1.4.0.md:
##########
@@ -0,0 +1,77 @@
+---
+title: "Apache Flink Kubernetes Operator 1.4.0 Release Announcement"
+date: "2023-02-27T08:00:00.000Z"
+authors:
+- gyfora:
+ name: "Gyula Fora"
+ twitter: "GyulaFora"
+- mxm:
+ name: "Max Michels"
+ twitter: "stadtlegende"
+aliases:
+- /news/2023/02/27/release-kubernetes-operator-1.4.0.html
+---
+
+We are proud to announce the latest stable release of the operator. In
addition to the expected stability improvements and fixes, the 1.4.0 release
introduces the first version of the long-awaited autoscaler module.
+
+## Flink Streaming Job Autoscaler
+
+A highly requested feature for Flink applications is the ability to scale the
pipeline based on incoming data load and other performance metrics. While Flink
has already provided some of the required building blocks, this feature has not
yet been realised in the open source ecosystem.
+
+With
[FLIP-271](https://cwiki.apache.org/confluence/display/FLINK/FLIP-271%3A+Autoscaling)
the community set out to build such autoscaler component as part of the
Kubernetes Operator subproject. The Kubernetes Operator proved to be a great
place for the autoscaler module as it already contains all the necessary bits
for managing and upgrading production streaming applications.
+
+Fast-forward to the 1.4.0 release and we now have the first fully functional
autoscaler implementation in the operator, ready to be tested and used in
production applications. For more, detailed information, please refer to the
[Autoscaler
Documentation](https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-release-1.4/docs/custom-resource/autoscaler/).
+
+### Overview
+
+The autoscaler uses Flink source and operator vertex metrics to efficiently
and independently scale the job vertexes of the streaming pipeline.
+
+The algorithm starts from the sources and recursively computes the required
processing capacity for each operator in the pipeline. At the source vertices,
target data rate (processing capacity) is equal to the data rate in Kafka.
+
+For other operators we compute it as the sum of the input (upstream) operators
output data rate.
+
+<p align="center">
+<img
src="/img/blog/2023-02-27-release-kubernetes-operator-1.4.0/autoscaler_fig1.png"
width="70%" height="70%">
+</p>
+
+Users can configure target utilization percentages for the operators in the
pipeline, and if the increased input rate (or slowed down processing) requires
operators will be scaled up to match the target:
+
+<p align="center">
+<img
src="/img/blog/2023-02-27-release-kubernetes-operator-1.4.0/autoscaler_fig2.png"
width="70%" height="70%">
+</p>
+
+Similarly as load decreases, the autoscaler adjusts individual operator
parallelism levels to match the current rate over time.
+
+<p align="center">
+<img
src="/img/blog/2023-02-27-release-kubernetes-operator-1.4.0/autoscaler_fig3.png"
width="70%" height="70%">
+</p>
+
+### Limitations
+
+While we are very happy with the progress we have made in the last few months,
we recognise that the autoscaler is still in an early stage of development.
Review Comment:
```suggestion
While we are very happy with the progress we have made in the last few
months, the autoscaler is still in an early stage of development.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
