knaufk commented on a change in pull request #15064: URL: https://github.com/apache/flink/pull/15064#discussion_r585585810
########## File path: docs/content/docs/deployment/elastic_scaling.md ########## @@ -0,0 +1,91 @@ +--- +title: Elastic Scaling +weight: 5 +type: docs + +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Elastic Scaling + +Flink allows you to adjust your cluster size dynamically to your workloads. This is possible by stopping a job with a savepoint and restarting it with a different parallelism. This page describes options where Flink automatically adjusts the parallelism. + +## Reactive Mode + +{{< hint danger >}} +Reactive mode is an experimental feature. The Flink community is actively looking for feedback by users through our mailing lists. Review comment: In the roadmap draft this is called an "MVP" feature. ########## File path: docs/content/docs/deployment/elastic_scaling.md ########## @@ -0,0 +1,91 @@ +--- +title: Elastic Scaling +weight: 5 +type: docs + +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Elastic Scaling + +Flink allows you to adjust your cluster size dynamically to your workloads. This is possible by stopping a job with a savepoint and restarting it with a different parallelism. This page describes options where Flink automatically adjusts the parallelism. + +## Reactive Mode + +{{< hint danger >}} +Reactive mode is an experimental feature. The Flink community is actively looking for feedback by users through our mailing lists. +{{< /hint >}} + +The Reactive Mode allows Flink users to implement a powerful autoscaling mechanism, by having an external service monitor certain metrics, such as consumer lag, aggregate CPU utilization, throughput or latency. As soon as these metrics are above or below a certain threshold, additional TaskManagers can be added or removed from the Flink cluster. This could be implemented through changing the [replica factor](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#replicas) of a Kubernetes deployment, or an [autoscaling](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html) group. + +### Getting started + +If you just want to try out Reactive Mode, follow these instructions. They assume that you are deploying Flink on one machine. + +```bash +# Put Job into lib/ directory +cp ./examples/streaming/TopSpeedWindowing.jar lib/ +# Submit Job in Application Mode +./bin/standalone-job.sh start -Drestart-strategy=fixeddelay -Drestart-strategy.fixed-delay.attempts=100000 -Djobmanager.scheduler=adaptive -Dscheduler-mode=reactive -j org.apache.flink.streaming.examples.windowing.TopSpeedWindowing +# Start first TaskManager +./bin/taskmanager.sh start +``` + +Let's quickly examine the used configuration parameters: +- `-Drestart-strategy=fixeddelay` and `-Drestart-strategy.fixed-delay.attempts=100000` configure the job to restart on failure. This is needed for supporting scale-down. +- `-Djobmanager.scheduler=adaptive` enables the Adaptive Scheduler, which is needed to use Reactive Mode. +- `-Dscheduler-mode=reactive` enables Reactive Mode. + +You have now started a Flink job in Application Mode. The [web interface on localhost:8081](http://localhost:8081) now shows that the job is running on one TaskManager. If you want to scale up the job, add another TaskManager to the cluster: Review comment: Application Mode? I mean, yes, but did you want to say "Reactive Mode". ########## File path: docs/content/docs/deployment/elastic_scaling.md ########## @@ -0,0 +1,91 @@ +--- +title: Elastic Scaling +weight: 5 +type: docs + +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Elastic Scaling + +Flink allows you to adjust your cluster size dynamically to your workloads. This is possible by stopping a job with a savepoint and restarting it with a different parallelism. This page describes options where Flink automatically adjusts the parallelism. Review comment: This introduction is confusing. Who is doing what? ########## File path: docs/content/docs/deployment/elastic_scaling.md ########## @@ -0,0 +1,91 @@ +--- +title: Elastic Scaling +weight: 5 +type: docs + +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Elastic Scaling + +Flink allows you to adjust your cluster size dynamically to your workloads. This is possible by stopping a job with a savepoint and restarting it with a different parallelism. This page describes options where Flink automatically adjusts the parallelism. + +## Reactive Mode + +{{< hint danger >}} +Reactive mode is an experimental feature. The Flink community is actively looking for feedback by users through our mailing lists. +{{< /hint >}} + +The Reactive Mode allows Flink users to implement a powerful autoscaling mechanism, by having an external service monitor certain metrics, such as consumer lag, aggregate CPU utilization, throughput or latency. As soon as these metrics are above or below a certain threshold, additional TaskManagers can be added or removed from the Flink cluster. This could be implemented through changing the [replica factor](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#replicas) of a Kubernetes deployment, or an [autoscaling](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html) group. + +### Getting started + +If you just want to try out Reactive Mode, follow these instructions. They assume that you are deploying Flink on one machine. + +```bash +# Put Job into lib/ directory +cp ./examples/streaming/TopSpeedWindowing.jar lib/ +# Submit Job in Application Mode +./bin/standalone-job.sh start -Drestart-strategy=fixeddelay -Drestart-strategy.fixed-delay.attempts=100000 -Djobmanager.scheduler=adaptive -Dscheduler-mode=reactive -j org.apache.flink.streaming.examples.windowing.TopSpeedWindowing +# Start first TaskManager +./bin/taskmanager.sh start +``` + +Let's quickly examine the used configuration parameters: +- `-Drestart-strategy=fixeddelay` and `-Drestart-strategy.fixed-delay.attempts=100000` configure the job to restart on failure. This is needed for supporting scale-down. +- `-Djobmanager.scheduler=adaptive` enables the Adaptive Scheduler, which is needed to use Reactive Mode. +- `-Dscheduler-mode=reactive` enables Reactive Mode. + +You have now started a Flink job in Application Mode. The [web interface on localhost:8081](http://localhost:8081) now shows that the job is running on one TaskManager. If you want to scale up the job, add another TaskManager to the cluster: +```bash +# Start additional TaskManager +./bin/taskmanager.sh start +``` + +To scale down, remove a TaskManager instance. +```bash +# Remove a TaskManager +./bin/taskmanager.sh stop +``` + +### Limitations + +Since Reactive Mode is a new, experimental feature, not all features supported by the default scheduler are also available with Reactive Mode (and its adaptive scheduler). The Flink community is working on addressing these limitations. + +- **Deployment is only supported as a standalone application** deployment. Active resource managers (such as native Kubernetes, YARN or Mesos) are explicitly not supported. Standalone session clusters are not supported either. The application deployment is limited to single job applications. +- **Streaming jobs only**: The first version of Reactive Mode runs with streaming jobs only. When submitting a batch job, then the default scheduler will be used. +- **No support for local recovery**: Local recovery is a feature that schedules tasks to machines so that the state on that machine gets re-used if possible. The lack of this feature means that Reactive Mode will always need to restore the entire state from the checkpoint storage. +- **No support for local failovers**: Local failover means that the scheduler is able to restart parts ("regions" in Flink's internals) of a failed job, instead of the entire job. This limitations impacts only recovery time of embarrassingly parallel jobs -- Flink's default scheduler can restart failed parts, while Reactive Mode will restart the entire job. Review comment: ```suggestion - **No support for local failover**: Local failover means that the scheduler is able to restart parts ("regions" in Flink's internals) of a failed job, instead of the entire job. This limitation impacts only recovery time of embarrassingly parallel jobs -- Flink's default scheduler can restart failed parts, while Reactive Mode will restart the entire job. ``` ########## File path: docs/content/docs/deployment/elastic_scaling.md ########## @@ -0,0 +1,91 @@ +--- +title: Elastic Scaling +weight: 5 +type: docs + +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Elastic Scaling + +Flink allows you to adjust your cluster size dynamically to your workloads. This is possible by stopping a job with a savepoint and restarting it with a different parallelism. This page describes options where Flink automatically adjusts the parallelism. + +## Reactive Mode + +{{< hint danger >}} +Reactive mode is an experimental feature. The Flink community is actively looking for feedback by users through our mailing lists. +{{< /hint >}} + +The Reactive Mode allows Flink users to implement a powerful autoscaling mechanism, by having an external service monitor certain metrics, such as consumer lag, aggregate CPU utilization, throughput or latency. As soon as these metrics are above or below a certain threshold, additional TaskManagers can be added or removed from the Flink cluster. This could be implemented through changing the [replica factor](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#replicas) of a Kubernetes deployment, or an [autoscaling](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html) group. + +### Getting started + +If you just want to try out Reactive Mode, follow these instructions. They assume that you are deploying Flink on one machine. + +```bash +# Put Job into lib/ directory +cp ./examples/streaming/TopSpeedWindowing.jar lib/ +# Submit Job in Application Mode +./bin/standalone-job.sh start -Drestart-strategy=fixeddelay -Drestart-strategy.fixed-delay.attempts=100000 -Djobmanager.scheduler=adaptive -Dscheduler-mode=reactive -j org.apache.flink.streaming.examples.windowing.TopSpeedWindowing +# Start first TaskManager +./bin/taskmanager.sh start +``` + +Let's quickly examine the used configuration parameters: +- `-Drestart-strategy=fixeddelay` and `-Drestart-strategy.fixed-delay.attempts=100000` configure the job to restart on failure. This is needed for supporting scale-down. +- `-Djobmanager.scheduler=adaptive` enables the Adaptive Scheduler, which is needed to use Reactive Mode. +- `-Dscheduler-mode=reactive` enables Reactive Mode. + +You have now started a Flink job in Application Mode. The [web interface on localhost:8081](http://localhost:8081) now shows that the job is running on one TaskManager. If you want to scale up the job, add another TaskManager to the cluster: +```bash +# Start additional TaskManager +./bin/taskmanager.sh start +``` + +To scale down, remove a TaskManager instance. +```bash +# Remove a TaskManager +./bin/taskmanager.sh stop +``` + +### Limitations + +Since Reactive Mode is a new, experimental feature, not all features supported by the default scheduler are also available with Reactive Mode (and its adaptive scheduler). The Flink community is working on addressing these limitations. + +- **Deployment is only supported as a standalone application** deployment. Active resource managers (such as native Kubernetes, YARN or Mesos) are explicitly not supported. Standalone session clusters are not supported either. The application deployment is limited to single job applications. +- **Streaming jobs only**: The first version of Reactive Mode runs with streaming jobs only. When submitting a batch job, then the default scheduler will be used. +- **No support for local recovery**: Local recovery is a feature that schedules tasks to machines so that the state on that machine gets re-used if possible. The lack of this feature means that Reactive Mode will always need to restore the entire state from the checkpoint storage. Review comment: Link to documentation on local recovery? ########## File path: docs/content/docs/deployment/elastic_scaling.md ########## @@ -0,0 +1,91 @@ +--- +title: Elastic Scaling +weight: 5 +type: docs + +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Elastic Scaling + +Flink allows you to adjust your cluster size dynamically to your workloads. This is possible by stopping a job with a savepoint and restarting it with a different parallelism. This page describes options where Flink automatically adjusts the parallelism. + +## Reactive Mode + +{{< hint danger >}} +Reactive mode is an experimental feature. The Flink community is actively looking for feedback by users through our mailing lists. +{{< /hint >}} + +The Reactive Mode allows Flink users to implement a powerful autoscaling mechanism, by having an external service monitor certain metrics, such as consumer lag, aggregate CPU utilization, throughput or latency. As soon as these metrics are above or below a certain threshold, additional TaskManagers can be added or removed from the Flink cluster. This could be implemented through changing the [replica factor](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#replicas) of a Kubernetes deployment, or an [autoscaling](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html) group. + +### Getting started + +If you just want to try out Reactive Mode, follow these instructions. They assume that you are deploying Flink on one machine. + +```bash +# Put Job into lib/ directory +cp ./examples/streaming/TopSpeedWindowing.jar lib/ +# Submit Job in Application Mode +./bin/standalone-job.sh start -Drestart-strategy=fixeddelay -Drestart-strategy.fixed-delay.attempts=100000 -Djobmanager.scheduler=adaptive -Dscheduler-mode=reactive -j org.apache.flink.streaming.examples.windowing.TopSpeedWindowing +# Start first TaskManager +./bin/taskmanager.sh start +``` + +Let's quickly examine the used configuration parameters: +- `-Drestart-strategy=fixeddelay` and `-Drestart-strategy.fixed-delay.attempts=100000` configure the job to restart on failure. This is needed for supporting scale-down. +- `-Djobmanager.scheduler=adaptive` enables the Adaptive Scheduler, which is needed to use Reactive Mode. +- `-Dscheduler-mode=reactive` enables Reactive Mode. + +You have now started a Flink job in Application Mode. The [web interface on localhost:8081](http://localhost:8081) now shows that the job is running on one TaskManager. If you want to scale up the job, add another TaskManager to the cluster: +```bash +# Start additional TaskManager +./bin/taskmanager.sh start +``` + +To scale down, remove a TaskManager instance. +```bash +# Remove a TaskManager +./bin/taskmanager.sh stop +``` + +### Limitations + +Since Reactive Mode is a new, experimental feature, not all features supported by the default scheduler are also available with Reactive Mode (and its adaptive scheduler). The Flink community is working on addressing these limitations. + +- **Deployment is only supported as a standalone application** deployment. Active resource managers (such as native Kubernetes, YARN or Mesos) are explicitly not supported. Standalone session clusters are not supported either. The application deployment is limited to single job applications. Review comment: Link to the Different Modes? ########## File path: docs/content/docs/deployment/elastic_scaling.md ########## @@ -0,0 +1,91 @@ +--- +title: Elastic Scaling +weight: 5 +type: docs + +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Elastic Scaling + +Flink allows you to adjust your cluster size dynamically to your workloads. This is possible by stopping a job with a savepoint and restarting it with a different parallelism. This page describes options where Flink automatically adjusts the parallelism. + +## Reactive Mode + +{{< hint danger >}} +Reactive mode is an experimental feature. The Flink community is actively looking for feedback by users through our mailing lists. +{{< /hint >}} + +The Reactive Mode allows Flink users to implement a powerful autoscaling mechanism, by having an external service monitor certain metrics, such as consumer lag, aggregate CPU utilization, throughput or latency. As soon as these metrics are above or below a certain threshold, additional TaskManagers can be added or removed from the Flink cluster. This could be implemented through changing the [replica factor](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#replicas) of a Kubernetes deployment, or an [autoscaling](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html) group. Review comment: Before this you might want to add a sentence or two explaining what the Reactive Mode is actually doing, i.e. that it always takes all the slots available in the clusters and scales the job to that parallelism. ########## File path: docs/content/docs/deployment/elastic_scaling.md ########## @@ -0,0 +1,91 @@ +--- +title: Elastic Scaling +weight: 5 +type: docs + +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Elastic Scaling + +Flink allows you to adjust your cluster size dynamically to your workloads. This is possible by stopping a job with a savepoint and restarting it with a different parallelism. This page describes options where Flink automatically adjusts the parallelism. + +## Reactive Mode + +{{< hint danger >}} +Reactive mode is an experimental feature. The Flink community is actively looking for feedback by users through our mailing lists. +{{< /hint >}} + +The Reactive Mode allows Flink users to implement a powerful autoscaling mechanism, by having an external service monitor certain metrics, such as consumer lag, aggregate CPU utilization, throughput or latency. As soon as these metrics are above or below a certain threshold, additional TaskManagers can be added or removed from the Flink cluster. This could be implemented through changing the [replica factor](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#replicas) of a Kubernetes deployment, or an [autoscaling](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html) group. + +### Getting started + +If you just want to try out Reactive Mode, follow these instructions. They assume that you are deploying Flink on one machine. + +```bash +# Put Job into lib/ directory +cp ./examples/streaming/TopSpeedWindowing.jar lib/ +# Submit Job in Application Mode +./bin/standalone-job.sh start -Drestart-strategy=fixeddelay -Drestart-strategy.fixed-delay.attempts=100000 -Djobmanager.scheduler=adaptive -Dscheduler-mode=reactive -j org.apache.flink.streaming.examples.windowing.TopSpeedWindowing +# Start first TaskManager +./bin/taskmanager.sh start +``` + +Let's quickly examine the used configuration parameters: +- `-Drestart-strategy=fixeddelay` and `-Drestart-strategy.fixed-delay.attempts=100000` configure the job to restart on failure. This is needed for supporting scale-down. +- `-Djobmanager.scheduler=adaptive` enables the Adaptive Scheduler, which is needed to use Reactive Mode. +- `-Dscheduler-mode=reactive` enables Reactive Mode. + +You have now started a Flink job in Application Mode. The [web interface on localhost:8081](http://localhost:8081) now shows that the job is running on one TaskManager. If you want to scale up the job, add another TaskManager to the cluster: +```bash +# Start additional TaskManager +./bin/taskmanager.sh start +``` + +To scale down, remove a TaskManager instance. +```bash +# Remove a TaskManager +./bin/taskmanager.sh stop +``` + +### Limitations + +Since Reactive Mode is a new, experimental feature, not all features supported by the default scheduler are also available with Reactive Mode (and its adaptive scheduler). The Flink community is working on addressing these limitations. + +- **Deployment is only supported as a standalone application** deployment. Active resource managers (such as native Kubernetes, YARN or Mesos) are explicitly not supported. Standalone session clusters are not supported either. The application deployment is limited to single job applications. +- **Streaming jobs only**: The first version of Reactive Mode runs with streaming jobs only. When submitting a batch job, then the default scheduler will be used. +- **No support for local recovery**: Local recovery is a feature that schedules tasks to machines so that the state on that machine gets re-used if possible. The lack of this feature means that Reactive Mode will always need to restore the entire state from the checkpoint storage. +- **No support for local failovers**: Local failover means that the scheduler is able to restart parts ("regions" in Flink's internals) of a failed job, instead of the entire job. This limitations impacts only recovery time of embarrassingly parallel jobs -- Flink's default scheduler can restart failed parts, while Reactive Mode will restart the entire job. +- **Limited integration with Flink's Web UI**: Reactive Mode allows that a job's parallelism can change over its lifetime. The web UI only shows the current parallelism of a job, not the historic evolution of the job. There might be other inconveniences such as a lack of the exception history, incorrect job status time-stamps and incorrect metrics. Review comment: Can we be more specific here? "There might be other inconveniences ... like incorrect metrics", does not sound good. We should know, how metrics are impacted and document this here. ########## File path: docs/content/docs/deployment/elastic_scaling.md ########## @@ -0,0 +1,91 @@ +--- +title: Elastic Scaling +weight: 5 +type: docs + +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Elastic Scaling + +Flink allows you to adjust your cluster size dynamically to your workloads. This is possible by stopping a job with a savepoint and restarting it with a different parallelism. This page describes options where Flink automatically adjusts the parallelism. + +## Reactive Mode Review comment: What other section do you expect here besides "Reactive Mode"? ########## File path: docs/content/docs/deployment/elastic_scaling.md ########## @@ -0,0 +1,91 @@ +--- +title: Elastic Scaling +weight: 5 +type: docs + +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Elastic Scaling + +Flink allows you to adjust your cluster size dynamically to your workloads. This is possible by stopping a job with a savepoint and restarting it with a different parallelism. This page describes options where Flink automatically adjusts the parallelism. + +## Reactive Mode + +{{< hint danger >}} +Reactive mode is an experimental feature. The Flink community is actively looking for feedback by users through our mailing lists. +{{< /hint >}} + +The Reactive Mode allows Flink users to implement a powerful autoscaling mechanism, by having an external service monitor certain metrics, such as consumer lag, aggregate CPU utilization, throughput or latency. As soon as these metrics are above or below a certain threshold, additional TaskManagers can be added or removed from the Flink cluster. This could be implemented through changing the [replica factor](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#replicas) of a Kubernetes deployment, or an [autoscaling](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html) group. + +### Getting started + +If you just want to try out Reactive Mode, follow these instructions. They assume that you are deploying Flink on one machine. + +```bash +# Put Job into lib/ directory +cp ./examples/streaming/TopSpeedWindowing.jar lib/ +# Submit Job in Application Mode +./bin/standalone-job.sh start -Drestart-strategy=fixeddelay -Drestart-strategy.fixed-delay.attempts=100000 -Djobmanager.scheduler=adaptive -Dscheduler-mode=reactive -j org.apache.flink.streaming.examples.windowing.TopSpeedWindowing +# Start first TaskManager +./bin/taskmanager.sh start +``` + +Let's quickly examine the used configuration parameters: +- `-Drestart-strategy=fixeddelay` and `-Drestart-strategy.fixed-delay.attempts=100000` configure the job to restart on failure. This is needed for supporting scale-down. +- `-Djobmanager.scheduler=adaptive` enables the Adaptive Scheduler, which is needed to use Reactive Mode. +- `-Dscheduler-mode=reactive` enables Reactive Mode. + +You have now started a Flink job in Application Mode. The [web interface on localhost:8081](http://localhost:8081) now shows that the job is running on one TaskManager. If you want to scale up the job, add another TaskManager to the cluster: +```bash +# Start additional TaskManager +./bin/taskmanager.sh start +``` + +To scale down, remove a TaskManager instance. +```bash +# Remove a TaskManager +./bin/taskmanager.sh stop +``` + +### Limitations + +Since Reactive Mode is a new, experimental feature, not all features supported by the default scheduler are also available with Reactive Mode (and its adaptive scheduler). The Flink community is working on addressing these limitations. + +- **Deployment is only supported as a standalone application** deployment. Active resource managers (such as native Kubernetes, YARN or Mesos) are explicitly not supported. Standalone session clusters are not supported either. The application deployment is limited to single job applications. Review comment: ```suggestion - **Application Mode Only **. In terms of Deplyoment Modes only Application Mode is supported. In consequence, active resource managers (such as native Kubernetes, YARN or Mesos) not supported. Generally, Application Mode supports to submit multiple Flink Jobs as part of one application. This is not supported: reactive Mode only supports applications consisting of one Flink Job. ``` ########## File path: docs/content/docs/deployment/elastic_scaling.md ########## @@ -0,0 +1,91 @@ +--- +title: Elastic Scaling +weight: 5 +type: docs + +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Elastic Scaling + +Flink allows you to adjust your cluster size dynamically to your workloads. This is possible by stopping a job with a savepoint and restarting it with a different parallelism. This page describes options where Flink automatically adjusts the parallelism. + +## Reactive Mode + +{{< hint danger >}} +Reactive mode is an experimental feature. The Flink community is actively looking for feedback by users through our mailing lists. +{{< /hint >}} + +The Reactive Mode allows Flink users to implement a powerful autoscaling mechanism, by having an external service monitor certain metrics, such as consumer lag, aggregate CPU utilization, throughput or latency. As soon as these metrics are above or below a certain threshold, additional TaskManagers can be added or removed from the Flink cluster. This could be implemented through changing the [replica factor](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#replicas) of a Kubernetes deployment, or an [autoscaling](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html) group. + +### Getting started + +If you just want to try out Reactive Mode, follow these instructions. They assume that you are deploying Flink on one machine. + +```bash +# Put Job into lib/ directory +cp ./examples/streaming/TopSpeedWindowing.jar lib/ +# Submit Job in Application Mode +./bin/standalone-job.sh start -Drestart-strategy=fixeddelay -Drestart-strategy.fixed-delay.attempts=100000 -Djobmanager.scheduler=adaptive -Dscheduler-mode=reactive -j org.apache.flink.streaming.examples.windowing.TopSpeedWindowing +# Start first TaskManager +./bin/taskmanager.sh start +``` + +Let's quickly examine the used configuration parameters: +- `-Drestart-strategy=fixeddelay` and `-Drestart-strategy.fixed-delay.attempts=100000` configure the job to restart on failure. This is needed for supporting scale-down. +- `-Djobmanager.scheduler=adaptive` enables the Adaptive Scheduler, which is needed to use Reactive Mode. +- `-Dscheduler-mode=reactive` enables Reactive Mode. + +You have now started a Flink job in Application Mode. The [web interface on localhost:8081](http://localhost:8081) now shows that the job is running on one TaskManager. If you want to scale up the job, add another TaskManager to the cluster: +```bash +# Start additional TaskManager +./bin/taskmanager.sh start +``` + +To scale down, remove a TaskManager instance. +```bash +# Remove a TaskManager +./bin/taskmanager.sh stop +``` + +### Limitations Review comment: Currently, there is only a "Getting Started" and "Limitations" section. Shouldn't there a section like "Configuration" or "Usage", too? ########## File path: docs/content/docs/deployment/elastic_scaling.md ########## @@ -0,0 +1,91 @@ +--- +title: Elastic Scaling +weight: 5 +type: docs + +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Elastic Scaling + +Flink allows you to adjust your cluster size dynamically to your workloads. This is possible by stopping a job with a savepoint and restarting it with a different parallelism. This page describes options where Flink automatically adjusts the parallelism. + +## Reactive Mode + +{{< hint danger >}} +Reactive mode is an experimental feature. The Flink community is actively looking for feedback by users through our mailing lists. +{{< /hint >}} + +The Reactive Mode allows Flink users to implement a powerful autoscaling mechanism, by having an external service monitor certain metrics, such as consumer lag, aggregate CPU utilization, throughput or latency. As soon as these metrics are above or below a certain threshold, additional TaskManagers can be added or removed from the Flink cluster. This could be implemented through changing the [replica factor](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#replicas) of a Kubernetes deployment, or an [autoscaling](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html) group. + +### Getting started + +If you just want to try out Reactive Mode, follow these instructions. They assume that you are deploying Flink on one machine. + +```bash +# Put Job into lib/ directory +cp ./examples/streaming/TopSpeedWindowing.jar lib/ +# Submit Job in Application Mode +./bin/standalone-job.sh start -Drestart-strategy=fixeddelay -Drestart-strategy.fixed-delay.attempts=100000 -Djobmanager.scheduler=adaptive -Dscheduler-mode=reactive -j org.apache.flink.streaming.examples.windowing.TopSpeedWindowing +# Start first TaskManager +./bin/taskmanager.sh start +``` + +Let's quickly examine the used configuration parameters: +- `-Drestart-strategy=fixeddelay` and `-Drestart-strategy.fixed-delay.attempts=100000` configure the job to restart on failure. This is needed for supporting scale-down. +- `-Djobmanager.scheduler=adaptive` enables the Adaptive Scheduler, which is needed to use Reactive Mode. +- `-Dscheduler-mode=reactive` enables Reactive Mode. + +You have now started a Flink job in Application Mode. The [web interface on localhost:8081](http://localhost:8081) now shows that the job is running on one TaskManager. If you want to scale up the job, add another TaskManager to the cluster: +```bash +# Start additional TaskManager +./bin/taskmanager.sh start +``` + +To scale down, remove a TaskManager instance. +```bash +# Remove a TaskManager +./bin/taskmanager.sh stop +``` + +### Limitations + +Since Reactive Mode is a new, experimental feature, not all features supported by the default scheduler are also available with Reactive Mode (and its adaptive scheduler). The Flink community is working on addressing these limitations. + +- **Deployment is only supported as a standalone application** deployment. Active resource managers (such as native Kubernetes, YARN or Mesos) are explicitly not supported. Standalone session clusters are not supported either. The application deployment is limited to single job applications. +- **Streaming jobs only**: The first version of Reactive Mode runs with streaming jobs only. When submitting a batch job, then the default scheduler will be used. +- **No support for local recovery**: Local recovery is a feature that schedules tasks to machines so that the state on that machine gets re-used if possible. The lack of this feature means that Reactive Mode will always need to restore the entire state from the checkpoint storage. +- **No support for local failovers**: Local failover means that the scheduler is able to restart parts ("regions" in Flink's internals) of a failed job, instead of the entire job. This limitations impacts only recovery time of embarrassingly parallel jobs -- Flink's default scheduler can restart failed parts, while Reactive Mode will restart the entire job. Review comment: Link to documentation on local failover? ########## File path: docs/content/docs/deployment/elastic_scaling.md ########## @@ -0,0 +1,91 @@ +--- +title: Elastic Scaling +weight: 5 +type: docs + +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Elastic Scaling + +Flink allows you to adjust your cluster size dynamically to your workloads. This is possible by stopping a job with a savepoint and restarting it with a different parallelism. This page describes options where Flink automatically adjusts the parallelism. + +## Reactive Mode + +{{< hint danger >}} +Reactive mode is an experimental feature. The Flink community is actively looking for feedback by users through our mailing lists. +{{< /hint >}} + +The Reactive Mode allows Flink users to implement a powerful autoscaling mechanism, by having an external service monitor certain metrics, such as consumer lag, aggregate CPU utilization, throughput or latency. As soon as these metrics are above or below a certain threshold, additional TaskManagers can be added or removed from the Flink cluster. This could be implemented through changing the [replica factor](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#replicas) of a Kubernetes deployment, or an [autoscaling](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html) group. + +### Getting started + +If you just want to try out Reactive Mode, follow these instructions. They assume that you are deploying Flink on one machine. + +```bash +# Put Job into lib/ directory +cp ./examples/streaming/TopSpeedWindowing.jar lib/ +# Submit Job in Application Mode +./bin/standalone-job.sh start -Drestart-strategy=fixeddelay -Drestart-strategy.fixed-delay.attempts=100000 -Djobmanager.scheduler=adaptive -Dscheduler-mode=reactive -j org.apache.flink.streaming.examples.windowing.TopSpeedWindowing +# Start first TaskManager +./bin/taskmanager.sh start +``` + +Let's quickly examine the used configuration parameters: +- `-Drestart-strategy=fixeddelay` and `-Drestart-strategy.fixed-delay.attempts=100000` configure the job to restart on failure. This is needed for supporting scale-down. +- `-Djobmanager.scheduler=adaptive` enables the Adaptive Scheduler, which is needed to use Reactive Mode. +- `-Dscheduler-mode=reactive` enables Reactive Mode. + +You have now started a Flink job in Application Mode. The [web interface on localhost:8081](http://localhost:8081) now shows that the job is running on one TaskManager. If you want to scale up the job, add another TaskManager to the cluster: +```bash +# Start additional TaskManager +./bin/taskmanager.sh start +``` + +To scale down, remove a TaskManager instance. +```bash +# Remove a TaskManager +./bin/taskmanager.sh stop +``` + +### Limitations + +Since Reactive Mode is a new, experimental feature, not all features supported by the default scheduler are also available with Reactive Mode (and its adaptive scheduler). The Flink community is working on addressing these limitations. + +- **Deployment is only supported as a standalone application** deployment. Active resource managers (such as native Kubernetes, YARN or Mesos) are explicitly not supported. Standalone session clusters are not supported either. The application deployment is limited to single job applications. +- **Streaming jobs only**: The first version of Reactive Mode runs with streaming jobs only. When submitting a batch job, then the default scheduler will be used. +- **No support for local recovery**: Local recovery is a feature that schedules tasks to machines so that the state on that machine gets re-used if possible. The lack of this feature means that Reactive Mode will always need to restore the entire state from the checkpoint storage. +- **No support for local failovers**: Local failover means that the scheduler is able to restart parts ("regions" in Flink's internals) of a failed job, instead of the entire job. This limitations impacts only recovery time of embarrassingly parallel jobs -- Flink's default scheduler can restart failed parts, while Reactive Mode will restart the entire job. +- **Limited integration with Flink's Web UI**: Reactive Mode allows that a job's parallelism can change over its lifetime. The web UI only shows the current parallelism of a job, not the historic evolution of the job. There might be other inconveniences such as a lack of the exception history, incorrect job status time-stamps and incorrect metrics. +- **No support for fine grained resource specifications**: Fine-grained resource specifications are ignored by Reactive Mode. Review comment: Like to documentation on fine-grained resource specifications. ########## File path: docs/content/docs/deployment/elastic_scaling.md ########## @@ -0,0 +1,91 @@ +--- +title: Elastic Scaling +weight: 5 +type: docs + +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Elastic Scaling + +Flink allows you to adjust your cluster size dynamically to your workloads. This is possible by stopping a job with a savepoint and restarting it with a different parallelism. This page describes options where Flink automatically adjusts the parallelism. + +## Reactive Mode + +{{< hint danger >}} +Reactive mode is an experimental feature. The Flink community is actively looking for feedback by users through our mailing lists. +{{< /hint >}} + +The Reactive Mode allows Flink users to implement a powerful autoscaling mechanism, by having an external service monitor certain metrics, such as consumer lag, aggregate CPU utilization, throughput or latency. As soon as these metrics are above or below a certain threshold, additional TaskManagers can be added or removed from the Flink cluster. This could be implemented through changing the [replica factor](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#replicas) of a Kubernetes deployment, or an [autoscaling](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html) group. + +### Getting started + +If you just want to try out Reactive Mode, follow these instructions. They assume that you are deploying Flink on one machine. + +```bash +# Put Job into lib/ directory +cp ./examples/streaming/TopSpeedWindowing.jar lib/ +# Submit Job in Application Mode +./bin/standalone-job.sh start -Drestart-strategy=fixeddelay -Drestart-strategy.fixed-delay.attempts=100000 -Djobmanager.scheduler=adaptive -Dscheduler-mode=reactive -j org.apache.flink.streaming.examples.windowing.TopSpeedWindowing +# Start first TaskManager +./bin/taskmanager.sh start +``` + +Let's quickly examine the used configuration parameters: +- `-Drestart-strategy=fixeddelay` and `-Drestart-strategy.fixed-delay.attempts=100000` configure the job to restart on failure. This is needed for supporting scale-down. +- `-Djobmanager.scheduler=adaptive` enables the Adaptive Scheduler, which is needed to use Reactive Mode. +- `-Dscheduler-mode=reactive` enables Reactive Mode. + +You have now started a Flink job in Application Mode. The [web interface on localhost:8081](http://localhost:8081) now shows that the job is running on one TaskManager. If you want to scale up the job, add another TaskManager to the cluster: Review comment: ```suggestion You have now started a Flink job in Application Mode. The [web interface](http://localhost:8081) shows that the job is running on one TaskManager. If you want to scale up the job, simply add another TaskManager to the cluster: ``` ########## File path: docs/content/docs/deployment/elastic_scaling.md ########## @@ -0,0 +1,91 @@ +--- +title: Elastic Scaling +weight: 5 +type: docs + +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Elastic Scaling + +Flink allows you to adjust your cluster size dynamically to your workloads. This is possible by stopping a job with a savepoint and restarting it with a different parallelism. This page describes options where Flink automatically adjusts the parallelism. + +## Reactive Mode + +{{< hint danger >}} +Reactive mode is an experimental feature. The Flink community is actively looking for feedback by users through our mailing lists. +{{< /hint >}} + +The Reactive Mode allows Flink users to implement a powerful autoscaling mechanism, by having an external service monitor certain metrics, such as consumer lag, aggregate CPU utilization, throughput or latency. As soon as these metrics are above or below a certain threshold, additional TaskManagers can be added or removed from the Flink cluster. This could be implemented through changing the [replica factor](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#replicas) of a Kubernetes deployment, or an [autoscaling](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html) group. + +### Getting started + +If you just want to try out Reactive Mode, follow these instructions. They assume that you are deploying Flink on one machine. + +```bash +# Put Job into lib/ directory +cp ./examples/streaming/TopSpeedWindowing.jar lib/ +# Submit Job in Application Mode +./bin/standalone-job.sh start -Drestart-strategy=fixeddelay -Drestart-strategy.fixed-delay.attempts=100000 -Djobmanager.scheduler=adaptive -Dscheduler-mode=reactive -j org.apache.flink.streaming.examples.windowing.TopSpeedWindowing +# Start first TaskManager +./bin/taskmanager.sh start +``` + +Let's quickly examine the used configuration parameters: +- `-Drestart-strategy=fixeddelay` and `-Drestart-strategy.fixed-delay.attempts=100000` configure the job to restart on failure. This is needed for supporting scale-down. +- `-Djobmanager.scheduler=adaptive` enables the Adaptive Scheduler, which is needed to use Reactive Mode. +- `-Dscheduler-mode=reactive` enables Reactive Mode. + +You have now started a Flink job in Application Mode. The [web interface on localhost:8081](http://localhost:8081) now shows that the job is running on one TaskManager. If you want to scale up the job, add another TaskManager to the cluster: +```bash +# Start additional TaskManager +./bin/taskmanager.sh start +``` + +To scale down, remove a TaskManager instance. +```bash +# Remove a TaskManager +./bin/taskmanager.sh stop +``` + +### Limitations + +Since Reactive Mode is a new, experimental feature, not all features supported by the default scheduler are also available with Reactive Mode (and its adaptive scheduler). The Flink community is working on addressing these limitations. + +- **Deployment is only supported as a standalone application** deployment. Active resource managers (such as native Kubernetes, YARN or Mesos) are explicitly not supported. Standalone session clusters are not supported either. The application deployment is limited to single job applications. +- **Streaming jobs only**: The first version of Reactive Mode runs with streaming jobs only. When submitting a batch job, then the default scheduler will be used. +- **No support for local recovery**: Local recovery is a feature that schedules tasks to machines so that the state on that machine gets re-used if possible. The lack of this feature means that Reactive Mode will always need to restore the entire state from the checkpoint storage. +- **No support for local failovers**: Local failover means that the scheduler is able to restart parts ("regions" in Flink's internals) of a failed job, instead of the entire job. This limitations impacts only recovery time of embarrassingly parallel jobs -- Flink's default scheduler can restart failed parts, while Reactive Mode will restart the entire job. +- **Limited integration with Flink's Web UI**: Reactive Mode allows that a job's parallelism can change over its lifetime. The web UI only shows the current parallelism of a job, not the historic evolution of the job. There might be other inconveniences such as a lack of the exception history, incorrect job status time-stamps and incorrect metrics. +- **No support for fine grained resource specifications**: Fine-grained resource specifications are ignored by Reactive Mode. +- **Rescaling causes downtime**: Rescaling of a job happens by restarting the job, thus jobs with large state might need a lot of resources and time to rescale. Rescaling a job causes downtime of your job, but no data loss. Review comment: ```suggestion - **Rescaling causes downtime**: Rescaling of a job happens by restarting the job, thus jobs with large state might need a lot of resources and time to rescale. Rescaling a job causes downtime of your job, but no data loss. ``` Should we add this as a limitation? This sounds more like a possible future optimization than a limitation. It is completely expected like this. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
