[
https://issues.apache.org/jira/browse/FLINK-34152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17822790#comment-17822790
]
Maximilian Michels edited comment on FLINK-34152 at 3/2/24 9:11 AM:
--------------------------------------------------------------------
{quote}What I'm trying to say is that I didn't notice any updated configuration
for the memory request and limit at the pod template level for the taskmanager.
Therefore, I assume that the pod's memory allocation won't automatically adjust
to reflect changes in the taskmanager's heap size, unless I've missed something.
{quote}
I would ask you to check again. When the tuning is applied, the pod resource
requests/limits of the TaskManager pods will be adjusted. So changes will be
directly reflected in terms of resource usage in Kubernetes.
{quote}Indeed, by implementing bin-packing, we can optimize resource
utilization, which is now clearer to me. However, its management becomes more
complex (K8s upgrade, daily node restart/eviction) for sure, especially when
there are other application components in the same Kubernetes cluster IMO
{quote}
You are right, there is more complexity to realize Flink Autoscaling benefits
end to end. However, there is also a great amount of resource savings and
convenience for the user that come out of it. We have seen 60% fewer nodes
after enabling Flink Autoscaling while maintaing the same amount of service and
drastically decreasing the maintaince for our users who would have to adjust
parallelism constantly to run cost-efficient. They usually did not want to do
that and thus all jobs ran very over-provisioned.
{quote}Can you take a look on it
https://issues.apache.org/jira/browse/FLINK-34563 and
[https://github.com/apache/flink-kubernetes-operator/pull/787] ? And tell me if
you think it's making sense, thanks :)
{quote}
Thank you, I'll review in the next days!
was (Author: mxm):
{quote}What I'm trying to say is that I didn't notice any updated configuration
for the memory request and limit at the pod template level for the taskmanager.
Therefore, I assume that the pod's memory allocation won't automatically adjust
to reflect changes in the taskmanager's heap size, unless I've missed something.
{quote}
I would ask you to check again. When the tuning is applied, the pod resource
requests/limits of the TaskManager pods will be adjusted. So changes will be
directly reflected in terms of resource usage in Kubernetes.
{quote}Indeed, by implementing bin-packing, we can optimize resource
utilization, which is now clearer to me. However, its management becomes more
complex (K8s upgrade, daily node restart/eviction) for sure, especially when
there are other application components in the same Kubernetes cluster IMO
{quote}
You are right, there is more complexity to realize Flink Autoscaling benefits
end to end. However, there is also a great amount of resource savings and
convenience for the user that come out of it. We have seen 60% fewer nodes
after enabling Flink Autoscaling while maintaing the same amount of service and
drastically decreasing the maintaince for our users who would have to adjust
parallelism constantly to run cost-efficient. They usually did not want to do
that and thus all jobs ran very over-provisioned.
{quote}Can you take a look on it
https://issues.apache.org/jira/browse/FLINK-34563 and
[https://github.com/apache/flink-kubernetes-operator/pull/787] ? And tell me if
you think it's making sense, thanks :)
{quote}
Thank you, I'll review in the next days. Generally, I wonder. This idea has
crossed my mind before. I wasn't really sure how exactly to adjust the
parallelism to fill the TaskManagers. Adjusting only the vertex with the
highest parallelism might be unfair to other vertices. I think spreading out
the unused task slots to the vertices with ther lowest parallelism might be
more beneficial for the stability. We have seen more instability with lower
parallelisms because the metrics are less precise.
> Tune TaskManager memory
> -----------------------
>
> Key: FLINK-34152
> URL: https://issues.apache.org/jira/browse/FLINK-34152
> Project: Flink
> Issue Type: Sub-task
> Components: Autoscaler, Kubernetes Operator
> Reporter: Maximilian Michels
> Assignee: Maximilian Michels
> Priority: Major
> Labels: pull-request-available
> Fix For: kubernetes-operator-1.8.0
>
>
> The current autoscaling algorithm adjusts the parallelism of the job task
> vertices according to the processing needs. By adjusting the parallelism, we
> systematically scale the amount of CPU for a task. At the same time, we also
> indirectly change the amount of memory tasks have at their dispense. However,
> there are some problems with this.
> # Memory is overprovisioned: On scale up we may add more memory than we
> actually need. Even on scale down, the memory / cpu ratio can still be off
> and too much memory is used.
> # Memory is underprovisioned: For stateful jobs, we risk running into
> OutOfMemoryErrors on scale down. Even before running out of memory, too
> little memory can have a negative impact on the effectiveness of the scaling.
> We lack the capability to tune memory proportionally to the processing needs.
> In the same way that we measure CPU usage and size the tasks accordingly, we
> need to evaluate memory usage and adjust the heap memory size.
> https://docs.google.com/document/d/19GXHGL_FvN6WBgFvLeXpDABog2H_qqkw1_wrpamkFSc/edit
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
