[
https://issues.apache.org/jira/browse/FLINK-27925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17649232#comment-17649232
]
ouyangwulin edited comment on FLINK-27925 at 12/19/22 12:31 PM:
----------------------------------------------------------------
In the case of large-scale start and stop jobs, constantly reading data from
etcd can cause a bottleneck in etcd performance. I agree with [~wangyang0918]
using informer will increase memory pressure. We can increase resourceversion=0
in watcher to reduce data read from etcd.
As
[https://kubernetes.io/docs/reference/using-api/api-concepts/#the-resourceversion-parameter]
describe and the screenshots of code，
1 ."resourceVersion unset" is means "Most Recent" ，The returned data must be
consistent (in detail: served from etcd via a quorum read).
2. "resourceVersion="0" is means "Any". Return data at any resource
version. The newest available resource version is preferred, but strong
consistency is not required; data at any resource version may be served. It is
possible for the request to return data at a much older resource version that
the client has previously observed, particularly in high availability
configurations, due to partitions or stale caches.
!image-2022-12-19-20-19-41-303.png!
was (Author: ouyangwuli):
In the case of large-scale start and stop jobs, constantly reading data from
etcd can cause a bottleneck in etcd performance. I agree with [~wangyang0918]
using informer will increase memory pressure. We can increase resourceversion=0
in watch to reduce data read from etcd.
> Avoid to create watcher without the resourceVersion
> ---------------------------------------------------
>
> Key: FLINK-27925
> URL: https://issues.apache.org/jira/browse/FLINK-27925
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / Kubernetes
> Reporter: Aitozi
> Priority: Major
> Labels: pull-request-available
> Attachments: image-2022-12-19-20-19-41-303.png
>
>
> Currently, we create the watcher in KubernetesResourceManager. But it do not
> pass the resourceVersion parameter, it will trigger a request to etcd. It
> will bring the burden to the etcd in large scale cluster (which have been
> seen in our internal k8s cluster). More detail can be found
> [here|https://kubernetes.io/docs/reference/using-api/api-concepts/#the-resourceversion-parameter]
>
> I think we could use the informer to improve it (which will spawn a
> list-watch and maintain the resourceVersion internally)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
