tangjiaxing669 opened a new issue #8565: URL: https://github.com/apache/skywalking/issues/8565
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/skywalking/issues?q=is%3Aissue) and found no similar issues. ### Apache SkyWalking Component Helm Chart (apache/skywalking-kubernetes) ### What happened Kubernetes Version: ```bash Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.5", GitCommit:"aea7bbadd2fc0cd689de94a54e5b7b758869d691", GitTreeState:"clean", BuildDate:"2021-09-15T21:10:45Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.5-eks-bc4871b", GitCommit:"5236faf39f1b7a7dabea8df12726f25608131aa9", GitTreeState:"clean", BuildDate:"2021-10-29T23:32:16Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"} ``` The helm chat reference is installed [here](https://github.com/apache/skywalking-kubernetes). The follow is values.yaml file: ```yaml serviceAccounts: oap: null imagePullSecrets: [] initContainer: image: busybox tag: '1.30' oap: name: oap dynamicConfigEnabled: false image: repository: skywalking.docker.scarf.sh/apache/skywalking-oap-server tag: 8.8.1 pullPolicy: IfNotPresent storageType: elasticsearch ports: grpc: 11800 rest: 12800 replicas: 2 service: type: ClusterIP javaOpts: '-Xmx2g -Xms2g' antiAffinity: soft nodeAffinity: {} nodeSelector: {} tolerations: [] resources: {} envoy: als: enabled: false istio: adapter: enabled: false env: null ui: name: ui replicas: 1 image: repository: skywalking.docker.scarf.sh/apache/skywalking-ui tag: 8.8.1 pullPolicy: IfNotPresent nodeAffinity: {} nodeSelector: {} tolerations: [] ingress: enabled: false annotations: {} path: / hosts: [] tls: [] service: type: LoadBalancer externalPort: 80 internalPort: 8080 annotations: {} esInit: nodeAffinity: {} nodeSelector: {} tolerations: [] elasticsearch: enabled: true config: port: http: 9200 host: elasticsearch user: xxx password: xxx clusterName: elasticsearch nodeGroup: master masterService: '' roles: master: 'true' ingest: 'true' data: 'true' replicas: 3 minimumMasterNodes: 2 esMajorVersion: '' esConfig: {} extraEnvs: [] secretMounts: [] image: docker.elastic.co/elasticsearch/elasticsearch imageTag: 6.8.6 imagePullPolicy: IfNotPresent podAnnotations: {} labels: {} esJavaOpts: '-Xmx1g -Xms1g' resources: requests: cpu: 100m memory: 2Gi limits: cpu: 1000m memory: 2Gi initResources: limits: cpu: 25m memory: 128Mi requests: cpu: 25m memory: 128Mi sidecarResources: limits: cpu: 25m memory: 128Mi requests: cpu: 25m memory: 128Mi networkHost: 0.0.0.0 volumeClaimTemplate: storageClassName: gp2 accessModes: - ReadWriteOnce resources: requests: storage: 30Gi rbac: create: false serviceAccountName: '' podSecurityPolicy: create: false name: '' spec: privileged: true fsGroup: rule: RunAsAny runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny volumes: - secret - configMap - persistentVolumeClaim persistence: enabled: false annotations: {} extraVolumes: '' extraVolumeMounts: '' extraInitContainers: '' priorityClassName: '' antiAffinityTopologyKey: kubernetes.io/hostname antiAffinity: hard nodeAffinity: {} podManagementPolicy: Parallel protocol: http httpPort: 9200 transportPort: 9300 service: labels: {} labelsHeadless: {} type: ClusterIP nodePort: '' annotations: {} httpPortName: http transportPortName: transport updateStrategy: RollingUpdate maxUnavailable: 1 podSecurityContext: fsGroup: 1000 runAsUser: 1000 fsGroup: '' securityContext: capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000 terminationGracePeriod: 120 sysctlVmMaxMapCount: 262144 readinessProbe: failureThreshold: 3 initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 3 timeoutSeconds: 5 clusterHealthCheckParams: wait_for_status=green&timeout=1s schedulerName: '' imagePullSecrets: [] nodeSelector: {} tolerations: [] ingress: enabled: false annotations: {} path: / hosts: - chart-example.local tls: [] nameOverride: '' fullnameOverride: '' masterTerminationFix: false lifecycle: {} sysctlInitContainer: enabled: true keystore: [] satellite: name: satellite replicas: 1 enabled: false image: repository: skywalking.docker.scarf.sh/apache/skywalking-satellite tag: null pullPolicy: IfNotPresent ports: grpc: 11800 prometheus: 1234 service: type: ClusterIP antiAffinity: soft nodeAffinity: {} nodeSelector: {} tolerations: [] resources: {} podAnnotations: null env: null nameOverride: '' fullnameOverride: '' ``` When I use java agent to inject into my business container, after a while, the log of the OAP container will print a lot of the following information: ``` 2022-02-21 02:19:13,434 io.kubernetes.client.informer.cache.ReflectorRunnable 235 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] INFO [] - Watch connection expired: too old resource version: 19202086 (19434506) 2022-02-21 02:19:13,480 io.kubernetes.client.informer.cache.ReflectorRunnable 235 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] INFO [] - Watch connection expired: too old resource version: 19202086 (19434506) 2022-02-21 02:19:13,524 io.kubernetes.client.informer.cache.ReflectorRunnable 235 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] INFO [] - Watch connection expired: too old resource version: 19202086 (19434506) 2022-02-21 02:19:13,525 io.kubernetes.client.informer.cache.ReflectorRunnable 235 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] INFO [] - Watch connection expired: too old resource version: 19202086 (19434506) 2022-02-21 02:19:13,568 io.kubernetes.client.informer.cache.ReflectorRunnable 235 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] INFO [] - Watch connection expired: too old resource version: 19202086 (19434506) 2022-02-21 02:19:13,569 io.kubernetes.client.informer.cache.ReflectorRunnable 235 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] INFO [] - Watch connection expired: too old resource version: 19202086 (19434506) 2022-02-21 02:19:13,612 io.kubernetes.client.informer.cache.ReflectorRunnable 235 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] INFO [] - Watch connection expired: too old resource version: 19202086 (19434506) 2022-02-21 02:19:13,656 io.kubernetes.client.informer.cache.ReflectorRunnable 235 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] INFO [] - Watch connection expired: too old resource version: 19202086 (19434506) 2022-02-21 02:19:13,657 io.kubernetes.client.informer.cache.ReflectorRunnable 235 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] INFO [] - Watch connection expired: too old resource version: 19202086 (19434506) 2022-02-21 02:19:13,700 io.kubernetes.client.informer.cache.ReflectorRunnable 235 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] INFO [] - Watch connection expired: too old resource version: 19202086 (19434506) 2022-02-21 02:19:13,701 io.kubernetes.client.informer.cache.ReflectorRunnable 235 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] INFO [] - Watch connection expired: too old resource version: 19202086 (19434506) 2022-02-21 02:19:13,702 io.kubernetes.client.informer.cache.ReflectorRunnable 235 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] INFO [] - Watch connection expired: too old resource version: 19202086 (19434506) 2022-02-21 02:19:13,748 io.kubernetes.client.informer.cache.ReflectorRunnable 235 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] INFO [] - Watch connection expired: too old resource version: 19202086 (19434506) 2022-02-21 02:19:13,754 io.kubernetes.client.informer.cache.ReflectorRunnable 235 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] INFO [] - Watch connection expired: too old resource version: 19202086 (19434506) 2022-02-21 02:19:13,756 io.kubernetes.client.informer.cache.ReflectorRunnable 235 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] INFO [] - Watch connection expired: too old resource version: 19202086 (19434506) 2022-02-21 02:19:13,803 io.kubernetes.client.informer.cache.ReflectorRunnable 235 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] INFO [] - Watch connection expired: too old resource version: 19202086 (19434506) 2022-02-21 02:19:13,806 io.kubernetes.client.informer.cache.ReflectorRunnable 235 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] INFO [] - Watch connection expired: too old resource version: 19202086 (19434506) 2022-02-21 02:19:13,808 io.kubernetes.client.informer.cache.ReflectorRunnable 235 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] INFO [] - Watch connection expired: too old resource version: 19202086 (19434506) 2022-02-21 02:19:13,810 io.kubernetes.client.informer.cache.ReflectorRunnable 235 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] INFO [] - Watch connection expired: too old resource version: 19202086 (19434506) 2022-02-21 02:19:13,852 io.kubernetes.client.informer.cache.ReflectorRunnable 235 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] INFO [] - Watch connection expired: too old resource version: 19202086 (19434506) 2022-02-21 02:19:13,853 io.kubernetes.client.informer.cache.ReflectorRunnable 235 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] INFO [] - Watch connection expired: too old resource version: 19202086 (19434506) 2022-02-21 02:19:13,896 io.kubernetes.client.informer.cache.ReflectorRunnable 235 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] INFO [] - Watch connection expired: too old resource version: 19202086 (19434506) 2022-02-21 02:19:13,940 io.kubernetes.client.informer.cache.ReflectorRunnable 235 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] INFO [] - Watch connection expired: too old resource version: 19202086 (19434506) 2022-02-21 02:19:13,942 io.kubernetes.client.informer.cache.ReflectorRunnable 235 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] INFO [] - Watch connection expired: too old resource version: 19202086 (19434506) 2022-02-21 02:19:13,984 io.kubernetes.client.informer.cache.ReflectorRunnable 235 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] INFO [] - Watch connection expired: too old resource version: 19202086 (19434506) 2022-02-21 02:19:13,985 io.kubernetes.client.informer.cache.ReflectorRunnable 235 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] INFO [] - Watch connection expired: too old resource version: 19202086 (19434506) 2022-02-21 02:19:14,028 io.kubernetes.client.informer.cache.ReflectorRunnable 235 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] INFO [] - Watch connection expired: too old resource version: 19202086 (19434506) 2022-02-21 02:19:14,029 io.kubernetes.client.informer.cache.ReflectorRunnable 235 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] INFO [] - Watch connection expired: too old resource version: 19202086 (19434506) 2022-02-21 02:19:14,071 io.kubernetes.client.informer.cache.ReflectorRunnable 235 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] INFO [] - Watch connection expired: too old resource version: 19202086 (19434506) 2022-02-21 02:19:14,116 io.kubernetes.client.informer.cache.ReflectorRunnable 235 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] INFO [] - Watch connection expired: too old resource version: 19202086 (19434506) 2022-02-21 02:19:14,117 io.kubernetes.client.informer.cache.ReflectorRunnable 235 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] INFO [] - Watch connection expired: too old resource version: 19202086 (19434506) ``` I'm not sure if this is a bug, but when there are a lot of these logs, the ui will not be able to get the latest trace and metric data, like the oap is blocked. ### What you expected to happen When I restart the OAP container, it will go back to normal, but after a while, the messages will appear again, dozens of messages per second. ### How to reproduce If you install in eks using helm I think the same problem occurs. ### Anything else _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
