Hi all,
I am using Prometheus and cAdvisor to monitor containers running in 4 nodes
Kubernetes cluster.
Let's say,
Master IP: 192.168.13.197
Worker-node 1: 192.168.13.198
Worker-node 2: 192.168.13.234
Worker-node 3: 192.168.13.235
I deployed cAdvisor as daemonset and scraping using below configuration in
prometheus.yml
- job_name: cadvisor
static_configs:
- targets: ['192.168.13.197:31223', '192.168.13.198:31223',
'192.168.13.234:31223', '192.168.13.235:31223']
I deployed a container busybox in Worker-node3 (192.168.13.235) and ensure
that the container is running as expected.
kube[root@master-node ~]# kubectl get pods --all-namespaces -o wide | grep
busybox
monitoring busybox 1/1
Running 0 29m 173.168.200.67 worker-node3 <none>
<none>
[root@master-node ~]#
When I query the statistics of this container only when I do below query
192.168.13.198:31223. But, I am expecting it should be available when I
query 192.168.13.235:31223.
container_network_transmit_packets_total{container_label_annotation_hash_operator_tigera_io_cni_config="",container_label_annotation_hash_operator_tigera_io_node_cert="",container_label_annotation_hash_operator_tigera_io_typha_ca="",container_label_annotation_hash_operator_tigera_io_typha_cert="",container_label_annotation_io_kubernetes_container_hash="",container_label_annotation_io_kubernetes_container_ports="",container_label_annotation_io_kubernetes_container_restartCount="",container_label_annotation_io_kubernetes_container_terminationMessagePath="",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="",container_label_annotation_kubectl_kubernetes_io_last_applied_configuration="{\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"annotations\":{},\"labels\":{\"run\":\"busybox\"},\"name\":\"busybox\",\"namespace\":\"monitoring\"},\"spec\":{\"containers\":[{\"args\":[\"sleep\",\"1000000\"],\"image\":\"busybox\",\"name\":\"busybox\"}]}}\n",container_label_annotation_kubernetes_io_config_seen="2021-01-12T17:16:41.579490144+05:30",container_label_annotation_kubernetes_io_config_source="api",container_label_app="",container_label_component="",container_label_controller_revision_hash="",container_label_description="",container_label_io_kubernetes_container_logpath="",container_label_io_kubernetes_container_name="POD",container_label_io_kubernetes_docker_type="podsandbox",container_label_io_kubernetes_pod_name="busybox",container_label_io_kubernetes_pod_namespace="monitoring",container_label_io_kubernetes_pod_uid="fb5da70b-f8db-4394-8725-4e28a4635ac3",container_label_io_kubernetes_sandbox_id="",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_run="busybox",container_label_summary="",container_label_vendor="",container_label_version="",id="/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb5da70b_f8db_4394_8725_4e28a4635ac3.slice/docker-2e56f2f1ffa9ffaf1f3078fa0f409ec08e6e81358ad44bf23e5eb3e1797f4fb4.scope",image="k8s.gcr.io/pause:3.2",interface="eth0",name="k8s_POD_busybox_monitoring_fb5da70b-f8db-4394-8725-4e28a4635ac3_0"}
8.973906e+06 1610452889584
container_network_transmit_packets_total{container_label_annotation_hash_operator_tigera_io_cni_config="",container_label_annotation_hash_operator_tigera_io_node_cert="",container_label_annotation_hash_operator_tigera_io_typha_ca="",container_label_annotation_hash_operator_tigera_io_typha_cert="",container_label_annotation_io_kubernetes_container_hash="",container_label_annotation_io_kubernetes_container_ports="",container_label_annotation_io_kubernetes_container_restartCount="",container_label_annotation_io_kubernetes_container_terminationMessagePath="",container_label_annotation_io_kubernetes_container_terminationMessagePolicy="",container_label_annotation_io_kubernetes_pod_terminationGracePeriod="",container_label_annotation_kubectl_kubernetes_io_last_applied_configuration="{\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"annotations\":{},\"labels\":{\"run\":\"busybox\"},\"name\":\"busybox\",\"namespace\":\"monitoring\"},\"spec\":{\"containers\":[{\"args\":[\"sleep\",\"1000000\"],\"image\":\"busybox\",\"name\":\"busybox\"}]}}\n",container_label_annotation_kubernetes_io_config_seen="2021-01-12T17:16:41.579490144+05:30",container_label_annotation_kubernetes_io_config_source="api",container_label_app="",container_label_component="",container_label_controller_revision_hash="",container_label_description="",container_label_io_kubernetes_container_logpath="",container_label_io_kubernetes_container_name="POD",container_label_io_kubernetes_docker_type="podsandbox",container_label_io_kubernetes_pod_name="busybox",container_label_io_kubernetes_pod_namespace="monitoring",container_label_io_kubernetes_pod_uid="fb5da70b-f8db-4394-8725-4e28a4635ac3",container_label_io_kubernetes_sandbox_id="",container_label_k8s_app="",container_label_maintainer="",container_label_name="",container_label_pod_template_generation="",container_label_pod_template_hash="",container_label_release="",container_label_run="busybox",container_label_summary="",container_label_vendor="",container_label_version="",id="/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb5da70b_f8db_4394_8725_4e28a4635ac3.slice/docker-2e56f2f1ffa9ffaf1f3078fa0f409ec08e6e81358ad44bf23e5eb3e1797f4fb4.scope",image="k8s.gcr.io/pause:3.2",interface="tunl0",name="k8s_POD_busybox_monitoring_fb5da70b-f8db-4394-8725-4e28a4635ac3_0"}
0 1610452889584
Can anyone tell me is it working as expected and my expectation is wrong?
Please help me to understand how this works.
Let me know if any additional details needed.
Thanks in advance.
Best regards,
Binu
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/5f67d657-6805-4d0b-9584-a976a508ed59n%40googlegroups.com.
