Hi All,
I am seeing some time series from the target are getting dropped and not
written to tsdb. Initially i thought it might be related to wrong metric
relabel configs, but i have removed all the relabel configs, even then the
time series is not being written.
Target is giving the time series.
Following the is curl output from the target:
istio_requests_total{response_code="200",reporter="destination",
*source_workload="ind-cpro"*,source_workload_namespace="csd-ns",source_principal="unknown",source_app="cpro",source_version="unknown",destination_workload="ind-nrfp",destination_workload_namespace="csd-ns",destination_principal="unknown",destination_app="nrfp",destination_version="unknown",destination_service="csd-nnrfp.csd-ns.svc.cluster.local",destination_service_name="csd-nnrfp",destination_service_namespace="csd-ns",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="none",source_canonical_service="cpro",destination_canonical_service="nrfp",source_canonical_revision="2.15.1",destination_canonical_revision="latest"}
3017
istio_requests_total{response_code="200",reporter="source",
*source_workload="ind-nrfp"*,source_workload_namespace="csd-ns",source_principal="unknown",source_app="nrfp",source_version="unknown",destination_workload="unknown",destination_workload_namespace="unknown",destination_principal="unknown",destination_app="unknown",destination_version="unknown",destination_service="
abc.default.svc.cluster.local",destination_service_name="abc.default.svc.cluster.local",destination_service_namespace="unknown",request_protocol="grpc",response_flags="-",grpc_response_status="0",connection_security_policy="unknown",source_canonical_service="nrfp",destination_canonical_service="unknown",source_canonical_revision="latest",destination_canonical_revision="latest"}
276
Prometheus job configuration is as follows(ip address is masked):
- honor_labels: true
job_name: chal
metrics_path: /stats/prometheus
static_configs:
- targets:
- abc.xyz.def.pqr:15090
In the above time series from target, the time series with
*source_workload="ind-nrfp"
*is not present in prometheus. There are no warnings/errors seen in the
debug logs
Can you let me know how to debug this issue?
Thanks n Regards,
Chalapathi.
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/CABXnQPtESH13%3DZUFBs714mTay71i6U02yCL4LLJ2g0c-236PZQ%40mail.gmail.com.
