As Stuart says, that looks correct, assuming your metrics don't have any
labels other than the ones you've excluded. You'd save a lot of typing just
by doing:
sum(scrape_samples_scraped)
which is expected to return a single value, with no labels (as it's summed
across all timeseries of this metric).
The value 7,525,871,918 does seem quite high - what was it before? You can
set an execution time for this query in the PromQL browser, or draw a graph
this expression over time, to see historical values.
You could also look at
count(scrape_samples_scraped)
or more simply
count(up)
and see if that has jumped up: it would imply that lots more targets have
been added (e.g. more pods are being monitored).
If not, then as well as Stuart's suggestion of graphing
"scrape_samples_scraped" by itself to see if one particular target is
generating way more metrics than usual, you could try different summary
variants like
sum by (instance,job) (scrape_samples_scraped)
sum by (clusterName) (scrape_samples_scraped)
... etc
and see if there's a spike in any of these. This may help you drill down
to the offending item(s).
On Thursday, 27 July 2023 at 15:51:24 UTC+1 Uvais Ibrahim wrote:
> Hi Brain,
>
> This is the query that I have used.
>
> sum(scrape_samples_scraped)without(app,app_kubernetes_io_managed_by,clusterName,release,environment,instance,job,k8s_cluster,kubernetes_name,kubernetes_namespace,ou,app_kubernetes_io_component,app_kubernetes_io_name,app_kubernetes_io_version,kustomize_toolkit_fluxcd_io_name,kustomize_toolkit_fluxcd_io_namespace,application,name,role,app_kubernetes_io_instance,app_kubernetes_io_part_of,control_plane,beta_kubernetes_io_arch,beta_kubernetes_io_instance_type,
>
> beta_kubernetes_io_os, failure_domain_beta_kubernetes_io_region,
> failure_domain_beta_kubernetes_io_zone,kubernetes_io_arch,
> kubernetes_io_hostname, kubernetes_io_os, node_kubernetes_io_instance_type,
> nodegroup, topology_kubernetes_io_region,
> topology_kubernetes_io_zone,chart,heritage,revised,transit,component,namespace,
>
> pod_name, pod_template_hash, security_istio_io_tlsMode,
> service_istio_io_canonical_name,
> service_istio_io_canonical_revision,k8s_app,kubernetes_io_cluster_service,kubernetes_io_name,route_reflector)
>
> Which simply excluded every label but still I am getting a result like this
>
> {} 7525871918
>
>
> It shouldn't return any results right?
>
> Prometheus version: 2.36.2
>
> By increased traffic I meant that, the prometheus servers are getting high
> traffic from a specific point of time. Currently prometheus is getting 13
> million packets earlier it was like 2 to 3 M packets on an average. And the
> prometheus endpoint is not public.
>
>
> On Thursday, July 27, 2023 at 6:06:10 PM UTC+5:30 Brian Candler wrote:
>
>> scrape_samples_scraped always has the labels which prometheus itself adds
>> (i.e. job and instance).
>>
>> Extraordinary claims require extraordinary evidence. Are you saying that
>> the PromQL query *scrape_samples_scraped{job="",instance=""}* returns a
>> result? If so, what's the number? What do you mean by "with increased
>> size" - increased as compared to what? And what version of prometheus are
>> you running?
>>
>> In any case, what you see with scrape_samples_scraped may be completely
>> unrelated to the "high traffic" issue. Is your prometheus server exposed
>> to the Internet? Maybe someone is accessing it remotely. Even if not, you
>> can use packet capture to work out where the traffic is going to and from.
>> A tool like https://www.sniffnet.net/ may be helpful.
>>
>> On Thursday, 27 July 2023 at 13:14:25 UTC+1 Uvais Ibrahim wrote:
>>
>>> Hi,
>>>
>>> Since last night, my Prometheus EC2 servers are getting high traffic
>>> unusually. When I was checking in Prometheus I can see this
>>> metric scrape_samples_scraped with with increased size but without any
>>> labels. What could be the reason?
>>>
>>>
>>> Thanks,
>>> Uvais Ibrahim
>>>
>>>
>>>
>>>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/811fba5c-1bd3-4677-b276-84116180a1acn%40googlegroups.com.
