Re: [prometheus-users] Re: Prometheus Performance issues

[adi garg]

Hey Ben, as per your suggestion, I have upgraded it to v2.16.0, but 
surprisingly my memory consumption increases after that. Now it's taking 
around 3kb per metric. Can you please help me with that?

On Wednesday, March 25, 2020 at 12:54:45 PM UTC+5:30, Ben Kochie wrote:
>
> Your Prometheus version is almost 2 years old. You need to upgrade to 
> 2.17.0.
>
> On Wed, Mar 25, 2020 at 8:08 AM adi garg > 
> wrote:
>
>> Please help?? Do I need to specify any other detail?
>>
>> On Wednesday, March 25, 2020 at 4:12:28 AM UTC+5:30, adi garg wrote:
>>>
>>> Hello experts, I was stress-testing Prometheus on the AWS cluster. 
>>> Prometheus version = 2.3
>>> Number of worker nodes in a cluster = 50 
>>> scrape_interval = 15s
>>> I am getting around 10,000 metrics from a node and I am running 
>>> Prometheus in a docker container with memory specified to 500MB.
>>> My avg metric size comes out to be around 2.3kb, which according to me 
>>> is unexpectedly higher. Moreover, the Docker container keeps getting burst 
>>> out every (10-15)s and then restarting again.
>>> Can somebody tells me what could be the reason for this?
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to promethe...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/prometheus-users/6073cdfc-5622-4a4f-a27f-7e9b0fe06339%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/4fa50403-80c9-43c0-ba3d-cfe46eb9cf5a%40googlegroups.com.

[prometheus-users] Re: Prometheus Performance issues

[Brian Candler]

If you run a PromQL query, it will automatically access data from wherever 
it's needed - in RAM, in stored chunks, or both, depending on what time 
range the query covers.

However if you query data outside the retention period then obviously it 
may not be there at all - depending on exactly when the old chunks are 
dropped.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/bdfd5184-b90f-47c1-bf98-1b0627c54588%40googlegroups.com.

[prometheus-users] Re: Prometheus Performance issues

[adi garg]

Thanks, Brian for the suggestion. I have one more doubt, assume our 
retention period is 2hrs and we fire a query to get data for 4hrs, so it's 
gonna fire two queries, one for the data in RAM and other for the external 
storage that we are using?
 

On Wednesday, March 25, 2020 at 1:21:35 PM UTC+5:30, Brian Candler wrote:
>
> You are getting 10,000 metrics from each node and there are 50 nodes, 
> that's 500,000 timeseries.
>
> Your prometheus server will need significantly more than 500MB of RAM to 
> handle that.  See:
>
> https://www.robustperception.io/how-much-ram-does-prometheus-2-x-need-for-cardinality-and-ingestion
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/b99bcb2e-2a95-431b-a070-09f67e622926%40googlegroups.com.

[prometheus-users] Re: Prometheus Performance issues

[Brian Candler]

You are getting 10,000 metrics from each node and there are 50 nodes, 
that's 500,000 timeseries.

Your prometheus server will need significantly more than 500MB of RAM to 
handle that.  See:
https://www.robustperception.io/how-much-ram-does-prometheus-2-x-need-for-cardinality-and-ingestion

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/111932ef-f883-4d7d-b360-8657d16c5f5c%40googlegroups.com.

Re: [prometheus-users] Re: Prometheus Performance issues

[Ben Kochie]

Your Prometheus version is almost 2 years old. You need to upgrade to
2.17.0.

On Wed, Mar 25, 2020 at 8:08 AM adi garg  wrote:

> Please help?? Do I need to specify any other detail?
>
> On Wednesday, March 25, 2020 at 4:12:28 AM UTC+5:30, adi garg wrote:
>>
>> Hello experts, I was stress-testing Prometheus on the AWS cluster.
>> Prometheus version = 2.3
>> Number of worker nodes in a cluster = 50
>> scrape_interval = 15s
>> I am getting around 10,000 metrics from a node and I am running
>> Prometheus in a docker container with memory specified to 500MB.
>> My avg metric size comes out to be around 2.3kb, which according to me is
>> unexpectedly higher. Moreover, the Docker container keeps getting burst out
>> every (10-15)s and then restarting again.
>> Can somebody tells me what could be the reason for this?
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/6073cdfc-5622-4a4f-a27f-7e9b0fe06339%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CABbyFmoc15QgkAmf9RkWLXimbdmhPhh9NJNU6PTC7591SCFYaQ%40mail.gmail.com.

[prometheus-users] Re: Prometheus Performance issues

[adi garg]

Please help?? Do I need to specify any other detail?

On Wednesday, March 25, 2020 at 4:12:28 AM UTC+5:30, adi garg wrote:
>
> Hello experts, I was stress-testing Prometheus on the AWS cluster. 
> Prometheus version = 2.3
> Number of worker nodes in a cluster = 50 
> scrape_interval = 15s
> I am getting around 10,000 metrics from a node and I am running Prometheus 
> in a docker container with memory specified to 500MB.
> My avg metric size comes out to be around 2.3kb, which according to me is 
> unexpectedly higher. Moreover, the Docker container keeps getting burst out 
> every (10-15)s and then restarting again.
> Can somebody tells me what could be the reason for this?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/6073cdfc-5622-4a4f-a27f-7e9b0fe06339%40googlegroups.com.