Perform the two halves of the query separately, i.e.
max_over_time(node_filesystem_avail_bytes{...}[1h]
max_over_time(node_filesystem_size_bytes{...}[1h]and then you'll see why they divide to give 48% instead of 97% I expect node_filesystem_size_bytes doesn't change much, so max_over_time doesn't do much for that. But max_over_time(node_filesystem_avail_bytes) will show the *largest* available space over that 1 hour window, and therefore you'll get the value for when the disk was *least full*. If you want to know the value when it was *most full* then it would be min_over_time(node_filesystem_avail_bytes). Note that you showed a graph, rather than a table. When you're graphing, you're repeating the same query at different evaluation times. So where the time axis show 04:00, the data point on the graph is for the 1 hour period from 03:00 to 04:00. Where the time axis is 04:45, the result is of your query covering the 1 hour from 03:45 to 04:45. Aside: in general, I'd advise keeping percentage queries simple by removing the factor of 100, so you get a fraction between 0 and 1 instead. This can be represented as a human-friendly percentage when rendered (e.g. Grafana can quite happily render 0-1 as 0-100%) On Friday 27 September 2024 at 06:01:16 UTC+1 mohan garden wrote: > Sorry for the double posting, image was corrupted, so reposting > > Thank you for the response Brian, > > I removed the $__ variables and tried viewing disk usage metrics from past > 1 hour in PromUI - > I tried the query in the Prometheus UI , and i was expecting value ~97% > with following query for past 1 hour metrics but the table view reports > 48%. > > [image: max_over_time.png] > I am not sure if i missed out on some thing within the query. > > i am under impression that max function works with multiple series, and > over time will generate stats from the values within the series. > Please advice. > > > > > > > On Friday, September 27, 2024 at 10:29:22 AM UTC+5:30 mohan garden wrote: > >> Thank you for the response Brian, >> >> I removed the $__ variables and tried viewing disk usage metrics from >> past 1 hour in PromUI - >> >> >> I tried the query in the Prometheus UI , and i was expecting value ~97% >> for past 1 hour metrics but the table view reports 48%. >> >> [image: max_over_time.png] >> I am not sure if i missed out on something within the query. >> >> i am under impression that max function works with multiple series, and >> over time will generate stats from the values within the series. >> Please advice. >> >> >> >> On Tuesday, September 24, 2024 at 8:12:29 PM UTC+5:30 Brian Candler wrote: >> >>> $__rate_interval is (roughly speaking) the interval between 2 adjacent >>> points in the graph, with a minimum of 4 times the configured scrape >>> interval. It's not the entire period over which Grafana is drawing the >>> graph. You probably want $__range or $__range_s. See: >>> >>> https://grafana.com/docs/grafana/latest/datasources/prometheus/template-variables/#use-__rate_interval >>> >>> https://grafana.com/docs/grafana/latest/dashboards/variables/add-template-variables/#global-variables >>> >>> However, questions about Grafana would be better off asked in the >>> Grafana community. Prometheus is not Grafana, and those variables are >>> Grafana-specific. >>> >>> > so you can see that avg|min|max_over_time functions return identical >>> values which dont make much sense >>> >>> It makes sense when you realise that the time period you're querying >>> over is very small; hence for a value that doesn't change rapidly, the >>> min/max/average over such a short time range will all be roughly the same. >>> >>> On Tuesday 24 September 2024 at 15:10:33 UTC+1 mohan garden wrote: >>> >>>> Hi , >>>> seems images in my previous post did not show up as expected. >>>> Sorry for the spam , reposting again - >>>> >>>> >>>> Hi , >>>> >>>> I am trying to analyse memory usage of a server for 2 specific months >>>> using Grafana and prometheus. but seems _over_time functions are returning >>>> unexpected results. >>>> >>>> Here is the data for the duration >>>> >>>> [image: one.png] >>>> >>>> Now, the summary table shows expected values >>>> [image: one.png] >>>> >>>> >>>> query - >>>> (( >>>> node_memory_MemAvailable_bytes{instance="$node",job="$job"}[$__rate_interval]) >>>> >>>> * 100 ) / >>>> node_memory_MemTotal_bytes{instance="$node",job="$job"}[$__rate_interval] >>>> >>>> >>>> Issue - when i am trying to create similar stats using PromQL at my end >>>> , i am facing issues . i fail to get the same stats when i use the >>>> following promql , example - >>>> >>>> [image: two.png] >>>> >>>> ( >>>> avg_over_time(node_memory_MemAvailable_bytes{instance="$node",job="$job"}[$__rate_interval]) >>>> >>>> / >>>> avg_over_time(node_memory_MemTotal_bytes{instance="$node",job="$job"}[$__rate_interval]) >>>> >>>> ) * 100 >>>> >>>> ( >>>> min_over_time(node_memory_MemAvailable_bytes{instance="$node",job="$job"}[$__rate_interval]) >>>> >>>> / >>>> min_over_time(node_memory_MemTotal_bytes{instance="$node",job="$job"}[$__rate_interval]) >>>> >>>> ) * 100 >>>> >>>> ( >>>> max_over_time(node_memory_MemAvailable_bytes{instance="$node",job="$job"}[$__rate_interval]) >>>> >>>> / >>>> max_over_time(node_memory_MemTotal_bytes{instance="$node",job="$job"}[$__rate_interval]) >>>> >>>> ) * 100 >>>> >>>> so you can see that avg|min|max_over_time functions return identical >>>> values which dont make much sense. I was using following setting >>>> >>>> [image: one.png] >>>> >>>> I tried changing from range -> instant, i see similar values >>>> [image: two.png] >>>> >>>> Where do i need to make modifications in PromQL so i can get the >>>> correct min/max/avg values in the gauges as correctly reported by the >>>> [image: one.png] >>>> >>>> >>>> for a specific duration , say - >>>> >>>> [image: one.png] >>>> >>>> please advice >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> On Tuesday, September 24, 2024 at 7:25:00 PM UTC+5:30 mohan garden >>>> wrote: >>>> >>>>> I am trying to analyse memory usage of a server for 2 specific months >>>>> using Grafana and prometheus. but seems _over_time functions are >>>>> returning >>>>> unexpected results. >>>>> >>>>> Here is the data for the duration >>>>> [image: image] >>>>> <https://private-user-images.githubusercontent.com/154580421/370293312-9755f80b-01f4-4785-8d94-5d4e55b3cb0b.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjcxODYwMjMsIm5iZiI6MTcyNzE4NTcyMywicGF0aCI6Ii8xNTQ1ODA0MjEvMzcwMjkzMzEyLTk3NTVmODBiLTAxZjQtNDc4NS04ZDk0LTVkNGU1NWIzY2IwYi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwOTI0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDkyNFQxMzQ4NDNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1iNmUzNjQ2Yzk5MjllZWQwYjhjOWZjZjk4MTNmYTFjN2ZkNDE3YzA2NDRiOWZiMzdkY2FkYWYyZGJkNDU4M2ZiJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.bVjNK3XE9QLGybgL6GUbJ-VHvUMk-0m4yJIsHEAeGo4> >>>>> the summary table shows expected values >>>>> [image: image] >>>>> <https://private-user-images.githubusercontent.com/154580421/370293517-6a3d94db-f0eb-4915-a0d5-b9d235e5a944.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjcxODYwMjMsIm5iZiI6MTcyNzE4NTcyMywicGF0aCI6Ii8xNTQ1ODA0MjEvMzcwMjkzNTE3LTZhM2Q5NGRiLWYwZWItNDkxNS1hMGQ1LWI5ZDIzNWU1YTk0NC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwOTI0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDkyNFQxMzQ4NDNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1kMzVhNTE5ZTQ1Nzg2MDhmMDcyYWI1YjI1MGQ0YTkwMDZhNDJlYjYyODQ2Mjg1MWRiNWJkNGVmOTMyNjE0MjA1JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.POaO2I4CLHlxSH9gzIbF8W2Hl3mAcorBj_FjQlR4SOQ> >>>>> query - >>>>> (( >>>>> node_memory_MemAvailable_bytes{instance="$node",job="$job"}[$__rate_interval]) >>>>> >>>>> * 100 ) / >>>>> node_memory_MemTotal_bytes{instance="$node",job="$job"}[$__rate_interval] >>>>> >>>>> >>>>> Issue - when i am trying to create similar stats using PromQL at my >>>>> end , i am facing issues . i fail to get the same values when i use >>>>> promql >>>>> , example - >>>>> >>>>> [image: image] >>>>> <https://private-user-images.githubusercontent.com/154580421/370295234-12eeea9c-a974-4a08-b21a-bb9b6bfbfc87.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjcxODYwMjMsIm5iZiI6MTcyNzE4NTcyMywicGF0aCI6Ii8xNTQ1ODA0MjEvMzcwMjk1MjM0LTEyZWVlYTljLWE5NzQtNGEwOC1iMjFhLWJiOWI2YmZiZmM4Ny5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwOTI0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDkyNFQxMzQ4NDNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT01NzY2MjE1MzNmMmM0ZjRkZWYyYWU3Y2ViOTFjNTk3NTJlYjNlN2Q3OTMzMTY0MzhiOTllODY2ZDIxNzExYjVkJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.WPjDSluZWsNcTbCVDVnoTB3wa3CSdFgBXRq7bxu5HS0> >>>>> >>>>> ( >>>>> avg_over_time(node_memory_MemAvailable_bytes{instance="$node",job="$job"}[$__rate_interval]) >>>>> >>>>> / >>>>> avg_over_time(node_memory_MemTotal_bytes{instance="$node",job="$job"}[$__rate_interval]) >>>>> >>>>> ) * 100 >>>>> >>>>> ( >>>>> min_over_time(node_memory_MemAvailable_bytes{instance="$node",job="$job"}[$__rate_interval]) >>>>> >>>>> / >>>>> min_over_time(node_memory_MemTotal_bytes{instance="$node",job="$job"}[$__rate_interval]) >>>>> >>>>> ) * 100 >>>>> >>>>> ( >>>>> max_over_time(node_memory_MemAvailable_bytes{instance="$node",job="$job"}[$__rate_interval]) >>>>> >>>>> / >>>>> max_over_time(node_memory_MemTotal_bytes{instance="$node",job="$job"}[$__rate_interval]) >>>>> >>>>> ) * 100 >>>>> >>>>> so you can see that avg|min|max_over_time functions return identical >>>>> values with following setting >>>>> [image: image] >>>>> <https://private-user-images.githubusercontent.com/154580421/370295926-18b0a6a3-fea5-48b9-8f09-715df5725f34.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjcxODYwMjMsIm5iZiI6MTcyNzE4NTcyMywicGF0aCI6Ii8xNTQ1ODA0MjEvMzcwMjk1OTI2LTE4YjBhNmEzLWZlYTUtNDhiOS04ZjA5LTcxNWRmNTcyNWYzNC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwOTI0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDkyNFQxMzQ4NDNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT03MjI5MjI5NDNhODA2ZjhkNjIwNTA3ZjAwM2VlNTlmM2I4NWM5ZjdiNGM5NjExYmE1ZTg2Njg2NzllOTM5YWFjJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.Nbpvcc5P80H3yC3uOHnYHGb7FwpbmMLV4xujhJ6YqR0> >>>>> >>>>> when i change from range -> instant, i see similar values >>>>> [image: image] >>>>> <https://private-user-images.githubusercontent.com/154580421/370296399-fbba4ab2-7efb-40d0-a067-3705ddbc111e.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjcxODYwMjMsIm5iZiI6MTcyNzE4NTcyMywicGF0aCI6Ii8xNTQ1ODA0MjEvMzcwMjk2Mzk5LWZiYmE0YWIyLTdlZmItNDBkMC1hMDY3LTM3MDVkZGJjMTExZS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwOTI0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDkyNFQxMzQ4NDNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0wYzk1OTczOGVmMGVlNWFjYjY2NDkwMzQ0MzVjNzc5NzI3YzcwNzk3MmRkYTJlMzhiMzJhNDNmNzI0NjlkYTI1JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.-q5VtuzgNYlX5_r9nui9UJWLyFUTMU3H7Q8I7l0giFk> >>>>> >>>>> Where do i need to make modifications in PromQL so i can get the >>>>> correct min/max/avg values in the gauges as reported by the >>>>> [image: image] >>>>> <https://private-user-images.githubusercontent.com/154580421/370293517-6a3d94db-f0eb-4915-a0d5-b9d235e5a944.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjcxODYwMjMsIm5iZiI6MTcyNzE4NTcyMywicGF0aCI6Ii8xNTQ1ODA0MjEvMzcwMjkzNTE3LTZhM2Q5NGRiLWYwZWItNDkxNS1hMGQ1LWI5ZDIzNWU1YTk0NC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwOTI0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDkyNFQxMzQ4NDNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1kMzVhNTE5ZTQ1Nzg2MDhmMDcyYWI1YjI1MGQ0YTkwMDZhNDJlYjYyODQ2Mjg1MWRiNWJkNGVmOTMyNjE0MjA1JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.POaO2I4CLHlxSH9gzIbF8W2Hl3mAcorBj_FjQlR4SOQ> >>>>> for a specific duration , say - >>>>> [image: image] >>>>> <https://private-user-images.githubusercontent.com/5935825/370298748-b4ecd557-f74d-44da-a33c-3df91a8c199d.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjcxODYwMjMsIm5iZiI6MTcyNzE4NTcyMywicGF0aCI6Ii81OTM1ODI1LzM3MDI5ODc0OC1iNGVjZDU1Ny1mNzRkLTQ0ZGEtYTMzYy0zZGY5MWE4YzE5OWQucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDkyNCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA5MjRUMTM0ODQzWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9YjVhZWJhNGM1MjE1MTUwODc5OTgxMmVhOGU2MDJhZTI5Y2EwNWQ3YmRhODRmN2I3MzRjN2E4ZTQzZTQ1MTgyOSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QifQ.Qw9GrH1ef_Gj1jAUNtYT2Bz05xZ1YUtVELfYJJP2Cds> >>>>> >>>>> please advice >>>>> >>>>> >>>>> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/105535b0-cde9-46cd-81b3-28ba246c99c9n%40googlegroups.com.
