Hi There, I am experiencing the following situation with a number of my prometheus targets.
1. A number of targets are "DOWN" with the error message "Context Deadline Exceeded" 2. My scrape timeout is set to 10s 3. My scrape duration starts out lower than 10s on re-deploy but over the course of a couple days steadily approaches the timeout 4. The scrape samples scraped is under 25k per scrape, but more importantly doesn't seem correlated with the approach to timeout. At times where the scrape_samples_scraped == 25k but memory is low, the scrape_duration remains low 5. According to my dashboard the item that seems to have a strong correlation with timeout's is memory usage. I.e after a redeploy memory usage steadily increases and then fluctuates in a range close to max usage My question is around the effect of low memory availability on a prometheus scrape. If a request is sent to the /prometheus_metrics endpoint, and there was a limited amount of memory would the generate_latest method ( python client ) hang and cause a timeout? Please let me know if there is anything else I can provide and cheers in advance ------------------------------This email is protected by Tessian <https://www.tessian.com/securethehumanlayer> -- Tessian Limited is a limited company registered in England and Wales with registered number 08358482. Our registered office is at 3 Finsbury Avenue, London, EC2M 2PA. This message is intended solely for the addressee and may contain confidential information. If you have received this message in error, please send it back to us, and immediately and permanently delete it. Do not use, copy or disclose the information contained in this message or in any attachment. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/12c71fb9-409d-43f7-8487-096ba0ab3e07%40googlegroups.com.
