Prometheus doesn't support storing historical data and samples with out-of-order timestamps. If you need writing such data, then take a look at other Prometheus-inspired solutions. See, for example, https://victoriametrics.github.io/#backfilling .
On Mon, Jul 6, 2020 at 10:31 AM Venkata Bhagavatula <[email protected]> wrote: > Hi All, > Can any one respond to my queries? Also we observed the following: > 1. If for eg timestamp(epoch) in the scrape is 12:00:00, then prometheus > is not scrapping the targets > 2. If for eg timestamp(epoch) in the scrape is 12:00:01, then prometheus > is scrapping the targets. > > Thanks & regards, > Chalapathi > > On Thu, Jul 2, 2020 at 3:03 PM Venkata Bhagavatula < > [email protected]> wrote: > >> Hi, >> >> We are using prometheus version 2.11.1, In our application, the scrape >> target has timestamp field. when timestamp field is present, then >> prometheus is not scrapping any metrics. >> Following is the output of the curl request for scrape target: >> >> - *cmd: curl http://<readacted>:24231/metrics* >> >> meas_gauge{id="Filtered",HOST="test",STREAM="Smoke_stream",NODE="MFE2"} >> 0.0 1593680400000 >> meas_gauge{id="Rejected",HOST="test",STREAM="Smoke_stream",NODE="MFE2"} >> 0.0 1593680400000 >> meas_gauge{id="ReprocessedIn",HOST="test",STREAM="Smoke_stream",NODE="MFE2"} >> 0.0 1593680400000 >> meas_gauge{id="Created",HOST="test",STREAM="Smoke_stream",NODE="MFE2"} >> 0.0 1593680400000 >> meas_gauge{id="Duplicated",HOST="test",STREAM="Smoke_stream",NODE="MFE2"} >> 0.0 1593680400000 >> meas_gauge{id="Stored",HOST="test",STREAM="Smoke_stream",NODE="MFE2"} >> 336.0 1593680400000 >> meas_gauge{id="Retrieved",HOST="test",STREAM="Smoke_stream",NODE="MFE2"} >> 354.0 1593680400000 >> meas_gauge{id="ReducedInMerging",HOST="test",STREAM="Smoke_stream",NODE="MFE2"} >> 0.0 1593680400000 >> >> >> >> - I checked that time is in sync between the prometheus node and the >> target node. >> - Following is the epoch time on the prometheus node: >> >> *cmd: date +'%s%3N'* >> *1593681793979* >> >> >> - Epoch difference between the prometheus node and the time stamp >> present in the sample is more than an hour. >> >> difference = ( 1593681793979 - 1593680400000) / 1000 = 1393sec = 23min >> >> Scrape_interval is configured as 300s >> honor_timestamps is set to true. >> >> Can you let us know why prometheus is not able to scrape the targets? Is >> it due to the timestamp difference between prometheus and target? >> How much difference will prometheus tolerate? >> >> Thanks n Regards, >> Chalapathi >> > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/CABXnQPuB5iWDhDw06OLOepmz5_XgC2a%3DC9uuVaDKcczR9B-%2BAA%40mail.gmail.com > <https://groups.google.com/d/msgid/prometheus-users/CABXnQPuB5iWDhDw06OLOepmz5_XgC2a%3DC9uuVaDKcczR9B-%2BAA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- Best Regards, Aliaksandr Valialkin, CTO VictoriaMetrics -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAPbKnmCdo2txis5J8vVRGbMGB4X1N%2BjQiUr9aujrf1sqdXmjPw%40mail.gmail.com.
