Hi All, Can any one respond to my queries? Also we observed the following: 1. If for eg timestamp(epoch) in the scrape is 12:00:00, then prometheus is not scrapping the targets 2. If for eg timestamp(epoch) in the scrape is 12:00:01, then prometheus is scrapping the targets.
Thanks & regards, Chalapathi On Thu, Jul 2, 2020 at 3:03 PM Venkata Bhagavatula <[email protected]> wrote: > Hi, > > We are using prometheus version 2.11.1, In our application, the scrape > target has timestamp field. when timestamp field is present, then > prometheus is not scrapping any metrics. > Following is the output of the curl request for scrape target: > > - *cmd: curl http://<readacted>:24231/metrics* > > meas_gauge{id="Filtered",HOST="test",STREAM="Smoke_stream",NODE="MFE2"} > 0.0 1593680400000 > meas_gauge{id="Rejected",HOST="test",STREAM="Smoke_stream",NODE="MFE2"} > 0.0 1593680400000 > meas_gauge{id="ReprocessedIn",HOST="test",STREAM="Smoke_stream",NODE="MFE2"} > 0.0 1593680400000 > meas_gauge{id="Created",HOST="test",STREAM="Smoke_stream",NODE="MFE2"} 0.0 > 1593680400000 > meas_gauge{id="Duplicated",HOST="test",STREAM="Smoke_stream",NODE="MFE2"} > 0.0 1593680400000 > meas_gauge{id="Stored",HOST="test",STREAM="Smoke_stream",NODE="MFE2"} > 336.0 1593680400000 > meas_gauge{id="Retrieved",HOST="test",STREAM="Smoke_stream",NODE="MFE2"} > 354.0 1593680400000 > meas_gauge{id="ReducedInMerging",HOST="test",STREAM="Smoke_stream",NODE="MFE2"} > 0.0 1593680400000 > > > > - I checked that time is in sync between the prometheus node and the > target node. > - Following is the epoch time on the prometheus node: > > *cmd: date +'%s%3N'* > *1593681793979* > > > - Epoch difference between the prometheus node and the time stamp > present in the sample is more than an hour. > > difference = ( 1593681793979 - 1593680400000) / 1000 = 1393sec = 23min > > Scrape_interval is configured as 300s > honor_timestamps is set to true. > > Can you let us know why prometheus is not able to scrape the targets? Is > it due to the timestamp difference between prometheus and target? > How much difference will prometheus tolerate? > > Thanks n Regards, > Chalapathi > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABXnQPuB5iWDhDw06OLOepmz5_XgC2a%3DC9uuVaDKcczR9B-%2BAA%40mail.gmail.com.
