Resolved.
Precision of timestamp must be millisecond but I've had put second.
On Tuesday, May 26, 2020 at 1:30:39 AM UTC+9, shimpeko wrote:
>
> Hello,
>
> Metrics are not saved when I put a timestamps with metrics.
> I've got through github issue and some of prometheus codes but not yet
> getting the root cause.
>
> I would be happy if I can get why I'm getting below errors.
>
> Errors
> *Error 1*
> I got below error when executing prometheus with prometheus.yml (v1).
> Please see blow for the prometheus.yml (v1).
>
> prometheus_1 | level=warn ts=2020-05-25T12:29:17.216Z caller=scrape.go:
> 1240 component="scrape manager" scrape_pool=my-app target=https://<internal
> server address>:443/metrics msg="Error on ingesting samples that are too
> old or are too far into the future" num_dropped=600
>
> *Error2*
> I got massive amount of below log when running prometheus with
> prometheus.yml (v2). Please see below for prometheus.yml (v2)
>
> prometheus_1 | level=info ts=2020-05-25T15:57:25.857Z caller=head.go:732
> component=tsdb msg="WAL checkpoint complete" first=1164 last=1165
> duration=2.1818ms
> prometheus_1 | level=info ts=2020-05-25T15:57:25.867Z caller=head.go:662
> component=tsdb msg="head GC completed" duration=509.9µs
> prometheus_1 | level=info ts=2020-05-25T15:57:25.879Z caller=head.go:662
> component=tsdb msg="head GC completed" duration=549.7µs
> prometheus_1 | level=info ts=2020-05-25T15:57:25.882Z caller=head.go:732
> component=tsdb msg="WAL checkpoint complete" first=1166 last=1167
> duration=2.7017ms
> prometheus_1 | level=info ts=2020-05-25T15:57:25.892Z caller=head.go:662
> component=tsdb msg="head GC completed" duration=564.7µs
> prometheus_1 | level=info ts=2020-05-25T15:57:25.903Z caller=head.go:662
> component=tsdb msg="head GC completed" duration=651.9µs
> prometheus_1 | level=info ts=2020-05-25T15:57:25.906Z caller=head.go:732
> component=tsdb msg="WAL checkpoint complete" first=1168 last=1169
> duration=2.8861ms
> prometheus_1 | level=info ts=2020-05-25T15:57:25.916Z caller=head.go:662
> component=tsdb msg="head GC completed" duration=562.8µs
> prometheus_1 | level=info ts=2020-05-25T15:57:25.925Z caller=head.go:662
> component=tsdb msg="head GC completed" duration=633µs
>
>
> Background
>
> I'm testing prometheus for custom app.
> I've created endpoint which expose metrics in text-based format for the
> app.
>
> Since metrics are calculated with batch job, I've put a timestamps to
> metrics to show when the metrics was calculated.
> The batch job is executed in every 5 min, so the timestamp will not be
> older than 5 min compared to the current timestamp.
>
> Here is an example of exposed data. There are about 600 lines in real data.
>
> #HELP processed_rows_total the total number of processed rows
> #TYPE processed_rows_total counter
> processed_rows_total{stream_name="a", result="success"} 93163 1590422899
> processed_rows_total{stream_name="a", result="error"} 0 1590422899
> processed_rows_total{stream_name="b", result="success"} 45695851 1590422899
>
> processed_rows_total{stream_name="b", result="error"} 0 1590422899
>
>
> Environment
>
> Running with docker.
>
> docker-compose.yml
>
> prometheus:
> image: prom/prometheus:v2.17.2
> entrypoint:
> - "/bin/prometheus"
> - "--log.level=debug"
> - "--config.file=/etc/prometheus/prometheus.yml"
> - "--storage.tsdb.path=/prometheus"
> - "--web.console.libraries=/usr/share/prometheus/console_libraries"
> - "--web.console.templates=/usr/share/prometheus/consoles"
> volumes:
> - ./prometheus.yml:/etc/prometheus/prometheus.yml
> ports:
> - "127.0.0.1:9090:9090"
>
> prometheus.yml (v1)
>
> # my global config
> global:
> scrape_interval: 15s # Set the scrape interval to every 15 seconds.
> Default is every 1 minute.
> evaluation_interval: 15s # Evaluate rules every 15 seconds. The default
> is every 1 minute.
>
> # A scrape configuration containing exactly one endpoint to scrape:
> # Here it's Prometheus itself.
> scrape_configs:
> # The job name is added as a label `job=<job_name>` to any timeseries
> scraped from this config.
> - job_name: 'prometheus'
> static_configs:
> - targets: ['localhost:9090']
>
> - job_name: 'my-app'
> metrics_path: '/metrics'
> scrape_interval: 1m
> scheme: 'https'
> static_configs:
> - targets: ['<internal server address>']
>
> prometheus.yml (v2)
>
> # my global config
> global:
> scrape_interval: 15s # Set the scrape interval to every 15 seconds.
> Default is every 1 minute.
> evaluation_interval: 15s # Evaluate rules every 15 seconds. The default
> is every 1 minute.
>
> # A scrape configuration containing exactly one endpoint to scrape:
> # Here it's Prometheus itself.
> scrape_configs:
> - job_name: 'my-app'
> metrics_path: '/metrics'
> scrape_interval: 1m
> scheme: 'https'
> static_configs:
> - targets: ['<internal server address>']
>
> Other notes
>
> With *honor-timestamps: false* configuration in prometheus.yml, metrics
> are saved correctly.
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/094ff2d0-5a38-4c17-a310-3e7096e58ea9%40googlegroups.com.
