Further analysis...
The records recieved from prometheus by the adapter proves that the records
are older than 24 hours
*Does prometheus return oldest metrics as well as part of the remote writes
?*
For ex: One of the records recieved from prometheus
"MeasureName":"fluentd_output_status_num_errors","MeasureValue":"162","MeasureValueType":"DOUBLE","Time":"
*1610130823809*","TimeUnit":"*MILLISECONDS*"}]}
The time converts to - *Fri Jan 08 2021 18:33:43* Where as we have the
retention on AWS timestream set to *24 hrs *
Any suggestions will be helpful.
Regards,
Vinod.
On Monday, January 11, 2021 at 1:32:59 PM UTC Vinod Sai Kumar wrote:
>
> Hi There ,
>
> We have integrated AWS timestream to prometheus via the remote write &
> read settings. We have the below setup,
> 1. Prometheus is a pod in EKS cluster
> 2. We have prometheus-timestream-adapter as well deployed in a pod
> 3. Prometheus remote read & write is configured to this adapter.
> 4. AWS timestream has a table setup with 24hrs memory & 7 days magnetic
> retention
>
> All connectivity, setup etc, looks good apart from the prometheus seems to
> be sending old record sets. Hence we get this below message in the adapter
> and it fails to store the data in timestream
>
> {"level":"warn","ts":1610368736.6555033,"caller":"prometheus-timestream-adapter/timestream.go:131","msg":"Error
>
> sending samples to remote storage","err":"RejectedRecordsException: One or
> more records have been rejected. See RejectedRecords for details.\n{\n
> RespMetadata: {\n StatusCode: 419,\n RequestID:
> \"AHHJ7P7BBH4TIUEGMKOPO42FQQ\"\n },\n Message_: \"One or more records
> have been rejected. See RejectedRecords for details.\",\n RejectedRecords:
> [\n {\n Reason: \"*The record timestamp is outside the time range
> *[2021-01-10T12:33:56.651Z, 2021-01-11T13:13:56.651Z) of the memory
> store.\",\n RecordIndex: 0\n }
>
> Looking forward for some suggestions ...
>
> Regards,
> Vinod.
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/8042a417-35d5-4247-83e3-b107c74941fbn%40googlegroups.com.
