Further .....
We created a temp table in AWS timestream and increased the retention to 7
days, this allowed the data to be written in the remote storage. What we
notice now is the records are always for the same time period that is in
our case its between
[2021-01-08 18:33:44.280000000 & 2021-01-08 18:35:14.390000000]Are we
missing any configuration somemwhere ?
Regards,
Vinod
On Monday, January 11, 2021 at 6:09:44 PM UTC Vinod Sai Kumar wrote:
> Further analysis...
>
> The records recieved from prometheus by the adapter proves that the
> records are older than 24 hours
>
> *Does prometheus return oldest metrics as well as part of the remote
> writes ?*
>
> For ex: One of the records recieved from prometheus
>
>
> "MeasureName":"fluentd_output_status_num_errors","MeasureValue":"162","MeasureValueType":"DOUBLE","Time":"
> *1610130823809*","TimeUnit":"*MILLISECONDS*"}]}
>
> The time converts to - *Fri Jan 08 2021 18:33:43* Where as we have the
> retention on AWS timestream set to *24 hrs *
>
> Any suggestions will be helpful.
>
> Regards,
> Vinod.
>
> On Monday, January 11, 2021 at 1:32:59 PM UTC Vinod Sai Kumar wrote:
>
>>
>> Hi There ,
>>
>> We have integrated AWS timestream to prometheus via the remote write &
>> read settings. We have the below setup,
>> 1. Prometheus is a pod in EKS cluster
>> 2. We have prometheus-timestream-adapter as well deployed in a pod
>> 3. Prometheus remote read & write is configured to this adapter.
>> 4. AWS timestream has a table setup with 24hrs memory & 7 days magnetic
>> retention
>>
>> All connectivity, setup etc, looks good apart from the prometheus seems
>> to be sending old record sets. Hence we get this below message in the
>> adapter and it fails to store the data in timestream
>>
>> {"level":"warn","ts":1610368736.6555033,"caller":"prometheus-timestream-adapter/timestream.go:131","msg":"Error
>>
>> sending samples to remote storage","err":"RejectedRecordsException: One or
>> more records have been rejected. See RejectedRecords for details.\n{\n
>> RespMetadata: {\n StatusCode: 419,\n RequestID:
>> \"AHHJ7P7BBH4TIUEGMKOPO42FQQ\"\n },\n Message_: \"One or more records
>> have been rejected. See RejectedRecords for details.\",\n RejectedRecords:
>> [\n {\n Reason: \"*The record timestamp is outside the time
>> range *[2021-01-10T12:33:56.651Z, 2021-01-11T13:13:56.651Z) of the
>> memory store.\",\n RecordIndex: 0\n }
>>
>> Looking forward for some suggestions ...
>>
>> Regards,
>> Vinod.
>>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/9d0096a1-d59a-4556-9a47-27608c52a605n%40googlegroups.com.
