Okay cool I'll start looking into CrateDB, although it's showing the CPU usage at roughly 10% the load averages for 1m,5m,15m are 8.5,8,9.
On Friday, 17 April 2020 13:23:59 UTC+1, Julius Volz wrote: > > "context deadline exceeded" means there's a timeout between the CrateDB > adapter and CrateDB itself. The "POST" is actually misleading, as the > adapter has been changed from using HTTP in its original version to using > the Postgres protocol via https://github.com/jackc/pgx. > > A timeout could either mean there's a network / reachability problem, or > that the CrateDB is too overloaded to process write requests in time (I > couldn't quickly figure out what the timeout value would be here, but I > guess it wouldn't be tiny). Have you tried sending tiny data (like just a > handful of series) first, to see if it works at all? > > On Fri, Apr 17, 2020 at 10:47 AM ryan woods <[email protected] > <javascript:>> wrote: > >> Hey guys, >> >> So I'm currently running a 3 node CrateDB cluster for Prometheus long >> term storage and using the CrateDB adapter provided by crateio. I've ran >> into an issue where Prometheus is attempting to POST data to the >> remote_write (crate adapter) but it's unable to. Logs are below from >> CrateDB adapter >> >> time="2020-04-17T08:46:31Z" level=error msg="Failed to POST inserts to >> Crate." err="context deadline exceeded" source="server.go:332" >> time="2020-04-17T08:46:31Z" level=error msg="Failed to POST inserts to >> Crate." err="context deadline exceeded" source="server.go:332" >> time="2020-04-17T08:46:31Z" level=error msg="Failed to POST inserts to >> Crate." err="context deadline exceeded" source="server.go:332" >> time="2020-04-17T08:46:31Z" level=error msg="Failed to POST inserts to >> Crate." err="context deadline exceeded" source="server.go:332" >> time="2020-04-17T08:46:31Z" level=error msg="Failed to POST inserts to >> Crate." err="context deadline exceeded" source="server.go:332" >> time="2020-04-17T08:46:31Z" level=error msg="Failed to POST inserts to >> Crate." err="context deadline exceeded" source="server.go:332" >> >> Although these were collected from the cratedb adapter I'm thinkig this >> is a prometheus issue/misconfiguration by myself. Any help would be great >> :) i've tried increasing the scrape_interval and it doesn't seem to help. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/b7ba1543-076d-4514-8d19-6dec69eee20c%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/b7ba1543-076d-4514-8d19-6dec69eee20c%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d6e1841c-c377-47e4-adb3-d7fb1a4577d3%40googlegroups.com.
