Great, thanks! In parallel, I opened a ticket with the vendor to report the findings and they referenced a future update which I've not yet moved to where this behavior may be corrected. I am going to run some tests and see where that gets me.
Thanks for the speedy replies everyone! I was racking my brain with this one, it's nice to have some closure! On Tue, Feb 2, 2021, 6:28 PM Julius Volz <[email protected]> wrote: > If it's really just emitting a single newline, then that is indeed an > incorrect HTTP response by your third-party vendor and not a problem in > Alertmanager. As far as Alertmanager is concerned, it's simply waiting for > the headers to continue. > > On Wed, Feb 3, 2021 at 12:02 AM Christopher Parsons < > [email protected]> wrote: > >> Thanks Julius! Is one newline character ever considered valid? I'm >> wondering if my next steps are to report this as a bug/feature request for >> the AlertManager project to support single newline responses or if I need >> to work with the third party vendor to update how their server is >> responding. I am pretty sure the service is just using standard Perl >> packages to handle the incoming HTTP messages so I'm not sure why they >> would ignore the 2nd newline character. >> >> On Tuesday, February 2, 2021 at 5:46:05 PM UTC-5 [email protected] >> wrote: >> >>> Yep, exactly, you need two newlines (2 times 0x0d0a) to generate a >>> completely empty line, which HTTP uses to signal that the header section is >>> over. >>> >>> On Tue, Feb 2, 2021 at 3:35 PM Christopher Parsons <[email protected]> >>> wrote: >>> >>>> It appears there's just a single CRLF (0d 0a) after the 200 OK line. >>>> Are you saying there needs to be another one in order for AlertManager to >>>> detect a valid 200 response? >>>> >>>> [image: mon101 response with hex.PNG] >>>> >>>> On Tuesday, February 2, 2021 at 4:25:13 AM UTC-5 [email protected] >>>> wrote: >>>> >>>>> Hi Christopher, >>>>> >>>>> It's not clear from your Wireshark stream, maybe show the data as hex >>>>> instead: does the problematic server insert an empty line after the >>>>> "HTTP/1.1 200 OK" line and then terminate the connection? So a CRLF after >>>>> the 200 line and another CRLF for the empty line. This is required for the >>>>> response to be read as a valid HTTP response (the body is optional). >>>>> >>>>> Julius >>>>> >>>>> On Tue, Feb 2, 2021 at 12:45 AM Christopher Parsons < >>>>> [email protected]> wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> I have a third party tool that is listening for incoming HTTP >>>>>> messages on a specific port. When we point AlertManager to this >>>>>> endpoint, >>>>>> notifications are delivered and consumed by the third party tool and >>>>>> inserted into our monitoring database successfully. However, >>>>>> AlertManager >>>>>> continuously retries the notification and slowly backs off on the >>>>>> retries. >>>>>> >>>>>> In order to try to figure out what's going on, we set up packet >>>>>> captures on AlertManager server and one on the endpoint server. As a >>>>>> control group, we also configured a simple node.js server which responds >>>>>> back with a 200. The only difference, as far as I can tell, from the TCP >>>>>> streams in Wireshark is the response headers. The server forcing >>>>>> AlertManager to continuously retry replies with a simple "*HTTP/1.1 >>>>>> 200 OK*". However, the node.js control group which does not force >>>>>> AlertManager to send subsequent replies is responding with something >>>>>> like... >>>>>> >>>>>> *HTTP/1.1 200 OK* >>>>>> >>>>>> *Date: Mon, 01 Feb 2021 22:56:09 GMT* >>>>>> >>>>>> *Connection: keep-alive* >>>>>> >>>>>> *Transfer-Encoding: chunked* >>>>>> >>>>>> Does the go-lang web server expect more than a simple "HTTP/1.1 200 >>>>>> OK" as a response? The third party tool is not open source but I may be >>>>>> able to get them to issue a future update if this is considered >>>>>> unexpected >>>>>> behavior. Any suggestions or workarounds would be greatly appreciated. >>>>>> I've also attached a couple of screenshots from the Wireshark TCP stream >>>>>> traces for reference. >>>>>> >>>>>> Normal behavior: >>>>>> >>>>>> [image: logserver normal behavior.PNG] >>>>>> >>>>>> Repeat notification behavior: >>>>>> [image: mon101 repeat behavior.PNG] >>>>>> >>>>>> Please let me know if there are any other details I can provide! >>>>>> >>>>>> Thanks, >>>>>> Chris >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Prometheus Users" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/prometheus-users/014b62ad-72bb-41af-b794-f3bbe61bf705n%40googlegroups.com >>>>>> <https://groups.google.com/d/msgid/prometheus-users/014b62ad-72bb-41af-b794-f3bbe61bf705n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> >>>>> >>>>> -- >>>>> Julius Volz >>>>> PromLabs - promlabs.com >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Prometheus Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> >>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/prometheus-users/83566c90-0119-4e7d-97fb-23d30c21299cn%40googlegroups.com >>>> <https://groups.google.com/d/msgid/prometheus-users/83566c90-0119-4e7d-97fb-23d30c21299cn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> >>> >>> -- >>> Julius Volz >>> PromLabs - promlabs.com >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/f3159722-8d91-4196-8888-9ab180da31aan%40googlegroups.com >> <https://groups.google.com/d/msgid/prometheus-users/f3159722-8d91-4196-8888-9ab180da31aan%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > > > -- > Julius Volz > PromLabs - promlabs.com > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABTsVToBZOEEaGsVA3JhkYxsWpYKccDM2nyw60yWmpfen5yQ5w%40mail.gmail.com.
