If it's really just emitting a single newline, then that is indeed an incorrect HTTP response by your third-party vendor and not a problem in Alertmanager. As far as Alertmanager is concerned, it's simply waiting for the headers to continue.
On Wed, Feb 3, 2021 at 12:02 AM Christopher Parsons <[email protected]> wrote: > Thanks Julius! Is one newline character ever considered valid? I'm > wondering if my next steps are to report this as a bug/feature request for > the AlertManager project to support single newline responses or if I need > to work with the third party vendor to update how their server is > responding. I am pretty sure the service is just using standard Perl > packages to handle the incoming HTTP messages so I'm not sure why they > would ignore the 2nd newline character. > > On Tuesday, February 2, 2021 at 5:46:05 PM UTC-5 [email protected] > wrote: > >> Yep, exactly, you need two newlines (2 times 0x0d0a) to generate a >> completely empty line, which HTTP uses to signal that the header section is >> over. >> >> On Tue, Feb 2, 2021 at 3:35 PM Christopher Parsons <[email protected]> >> wrote: >> >>> It appears there's just a single CRLF (0d 0a) after the 200 OK line. >>> Are you saying there needs to be another one in order for AlertManager to >>> detect a valid 200 response? >>> >>> [image: mon101 response with hex.PNG] >>> >>> On Tuesday, February 2, 2021 at 4:25:13 AM UTC-5 [email protected] >>> wrote: >>> >>>> Hi Christopher, >>>> >>>> It's not clear from your Wireshark stream, maybe show the data as hex >>>> instead: does the problematic server insert an empty line after the >>>> "HTTP/1.1 200 OK" line and then terminate the connection? So a CRLF after >>>> the 200 line and another CRLF for the empty line. This is required for the >>>> response to be read as a valid HTTP response (the body is optional). >>>> >>>> Julius >>>> >>>> On Tue, Feb 2, 2021 at 12:45 AM Christopher Parsons < >>>> [email protected]> wrote: >>>> >>>>> Hello, >>>>> >>>>> I have a third party tool that is listening for incoming HTTP messages >>>>> on a specific port. When we point AlertManager to this endpoint, >>>>> notifications are delivered and consumed by the third party tool and >>>>> inserted into our monitoring database successfully. However, AlertManager >>>>> continuously retries the notification and slowly backs off on the retries. >>>>> >>>>> In order to try to figure out what's going on, we set up packet >>>>> captures on AlertManager server and one on the endpoint server. As a >>>>> control group, we also configured a simple node.js server which responds >>>>> back with a 200. The only difference, as far as I can tell, from the TCP >>>>> streams in Wireshark is the response headers. The server forcing >>>>> AlertManager to continuously retry replies with a simple "*HTTP/1.1 >>>>> 200 OK*". However, the node.js control group which does not force >>>>> AlertManager to send subsequent replies is responding with something >>>>> like... >>>>> >>>>> *HTTP/1.1 200 OK* >>>>> >>>>> *Date: Mon, 01 Feb 2021 22:56:09 GMT* >>>>> >>>>> *Connection: keep-alive* >>>>> >>>>> *Transfer-Encoding: chunked* >>>>> >>>>> Does the go-lang web server expect more than a simple "HTTP/1.1 200 >>>>> OK" as a response? The third party tool is not open source but I may be >>>>> able to get them to issue a future update if this is considered unexpected >>>>> behavior. Any suggestions or workarounds would be greatly appreciated. >>>>> I've also attached a couple of screenshots from the Wireshark TCP stream >>>>> traces for reference. >>>>> >>>>> Normal behavior: >>>>> >>>>> [image: logserver normal behavior.PNG] >>>>> >>>>> Repeat notification behavior: >>>>> [image: mon101 repeat behavior.PNG] >>>>> >>>>> Please let me know if there are any other details I can provide! >>>>> >>>>> Thanks, >>>>> Chris >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Prometheus Users" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/prometheus-users/014b62ad-72bb-41af-b794-f3bbe61bf705n%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/prometheus-users/014b62ad-72bb-41af-b794-f3bbe61bf705n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> >>>> >>>> -- >>>> Julius Volz >>>> PromLabs - promlabs.com >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> >> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/83566c90-0119-4e7d-97fb-23d30c21299cn%40googlegroups.com >>> <https://groups.google.com/d/msgid/prometheus-users/83566c90-0119-4e7d-97fb-23d30c21299cn%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> >> >> -- >> Julius Volz >> PromLabs - promlabs.com >> > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/f3159722-8d91-4196-8888-9ab180da31aan%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/f3159722-8d91-4196-8888-9ab180da31aan%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- Julius Volz PromLabs - promlabs.com -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAObpH5xOA8s9DhaO2dT71tTqoL2GEKvoEks9oYZZn-gG88Acpw%40mail.gmail.com.
