Thanks Julius! Is one newline character ever considered valid? I'm wondering if my next steps are to report this as a bug/feature request for the AlertManager project to support single newline responses or if I need to work with the third party vendor to update how their server is responding. I am pretty sure the service is just using standard Perl packages to handle the incoming HTTP messages so I'm not sure why they would ignore the 2nd newline character.
On Tuesday, February 2, 2021 at 5:46:05 PM UTC-5 [email protected] wrote: > Yep, exactly, you need two newlines (2 times 0x0d0a) to generate a > completely empty line, which HTTP uses to signal that the header section is > over. > > On Tue, Feb 2, 2021 at 3:35 PM Christopher Parsons <[email protected]> > wrote: > >> It appears there's just a single CRLF (0d 0a) after the 200 OK line. Are >> you saying there needs to be another one in order for AlertManager to >> detect a valid 200 response? >> >> [image: mon101 response with hex.PNG] >> >> On Tuesday, February 2, 2021 at 4:25:13 AM UTC-5 [email protected] >> wrote: >> >>> Hi Christopher, >>> >>> It's not clear from your Wireshark stream, maybe show the data as hex >>> instead: does the problematic server insert an empty line after the >>> "HTTP/1.1 200 OK" line and then terminate the connection? So a CRLF after >>> the 200 line and another CRLF for the empty line. This is required for the >>> response to be read as a valid HTTP response (the body is optional). >>> >>> Julius >>> >>> On Tue, Feb 2, 2021 at 12:45 AM Christopher Parsons <[email protected]> >>> wrote: >>> >>>> Hello, >>>> >>>> I have a third party tool that is listening for incoming HTTP messages >>>> on a specific port. When we point AlertManager to this endpoint, >>>> notifications are delivered and consumed by the third party tool and >>>> inserted into our monitoring database successfully. However, AlertManager >>>> continuously retries the notification and slowly backs off on the retries. >>>> >>>> In order to try to figure out what's going on, we set up packet >>>> captures on AlertManager server and one on the endpoint server. As a >>>> control group, we also configured a simple node.js server which responds >>>> back with a 200. The only difference, as far as I can tell, from the TCP >>>> streams in Wireshark is the response headers. The server forcing >>>> AlertManager to continuously retry replies with a simple "*HTTP/1.1 >>>> 200 OK*". However, the node.js control group which does not force >>>> AlertManager to send subsequent replies is responding with something >>>> like... >>>> >>>> *HTTP/1.1 200 OK* >>>> >>>> *Date: Mon, 01 Feb 2021 22:56:09 GMT* >>>> >>>> *Connection: keep-alive* >>>> >>>> *Transfer-Encoding: chunked* >>>> >>>> Does the go-lang web server expect more than a simple "HTTP/1.1 200 OK" >>>> as a response? The third party tool is not open source but I may be able >>>> to get them to issue a future update if this is considered unexpected >>>> behavior. Any suggestions or workarounds would be greatly appreciated. >>>> I've also attached a couple of screenshots from the Wireshark TCP stream >>>> traces for reference. >>>> >>>> Normal behavior: >>>> >>>> [image: logserver normal behavior.PNG] >>>> >>>> Repeat notification behavior: >>>> [image: mon101 repeat behavior.PNG] >>>> >>>> Please let me know if there are any other details I can provide! >>>> >>>> Thanks, >>>> Chris >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Prometheus Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/prometheus-users/014b62ad-72bb-41af-b794-f3bbe61bf705n%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/prometheus-users/014b62ad-72bb-41af-b794-f3bbe61bf705n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> >>> >>> -- >>> Julius Volz >>> PromLabs - promlabs.com >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/83566c90-0119-4e7d-97fb-23d30c21299cn%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/83566c90-0119-4e7d-97fb-23d30c21299cn%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > > > -- > Julius Volz > PromLabs - promlabs.com > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f3159722-8d91-4196-8888-9ab180da31aan%40googlegroups.com.
