Re: Observations about reloads and DNS SRV records
Hi Tait, 1. Reloading with SRV records ignores server-state-file > Can you tell me more about this one. How do you see that? I mean, I have a conf similar to yours and I can see HAProxy parsing the server state file (and returning a bunch of warning I'm working on about backend name and ID mismatch). ==> while writing this mail, I am able to reproduce the issue I think: - start HAProxy with SRV records - dump servers state - update haproxy conf to prevent dns resolution at runtime - reload haproxy ==> my servers are now "unconfigured"... (no IP, no FQDN, nothing from the servers state file) - While this is not a huge deal, it does mean that the backend in > question becomes unavailable when the proxy is reloaded until the SRV and > subsequent A records are resolved > Well, this is the role of the server state file: populating server info at configuration parsing time so you don't have to wait until first DNS resolution to get servers' information. ==> this might be caused by the bug found above. 2. Additional record responses from the nameserver are not parsed > - This just means that any servers that are populated from the SRV > records require a second round of querying for each of the hosts after the > fqdn is stored. It might be more efficient if these records are also parsed > but I can see that it might be pretty challenging in the current DNS > resolver > - Only reason I thought of this was to try and reduce up the time it > takes to populate the backend servers with addresses in an effort to lessen > the effects of #1 > > I'll work on this one as soon as I fixed the bug above/ Baptiste
Re: Observations about reloads and DNS SRV records
> > I'm a little swamped with other work at the moment, but when I get a > chance I would be able to provide a DNS server (written in Go) that returns > additional records to test with if that helps. > Thanks for giving me this idea! I wrote a quick and inflexible one here: https://github.com/bedis/dnsserver So feel free to contribute to it or write your own :) I'm going to use it to troubleshoot the issue you reported. That said, nothing is better than other real DNS servers (bind / unbound / powerdns and others) for real production. Baptiste
Re: haproxy 1.9 status update
Hi Willy, I don't see anywhere DNS over TCP mentioned. >From my point of view (and integration with consul / kubernetes), it's an important topic and I'd like to get it done by 1.9, ideally. I have not checked yet how this could be implemented in HAProxy, but I don't really feel comfortable to do it, I'm afraid it's a huge task and I won't have enough time to do it. Baptiste
[Feature request] Call fan-out to all endpoints.
I found myself needing the options to do "fantout" for a call. Meaning making 1 call to haproxy and have it pass that call to *all* of the endpoint currently active. I don't mind implementing this myself and push to code review Is this a feature you would be interested in ? Thanks, Amotz
Re: haproxy 1.9 status update
Hi Baptiste, On Sun, Jun 10, 2018 at 09:27:09AM +0200, Baptiste wrote: > Hi Willy, > > I don't see anywhere DNS over TCP mentioned. I have reported what I'm aware people are currently working on, as you know I don't want to speculate anymore about what would be nice to have if someone was willing to work on it, since the roadmap file is already full of this. > From my point of view (and integration with consul / kubernetes), it's an > important topic and I'd like to get it done by 1.9, ideally. > I have not checked yet how this could be implemented in HAProxy, but I > don't really feel comfortable to do it, I'm afraid it's a huge task and I > won't have enough time to do it. I certainly can understand, but first it means that someone will have to step up saying "I'm willing to work on this" and second you'll have to find at least enough time to help this person and go back and forth with the review, and later for the debugging. As you know this is not a negligible part of the job at all! Cheers, Willy
Re: [Feature request] Call fan-out to all endpoints.
On 10 June 2018 at 08:44, amotz wrote: > I found myself needing the options to do "fantout" for a call. Meaning > making 1 call to haproxy and have it pass that call to all of the endpoint > currently active. > I don't mind implementing this myself and push to code review Is this a > feature you would be interested in ? Hey Amotz, I'm merely an haproxy user (not a dev and nothing to do with the project from a feature/code/merging point of view), but I'd be interested in using this. I feel like an important part of it would be how you'd handle the merge of the different server responses. I.e. the fan-in part. I can see various merge strategies which would be useful in different situations. e.g. "Reply with *this* backend's response but totally ignore this other backend's response" could be useful for in a logging/audit scenario. "Merge the response bodies in this defined order" could be useful for structured data/responses being assembled. "Merge the response bodies in any order, so long as they gave an HTTP response code in the range of X-Y" could be useful for unstructured or self-contained data (e.g. a catalog API). "Merge these N distinct JSON documents into one properly formed JSON response" could be really handy, but would obviously move haproxy's job up the stack somewhat, and might well be an anti-feature! I could have used all the above strategies at various points in my career. I think all but the first strategy might well be harder to implement, as you'll have to cater for a situation where you've received a response but the admin's configured merging strategy dictates that you can't serve the response to the requestor yet. You'll have to find somewhere to cache entire individual response bodies for an amount of time. I don't have any insight into doing that - I can just see that it might be ... interesting :-) If Willy and the rest of the folks who'd have to support this in the future feel like this feature is worth it, please take this as an enthusiastic "yes please!" from a user! Jonathan
Re: [Feature request] Call fan-out to all endpoints.
On Sun, Jun 10, 2018 at 12:36 PM, Jonathan Matthews wrote: > On 10 June 2018 at 08:44, amotz wrote: > > I found myself needing the options to do "fantout" for a call. Meaning > > making 1 call to haproxy and have it pass that call to all of the > endpoint > > currently active. > > I don't mind implementing this myself and push to code review Is this a > > feature you would be interested in ? > > Hey Amotz, > > I'm merely an haproxy user (not a dev and nothing to do with the > project from a feature/code/merging point of view), but I'd be > interested in using this. > > I feel like an important part of it would be how you'd handle the > merge of the different server responses. I.e. the fan-in part. > > I can see various merge strategies which would be useful in different > situations. > > e.g. "Reply with *this* backend's response but totally ignore this > other backend's response" could be useful for in a logging/audit > scenario. > > "Merge the response bodies in this defined order" could be useful for > structured data/responses being assembled. > > "Merge the response bodies in any order, so long as they gave an HTTP > response code in the range of X-Y" could be useful for unstructured or > self-contained data (e.g. a catalog API). > > "Merge these N distinct JSON documents into one properly formed JSON > response" could be really handy, but would obviously move haproxy's > job up the stack somewhat, and might well be an anti-feature! > > I could have used all the above strategies at various points in my career. > > I think all but the first strategy might well be harder to implement, > as you'll have to cater for a situation where you've received a > response but the admin's configured merging strategy dictates that you > can't serve the response to the requestor yet. You'll have to find > somewhere to cache entire individual response bodies for an amount of > time. I don't have any insight into doing that - I can just see that > it might be ... interesting :-) > > If Willy and the rest of the folks who'd have to support this in the > future feel like this feature is worth it, please take this as an > enthusiastic "yes please!" from a user! > > Jonathan > > Hi, what's the use case? Is this API gateway kind of thing? Baptiste
Re: Observations about reloads and DNS SRV records
Hey Baptiste, On Sun, Jun 10, 2018 at 3:19 AM Baptiste wrote: > > ==> while writing this mail, I am able to reproduce the issue I think: > - start HAProxy with SRV records > - dump servers state > - update haproxy conf to prevent dns resolution at runtime > - reload haproxy > ==> my servers are now "unconfigured"... (no IP, no FQDN, nothing from the > servers state file) > > That is exactly the issue I was seeing, my apologies for not describing the actual issue I was seeing when I opened this thread. >> > 2. Additional record responses from the nameserver are not parsed >> - This just means that any servers that are populated from the SRV >> records require a second round of querying for each of the hosts after the >> fqdn is stored. It might be more efficient if these records are also parsed >> but I can see that it might be pretty challenging in the current DNS >> resolver >> - Only reason I thought of this was to try and reduce up the time it >> takes to populate the backend servers with addresses in an effort to lessen >> the effects of #1 >> >> > I'll work on this one as soon as I fixed the bug above/ > Great! The first one will go a long way in making reloads pretty seamless for DNS, at least for my use case :). Thanks again, Tait
Re: Observations about reloads and DNS SRV records
On Sun, Jun 10, 2018 at 3:22 AM Baptiste wrote: > > Thanks for giving me this idea! > I wrote a quick and inflexible one here: > https://github.com/bedis/dnsserver > So feel free to contribute to it or write your own :) > > I'm going to use it to troubleshoot the issue you reported. That said, > nothing is better than other real DNS servers (bind / unbound / powerdns > and others) for real production. > Looks good! My application is using the same DNS library (miekg/dns) which seems to be the most well-rounded and mature for go. If I think of anything to add I'll open a PR but it is currently exactly how I am returning my results. > > Baptiste >
Re: [Feature request] Call fan-out to all endpoints.
>From my experience this is mostly needed for operations/management API. Some examples: getStaus (i.e get the status/health from all endpoint) flashCache (make all endpoint flash their cache) setConfig (you get the point ...) more... with regard to the fan-in question by Jonathan. Maybe return 207 (multi-status) https://httpstatuses.com/207 ? IMO, the most intuitive response would be a json array of all the endpoints responses, but I'm open for suggestions. Thanks, Amotz בתאריך יום א׳, 10 ביוני 2018 ב-14:23 מאת Baptiste <bed...@gmail.com >: > > > On Sun, Jun 10, 2018 at 12:36 PM, Jonathan Matthews < > cont...@jpluscplusm.com> wrote: > >> On 10 June 2018 at 08:44, amotz wrote: >> > I found myself needing the options to do "fantout" for a call. Meaning >> > making 1 call to haproxy and have it pass that call to all of the >> endpoint >> > currently active. >> > I don't mind implementing this myself and push to code review Is this a >> > feature you would be interested in ? >> >> Hey Amotz, >> >> I'm merely an haproxy user (not a dev and nothing to do with the >> project from a feature/code/merging point of view), but I'd be >> interested in using this. >> >> I feel like an important part of it would be how you'd handle the >> merge of the different server responses. I.e. the fan-in part. >> >> I can see various merge strategies which would be useful in different >> situations. >> >> e.g. "Reply with *this* backend's response but totally ignore this >> other backend's response" could be useful for in a logging/audit >> scenario. >> >> "Merge the response bodies in this defined order" could be useful for >> structured data/responses being assembled. >> >> "Merge the response bodies in any order, so long as they gave an HTTP >> response code in the range of X-Y" could be useful for unstructured or >> self-contained data (e.g. a catalog API). >> >> "Merge these N distinct JSON documents into one properly formed JSON >> response" could be really handy, but would obviously move haproxy's >> job up the stack somewhat, and might well be an anti-feature! >> >> I could have used all the above strategies at various points in my career. >> >> I think all but the first strategy might well be harder to implement, >> as you'll have to cater for a situation where you've received a >> response but the admin's configured merging strategy dictates that you >> can't serve the response to the requestor yet. You'll have to find >> somewhere to cache entire individual response bodies for an amount of >> time. I don't have any insight into doing that - I can just see that >> it might be ... interesting :-) >> >> If Willy and the rest of the folks who'd have to support this in the >> future feel like this feature is worth it, please take this as an >> enthusiastic "yes please!" from a user! >> >> Jonathan >> >> > > Hi, > > what's the use case? > Is this API gateway kind of thing? > > Baptiste >
Re: [Feature request] Call fan-out to all endpoints.
Hi. On 10/06/2018 17:56, amotz wrote: Baptiste wrote: Hi, what's the use case? Is this API gateway kind of thing? Baptiste From my experience this is mostly needed for operations/management API. Some examples: getStaus (i.e get the status/health from all endpoint) flashCache (make all endpoint flash their cache) setConfig (you get the point ...) more... with regard to the fan-in question by Jonathan. Maybe return 207 (multi-status) https://httpstatuses.com/207 ? IMO, the most intuitive response would be a json array of all the endpoints responses, but I'm open for suggestions. Let's say you have a `option allendpoints /_fan-out`. When you now call `curl -sS https://haproxy:8080/_fan-out/health` then you will receive a json from *all active* endpoints (pods, app-server,...) with the result of there `/health`, something like this? That sounds interesting. Maybe it's possible with a `external-check command ` or some lua code? https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#external-check%20%28Alphabetically%20sorted%20keywords%20reference%29 Thanks, Amotz Best regards Aleks z בתאריך יום א׳, 10 ביוני 2018 ב-14:23 מאת Baptiste <bed...@gmail.com >: On Sun, Jun 10, 2018 at 12:36 PM, Jonathan Matthews < cont...@jpluscplusm.com> wrote: On 10 June 2018 at 08:44, amotz wrote: > I found myself needing the options to do "fantout" for a call. Meaning > making 1 call to haproxy and have it pass that call to all of the endpoint > currently active. > I don't mind implementing this myself and push to code review Is this a > feature you would be interested in ? Hey Amotz, I'm merely an haproxy user (not a dev and nothing to do with the project from a feature/code/merging point of view), but I'd be interested in using this. I feel like an important part of it would be how you'd handle the merge of the different server responses. I.e. the fan-in part. I can see various merge strategies which would be useful in different situations. e.g. "Reply with *this* backend's response but totally ignore this other backend's response" could be useful for in a logging/audit scenario. "Merge the response bodies in this defined order" could be useful for structured data/responses being assembled. "Merge the response bodies in any order, so long as they gave an HTTP response code in the range of X-Y" could be useful for unstructured or self-contained data (e.g. a catalog API). "Merge these N distinct JSON documents into one properly formed JSON response" could be really handy, but would obviously move haproxy's job up the stack somewhat, and might well be an anti-feature! I could have used all the above strategies at various points in my career. I think all but the first strategy might well be harder to implement, as you'll have to cater for a situation where you've received a response but the admin's configured merging strategy dictates that you can't serve the response to the requestor yet. You'll have to find somewhere to cache entire individual response bodies for an amount of time. I don't have any insight into doing that - I can just see that it might be ... interesting :-) If Willy and the rest of the folks who'd have to support this in the future feel like this feature is worth it, please take this as an enthusiastic "yes please!" from a user! Jonathan
Re: [Feature request] Call fan-out to all endpoints.
On 2018/6/10 13:27, Aleksandar Lazic wrote: > Hi. > > On 10/06/2018 17:56, amotz wrote: >> Baptiste wrote: >>> Hi, >>> >>> what's the use case? >>> Is this API gateway kind of thing? >>> >>> Baptiste >> >> From my experience this is mostly needed for operations/management API. >> >> Some examples: >> getStaus (i.e get the status/health from all endpoint) >> flashCache (make all endpoint flash their cache) >> setConfig (you get the point ...) >> more... >> >> with regard to the fan-in question by Jonathan. >> Maybe return 207 (multi-status) https://httpstatuses.com/207 ? >> IMO, the most intuitive response would be a json array of all the >> endpoints >> responses, but I'm open for suggestions. > > Let's say you have a `option allendpoints /_fan-out`. > > When you now call `curl -sS https://haproxy:8080/_fan-out/health` then > you will receive a json from *all active* endpoints (pods, > app-server,...) with the result of there `/health`, something like this? > > That sounds interesting. Maybe it's possible with a > `external-check command ` or some lua code? > > https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#external-check%20%28Alphabetically%20sorted%20keywords%20reference%29 > Just throwing out my own $0.02, I don't think this is a good responsibility for haproxy to support. This is very specific application level logic. Haproxy don't care about content types (json). What if I want to use this feature, but with some other encoding? How should haproxy respond if a server sends a 1GB response? It can't buffer the whole thing in memory so it can encode it and add it to the response message. What about the non-happy-path cases? What if one of the servers times out, what should haproxy put in the response? What if a server sends a partial response? How should the headers from a server response be encoded? This is basically the invention of a new protocol. Don't get me wrong, the underlying goal, having a client send a single request and that request getting duplicated amongst the servers, is a good one. In fact we do this at my work. But we use a custom application that is specifically designed to handle the protocol we are wrapping. I think this might be reasonable to do in LUA, and maybe even possible already, but there's still going to be lots of the fore-mentioned difficulties. However to put some measure of positive spin on things, I think HTTP/2 would fit very well with this use case. HTTP/2 supports server push messages. Meaning it's built in to the protocol that the client can send a single request, and receive multiple responses. Haproxy doesn't support fully H2 passthrough right now, but that may not be necessary. I think LUA really only needs a few things to be able to support this: The ability to receive H2 requests & generate responses (LUA already has http/1.1 response capabilities, but I have no idea if they work with H2 requests), and then the ability to trigger a request to a server, and have that sent back to the client as a server-push message. -Patrick
