Re: Using haproxy together with NFS
Hi, You might want to have a look at IPVS for instance in combination with Keepalived. You can then even use udp mounts if you want. Just my 2 cents. Regards, Sander > On 2 Aug 2018, at 18:40, Lucas Rolff wrote: > > I indeed removed the send-proxy - then I had to put the IP of haproxy in the > NFS exports file instead to be able to mount the share (which makes sense > seen from a NFS perspective). > > Making the NFS server support proxy protocol, isn't something I think will > happen - I rely on the upstream packages (CentOS 7 packages in this case). > > And using transparency mode - I think relying on stuff going via haproxy for > routing won't be a possibility in this case - so I guess I have to drop my > wish about haproxy + NFS in this case, I'd like something that is fairly > standard without too much modifications on the current NFS infrastructure > (since it would introduce more complexity). > > Thanks for your replies both of you! > > Best Regards, > > On 02/08/2018, 18.09, "Willy Tarreau" wrote: > >>On Thu, Aug 02, 2018 at 04:05:24AM +, Lucas Rolff wrote: >> Hi michael, >> >> Without the send-proxy, the client IP in the export would have to be the >> haproxy server in that case right? > >That's it. But Michael is absolutely right, your NFS server doesn't support >the proxy protocol, and the lines it emits below indicate it : > > Aug 01 21:44:44 nfs-server-f8209dc4-a1a6-4baf-86fa-eba0b0254bc9 kernel: > RPC: fragment too large: 1347571544 > Aug 01 21:44:44 nfs-server-f8209dc4-a1a6-4baf-86fa-eba0b0254bc9 kernel: > RPC: fragment too large: 1347571544 > Aug 01 21:44:44 nfs-server-f8209dc4-a1a6-4baf-86fa-eba0b0254bc9 kernel: > RPC: fragment too large: 1347571544 > Aug 01 21:44:45 nfs-server-f8209dc4-a1a6-4baf-86fa-eba0b0254bc9 kernel: > RPC: fragment too large: 1347571544 > >This fragment size (1347571544) is "PROX" encoded in big endian, which are >the first 4 chars of the proxy protocol header :-) > >> The issue there is then, that I end up with all clients having access to >> haproxy can suddenly mount all shares in nfs, which I would like to prevent > >Maybe you can modify your NFS server to support the proxy protocol, that >could possibly make sense for your use case ? Otherwise on Linux you may >be able to configure haproxy to work in transparent mode using "source >0.0.0.0 usesrc clientip" but beware that it requires some specific iptables >rules to divert the traffic and send it back to haproxy. It will also > require >that all your NFS servers route the clients via haproxy for the response >traffic. This is not always very convenient. > >Regards, >Willy > >
Re: Using haproxy together with NFS
I indeed removed the send-proxy - then I had to put the IP of haproxy in the NFS exports file instead to be able to mount the share (which makes sense seen from a NFS perspective). Making the NFS server support proxy protocol, isn't something I think will happen - I rely on the upstream packages (CentOS 7 packages in this case). And using transparency mode - I think relying on stuff going via haproxy for routing won't be a possibility in this case - so I guess I have to drop my wish about haproxy + NFS in this case, I'd like something that is fairly standard without too much modifications on the current NFS infrastructure (since it would introduce more complexity). Thanks for your replies both of you! Best Regards, On 02/08/2018, 18.09, "Willy Tarreau" wrote: On Thu, Aug 02, 2018 at 04:05:24AM +, Lucas Rolff wrote: > Hi michael, > > Without the send-proxy, the client IP in the export would have to be the > haproxy server in that case right? That's it. But Michael is absolutely right, your NFS server doesn't support the proxy protocol, and the lines it emits below indicate it : Aug 01 21:44:44 nfs-server-f8209dc4-a1a6-4baf-86fa-eba0b0254bc9 kernel: RPC: fragment too large: 1347571544 Aug 01 21:44:44 nfs-server-f8209dc4-a1a6-4baf-86fa-eba0b0254bc9 kernel: RPC: fragment too large: 1347571544 Aug 01 21:44:44 nfs-server-f8209dc4-a1a6-4baf-86fa-eba0b0254bc9 kernel: RPC: fragment too large: 1347571544 Aug 01 21:44:45 nfs-server-f8209dc4-a1a6-4baf-86fa-eba0b0254bc9 kernel: RPC: fragment too large: 1347571544 This fragment size (1347571544) is "PROX" encoded in big endian, which are the first 4 chars of the proxy protocol header :-) > The issue there is then, that I end up with all clients having access to > haproxy can suddenly mount all shares in nfs, which I would like to prevent Maybe you can modify your NFS server to support the proxy protocol, that could possibly make sense for your use case ? Otherwise on Linux you may be able to configure haproxy to work in transparent mode using "source 0.0.0.0 usesrc clientip" but beware that it requires some specific iptables rules to divert the traffic and send it back to haproxy. It will also require that all your NFS servers route the clients via haproxy for the response traffic. This is not always very convenient. Regards, Willy
Re: Using haproxy together with NFS
On Thu, Aug 02, 2018 at 04:05:24AM +, Lucas Rolff wrote: > Hi michael, > > Without the send-proxy, the client IP in the export would have to be the > haproxy server in that case right? That's it. But Michael is absolutely right, your NFS server doesn't support the proxy protocol, and the lines it emits below indicate it : Aug 01 21:44:44 nfs-server-f8209dc4-a1a6-4baf-86fa-eba0b0254bc9 kernel: RPC: fragment too large: 1347571544 Aug 01 21:44:44 nfs-server-f8209dc4-a1a6-4baf-86fa-eba0b0254bc9 kernel: RPC: fragment too large: 1347571544 Aug 01 21:44:44 nfs-server-f8209dc4-a1a6-4baf-86fa-eba0b0254bc9 kernel: RPC: fragment too large: 1347571544 Aug 01 21:44:45 nfs-server-f8209dc4-a1a6-4baf-86fa-eba0b0254bc9 kernel: RPC: fragment too large: 1347571544 This fragment size (1347571544) is "PROX" encoded in big endian, which are the first 4 chars of the proxy protocol header :-) > The issue there is then, that I end up with all clients having access to > haproxy can suddenly mount all shares in nfs, which I would like to prevent Maybe you can modify your NFS server to support the proxy protocol, that could possibly make sense for your use case ? Otherwise on Linux you may be able to configure haproxy to work in transparent mode using "source 0.0.0.0 usesrc clientip" but beware that it requires some specific iptables rules to divert the traffic and send it back to haproxy. It will also require that all your NFS servers route the clients via haproxy for the response traffic. This is not always very convenient. Regards, Willy
Re: Using haproxy together with NFS
Hi michael, Without the send-proxy, the client IP in the export would have to be the haproxy server in that case right? The issue there is then, that I end up with all clients having access to haproxy can suddenly mount all shares in nfs, which I would like to prevent There’s still different shares that different servers need access to I’ll try not the sample config from the link above! Thanks! Get Outlook for iOS<https://aka.ms/o0ukef> From: Michael Ezzell Sent: Thursday, August 2, 2018 2:38:06 AM To: Lucas Rolff Cc: HAproxy Mailing Lists Subject: Re: Using haproxy together with NFS On Wed, Aug 1, 2018, 16:00 Lucas Rolff mailto:lu...@lucasrolff.com>> wrote: I use the “send-proxy” to let the NFS Server see the actual source IP, instead of the haproxy machine IP. You'll probably need remove that. Unless the destination service explicitly supports the Proxy Protocol (in which case, it must not, by definition, process connections where the protocol's preamble is *absent* from the stream), then this would just look like corrupt data. This option doesn't actually change the source address. HAProxy in TCP mode should work fine with NFS -- at least, it does with NFS4.1 as implemented in Amazon Elastic File System -- which is the only version I've tested against. https://serverfault.com/a/799213/153161
Re: Using haproxy together with NFS
On Wed, Aug 1, 2018, 16:00 Lucas Rolff wrote: > > I use the “send-proxy” to let the NFS Server see the actual source IP, > instead of the haproxy machine IP. > You'll probably need remove that. Unless the destination service explicitly supports the Proxy Protocol (in which case, it must not, by definition, process connections where the protocol's preamble is *absent* from the stream), then this would just look like corrupt data. This option doesn't actually change the source address. HAProxy in TCP mode should work fine with NFS -- at least, it does with NFS4.1 as implemented in Amazon Elastic File System -- which is the only version I've tested against. https://serverfault.com/a/799213/153161
Using haproxy together with NFS
Hi guys, I’ve been playing around today with two NFS servers (each on their own storage array), synced by Unison to provide a bit higher uptime. To allow NFS clients to use a single IP, I’ve configured an haproxy install (1 now, two when in prod), where I want to talk over tcp mode to the NFS servers. My idea is that all traffic is directed based on the source IP balancing, so the traffic will be somewhat 50/50 on each NFS server. My question is if anyone have actually ever got a setup like this to work, I’m using NFS 4.0, and whenever I try to mount the NFS mount on the client, it does communicate with haproxy, and I do see traffic on the NFS server itself, meaning the communication seems to work. The issue I’m facing, is that the mounting will actually never complete due to some weird behavior when I go through haproxy: Aug 01 21:44:44 nfs-server-f8209dc4-a1a6-4baf-86fa-eba0b0254bc9 kernel: RPC: fragment too large: 1347571544 Aug 01 21:44:44 nfs-server-f8209dc4-a1a6-4baf-86fa-eba0b0254bc9 kernel: RPC: fragment too large: 1347571544 Aug 01 21:44:44 nfs-server-f8209dc4-a1a6-4baf-86fa-eba0b0254bc9 kernel: RPC: fragment too large: 1347571544 Aug 01 21:44:45 nfs-server-f8209dc4-a1a6-4baf-86fa-eba0b0254bc9 kernel: RPC: fragment too large: 1347571544 It will continue to give this “fragment too large”. If I bypass haproxy it works completely fine, so I know the NFS Server is configured correctly for the client to connect. My haproxy configuration looks like this: global log 127.0.0.1 local1 debug nbproc 4 user haproxy group haproxy daemon chroot /var/lib/haproxy pidfile /var/run/haproxy.pid defaults mode tcp log global option tcplog timeout client 1m timeout server 1m timeout connect 10s balance source frontend nfs-in1 bind *:2049 use_backend nfs_backend1 frontend nfs-in2 bind *:111 use_backend nfs_backend2 frontend nfs-in3 bind *:46716 use_backend nfs_backend3 frontend nfs-in4 bind *:36856 use_backend nfs_backend4 backend nfs_backend1 server nfs1 217.xx.xx.xx:2049 send-proxy backend nfs_backend2 server nfs1 217.xx.xx.xx:111 send-proxy backend nfs_backend3 server nfs1 217.xx.xx.xx:46716 send-proxy backend nfs_backend4 server nfs1 217.xx.xx.xx:36856 send-proxy I use the “send-proxy” to let the NFS Server see the actual source IP, instead of the haproxy machine IP. If anyone has any idea what can be the cause of the “fragment too large” when going via haproxy, or an actual working config for haproxy to do NFS 4.0 or 4.1 traffic – then please let me know! Best Regards, Lucas Rolff
