To summarize the resolution: Solution: available in v1.7 Workaround: iptables rule on k8s nodes: -A POSTROUTING -d 10.96.0.0/24 -m comment --comment "manjiri: SNAT for outbound traffic from cluster" -m addrtype ! --dst-type LOCAL -j MASQUERADE
Thank you, Tim! -Manjiri On Monday, May 22, 2017 at 8:25:23 AM UTC-7, Tim Hockin wrote: > > On Sun, May 21, 2017 at 9:28 AM, Manjiri Gadagkar > <manjiri....@ruckuswireless.com <javascript:>> wrote: > > Tim, > > > > Thank you for a quick response! > > > > Losing the original pod's IP is not a big concern for me, so this track > will > > be OK. My hope is to be able to use GKE without any > > customizations/workarounds/hacks. Hence, the v1.7 changes are welcome! > > You and me both. > > > 1. Can you confirm that my workaround will not break anything (as far as > you > > know) ? > > Should be fine. > > > 2. Can you point me to docs on "IP aliases support" that you mention > below? > > https://github.com/kubernetes/kubernetes/pull/42147 > > It's gone beta now. > > > 3. The problem I describe here is the pod-to-vpcsvc access. The other > side > > of this coin is vpcsvc-to-k8ssvc. There, I used a Google Internal Load > > Balancer** whose IP address is in subnet1 and accessible from subnet2 > (as > > expected). This is the complete use-case: "external" service is private > and > > K8S service is exposed on a 10.x.x.x network. I am curious about the > > thoughts around this use-case in terms of abstraction/configuration > options. > > ILB should be in 1.7 also. > > > ** Internal Load Balancer (ILB): I have a NodePort service (portA) whose > > node:portB are the backend. The ILB only works if the frontend port is > portB > > i.e. same as backend. I suppose this because of the "forwarder, not > proxy" > > architecture of ILB. My goal is to expose this service to VMs in subnet2 > on > > portA. I am able to do that by running HAproxy in subnet1. Is service > > type=InternalLoadBalancer (or equivalent with annotation or whatever) on > the > > GKE roadmap? Will the TCP+proxy mode be supported by the ILB? I bring > this > > up here because my use-case as a whole needs this piece as well and I > would > > like to understand if the GKE roadmap addresses the masquerade piece > > separately or this entire use-case is going to be supported. > > > > Thank you! > > -Manjiri > > > > On Saturday, May 20, 2017 at 8:56:52 PM UTC-7, Tim Hockin wrote: > >> > >> The VPC peerings are dropping non-VM packets. That's unfortunately > >> the state of things for now. The solution, as you show here, is to > >> masquerade. The downside is that you lose the original pod's IP in > >> the packet. > >> > >> We've added support for IP aliases, which should be beta in OSS in > >> v1.7 (and probably alpha in GKE at first), which will remove this > >> restriction. We're also working on a more configurable way to manage > >> the masquerade rules, so you can tweak it in small way, instead of the > >> single kubelet flag we have today. > >> > >> On Sat, May 20, 2017 at 8:41 PM, Manjiri Gadagkar > >> <manjiri....@ruckuswireless.com> wrote: > >> > Hello! > >> > > >> > My goal is to set up a cluster in Google Container Engine in a > private > >> > network (nw1:subnet1) that can access services in another private > >> > network > >> > (nw2:subnet2). The latter (subnet2) is an existing deployment of > >> > services > >> > running on Google Compute Engine instances (VMs). I have set up VPC > >> > Peering > >> > between nw1 and nw2. > >> > > >> > Master version: 1.5.7 > >> > nw1: > >> > subnet1: 10.224.0.0/11 > >> > Container address range: 10.164.0.0/14 > >> > PodCIDR: 10.164.2.0/24 > >> > nw2: > >> > subnet2: 10.96.0.0/24 (existing) > >> > subnet3: 192.168.0.0/24 (testing) > >> > subnet4: 172.16.0.0/24 (testing) > >> > > >> > I found that subnet3 and subnet4 are accessible from the pods. My > >> > problem is > >> > that I am not able to access subnet2 from the pods. > >> > > >> > I looked at the iptables on the GKE node, and found this rule: > >> > > >> > -A POSTROUTING ! -d 10.0.0.0/8 -m comment --comment "kubenet: > SNAT > >> > for > >> > outbound traffic from cluster" -m addrtype ! --dst-type LOCAL -j > >> > MASQUERADE > >> > > >> > I added a SNAT rule for subnet2 destination as follows. I don't know > if > >> > it > >> > breaks something else, but it appears to work around the issue. > >> > > >> > -A POSTROUTING -m comment --comment "kubernetes postrouting rules" > -j > >> > KUBE-POSTROUTING > >> > -A POSTROUTING -d 10.96.0.0/24 -m comment --comment "manjiri: SNAT > for > >> > outbound traffic from cluster" -m addrtype ! --dst-type LOCAL -j > >> > MASQUERADE > >> > -A POSTROUTING ! -d 10.0.0.0/8 -m comment --comment "kubenet: SNAT > for > >> > outbound traffic from cluster" -m addrtype ! --dst-type LOCAL -j > >> > MASQUERADE > >> > -A POSTROUTING -s 127.0.0.0/8 -o cbr0 -m comment --comment "SNAT > for > >> > localhost access to hostports" -j MASQUERADE > >> > > >> > Is my use-case supported in GKE? Is there a configuration parameter > that > >> > I > >> > could tweak? Anything else I could try? > >> > > >> > Thanks in advance! > >> > -Manjiri > >> > > >> > > >> > > >> > -- > >> > You received this message because you are subscribed to the Google > >> > Groups > >> > "Kubernetes user discussion and Q&A" group. > >> > To unsubscribe from this group and stop receiving emails from it, > send > >> > an > >> > email to kubernetes-use...@googlegroups.com. > >> > To post to this group, send email to kubernet...@googlegroups.com. > >> > Visit this group at https://groups.google.com/group/kubernetes-users. > > >> > For more options, visit https://groups.google.com/d/optout. > > > > -- > > You received this message because you are subscribed to the Google > Groups > > "Kubernetes user discussion and Q&A" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to kubernetes-use...@googlegroups.com <javascript:>. > > To post to this group, send email to kubernet...@googlegroups.com > <javascript:>. > > Visit this group at https://groups.google.com/group/kubernetes-users. > > For more options, visit https://groups.google.com/d/optout. > On Monday, May 22, 2017 at 8:25:23 AM UTC-7, Tim Hockin wrote: > > On Sun, May 21, 2017 at 9:28 AM, Manjiri Gadagkar > <manjiri....@ruckuswireless.com <javascript:>> wrote: > > Tim, > > > > Thank you for a quick response! > > > > Losing the original pod's IP is not a big concern for me, so this track > will > > be OK. My hope is to be able to use GKE without any > > customizations/workarounds/hacks. Hence, the v1.7 changes are welcome! > > You and me both. > > > 1. Can you confirm that my workaround will not break anything (as far as > you > > know) ? > > Should be fine. > > > 2. Can you point me to docs on "IP aliases support" that you mention > below? > > https://github.com/kubernetes/kubernetes/pull/42147 > > It's gone beta now. > > > 3. The problem I describe here is the pod-to-vpcsvc access. The other > side > > of this coin is vpcsvc-to-k8ssvc. There, I used a Google Internal Load > > Balancer** whose IP address is in subnet1 and accessible from subnet2 > (as > > expected). This is the complete use-case: "external" service is private > and > > K8S service is exposed on a 10.x.x.x network. I am curious about the > > thoughts around this use-case in terms of abstraction/configuration > options. > > ILB should be in 1.7 also. > > > ** Internal Load Balancer (ILB): I have a NodePort service (portA) whose > > node:portB are the backend. The ILB only works if the frontend port is > portB > > i.e. same as backend. I suppose this because of the "forwarder, not > proxy" > > architecture of ILB. My goal is to expose this service to VMs in subnet2 > on > > portA. I am able to do that by running HAproxy in subnet1. Is service > > type=InternalLoadBalancer (or equivalent with annotation or whatever) on > the > > GKE roadmap? Will the TCP+proxy mode be supported by the ILB? I bring > this > > up here because my use-case as a whole needs this piece as well and I > would > > like to understand if the GKE roadmap addresses the masquerade piece > > separately or this entire use-case is going to be supported. > > > > Thank you! > > -Manjiri > > > > On Saturday, May 20, 2017 at 8:56:52 PM UTC-7, Tim Hockin wrote: > >> > >> The VPC peerings are dropping non-VM packets. That's unfortunately > >> the state of things for now. The solution, as you show here, is to > >> masquerade. The downside is that you lose the original pod's IP in > >> the packet. > >> > >> We've added support for IP aliases, which should be beta in OSS in > >> v1.7 (and probably alpha in GKE at first), which will remove this > >> restriction. We're also working on a more configurable way to manage > >> the masquerade rules, so you can tweak it in small way, instead of the > >> single kubelet flag we have today. > >> > >> On Sat, May 20, 2017 at 8:41 PM, Manjiri Gadagkar > >> <manjiri....@ruckuswireless.com> wrote: > >> > Hello! > >> > > >> > My goal is to set up a cluster in Google Container Engine in a > private > >> > network (nw1:subnet1) that can access services in another private > >> > network > >> > (nw2:subnet2). The latter (subnet2) is an existing deployment of > >> > services > >> > running on Google Compute Engine instances (VMs). I have set up VPC > >> > Peering > >> > between nw1 and nw2. > >> > > >> > Master version: 1.5.7 > >> > nw1: > >> > subnet1: 10.224.0.0/11 > >> > Container address range: 10.164.0.0/14 > >> > PodCIDR: 10.164.2.0/24 > >> > nw2: > >> > subnet2: 10.96.0.0/24 (existing) > >> > subnet3: 192.168.0.0/24 (testing) > >> > subnet4: 172.16.0.0/24 (testing) > >> > > >> > I found that subnet3 and subnet4 are accessible from the pods. My > >> > problem is > >> > that I am not able to access subnet2 from the pods. > >> > > >> > I looked at the iptables on the GKE node, and found this rule: > >> > > >> > -A POSTROUTING ! -d 10.0.0.0/8 -m comment --comment "kubenet: > SNAT > >> > for > >> > outbound traffic from cluster" -m addrtype ! --dst-type LOCAL -j > >> > MASQUERADE > >> > > >> > I added a SNAT rule for subnet2 destination as follows. I don't know > if > >> > it > >> > breaks something else, but it appears to work around the issue. > >> > > >> > -A POSTROUTING -m comment --comment "kubernetes postrouting rules" > -j > >> > KUBE-POSTROUTING > >> > -A POSTROUTING -d 10.96.0.0/24 -m comment --comment "manjiri: SNAT > for > >> > outbound traffic from cluster" -m addrtype ! --dst-type LOCAL -j > >> > MASQUERADE > >> > -A POSTROUTING ! -d 10.0.0.0/8 -m comment --comment "kubenet: SNAT > for > >> > outbound traffic from cluster" -m addrtype ! --dst-type LOCAL -j > >> > MASQUERADE > >> > -A POSTROUTING -s 127.0.0.0/8 -o cbr0 -m comment --comment "SNAT > for > >> > localhost access to hostports" -j MASQUERADE > >> > > >> > Is my use-case supported in GKE? Is there a configuration parameter > that > >> > I > >> > could tweak? Anything else I could try? > >> > > >> > Thanks in advance! > >> > -Manjiri > >> > > >> > > >> > > >> > -- > >> > You received this message because you are subscribed to the Google > >> > Groups > >> > "Kubernetes user discussion and Q&A" group. > >> > To unsubscribe from this group and stop receiving emails from it, > send > >> > an > >> > email to kubernetes-use...@googlegroups.com. > >> > To post to this group, send email to kubernet...@googlegroups.com. > >> > Visit this group at https://groups.google.com/group/kubernetes-users. > > >> > For more options, visit https://groups.google.com/d/optout. > > > > -- > > You received this message because you are subscribed to the Google > Groups > > "Kubernetes user discussion and Q&A" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to kubernetes-use...@googlegroups.com <javascript:>. > > To post to this group, send email to kubernet...@googlegroups.com > <javascript:>. > > Visit this group at https://groups.google.com/group/kubernetes-users. > > For more options, visit https://groups.google.com/d/optout. > On Monday, May 22, 2017 at 8:25:23 AM UTC-7, Tim Hockin wrote: > > On Sun, May 21, 2017 at 9:28 AM, Manjiri Gadagkar > <manjiri....@ruckuswireless.com <javascript:>> wrote: > > Tim, > > > > Thank you for a quick response! > > > > Losing the original pod's IP is not a big concern for me, so this track > will > > be OK. My hope is to be able to use GKE without any > > customizations/workarounds/hacks. Hence, the v1.7 changes are welcome! > > You and me both. > > > 1. Can you confirm that my workaround will not break anything (as far as > you > > know) ? > > Should be fine. > > > 2. Can you point me to docs on "IP aliases support" that you mention > below? > > https://github.com/kubernetes/kubernetes/pull/42147 > > It's gone beta now. > > > 3. The problem I describe here is the pod-to-vpcsvc access. The other > side > > of this coin is vpcsvc-to-k8ssvc. There, I used a Google Internal Load > > Balancer** whose IP address is in subnet1 and accessible from subnet2 > (as > > expected). This is the complete use-case: "external" service is private > and > > K8S service is exposed on a 10.x.x.x network. I am curious about the > > thoughts around this use-case in terms of abstraction/configuration > options. > > ILB should be in 1.7 also. > > > ** Internal Load Balancer (ILB): I have a NodePort service (portA) whose > > node:portB are the backend. The ILB only works if the frontend port is > portB > > i.e. same as backend. I suppose this because of the "forwarder, not > proxy" > > architecture of ILB. My goal is to expose this service to VMs in subnet2 > on > > portA. I am able to do that by running HAproxy in subnet1. Is service > > type=InternalLoadBalancer (or equivalent with annotation or whatever) on > the > > GKE roadmap? Will the TCP+proxy mode be supported by the ILB? I bring > this > > up here because my use-case as a whole needs this piece as well and I > would > > like to understand if the GKE roadmap addresses the masquerade piece > > separately or this entire use-case is going to be supported. > > > > Thank you! > > -Manjiri > > > > On Saturday, May 20, 2017 at 8:56:52 PM UTC-7, Tim Hockin wrote: > >> > >> The VPC peerings are dropping non-VM packets. That's unfortunately > >> the state of things for now. The solution, as you show here, is to > >> masquerade. The downside is that you lose the original pod's IP in > >> the packet. > >> > >> We've added support for IP aliases, which should be beta in OSS in > >> v1.7 (and probably alpha in GKE at first), which will remove this > >> restriction. We're also working on a more configurable way to manage > >> the masquerade rules, so you can tweak it in small way, instead of the > >> single kubelet flag we have today. > >> > >> On Sat, May 20, 2017 at 8:41 PM, Manjiri Gadagkar > >> <manjiri....@ruckuswireless.com> wrote: > >> > Hello! > >> > > >> > My goal is to set up a cluster in Google Container Engine in a > private > >> > network (nw1:subnet1) that can access services in another private > >> > network > >> > (nw2:subnet2). The latter (subnet2) is an existing deployment of > >> > services > >> > running on Google Compute Engine instances (VMs). I have set up VPC > >> > Peering > >> > between nw1 and nw2. > >> > > >> > Master version: 1.5.7 > >> > nw1: > >> > subnet1: 10.224.0.0/11 > >> > Container address range: 10.164.0.0/14 > >> > PodCIDR: 10.164.2.0/24 > >> > nw2: > >> > subnet2: 10.96.0.0/24 (existing) > >> > subnet3: 192.168.0.0/24 (testing) > >> > subnet4: 172.16.0.0/24 (testing) > >> > > >> > I found that subnet3 and subnet4 are accessible from the pods. My > >> > problem is > >> > that I am not able to access subnet2 from the pods. > >> > > >> > I looked at the iptables on the GKE node, and found this rule: > >> > > >> > -A POSTROUTING ! -d 10.0.0.0/8 -m comment --comment "kubenet: > SNAT > >> > for > >> > outbound traffic from cluster" -m addrtype ! --dst-type LOCAL -j > >> > MASQUERADE > >> > > >> > I added a SNAT rule for subnet2 destination as follows. I don't know > if > >> > it > >> > breaks something else, but it appears to work around the issue. > >> > > >> > -A POSTROUTING -m comment --comment "kubernetes postrouting rules" > -j > >> > KUBE-POSTROUTING > >> > -A POSTROUTING -d 10.96.0.0/24 -m comment --comment "manjiri: SNAT > for > >> > outbound traffic from cluster" -m addrtype ! --dst-type LOCAL -j > >> > MASQUERADE > >> > -A POSTROUTING ! -d 10.0.0.0/8 -m comment --comment "kubenet: SNAT > for > >> > outbound traffic from cluster" -m addrtype ! --dst-type LOCAL -j > >> > MASQUERADE > >> > -A POSTROUTING -s 127.0.0.0/8 -o cbr0 -m comment --comment "SNAT > for > >> > localhost access to hostports" -j MASQUERADE > >> > > >> > Is my use-case supported in GKE? Is there a configuration parameter > that > >> > I > >> > could tweak? Anything else I could try? > >> > > >> > Thanks in advance! > >> > -Manjiri > >> > > >> > > >> > > >> > -- > >> > You received this message because you are subscribed to the Google > >> > Groups > >> > "Kubernetes user discussion and Q&A" group. > >> > To unsubscribe from this group and stop receiving emails from it, > send > >> > an > >> > email to kubernetes-use...@googlegroups.com. > >> > To post to this group, send email to kubernet...@googlegroups.com. > >> > Visit this group at https://groups.google.com/group/kubernetes-users. > > >> > For more options, visit https://groups.google.com/d/optout. > > > > -- > > You received this message because you are subscribed to the Google > Groups > > "Kubernetes user discussion and Q&A" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to kubernetes-use...@googlegroups.com <javascript:>. > > To post to this group, send email to kubernet...@googlegroups.com > <javascript:>. > > Visit this group at https://groups.google.com/group/kubernetes-users. > > For more options, visit https://groups.google.com/d/optout. > On Monday, May 22, 2017 at 8:25:23 AM UTC-7, Tim Hockin wrote: > > On Sun, May 21, 2017 at 9:28 AM, Manjiri Gadagkar > <manjiri....@ruckuswireless.com <javascript:>> wrote: > > Tim, > > > > Thank you for a quick response! > > > > Losing the original pod's IP is not a big concern for me, so this track > will > > be OK. My hope is to be able to use GKE without any > > customizations/workarounds/hacks. Hence, the v1.7 changes are welcome! > > You and me both. > > > 1. Can you confirm that my workaround will not break anything (as far as > you > > know) ? > > Should be fine. > > > 2. Can you point me to docs on "IP aliases support" that you mention > below? > > https://github.com/kubernetes/kubernetes/pull/42147 > > It's gone beta now. > > > 3. The problem I describe here is the pod-to-vpcsvc access. The other > side > > of this coin is vpcsvc-to-k8ssvc. There, I used a Google Internal Load > > Balancer** whose IP address is in subnet1 and accessible from subnet2 > (as > > expected). This is the complete use-case: "external" service is private > and > > K8S service is exposed on a 10.x.x.x network. I am curious about the > > thoughts around this use-case in terms of abstraction/configuration > options. > > ILB should be in 1.7 also. > > > ** Internal Load Balancer (ILB): I have a NodePort service (portA) whose > > node:portB are the backend. The ILB only works if the frontend port is > portB > > i.e. same as backend. I suppose this because of the "forwarder, not > proxy" > > architecture of ILB. My goal is to expose this service to VMs in subnet2 > on > > portA. I am able to do that by running HAproxy in subnet1. Is service > > type=InternalLoadBalancer (or equivalent with annotation or whatever) on > the > > GKE roadmap? Will the TCP+proxy mode be supported by the ILB? I bring > this > > up here because my use-case as a whole needs this piece as well and I > would > > like to understand if the GKE roadmap addresses the masquerade piece > > separately or this entire use-case is going to be supported. > > > > Thank you! > > -Manjiri > > > > On Saturday, May 20, 2017 at 8:56:52 PM UTC-7, Tim Hockin wrote: > >> > >> The VPC peerings are dropping non-VM packets. That's unfortunately > >> the state of things for now. The solution, as you show here, is to > >> masquerade. The downside is that you lose the original pod's IP in > >> the packet. > >> > >> We've added support for IP aliases, which should be beta in OSS in > >> v1.7 (and probably alpha in GKE at first), which will remove this > >> restriction. We're also working on a more configurable way to manage > >> the masquerade rules, so you can tweak it in small way, instead of the > >> single kubelet flag we have today. > >> > >> On Sat, May 20, 2017 at 8:41 PM, Manjiri Gadagkar > >> <manjiri....@ruckuswireless.com> wrote: > >> > Hello! > >> > > >> > My goal is to set up a cluster in Google Container Engine in a > private > >> > network (nw1:subnet1) that can access services in another private > >> > network > >> > (nw2:subnet2). The latter (subnet2) is an existing deployment of > >> > services > >> > running on Google Compute Engine instances (VMs). I have set up VPC > >> > Peering > >> > between nw1 and nw2. > >> > > >> > Master version: 1.5.7 > >> > nw1: > >> > subnet1: 10.224.0.0/11 > >> > Container address range: 10.164.0.0/14 > >> > PodCIDR: 10.164.2.0/24 > >> > nw2: > >> > subnet2: 10.96.0.0/24 (existing) > >> > subnet3: 192.168.0.0/24 (testing) > >> > subnet4: 172.16.0.0/24 (testing) > >> > > >> > I found that subnet3 and subnet4 are accessible from the pods. My > >> > problem is > >> > that I am not able to access subnet2 from the pods. > >> > > >> > I looked at the iptables on the GKE node, and found this rule: > >> > > >> > -A POSTROUTING ! -d 10.0.0.0/8 -m comment --comment "kubenet: > SNAT > >> > for > >> > outbound traffic from cluster" -m addrtype ! --dst-type LOCAL -j > >> > MASQUERADE > >> > > >> > I added a SNAT rule for subnet2 destination as follows. I don't know > if > >> > it > >> > breaks something else, but it appears to work around the issue. > >> > > >> > -A POSTROUTING -m comment --comment "kubernetes postrouting rules" > -j > >> > KUBE-POSTROUTING > >> > -A POSTROUTING -d 10.96.0.0/24 -m comment --comment "manjiri: SNAT > for > >> > outbound traffic from cluster" -m addrtype ! --dst-type LOCAL -j > >> > MASQUERADE > >> > -A POSTROUTING ! -d 10.0.0.0/8 -m comment --comment "kubenet: SNAT > for > >> > outbound traffic from cluster" -m addrtype ! --dst-type LOCAL -j > >> > MASQUERADE > >> > -A POSTROUTING -s 127.0.0.0/8 -o cbr0 -m comment --comment "SNAT > for > >> > localhost access to hostports" -j MASQUERADE > >> > > >> > Is my use-case supported in GKE? Is there a configuration parameter > that > >> > I > >> > could tweak? Anything else I could try? > >> > > >> > Thanks in advance! > >> > -Manjiri > >> > > >> > > >> > > >> > -- > >> > You received this message because you are subscribed to the Google > >> > Groups > >> > "Kubernetes user discussion and Q&A" group. > >> > To unsubscribe from this group and stop receiving emails from it, > send > >> > an > >> > email to kubernetes-use...@googlegroups.com. > >> > To post to this group, send email to kubernet...@googlegroups.com. > >> > Visit this group at https://groups.google.com/group/kubernetes-users. > > >> > For more options, visit https://groups.google.com/d/optout. > > > > -- > > You received this message because you are subscribed to the Google > Groups > > "Kubernetes user discussion and Q&A" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to kubernetes-use...@googlegroups.com <javascript:>. > > To post to this group, send email to kubernet...@googlegroups.com > <javascript:>. > > Visit this group at https://groups.google.com/group/kubernetes-users. > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
