GKE is our "new cluster".
Before GKE, our stack ran on mesos+marathon cluster on GCE with weave for
networking.
We still have some services "left behind" on the "old cluster", so I ran
into this when trying to access an "old service" from a "new service".
Not surprisingly, this does not work when the containers in both clusters
have overlapping intra-cluster address-spaces...
On Wed, Jul 5, 2017 at 1:22 AM 'Tim Hockin' via Kubernetes user discussion
and Q&A <kubernetes-users@googlegroups.com> wrote:
> We're you testing weave or running g it for real?
>
> On Jul 4, 2017 7:28 AM, "Itamar O" <itamar...@gmail.com> wrote:
>
>> solved it.
>> I found a collision in the routing table of the 10.240.0.2 host...
>> # route
>> Kernel IP routing table
>> Destination Gateway Genmask Flags Metric Ref Use
>> Iface
>> default 10.240.0.1 0.0.0.0 UG 0 0 0
>> eth0
>> *10.32.0.0 * 255.240.0.0 U 0 0 0
>> weave*
>> 10.240.0.1 * 255.255.255.255 UH 0 0 0
>> eth0
>> 172.17.0.0 * 255.255.0.0 U 0 0 0
>> docker0
>> 172.18.0.0 * 255.255.0.0 U 0 0 0
>> docker_gwbridge
>>
>> thanks.
>>
>> On Mon, Jul 3, 2017 at 10:45 AM Itamar O <itamar...@gmail.com> wrote:
>>
>>> here's a slightly redacted snippet of my GCP routes:
>>>
>>> > gcloud compute routes list
>>> NAME NETWORK
>>> DEST_RANGE NEXT_HOP
>>> PRIORITY
>>> default-route-2aab1c1d780bf8ed datalab 10.240.0.0/24 1000
>>> default-route-91098cb127e5377b default 10.240.0.0/16 1000
>>> gke-foo1 default 10.44.4.0/24
>>> us-central1-f/instances/gke-foo-pool1-a 1000
>>> gke-foo2 default 10.44.7.0/24
>>> us-central1-f/instances/gke-foo-pool1-b 1000
>>> gke-foo3 default 10.44.5.0/24
>>> us-central1-f/instances/gke-foo-pool1-c 1000
>>> gke-foo4 default 10.44.2.0/24
>>> us-central1-f/instances/gke-foo-pool1-d 1000
>>>
>>> I don't see overlaps in the 10.44.* space, but there's something with
>>> 10.240.*.
>>> Removing the "datalab" route didn't change anything,
>>> and I'm also not sure how it would explain the inconsistent behavior
>>> between 10.240.0.2 & 10.240.0.35.
>>>
>>> On Sun, Jul 2, 2017 at 7:21 PM 'Tim Hockin' via Kubernetes user
>>> discussion and Q&A <kubernetes-users@googlegroups.com> wrote:
>>>
>>>> Check for duplicate or overlapping routes in the cloud console?
>>>>
>>>> On Jul 2, 2017 9:14 AM, "Itamar O" <itamar...@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>> I'm investigating a weird routing behavior on our production GKE
>>>>> cluster (nodes & master running 1.6.6), not quite sure how to proceed at
>>>>> this point.
>>>>>
>>>>> The essence of the issue: containers on the GKE cluster can
>>>>> communicate with *some* GCE instances, but *not others*, and there's no
>>>>> (apparent) difference between the accessible and inaccessible GCE
>>>>> instances.
>>>>>
>>>>> More details:
>>>>> All GCE instances and GKE clusters co-exist happily within 10.240.*.*
>>>>> Pods on the GKE cluster are assigned addresses in the 10.44.*.* space.
>>>>> I noticed one of the services on GKE misbehaving, and the logs
>>>>> indicated that it is timing out on accessing a service that we have
>>>>> serving
>>>>> from a GCE instance (at 10.240.0.2 for that matter).
>>>>> I `kubectl exec` into a bash session in that pod, and indeed I saw
>>>>> that `ping 10.240.0.2` is timing out, but `ping 10.240.0.35` is working
>>>>> just fine (10.240.0.35 being another GCE machine, being 100% equivalent to
>>>>> 10.240.0.2 AFAIK).
>>>>> We also have another, older, GKE cluster up ("previous prod"), from
>>>>> which I tried the same pings with no such issues. The differences I can
>>>>> map
>>>>> include the previous cluster running k8s 1.6.2, and pods assigned under
>>>>> 10.88.*.*.
>>>>> I could not notice any meaningful related rule in GCE Routing or FW
>>>>> (they seem to be equivalent between the old and new GKE clusters).
>>>>>
>>>>> Any ideas on how to proceed with investigating this?
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Kubernetes user discussion and Q&A" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to kubernetes-users+unsubscr...@googlegroups.com.
>>>>> To post to this group, send email to kubernetes-users@googlegroups.com
>>>>> .
>>>>> Visit this group at https://groups.google.com/group/kubernetes-users.
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Kubernetes user discussion and Q&A" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to kubernetes-users+unsubscr...@googlegroups.com.
>>>> To post to this group, send email to kubernetes-users@googlegroups.com.
>>>> Visit this group at https://groups.google.com/group/kubernetes-users.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>> --
>> You received this message because you are subscribed to the Google Groups
>> "Kubernetes user discussion and Q&A" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to kubernetes-users+unsubscr...@googlegroups.com.
>> To post to this group, send email to kubernetes-users@googlegroups.com.
>> Visit this group at https://groups.google.com/group/kubernetes-users.
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Kubernetes user discussion and Q&A" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kubernetes-users+unsubscr...@googlegroups.com.
> To post to this group, send email to kubernetes-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/kubernetes-users.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to kubernetes-users+unsubscr...@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.
