This won't be super helpful - it's really hard to debug this sort of thing
remotely - but I will try.
First, I'd look at tcpdump from the root of the client machine to rule out
local latency. I can't fathom how 40-80 ms would be injected by the
iptables or bridging. Best to rule it out.
Second, I'd look at iptables on the server machine. Did the latency
disappear into the network or does it arrive quickly and disappear on the
server.
Third, pay attention to client IP through this. Is the server trying to do
reverse DNS lookups? Is it doing anything strange?
Fourth, if there are still no clues, try installing an iptables rule on the
client, like:
iptables -t nat -I POSTROUTING -j MASQUERADE
This will change the client IP to the VM's IP. Could be network infra
getting an IP it doesn't know?
Fifth, can you repro the case with all other variables removed? Using
'netcat' - no SSL, no auth, no handshakes, etc.
On Thursday, July 20, 2017 at 6:33:07 AM UTC-7, theairkit wrote:
>
> Hi
>
> I've encountered a pod network latency problem.
>
> I run simple mysql query ('time mysql -e "..."') to outer
> (not in k8s cluster) host and got following results:
>
> 1. From k8s-pod to outer host:
> avg time: 60ms
> worst time: 100ms
> (So, running applications in pods has same latency when works to outer
> resources)
>
> 2. From k8s-host (which contains pod above) to outer host:
> avg time: 10ms
> worst time: 20ms
>
> Also:
> - There are no any dependency on DNS: avg and worst times not changed
> when mysql-cli connects to IP-address or hostname of outer host.
> - There are enough free resources: typical k8s node have 56-cores CPU
> with LA ~ 10 (CPU load only and no any I/O load), more than half free
> memory,
> and no threads/processes (user/kernel), which cosume up to 100% cpu time.
>
> As a possible solution I divided CPUs between kernel and user processes
> (via kernel parameter isolcpus and cgroups), but latency did not changes.
>
> For now I perform more complex measurements, and trying to investigate
> which subsytem (k8s, cni, iptables etc.) may leads to this behaviour.
>
> Have you encountered such or similar behavior?
> May you provide any tip for my investigations?
>
> Would be very grateful for your help.
>
>
> My k8s cluster:
>
> kubelet version: 1.5.7
> network type: calico, image: calico/cni:v1.5.5, quay.io/calico/node:v1.0.0
> num of nodes: ~20
> pods per node: ~30
>
> OS: Debian GNU/Linux 8.6 (jessie)
> Kernel: 4.9.0-0.bpo.2-amd64
>
>
> Thanks!
>
--
You received this message because you are subscribed to the Google Groups
"Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to kubernetes-users+unsubscr...@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.
