Just wanna note we're seeing this on our end too. Narrowed it down to being the GCE Ingress by testing our deployment behind the older style LoadBalancer service, where we are seeing sub-200ms responses. Also confirmed Cloudflare wasn't the issue by preventing the host from being resolved via SSL and removed DNS from the equation. At least for us all signs point to GCE Ingress being the problem.
I also tried using what was suggested to Dave and adjusted our IdleTimeout/keepalive options and had no changes. On Monday, May 21, 2018 at 5:59:21 PM UTC-5, Dave Jensen wrote: > The issue appears to be back. For the past few nights I've been seeing > sporadic 4-5 second response times on calls. Again, it's calls like OPTIONS > that really stand out. > > > On Thursday, May 17, 2018 at 11:37:28 AM UTC-7, Dave Jensen wrote: > > Thank you, I think this solved the issue. We set the IdleTimeout (in Golang) > to 620s and, in our staging environment, I have not seen a request take > longer than 200ms. > > On Wednesday, May 16, 2018 at 2:41:32 PM UTC-7, Nicks wrote: > I created an HTTP LB setup on GCP using a golang HTTP server without > kubernetes and was able to see rare long-tail latencies in >1 second. After I > set `IdleTimeout` to larger than ten minutes, I stopped seeing those slow > responses. The echoheaders image uses nginx and doesn't set > `keepalive_timeout` (sent PR to update this). > > > > https://cloud.google.com/compute/docs/load-balancing/http/#timeouts_and_retries > > > > > On Wednesday, May 16, 2018 at 8:08:26 AM UTC-7, ama...@upstream.tech wrote:On > Monday, May 14, 2018 at 5:10:55 PM UTC-4, francois...@polynom.io wrote: > > > Le jeudi 19 avril 2018 02:14:28 UTC+2, Dave Jensen a écrit : > > > > We have what I believe to be a very straightforward ingress setup on GKE. > > > However, we started noticing random slowdowns almost immediately. On > > > further investigation it looked like the time to first byte (TTFB) was > > > very slow (1 - 3 seconds) but sporadically. Sometimes it would be a > > > pre-flight OPTIONS request, sometimes an application request, and other > > > times on static files. Even the echoserver would sporadically have a long > > > TTFB. > > > > > > > > > > > > I set up a portforward to one of the pods serving up our REST API server. > > > Sure enough, the slowdown was eliminated. > > > > > > > > > > > > Before I go down the rabbit hole of trying other ingress controllers, I > > > figured I'd ask the community if I was doing something wrong. > > > > > > > > > > > > ingress.yaml > > > > > > > > > > > > > > > > > > > > > > > > apiVersion: extensions/v1beta1 > > > > kind: Ingress > > > > metadata: > > > > name: brewd-ingress > > > > spec: > > > > tls: > > > > - hosts: > > > > - stage-api2.example.com > > > > - stage-app2.example.com > > > > - stage-echo.example.com > > > > secretName: redacted > > > > rules: > > > > - host: stage-api2.example.com > > > > http: > > > > paths: > > > > - backend: > > > > serviceName: gateway-service > > > > servicePort: 7000 > > > > - host: stage-app2.example.com > > > > http: > > > > paths: > > > > - backend: > > > > serviceName: web-service > > > > servicePort: 8080 > > > > - host: stage-echo.example.com > > > > http: > > > > paths: > > > > - backend: > > > > serviceName: echoserver > > > > servicePort: 8080 > > > > gateway-service.yaml > > > > > > > > > > > > > > > > apiVersion: v1 > > > > kind: Service > > > > metadata: > > > > name: gateway-service > > > > labels: > > > > app: gateway > > > > spec: > > > > type: NodePort > > > > ports: > > > > - port: 7000 > > > > selector: > > > > app: gateway > > > > --- > > > > apiVersion: apps/v1beta2 > > > > kind: Deployment > > > > metadata: > > > > name: gateway-deployment > > > > spec: > > > > selector: > > > > matchLabels: > > > > app: gateway > > > > replicas: 1 > > > > template: > > > > metadata: > > > > labels: > > > > app: gateway > > > > spec: > > > > containers: > > > > - name: gateway > > > > image: us.gcr.io/redacted/gateway:1.3.0-rc.0 > > > > imagePullPolicy: Always > > > > ports: > > > > - containerPort: 7000 > > > > env: > > > > - name: REDACTED_ENV > > > > value: stage > > > > The web-service yaml looks almost exactly the same as the above. > > > > > > Hi guyz, I have the exact same problem. If I deploy the service as LB, it > > is blazing fast, but if I use the basic ingress on Google Ingress, I have > > random latency whith response time ranging from 50ms to 5s. > > > > Hi Francois, > > > > We are having a similar issue. Could you give more information on how you > deployed as a LB? Want to get that setup while we continue looking for a > solution to this. -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
