Re: [kubernetes-users] Re: Slow response times using default Ingress on GKE
This looks something to do with google load balancer or the kube-proxy. I setup a google Http load balancer pointing to the NodePort and see the same issue. If I open up the service as LoadBalancer and access it directly I don’t see any issues. On 5/21/18, 10:07 PM, "kubernetes-users@googlegroups.com on behalf of rmur...@texastribune.org" wrote: Just wanna note we're seeing this on our end too. Narrowed it down to being the GCE Ingress by testing our deployment behind the older style LoadBalancer service, where we are seeing sub-200ms responses. Also confirmed Cloudflare wasn't the issue by preventing the host from being resolved via SSL and removed DNS from the equation. At least for us all signs point to GCE Ingress being the problem. I also tried using what was suggested to Dave and adjusted our IdleTimeout/keepalive options and had no changes. On Monday, May 21, 2018 at 5:59:21 PM UTC-5, Dave Jensen wrote: > The issue appears to be back. For the past few nights I've been seeing sporadic 4-5 second response times on calls. Again, it's calls like OPTIONS that really stand out. > > > On Thursday, May 17, 2018 at 11:37:28 AM UTC-7, Dave Jensen wrote: > > Thank you, I think this solved the issue. We set the IdleTimeout (in Golang) to 620s and, in our staging environment, I have not seen a request take longer than 200ms. > > On Wednesday, May 16, 2018 at 2:41:32 PM UTC-7, Nicks wrote: > I created an HTTP LB setup on GCP using a golang HTTP server without kubernetes and was able to see rare long-tail latencies in >1 second. After I set `IdleTimeout` to larger than ten minutes, I stopped seeing those slow responses. The echoheaders image uses nginx and doesn't set `keepalive_timeout` (sent PR to update this). > > > > https://cloud.google.com/compute/docs/load-balancing/http/#timeouts_and_retries > > > > > On Wednesday, May 16, 2018 at 8:08:26 AM UTC-7, ama...@upstream.tech wrote:On Monday, May 14, 2018 at 5:10:55 PM UTC-4, francois...@polynom.io wrote: > > > Le jeudi 19 avril 2018 02:14:28 UTC+2, Dave Jensen a écrit : > > > > We have what I believe to be a very straightforward ingress setup on GKE. However, we started noticing random slowdowns almost immediately. On further investigation it looked like the time to first byte (TTFB) was very slow (1 - 3 seconds) but sporadically. Sometimes it would be a pre-flight OPTIONS request, sometimes an application request, and other times on static files. Even the echoserver would sporadically have a long TTFB. > > > > > > > > > > > > I set up a portforward to one of the pods serving up our REST API server. Sure enough, the slowdown was eliminated. > > > > > > > > > > > > Before I go down the rabbit hole of trying other ingress controllers, I figured I'd ask the community if I was doing something wrong. > > > > > > > > > > > > ingress.yaml > > > > > > > > > > > > > > > > > > > > > > > > apiVersion: extensions/v1beta1 > > > > kind: Ingress > > > > metadata: > > > > name: brewd-ingress > > > > spec: > > > > tls: > > > > - hosts: > > > > - stage-api2.example.com > > > > - stage-app2.example.com > > > > - stage-echo.example.com > > > > secretName: redacted > > > > rules: > > > > - host: stage-api2.example.com > > > > http: > > > > paths: > > > > - backend: > > > > serviceName: gateway-service > > > > servicePort: 7000 > > > > - host: stage-app2.example.com > > > > http: > > > > paths: > > > > - backend: > > > > serviceName: web-service > > > > servicePort: 8080 > > > > - host: stage-echo.example.com > > > > http: > > > > paths: > > > > - backend: > > > > serviceName: echoserver > > > > servicePort: 8080 > > > > gateway-service.yaml > > > > > > > > > > > > > > > > apiVersion: v1 > > > > kind: Service > > > > metadata: > > > > name: gateway-service > > > > labels: > > > > app: gateway > > > > spec: > > > > type: NodePort > > > > ports: > > > > - port: 7000 > > > > selector: > > > > app: gateway > > > > --- > > > > apiVersion: apps/v1beta2 > > > > kind: Deployment > > > > metadata: > > > > name: gateway-deployment > > > > spec: > > > > selector:
[kubernetes-users] Re: Slow response times using default Ingress on GKE
Just wanna note we're seeing this on our end too. Narrowed it down to being the GCE Ingress by testing our deployment behind the older style LoadBalancer service, where we are seeing sub-200ms responses. Also confirmed Cloudflare wasn't the issue by preventing the host from being resolved via SSL and removed DNS from the equation. At least for us all signs point to GCE Ingress being the problem. I also tried using what was suggested to Dave and adjusted our IdleTimeout/keepalive options and had no changes. On Monday, May 21, 2018 at 5:59:21 PM UTC-5, Dave Jensen wrote: > The issue appears to be back. For the past few nights I've been seeing > sporadic 4-5 second response times on calls. Again, it's calls like OPTIONS > that really stand out. > > > On Thursday, May 17, 2018 at 11:37:28 AM UTC-7, Dave Jensen wrote: > > Thank you, I think this solved the issue. We set the IdleTimeout (in Golang) > to 620s and, in our staging environment, I have not seen a request take > longer than 200ms. > > On Wednesday, May 16, 2018 at 2:41:32 PM UTC-7, Nicks wrote: > I created an HTTP LB setup on GCP using a golang HTTP server without > kubernetes and was able to see rare long-tail latencies in >1 second. After I > set `IdleTimeout` to larger than ten minutes, I stopped seeing those slow > responses. The echoheaders image uses nginx and doesn't set > `keepalive_timeout` (sent PR to update this). > > > > https://cloud.google.com/compute/docs/load-balancing/http/#timeouts_and_retries > > > > > On Wednesday, May 16, 2018 at 8:08:26 AM UTC-7, ama...@upstream.tech wrote:On > Monday, May 14, 2018 at 5:10:55 PM UTC-4, francois...@polynom.io wrote: > > > Le jeudi 19 avril 2018 02:14:28 UTC+2, Dave Jensen a écrit : > > > > We have what I believe to be a very straightforward ingress setup on GKE. > > > However, we started noticing random slowdowns almost immediately. On > > > further investigation it looked like the time to first byte (TTFB) was > > > very slow (1 - 3 seconds) but sporadically. Sometimes it would be a > > > pre-flight OPTIONS request, sometimes an application request, and other > > > times on static files. Even the echoserver would sporadically have a long > > > TTFB. > > > > > > > > > > > > I set up a portforward to one of the pods serving up our REST API server. > > > Sure enough, the slowdown was eliminated. > > > > > > > > > > > > Before I go down the rabbit hole of trying other ingress controllers, I > > > figured I'd ask the community if I was doing something wrong. > > > > > > > > > > > > ingress.yaml > > > > > > > > > > > > > > > > > > > > > > > > apiVersion: extensions/v1beta1 > > > > kind: Ingress > > > > metadata: > > > > name: brewd-ingress > > > > spec: > > > > tls: > > > > - hosts: > > > > - stage-api2.example.com > > > > - stage-app2.example.com > > > > - stage-echo.example.com > > > > secretName: redacted > > > > rules: > > > > - host: stage-api2.example.com > > > > http: > > > > paths: > > > > - backend: > > > > serviceName: gateway-service > > > > servicePort: 7000 > > > > - host: stage-app2.example.com > > > > http: > > > > paths: > > > > - backend: > > > > serviceName: web-service > > > > servicePort: 8080 > > > > - host: stage-echo.example.com > > > > http: > > > > paths: > > > > - backend: > > > > serviceName: echoserver > > > > servicePort: 8080 > > > > gateway-service.yaml > > > > > > > > > > > > > > > > apiVersion: v1 > > > > kind: Service > > > > metadata: > > > > name: gateway-service > > > > labels: > > > > app: gateway > > > > spec: > > > > type: NodePort > > > > ports: > > > > - port: 7000 > > > > selector: > > > > app: gateway > > > > --- > > > > apiVersion: apps/v1beta2 > > > > kind: Deployment > > > > metadata: > > > > name: gateway-deployment > > > > spec: > > > > selector: > > > > matchLabels: > > > > app: gateway > > > > replicas: 1 > > > > template: > > > > metadata: > > > > labels: > > > > app: gateway > > > > spec: > > > > containers: > > > > - name: gateway > > > > image: us.gcr.io/redacted/gateway:1.3.0-rc.0 > > > > imagePullPolicy: Always > > > > ports: > > > > - containerPort: 7000 > > > > env: > > > > - name: REDACTED_ENV > > > > value: stage > > > > The web-service yaml looks almost exactly the same as the above. > > > > > > Hi guyz, I have the exact same problem. If I deploy the service as LB, it > > is blazing fast, but if I use the basic ingress on Google Ingress, I have > > random latency whith response time ranging from 50ms to 5s. > > > > Hi Francois, > > > > We are having a similar issue. Could you give more information on how you > deployed a
[kubernetes-users] Re: Slow response times using default Ingress on GKE
The issue appears to be back. For the past few nights I've been seeing sporadic 4-5 second response times on calls. Again, it's calls like OPTIONS that really stand out. On Thursday, May 17, 2018 at 11:37:28 AM UTC-7, Dave Jensen wrote: > Thank you, I think this solved the issue. We set the IdleTimeout (in > Golang) to 620s and, in our staging environment, I have not seen a request > take longer than 200ms. > > On Wednesday, May 16, 2018 at 2:41:32 PM UTC-7, Nicks wrote: >> >> I created an HTTP LB setup on GCP using a golang HTTP server without >> kubernetes and was able to see rare long-tail latencies in >1 second. After >> I set `IdleTimeout` to larger than ten minutes, I stopped seeing those slow >> responses. The echoheaders image uses nginx and doesn't set >> `keepalive_timeout` (sent PR to update this). >> >> >> https://cloud.google.com/compute/docs/load-balancing/http/#timeouts_and_retries >> >> >> On Wednesday, May 16, 2018 at 8:08:26 AM UTC-7, ama...@upstream.tech >> wrote: >>> >>> On Monday, May 14, 2018 at 5:10:55 PM UTC-4, francois...@polynom.io >>> wrote: >>> > Le jeudi 19 avril 2018 02:14:28 UTC+2, Dave Jensen a écrit : >>> > > We have what I believe to be a very straightforward ingress setup on >>> GKE. However, we started noticing random slowdowns almost immediately. On >>> further investigation it looked like the time to first byte (TTFB) was very >>> slow (1 - 3 seconds) but sporadically. Sometimes it would be a pre-flight >>> OPTIONS request, sometimes an application request, and other times on >>> static files. Even the echoserver would sporadically have a long TTFB. >>> > > >>> > > >>> > > I set up a portforward to one of the pods serving up our REST API >>> server. Sure enough, the slowdown was eliminated. >>> > > >>> > > >>> > > Before I go down the rabbit hole of trying other ingress >>> controllers, I figured I'd ask the community if I was doing something >>> wrong. >>> > > >>> > > >>> > > ingress.yaml >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > apiVersion: extensions/v1beta1 >>> > > kind: Ingress >>> > > metadata: >>> > > name: brewd-ingress >>> > > spec: >>> > > tls: >>> > > - hosts: >>> > > - stage-api2.example.com >>> > > - stage-app2.example.com >>> > > - stage-echo.example.com >>> > > secretName: redacted >>> > > rules: >>> > > - host: stage-api2.example.com >>> > > http: >>> > > paths: >>> > > - backend: >>> > > serviceName: gateway-service >>> > > servicePort: 7000 >>> > > - host: stage-app2.example.com >>> > > http: >>> > > paths: >>> > > - backend: >>> > > serviceName: web-service >>> > > servicePort: 8080 >>> > > - host: stage-echo.example.com >>> > > http: >>> > > paths: >>> > > - backend: >>> > > serviceName: echoserver >>> > > servicePort: 8080 >>> > > gateway-service.yaml >>> > > >>> > > >>> > > >>> > > apiVersion: v1 >>> > > kind: Service >>> > > metadata: >>> > > name: gateway-service >>> > > labels: >>> > > app: gateway >>> > > spec: >>> > > type: NodePort >>> > > ports: >>> > > - port: 7000 >>> > > selector: >>> > > app: gateway >>> > > --- >>> > > apiVersion: apps/v1beta2 >>> > > kind: Deployment >>> > > metadata: >>> > > name: gateway-deployment >>> > > spec: >>> > > selector: >>> > > matchLabels: >>> > > app: gateway >>> > > replicas: 1 >>> > > template: >>> > > metadata: >>> > > labels: >>> > > app: gateway >>> > > spec: >>> > > containers: >>> > > - name: gateway >>> > > image: us.gcr.io/redacted/gateway:1.3.0-rc.0 >>> > > imagePullPolicy: Always >>> > > ports: >>> > > - containerPort: 7000 >>> > > env: >>> > > - name: REDACTED_ENV >>> > > value: stage >>> > > The web-service yaml looks almost exactly the same as the above. >>> > >>> > Hi guyz, I have the exact same problem. If I deploy the service as LB, >>> it is blazing fast, but if I use the basic ingress on Google Ingress, I >>> have random latency whith response time ranging from 50ms to 5s. >>> >>> Hi Francois, >>> >>> We are having a similar issue. Could you give more information on how >>> you deployed as a LB? Want to get that setup while we continue looking for >>> a solution to this. >> >> -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
[kubernetes-users] Re: Slow response times using default Ingress on GKE
Thank you, I think this solved the issue. We set the IdleTimeout (in Golang) to 620s and, in our staging environment, I have not seen a request take longer than 200ms. On Wednesday, May 16, 2018 at 2:41:32 PM UTC-7, Nicks wrote: > > I created an HTTP LB setup on GCP using a golang HTTP server without > kubernetes and was able to see rare long-tail latencies in >1 second. After > I set `IdleTimeout` to larger than ten minutes, I stopped seeing those slow > responses. The echoheaders image uses nginx and doesn't set > `keepalive_timeout` (sent PR to update this). > > > https://cloud.google.com/compute/docs/load-balancing/http/#timeouts_and_retries > > > On Wednesday, May 16, 2018 at 8:08:26 AM UTC-7, ama...@upstream.tech wrote: >> >> On Monday, May 14, 2018 at 5:10:55 PM UTC-4, francois...@polynom.io >> wrote: >> > Le jeudi 19 avril 2018 02:14:28 UTC+2, Dave Jensen a écrit : >> > > We have what I believe to be a very straightforward ingress setup on >> GKE. However, we started noticing random slowdowns almost immediately. On >> further investigation it looked like the time to first byte (TTFB) was very >> slow (1 - 3 seconds) but sporadically. Sometimes it would be a pre-flight >> OPTIONS request, sometimes an application request, and other times on >> static files. Even the echoserver would sporadically have a long TTFB. >> > > >> > > >> > > I set up a portforward to one of the pods serving up our REST API >> server. Sure enough, the slowdown was eliminated. >> > > >> > > >> > > Before I go down the rabbit hole of trying other ingress controllers, >> I figured I'd ask the community if I was doing something wrong. >> > > >> > > >> > > ingress.yaml >> > > >> > > >> > > >> > > >> > > >> > > apiVersion: extensions/v1beta1 >> > > kind: Ingress >> > > metadata: >> > > name: brewd-ingress >> > > spec: >> > > tls: >> > > - hosts: >> > > - stage-api2.example.com >> > > - stage-app2.example.com >> > > - stage-echo.example.com >> > > secretName: redacted >> > > rules: >> > > - host: stage-api2.example.com >> > > http: >> > > paths: >> > > - backend: >> > > serviceName: gateway-service >> > > servicePort: 7000 >> > > - host: stage-app2.example.com >> > > http: >> > > paths: >> > > - backend: >> > > serviceName: web-service >> > > servicePort: 8080 >> > > - host: stage-echo.example.com >> > > http: >> > > paths: >> > > - backend: >> > > serviceName: echoserver >> > > servicePort: 8080 >> > > gateway-service.yaml >> > > >> > > >> > > >> > > apiVersion: v1 >> > > kind: Service >> > > metadata: >> > > name: gateway-service >> > > labels: >> > > app: gateway >> > > spec: >> > > type: NodePort >> > > ports: >> > > - port: 7000 >> > > selector: >> > > app: gateway >> > > --- >> > > apiVersion: apps/v1beta2 >> > > kind: Deployment >> > > metadata: >> > > name: gateway-deployment >> > > spec: >> > > selector: >> > > matchLabels: >> > > app: gateway >> > > replicas: 1 >> > > template: >> > > metadata: >> > > labels: >> > > app: gateway >> > > spec: >> > > containers: >> > > - name: gateway >> > > image: us.gcr.io/redacted/gateway:1.3.0-rc.0 >> > > imagePullPolicy: Always >> > > ports: >> > > - containerPort: 7000 >> > > env: >> > > - name: REDACTED_ENV >> > > value: stage >> > > The web-service yaml looks almost exactly the same as the above. >> > >> > Hi guyz, I have the exact same problem. If I deploy the service as LB, >> it is blazing fast, but if I use the basic ingress on Google Ingress, I >> have random latency whith response time ranging from 50ms to 5s. >> >> Hi Francois, >> >> We are having a similar issue. Could you give more information on how you >> deployed as a LB? Want to get that setup while we continue looking for a >> solution to this. > > -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
[kubernetes-users] Re: Slow response times using default Ingress on GKE
I created an HTTP LB setup on GCP using a golang HTTP server without kubernetes and was able to see rare long-tail latencies in >1 second. After I set `IdleTimeout` to larger than ten minutes, I stopped seeing those slow responses. The echoheaders image uses nginx and doesn't set `keepalive_timeout` (sent PR to update this). https://cloud.google.com/compute/docs/load-balancing/http/#timeouts_and_retries On Wednesday, May 16, 2018 at 8:08:26 AM UTC-7, ama...@upstream.tech wrote: > > On Monday, May 14, 2018 at 5:10:55 PM UTC-4, francois...@polynom.io > wrote: > > Le jeudi 19 avril 2018 02:14:28 UTC+2, Dave Jensen a écrit : > > > We have what I believe to be a very straightforward ingress setup on > GKE. However, we started noticing random slowdowns almost immediately. On > further investigation it looked like the time to first byte (TTFB) was very > slow (1 - 3 seconds) but sporadically. Sometimes it would be a pre-flight > OPTIONS request, sometimes an application request, and other times on > static files. Even the echoserver would sporadically have a long TTFB. > > > > > > > > > I set up a portforward to one of the pods serving up our REST API > server. Sure enough, the slowdown was eliminated. > > > > > > > > > Before I go down the rabbit hole of trying other ingress controllers, > I figured I'd ask the community if I was doing something wrong. > > > > > > > > > ingress.yaml > > > > > > > > > > > > > > > > > > apiVersion: extensions/v1beta1 > > > kind: Ingress > > > metadata: > > > name: brewd-ingress > > > spec: > > > tls: > > > - hosts: > > > - stage-api2.example.com > > > - stage-app2.example.com > > > - stage-echo.example.com > > > secretName: redacted > > > rules: > > > - host: stage-api2.example.com > > > http: > > > paths: > > > - backend: > > > serviceName: gateway-service > > > servicePort: 7000 > > > - host: stage-app2.example.com > > > http: > > > paths: > > > - backend: > > > serviceName: web-service > > > servicePort: 8080 > > > - host: stage-echo.example.com > > > http: > > > paths: > > > - backend: > > > serviceName: echoserver > > > servicePort: 8080 > > > gateway-service.yaml > > > > > > > > > > > > apiVersion: v1 > > > kind: Service > > > metadata: > > > name: gateway-service > > > labels: > > > app: gateway > > > spec: > > > type: NodePort > > > ports: > > > - port: 7000 > > > selector: > > > app: gateway > > > --- > > > apiVersion: apps/v1beta2 > > > kind: Deployment > > > metadata: > > > name: gateway-deployment > > > spec: > > > selector: > > > matchLabels: > > > app: gateway > > > replicas: 1 > > > template: > > > metadata: > > > labels: > > > app: gateway > > > spec: > > > containers: > > > - name: gateway > > > image: us.gcr.io/redacted/gateway:1.3.0-rc.0 > > > imagePullPolicy: Always > > > ports: > > > - containerPort: 7000 > > > env: > > > - name: REDACTED_ENV > > > value: stage > > > The web-service yaml looks almost exactly the same as the above. > > > > Hi guyz, I have the exact same problem. If I deploy the service as LB, > it is blazing fast, but if I use the basic ingress on Google Ingress, I > have random latency whith response time ranging from 50ms to 5s. > > Hi Francois, > > We are having a similar issue. Could you give more information on how you > deployed as a LB? Want to get that setup while we continue looking for a > solution to this. -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
[kubernetes-users] Re: Slow response times using default Ingress on GKE
On Monday, May 14, 2018 at 5:10:55 PM UTC-4, francois...@polynom.io wrote: > Le jeudi 19 avril 2018 02:14:28 UTC+2, Dave Jensen a écrit : > > We have what I believe to be a very straightforward ingress setup on GKE. > > However, we started noticing random slowdowns almost immediately. On > > further investigation it looked like the time to first byte (TTFB) was very > > slow (1 - 3 seconds) but sporadically. Sometimes it would be a pre-flight > > OPTIONS request, sometimes an application request, and other times on > > static files. Even the echoserver would sporadically have a long TTFB. > > > > > > I set up a portforward to one of the pods serving up our REST API server. > > Sure enough, the slowdown was eliminated. > > > > > > Before I go down the rabbit hole of trying other ingress controllers, I > > figured I'd ask the community if I was doing something wrong. > > > > > > ingress.yaml > > > > > > > > > > > > apiVersion: extensions/v1beta1 > > kind: Ingress > > metadata: > > name: brewd-ingress > > spec: > > tls: > > - hosts: > > - stage-api2.example.com > > - stage-app2.example.com > > - stage-echo.example.com > > secretName: redacted > > rules: > > - host: stage-api2.example.com > > http: > > paths: > > - backend: > > serviceName: gateway-service > > servicePort: 7000 > > - host: stage-app2.example.com > > http: > > paths: > > - backend: > > serviceName: web-service > > servicePort: 8080 > > - host: stage-echo.example.com > > http: > > paths: > > - backend: > > serviceName: echoserver > > servicePort: 8080 > > gateway-service.yaml > > > > > > > > apiVersion: v1 > > kind: Service > > metadata: > > name: gateway-service > > labels: > > app: gateway > > spec: > > type: NodePort > > ports: > > - port: 7000 > > selector: > > app: gateway > > --- > > apiVersion: apps/v1beta2 > > kind: Deployment > > metadata: > > name: gateway-deployment > > spec: > > selector: > > matchLabels: > > app: gateway > > replicas: 1 > > template: > > metadata: > > labels: > > app: gateway > > spec: > > containers: > > - name: gateway > > image: us.gcr.io/redacted/gateway:1.3.0-rc.0 > > imagePullPolicy: Always > > ports: > > - containerPort: 7000 > > env: > > - name: REDACTED_ENV > > value: stage > > The web-service yaml looks almost exactly the same as the above. > > Hi guyz, I have the exact same problem. If I deploy the service as LB, it is > blazing fast, but if I use the basic ingress on Google Ingress, I have random > latency whith response time ranging from 50ms to 5s. Hi Francois, We are having a similar issue. Could you give more information on how you deployed as a LB? Want to get that setup while we continue looking for a solution to this. -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
[kubernetes-users] Re: Slow response times using default Ingress on GKE
Le jeudi 19 avril 2018 02:14:28 UTC+2, Dave Jensen a écrit : > We have what I believe to be a very straightforward ingress setup on GKE. > However, we started noticing random slowdowns almost immediately. On further > investigation it looked like the time to first byte (TTFB) was very slow (1 - > 3 seconds) but sporadically. Sometimes it would be a pre-flight OPTIONS > request, sometimes an application request, and other times on static files. > Even the echoserver would sporadically have a long TTFB. > > > I set up a portforward to one of the pods serving up our REST API server. > Sure enough, the slowdown was eliminated. > > > Before I go down the rabbit hole of trying other ingress controllers, I > figured I'd ask the community if I was doing something wrong. > > > ingress.yaml > > > > > > apiVersion: extensions/v1beta1 > kind: Ingress > metadata: > name: brewd-ingress > spec: > tls: > - hosts: > - stage-api2.example.com > - stage-app2.example.com > - stage-echo.example.com > secretName: redacted > rules: > - host: stage-api2.example.com > http: > paths: > - backend: > serviceName: gateway-service > servicePort: 7000 > - host: stage-app2.example.com > http: > paths: > - backend: > serviceName: web-service > servicePort: 8080 > - host: stage-echo.example.com > http: > paths: > - backend: > serviceName: echoserver > servicePort: 8080 > gateway-service.yaml > > > > apiVersion: v1 > kind: Service > metadata: > name: gateway-service > labels: > app: gateway > spec: > type: NodePort > ports: > - port: 7000 > selector: > app: gateway > --- > apiVersion: apps/v1beta2 > kind: Deployment > metadata: > name: gateway-deployment > spec: > selector: > matchLabels: > app: gateway > replicas: 1 > template: > metadata: > labels: > app: gateway > spec: > containers: > - name: gateway > image: us.gcr.io/redacted/gateway:1.3.0-rc.0 > imagePullPolicy: Always > ports: > - containerPort: 7000 > env: > - name: REDACTED_ENV > value: stage > The web-service yaml looks almost exactly the same as the above. Hi guyz, I have the exact same problem. If I deploy the service as LB, it is blazing fast, but if I use the basic ingress on Google Ingress, I have random latency whith response time ranging from 50ms to 5s. -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
[kubernetes-users] Re: Slow response times using default Ingress on GKE
Hi Dave, What version of GKE (master) are you currently using? Is there a URL that can be tested/probed? (i.e. repro case)? As a start, it would be useful to post an anonymized `tcpdump` of the HTTP session that exhibits the delay, the output of curl with timing (something like: https://stackoverflow.com/questions/18215389/how-do-i-measure-request-and-response-times-at-once-using-curl), or the output of the tool you are using to measure TTFB. Bowei On Wednesday, April 18, 2018 at 5:14:28 PM UTC-7, Dave Jensen wrote: > > We have what I believe to be a very straightforward ingress setup on GKE. > However, we started noticing random slowdowns almost immediately. On > further investigation it looked like the time to first byte (TTFB) was very > slow (1 - 3 seconds) but sporadically. Sometimes it would be a pre-flight > OPTIONS request, sometimes an application request, and other times on > static files. Even the echoserver would sporadically have a long TTFB. > > I set up a portforward to one of the pods serving up our REST API server. > Sure enough, the slowdown was eliminated. > > Before I go down the rabbit hole of trying other ingress controllers, I > figured I'd ask the community if I was doing something wrong. > > ingress.yaml > > apiVersion: extensions/v1beta1 > kind: Ingress > metadata: > name: brewd-ingress > spec: > tls: > - hosts: > - stage-api2.example.com > - stage-app2.example.com > - stage-echo.example.com > secretName: redacted > rules: > - host: stage-api2.example.com > http: > paths: > - backend: > serviceName: gateway-service > servicePort: 7000 > - host: stage-app2.example.com > http: > paths: > - backend: > serviceName: web-service > servicePort: 8080 > - host: stage-echo.example.com > http: > paths: > - backend: > serviceName: echoserver > servicePort: 8080 > > gateway-service.yaml > apiVersion: v1 > kind: Service > metadata: > name: gateway-service > labels: > app: gateway > spec: > type: NodePort > ports: > - port: 7000 > selector: > app: gateway > --- > apiVersion: apps/v1beta2 > kind: Deployment > metadata: > name: gateway-deployment > spec: > selector: > matchLabels: > app: gateway > replicas: 1 > template: > metadata: > labels: > app: gateway > spec: > containers: > - name: gateway > image: us.gcr.io/redacted/gateway:1.3.0-rc.0 > imagePullPolicy: Always > ports: > - containerPort: 7000 > env: > - name: REDACTED_ENV > value: stage > > The web-service yaml looks almost exactly the same as the above. > -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
[kubernetes-users] Re: Slow response times using default Ingress on GKE
Hi Dave, we are experiencing identical conditions with a very similar setup. Please let us know if you learn anything and we will do the same. -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
[kubernetes-users] Re: Slow response times using default Ingress on GKE
> Have you tried using the service nodeport? You can do that and send traffic to one node, this will be round robin to all pods anyways. To see if it happens too. I just tried a portforward to the service (instead of a single pod) and it's blazing fast. Either the GCP LB is bad or the ingress-gce is bad. It's just super frustrating because no matter what channel I take, there is no support. I realize this is the Kubernetes mailing list but there is no dedicated GCP mailing list all the GCP docs point here. Given our super simple setup ... by following tutorials, something is wrong. On Tuesday, May 1, 2018 at 4:08:48 PM UTC-7, Dave Jensen wrote: > > Hello fa..., > > We followed the tutorial [3] and it technically works but it does not > perform to expectations. > > Dave > > On Sunday, April 22, 2018 at 11:15:59 AM UTC-7, fa...@google.com wrote: >> >> >> Hello Dave, >> >> Since your question is technical, you may get help from community experts >> and enthusiasts at serverfault.com. Make sure you include proper tags >> when you ask your question. For example, if using Google Cloud Kubernetes >> Engine [1] you may tag it as [google-kubernetes-engine], and [kubernetes]; >> but if using a Kubernetes cluster on Compute Engine, you may tag it as >> [kubernetes], and [google-compute-engine]. >> >> On the other hand, while I was checking the Ingress prerequisites [2], I >> understood that “Google Kubernetes Engine deploys an ingress controller on >> the master” for you, and like this tutorial [3] it creates the load >> balancer. According to the prerequisites, and this github page [4], it is >> still in beta, and you may check the limitations and expectations [5] with >> latency on the same page. >> That said, and if you suspect a defect with the Ingress Controller/ >> Google Cloud Load Balancer you may open a report through issue tracker [6]. >> But after verifying the correct settings with the community, and reviewing >> the above as not an expected behavior. I hope that helps. >> >> [1] https://console.cloud.google.com/kubernetes >> [2] >> https://kubernetes.io/docs/concepts/services-networking/ingress/#prerequisites >> >> [3] >> https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer >> [4] >> https://github.com/kubernetes/ingress-gce/blob/master/BETA_LIMITATIONS.md#glbc-beta-limitations >> >> [5] >> https://github.com/kubernetes/ingress-gce/blob/master/BETA_LIMITATIONS.md#latency >> >> [6] https://cloud.google.com/support/docs/issue-trackers > > -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
[kubernetes-users] Re: Slow response times using default Ingress on GKE
Hello fa..., We followed the tutorial [3] and it technically works but it does not perform to expectations. Dave On Sunday, April 22, 2018 at 11:15:59 AM UTC-7, fa...@google.com wrote: > > > Hello Dave, > > Since your question is technical, you may get help from community experts > and enthusiasts at serverfault.com. Make sure you include proper tags > when you ask your question. For example, if using Google Cloud Kubernetes > Engine [1] you may tag it as [google-kubernetes-engine], and [kubernetes]; > but if using a Kubernetes cluster on Compute Engine, you may tag it as > [kubernetes], and [google-compute-engine]. > > On the other hand, while I was checking the Ingress prerequisites [2], I > understood that “Google Kubernetes Engine deploys an ingress controller on > the master” for you, and like this tutorial [3] it creates the load > balancer. According to the prerequisites, and this github page [4], it is > still in beta, and you may check the limitations and expectations [5] with > latency on the same page. > That said, and if you suspect a defect with the Ingress Controller/ Google > Cloud Load Balancer you may open a report through issue tracker [6]. But > after verifying the correct settings with the community, and reviewing the > above as not an expected behavior. I hope that helps. > > [1] https://console.cloud.google.com/kubernetes > [2] > https://kubernetes.io/docs/concepts/services-networking/ingress/#prerequisites > > [3] > https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer > [4] > https://github.com/kubernetes/ingress-gce/blob/master/BETA_LIMITATIONS.md#glbc-beta-limitations > > [5] > https://github.com/kubernetes/ingress-gce/blob/master/BETA_LIMITATIONS.md#latency > > [6] https://cloud.google.com/support/docs/issue-trackers -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
Re: [kubernetes-users] Re: Slow response times using default Ingress on GKE
The @GCPcloud twitter account said that I would receive a follow-up from support early last week. There has been no response from them. I've decided to file an issue on the github repo for the project. On Monday, April 23, 2018 at 9:57:26 AM UTC-7, Ahmet Alp Balkan wrote: > > I suspect this issue is rooted in the GCP infrastructure and does not have > much to do with the open source part of the system. There's a good #gke > community on kubernetes slack channel, however I don't think anyone will be > able to help with this particular issue. > > I recommend reaching out to the Google Cloud Platform Support. They should > be able to help by routing your issue through the right channels. > > I also recommend you to come up with a good way to reproduce and measure > the TTFB and provide some endpoints that Google folks can reproduce on > their end. To measure tail latency etc. You can use tools like Apache AB or > https://github.com/rakyll/hey to do send some load and see the tail > latencies. > > On Sun, Apr 22, 2018 at 11:16 AM fadya via Kubernetes user discussion and > Q&A > wrote: > >> >> Hello Dave, >> >> Since your question is technical, you may get help from community experts >> and enthusiasts at serverfault.com. Make sure you include proper tags >> when you ask your question. For example, if using Google Cloud Kubernetes >> Engine [1] you may tag it as [google-kubernetes-engine], and [kubernetes]; >> but if using a Kubernetes cluster on Compute Engine, you may tag it as >> [kubernetes], and [google-compute-engine]. >> >> On the other hand, while I was checking the Ingress prerequisites [2], I >> understood that “Google Kubernetes Engine deploys an ingress controller on >> the master” for you, and like this tutorial [3] it creates the load >> balancer. According to the prerequisites, and this github page [4], it is >> still in beta, and you may check the limitations and expectations [5] with >> latency on the same page. >> That said, and if you suspect a defect with the Ingress Controller/ >> Google Cloud Load Balancer you may open a report through issue tracker [6]. >> But after verifying the correct settings with the community, and reviewing >> the above as not an expected behavior. I hope that helps. >> >> [1] https://console.cloud.google.com/kubernetes >> [2] >> https://kubernetes.io/docs/concepts/services-networking/ingress/#prerequisites >> [3] >> https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer >> [4] >> https://github.com/kubernetes/ingress-gce/blob/master/BETA_LIMITATIONS.md#glbc-beta-limitations >> [5] >> https://github.com/kubernetes/ingress-gce/blob/master/BETA_LIMITATIONS.md#latency >> [6] https://cloud.google.com/support/docs/issue-trackers >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Kubernetes user discussion and Q&A" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to kubernetes-use...@googlegroups.com . >> To post to this group, send email to kubernet...@googlegroups.com >> . >> Visit this group at https://groups.google.com/group/kubernetes-users. >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
Re: [kubernetes-users] Re: Slow response times using default Ingress on GKE
I suspect this issue is rooted in the GCP infrastructure and does not have much to do with the open source part of the system. There's a good #gke community on kubernetes slack channel, however I don't think anyone will be able to help with this particular issue. I recommend reaching out to the Google Cloud Platform Support. They should be able to help by routing your issue through the right channels. I also recommend you to come up with a good way to reproduce and measure the TTFB and provide some endpoints that Google folks can reproduce on their end. To measure tail latency etc. You can use tools like Apache AB or https://github.com/rakyll/hey to do send some load and see the tail latencies. On Sun, Apr 22, 2018 at 11:16 AM fadya via Kubernetes user discussion and Q&A wrote: > > Hello Dave, > > Since your question is technical, you may get help from community experts > and enthusiasts at serverfault.com. Make sure you include proper tags > when you ask your question. For example, if using Google Cloud Kubernetes > Engine [1] you may tag it as [google-kubernetes-engine], and [kubernetes]; > but if using a Kubernetes cluster on Compute Engine, you may tag it as > [kubernetes], and [google-compute-engine]. > > On the other hand, while I was checking the Ingress prerequisites [2], I > understood that “Google Kubernetes Engine deploys an ingress controller on > the master” for you, and like this tutorial [3] it creates the load > balancer. According to the prerequisites, and this github page [4], it is > still in beta, and you may check the limitations and expectations [5] with > latency on the same page. > That said, and if you suspect a defect with the Ingress Controller/ Google > Cloud Load Balancer you may open a report through issue tracker [6]. But > after verifying the correct settings with the community, and reviewing the > above as not an expected behavior. I hope that helps. > > [1] https://console.cloud.google.com/kubernetes > [2] > https://kubernetes.io/docs/concepts/services-networking/ingress/#prerequisites > [3] > https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer > [4] > https://github.com/kubernetes/ingress-gce/blob/master/BETA_LIMITATIONS.md#glbc-beta-limitations > [5] > https://github.com/kubernetes/ingress-gce/blob/master/BETA_LIMITATIONS.md#latency > [6] https://cloud.google.com/support/docs/issue-trackers > > -- > You received this message because you are subscribed to the Google Groups > "Kubernetes user discussion and Q&A" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to kubernetes-users+unsubscr...@googlegroups.com. > To post to this group, send email to kubernetes-users@googlegroups.com. > Visit this group at https://groups.google.com/group/kubernetes-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
[kubernetes-users] Re: Slow response times using default Ingress on GKE
Hello Dave, Since your question is technical, you may get help from community experts and enthusiasts at serverfault.com. Make sure you include proper tags when you ask your question. For example, if using Google Cloud Kubernetes Engine [1] you may tag it as [google-kubernetes-engine], and [kubernetes]; but if using a Kubernetes cluster on Compute Engine, you may tag it as [kubernetes], and [google-compute-engine]. On the other hand, while I was checking the Ingress prerequisites [2], I understood that “Google Kubernetes Engine deploys an ingress controller on the master” for you, and like this tutorial [3] it creates the load balancer. According to the prerequisites, and this github page [4], it is still in beta, and you may check the limitations and expectations [5] with latency on the same page. That said, and if you suspect a defect with the Ingress Controller/ Google Cloud Load Balancer you may open a report through issue tracker [6]. But after verifying the correct settings with the community, and reviewing the above as not an expected behavior. I hope that helps. [1] https://console.cloud.google.com/kubernetes [2] https://kubernetes.io/docs/concepts/services-networking/ingress/#prerequisites [3] https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer [4] https://github.com/kubernetes/ingress-gce/blob/master/BETA_LIMITATIONS.md#glbc-beta-limitations [5] https://github.com/kubernetes/ingress-gce/blob/master/BETA_LIMITATIONS.md#latency [6] https://cloud.google.com/support/docs/issue-trackers -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
[kubernetes-users] Re: Slow response times using default Ingress on GKE
Is there a community of Kubernetes/GCP users that is more active than this Google Group? On Wednesday, April 18, 2018 at 5:14:28 PM UTC-7, Dave Jensen wrote: > > We have what I believe to be a very straightforward ingress setup on GKE. > However, we started noticing random slowdowns almost immediately. On > further investigation it looked like the time to first byte (TTFB) was very > slow (1 - 3 seconds) but sporadically. Sometimes it would be a pre-flight > OPTIONS request, sometimes an application request, and other times on > static files. Even the echoserver would sporadically have a long TTFB. > > I set up a portforward to one of the pods serving up our REST API server. > Sure enough, the slowdown was eliminated. > > Before I go down the rabbit hole of trying other ingress controllers, I > figured I'd ask the community if I was doing something wrong. > > ingress.yaml > > apiVersion: extensions/v1beta1 > kind: Ingress > metadata: > name: brewd-ingress > spec: > tls: > - hosts: > - stage-api2.example.com > - stage-app2.example.com > - stage-echo.example.com > secretName: redacted > rules: > - host: stage-api2.example.com > http: > paths: > - backend: > serviceName: gateway-service > servicePort: 7000 > - host: stage-app2.example.com > http: > paths: > - backend: > serviceName: web-service > servicePort: 8080 > - host: stage-echo.example.com > http: > paths: > - backend: > serviceName: echoserver > servicePort: 8080 > > gateway-service.yaml > apiVersion: v1 > kind: Service > metadata: > name: gateway-service > labels: > app: gateway > spec: > type: NodePort > ports: > - port: 7000 > selector: > app: gateway > --- > apiVersion: apps/v1beta2 > kind: Deployment > metadata: > name: gateway-deployment > spec: > selector: > matchLabels: > app: gateway > replicas: 1 > template: > metadata: > labels: > app: gateway > spec: > containers: > - name: gateway > image: us.gcr.io/redacted/gateway:1.3.0-rc.0 > imagePullPolicy: Always > ports: > - containerPort: 7000 > env: > - name: REDACTED_ENV > value: stage > > The web-service yaml looks almost exactly the same as the above. > -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
