How big is your etcd working set in terms of number of keys?  How many
namespaces?  If keys <50k then i would suspect software, hardware, or
network issue in between masters and etcd.  Http etcd failures should only
happen when the master is losing elections and being turned over, or the
election/heartbeat timeout is too low for your actual network latency.
Double check the etcd logs and verify that you aren't seen any election
failures or turnover.

What metrics are the apiserver side returning related to etcd latencies and
failures?

On Aug 12, 2017, at 11:07 AM, Andrew Lau <and...@andrewklau.com> wrote:

etcd data is on dedicated drives and aws reports idle and burst capacity
around 90%

On Sun, 13 Aug 2017 at 00:28 Clayton Coleman <ccole...@redhat.com> wrote:

> Check how much IO is being used by etcd and how much you have provisioned.
>
>
> > On Aug 12, 2017, at 5:32 AM, Andrew Lau <and...@andrewklau.com> wrote:
> >
> > Post upgrade to 3.6 I'm noticing the API server seems to be responding a
> lot slower and my etcd metrics etcd_http_failed_total is returning a large
> number of failed GET requests.
> >
> > Has anyone seen this?
> > _______________________________________________
> > users mailing list
> > users@lists.openshift.redhat.com
> > http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>
_______________________________________________
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Reply via email to