Host to host on the same VLAN was always far less than 1ms RTT. I never once saw it go over that. It was usually far less. We only saw the problem when going from a host in VLAN A to a host in VLAN B, never the other way around. I thought this was a problem on the host in VLAN B, but any other server in the same VLAN could ping it with no latency problems at all.
On Thu, Sep 26, 2013 at 9:12 PM, Fwissue <[email protected]> wrote: > I would try host to host on the same vlan, then consider flow-control > impact > > Thanks > > ~mike > > On Sep 26, 2013, at 8:18 AM, John Neiberger <[email protected]> wrote: > > > It was host to host, so it was really Host A to Host B and vice versa. > The > > expected RTT was less than a millisecond, which is what they often got, > but > > the latency would spike regularly up to as high as 24 ms. I initially > > thought it was a problem on one of the hosts but they can ping to and > from > > devices on the same vlan with no variable latency. The latency only > occurs > > in one direction when going from one vlan to the other. We manipulated > the > > HSRP configs to shift traffic to different routers and switches but the > > behavior didn't change. From Host A to Host B we saw variable latency, > but > > never ever did we see it if the ping originated from Host B even though, > > depending on the HSRP configuration, the packets were traversing exactly > > the same path. It has me completely stumped. > > > > > > On Thu, Sep 26, 2013 at 9:04 AM, Blake Dunlap <[email protected]> wrote: > > > >> This may seem like a stupid question, but when you were pinging, were > you > >> pinging from hosts, or from the routers? > >> > >> -Blake > >> > >> > >> On Thu, Sep 26, 2013 at 9:38 AM, John Neiberger <[email protected] > >wrote: > >> > >>> Thanks! I talked to our Cisco NCE about this and he gave me these > >>> commands: > >>> > >>> show qos interface gigabitEthernet x/y- will show you 4 queues and > also > >>> whether QoS is disabled or not > >>> > >>> sh int gi x/y counters detail -you will see packet counters in queue > #1-4 > >>> incrementing > >>> > >>> Sh platform hardware interface g x/y stat | in TxB > >>> > >>> > >>> I'm nearly certain that this big buffer issue is the answer to my high > >>> variable latency problem, but there is still one mystery about this > that > >>> has me completely perplexed. The high variable latency was only > occurring > >>> in one direction (from VLAN A to VLAN B) but not in the other (from > VLAN B > >>> to VLAN A). What really makes that weird is that because of some hsrp > >>> differences, we really had a circular topology for a bit. The path was > >>> *exactly* the same no matter which direction you were pinging. The ICMP > >>> packets had to travel around the same circle through the same devices > and > >>> interfaces. So if we have big buffers on congested interfaces that are > >>> introducing variable latency, why would we only see it in one > direction? > >>> > >>> > >>> When VLAN A pings VLAN B, it is the initial ICMP packet that would have > >>> been delayed, while the response would come in on a different interface > >>> that wasn't congested. When VLAN B pings VLAN A, the initial ping would > >>> not > >>> hit congested interfaces but the ping reply would. The total round trip > >>> time should have been similar, but it never was. I'm completely > stumped by > >>> that. I even had Cisco HTTS on this for a couple of days and they > couldn't > >>> figure it out. > >>> > >>> > >>> Thanks, > >>> > >>> John > >>> > >>> > >>> On Thu, Sep 26, 2013 at 1:50 AM, Terebizh, Evgeny <[email protected]> > >>> wrote: > >>> > >>>> Try also > >>>> "show platform hardware interface gigabitEthernet 1/1 tx-queue". > >>>> I guess it's gonna show the actual values for queue utilisation. > >>>> Please let me know if this helps. > >>>> > >>>> /ET > >>>> > >>>> > >>>> > >>>> > >>>> On 9/24/13 11:17 PM, "John Neiberger" <[email protected]> wrote: > >>>> > >>>>> I've been helping to troubleshoot an interesting problem with > variable > >>>>> latency through a 4948. I haven't run into this before. I usually > have > >>>>> seen > >>>>> really low latency through 4948s, but this particular application > >>> requires > >>>>> consistent low latency and they've been noticing that latency goes up > >>> on > >>>>> average as load goes up. It didn't seem to be a problem on their > >>> servers, > >>>>> but communication through busy interfaces seemed to dramatically > >>> increase > >>>>> the latency. They were used to <1ms latency and it was bouncing up to > >>> 20+ > >>>>> ms at times. I'm starting to think this is due to the shared output > >>> buffer > >>>>> in the 4948 causing the output buffer on the uplink to dynamically > get > >>>>> bigger. > >>>>> > >>>>> I've been trying to find more details on how the 4948 handles its > >>> shared > >>>>> output queue space, but I haven't been able to find anything. Do any > of > >>>>> you > >>>>> know more about how this works and what commands I could use to > >>>>> troubleshoot? I can't find anything that might show how much buffer > >>> space > >>>>> a > >>>>> particular interface is using at any given time, or if it even makes > >>> sense > >>>>> to think of it that way. If I knew the size of the buffer at any > given > >>>>> moment, I could calculate the expected latency and prove whether or > not > >>>>> that was the problem. > >>>>> > >>>>> Thanks! > >>>>> John > >>>>> _______________________________________________ > >>>>> cisco-nsp mailing list [email protected] > >>>>> https://puck.nether.net/mailman/listinfo/cisco-nsp > >>>>> archive at http://puck.nether.net/pipermail/cisco-nsp/ > >>> _______________________________________________ > >>> cisco-nsp mailing list [email protected] > >>> https://puck.nether.net/mailman/listinfo/cisco-nsp > >>> archive at http://puck.nether.net/pipermail/cisco-nsp/ > > _______________________________________________ > > cisco-nsp mailing list [email protected] > > https://puck.nether.net/mailman/listinfo/cisco-nsp > > archive at http://puck.nether.net/pipermail/cisco-nsp/ > _______________________________________________ cisco-nsp mailing list [email protected] https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
