I would try host to host on the same vlan, then consider flow-control impact
Thanks ~mike On Sep 26, 2013, at 8:18 AM, John Neiberger <[email protected]> wrote: > It was host to host, so it was really Host A to Host B and vice versa. The > expected RTT was less than a millisecond, which is what they often got, but > the latency would spike regularly up to as high as 24 ms. I initially > thought it was a problem on one of the hosts but they can ping to and from > devices on the same vlan with no variable latency. The latency only occurs > in one direction when going from one vlan to the other. We manipulated the > HSRP configs to shift traffic to different routers and switches but the > behavior didn't change. From Host A to Host B we saw variable latency, but > never ever did we see it if the ping originated from Host B even though, > depending on the HSRP configuration, the packets were traversing exactly > the same path. It has me completely stumped. > > > On Thu, Sep 26, 2013 at 9:04 AM, Blake Dunlap <[email protected]> wrote: > >> This may seem like a stupid question, but when you were pinging, were you >> pinging from hosts, or from the routers? >> >> -Blake >> >> >> On Thu, Sep 26, 2013 at 9:38 AM, John Neiberger <[email protected]>wrote: >> >>> Thanks! I talked to our Cisco NCE about this and he gave me these >>> commands: >>> >>> show qos interface gigabitEthernet x/y- will show you 4 queues and also >>> whether QoS is disabled or not >>> >>> sh int gi x/y counters detail -you will see packet counters in queue #1-4 >>> incrementing >>> >>> Sh platform hardware interface g x/y stat | in TxB >>> >>> >>> I'm nearly certain that this big buffer issue is the answer to my high >>> variable latency problem, but there is still one mystery about this that >>> has me completely perplexed. The high variable latency was only occurring >>> in one direction (from VLAN A to VLAN B) but not in the other (from VLAN B >>> to VLAN A). What really makes that weird is that because of some hsrp >>> differences, we really had a circular topology for a bit. The path was >>> *exactly* the same no matter which direction you were pinging. The ICMP >>> packets had to travel around the same circle through the same devices and >>> interfaces. So if we have big buffers on congested interfaces that are >>> introducing variable latency, why would we only see it in one direction? >>> >>> >>> When VLAN A pings VLAN B, it is the initial ICMP packet that would have >>> been delayed, while the response would come in on a different interface >>> that wasn't congested. When VLAN B pings VLAN A, the initial ping would >>> not >>> hit congested interfaces but the ping reply would. The total round trip >>> time should have been similar, but it never was. I'm completely stumped by >>> that. I even had Cisco HTTS on this for a couple of days and they couldn't >>> figure it out. >>> >>> >>> Thanks, >>> >>> John >>> >>> >>> On Thu, Sep 26, 2013 at 1:50 AM, Terebizh, Evgeny <[email protected]> >>> wrote: >>> >>>> Try also >>>> "show platform hardware interface gigabitEthernet 1/1 tx-queue". >>>> I guess it's gonna show the actual values for queue utilisation. >>>> Please let me know if this helps. >>>> >>>> /ET >>>> >>>> >>>> >>>> >>>> On 9/24/13 11:17 PM, "John Neiberger" <[email protected]> wrote: >>>> >>>>> I've been helping to troubleshoot an interesting problem with variable >>>>> latency through a 4948. I haven't run into this before. I usually have >>>>> seen >>>>> really low latency through 4948s, but this particular application >>> requires >>>>> consistent low latency and they've been noticing that latency goes up >>> on >>>>> average as load goes up. It didn't seem to be a problem on their >>> servers, >>>>> but communication through busy interfaces seemed to dramatically >>> increase >>>>> the latency. They were used to <1ms latency and it was bouncing up to >>> 20+ >>>>> ms at times. I'm starting to think this is due to the shared output >>> buffer >>>>> in the 4948 causing the output buffer on the uplink to dynamically get >>>>> bigger. >>>>> >>>>> I've been trying to find more details on how the 4948 handles its >>> shared >>>>> output queue space, but I haven't been able to find anything. Do any of >>>>> you >>>>> know more about how this works and what commands I could use to >>>>> troubleshoot? I can't find anything that might show how much buffer >>> space >>>>> a >>>>> particular interface is using at any given time, or if it even makes >>> sense >>>>> to think of it that way. If I knew the size of the buffer at any given >>>>> moment, I could calculate the expected latency and prove whether or not >>>>> that was the problem. >>>>> >>>>> Thanks! >>>>> John >>>>> _______________________________________________ >>>>> cisco-nsp mailing list [email protected] >>>>> https://puck.nether.net/mailman/listinfo/cisco-nsp >>>>> archive at http://puck.nether.net/pipermail/cisco-nsp/ >>> _______________________________________________ >>> cisco-nsp mailing list [email protected] >>> https://puck.nether.net/mailman/listinfo/cisco-nsp >>> archive at http://puck.nether.net/pipermail/cisco-nsp/ > _______________________________________________ > cisco-nsp mailing list [email protected] > https://puck.nether.net/mailman/listinfo/cisco-nsp > archive at http://puck.nether.net/pipermail/cisco-nsp/ _______________________________________________ cisco-nsp mailing list [email protected] https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
