Re: NTT communications horrible routing, unresponsive NOC

[Jared Mauch]

Hello,

Could you send me some more details in private about who you spoke with 
including email addresses and ticket numbers with our NOC if any?

I'd like to understand what happened here as this is an uncharacteristic 
outcome. I know there was planned work in Seattle last night for a software 
upgrade combined with hardware work, but an operational issue like this should 
have been resolved. 

I'd like to understand the timeline and what broke down if anything. 

Thanks,

Jared Mauch

> On Mar 23, 2016, at 7:39 PM, Paras Jha  wrote:
> 
> Hi all,
> 
> I've been trying to get this issue resolved for the entire day now, but NTT
> has been pretty unreceptive here.
> 
> We're announcing a large prefix for a client across our network, and we
> discovered some insanely high latency.
> 
> After tracking down the issue, we determined it to be something wrong with
> NTT at their Seattle location. We anycast this prefix, but no matter where
> in the world traffic is originating from, it's going to Seattle and then to
> Atlanta. Example: Rotterdam in the Netherlands routes from Europe -> east
> coast -> west coast Seattle -> los angeles -> atlanta. The gist of it is
> that something is seriously messed up at NTT in Seattle.
> 
> We contacted our transit provider to try and carry the issue upstream, and
> what they told us was
> 
> Sorry for delay, I've asked NTT to clear the more specific for this one as
> well.
> The problem seems to be a bug on the NTT side which keeps stale routes in
> the routing table for more specifics ( at random ).
> If you have more routes affected please notify us of the routes and I will
> ask them to clear the routing table for these routes.
> NTT is working on this with their vendor to get this resolved as soon as
> possible.
> 
> 
> I had spoken to a sales rep for NTT a few weeks prior, and they assured me
> that the NOC was top notch, and that all routes were redundant, and they
> guaranteed less than 50ms in the US, and all kinds of marketing. However,
> it looks like it's all marketing - for this entire day this router has been
> causing tons of issues for our clients.
> 
> No-exporting it to NTT does not even solve the problem, as NTT's router in
> Seattle apparently just decides to keep random small prefixes in it,
> causing traffic to go there.
> 
> At a loss as to what to do now, since their NOC isn't receptive. Anyone
> have someone I can contact off-list to get this issue resolved? It's
> especially frustrating because the problem absolutely cannot be resolved on
> our end, even with a no-export since NTT is keeping the routes in their
> router.

Re: NTT communications horrible routing, unresponsive NOC

[James Bensley]

On 23 March 2016 at 23:39, Paras Jha  wrote:
> At a loss as to what to do now, since their NOC isn't receptive. Anyone
> have someone I can contact off-list to get this issue resolved? It's
> especially frustrating because the problem absolutely cannot be resolved on
> our end, even with a no-export since NTT is keeping the routes in their
> router.

It would be helpful to show how you came to these conclusions about
prefix caching and routing issues etc (share the data you gathered).

If you are an NTT customer (it's sounds a bit like you're a customer
of an NTT customer, it's not clear) then this should be account
management 101, you should have an escalations process/matrix, kick
off that process now.

There are NTT dudes on this list so probably they will reach out to
you off list, if not post back requesting an NTT contact.

James.

Re: NTT communications horrible routing, unresponsive NOC

[Christopher Morrow]

which route and did you look at their lookingglass during the period of
time when things were/are bad to see what the LG saw?

https://www.us.ntt.net/support/looking-glass/

shouting into the wind doesn't really help, information though could be
helpful.

On Wed, Mar 23, 2016 at 7:39 PM, Paras Jha 
wrote:

> Hi all,
>
> I've been trying to get this issue resolved for the entire day now, but NTT
> has been pretty unreceptive here.
>
> We're announcing a large prefix for a client across our network, and we
> discovered some insanely high latency.
>
> After tracking down the issue, we determined it to be something wrong with
> NTT at their Seattle location. We anycast this prefix, but no matter where
> in the world traffic is originating from, it's going to Seattle and then to
> Atlanta. Example: Rotterdam in the Netherlands routes from Europe -> east
> coast -> west coast Seattle -> los angeles -> atlanta. The gist of it is
> that something is seriously messed up at NTT in Seattle.
>
> We contacted our transit provider to try and carry the issue upstream, and
> what they told us was
>
> Sorry for delay, I've asked NTT to clear the more specific for this one as
> well.
> The problem seems to be a bug on the NTT side which keeps stale routes in
> the routing table for more specifics ( at random ).
> If you have more routes affected please notify us of the routes and I will
> ask them to clear the routing table for these routes.
> NTT is working on this with their vendor to get this resolved as soon as
> possible.
>
>
> I had spoken to a sales rep for NTT a few weeks prior, and they assured me
> that the NOC was top notch, and that all routes were redundant, and they
> guaranteed less than 50ms in the US, and all kinds of marketing. However,
> it looks like it's all marketing - for this entire day this router has been
> causing tons of issues for our clients.
>
> No-exporting it to NTT does not even solve the problem, as NTT's router in
> Seattle apparently just decides to keep random small prefixes in it,
> causing traffic to go there.
>
> At a loss as to what to do now, since their NOC isn't receptive. Anyone
> have someone I can contact off-list to get this issue resolved? It's
> especially frustrating because the problem absolutely cannot be resolved on
> our end, even with a no-export since NTT is keeping the routes in their
> router.
>

NTT communications horrible routing, unresponsive NOC

[Paras Jha]

Hi all,

I've been trying to get this issue resolved for the entire day now, but NTT
has been pretty unreceptive here.

We're announcing a large prefix for a client across our network, and we
discovered some insanely high latency.

After tracking down the issue, we determined it to be something wrong with
NTT at their Seattle location. We anycast this prefix, but no matter where
in the world traffic is originating from, it's going to Seattle and then to
Atlanta. Example: Rotterdam in the Netherlands routes from Europe -> east
coast -> west coast Seattle -> los angeles -> atlanta. The gist of it is
that something is seriously messed up at NTT in Seattle.

We contacted our transit provider to try and carry the issue upstream, and
what they told us was

Sorry for delay, I've asked NTT to clear the more specific for this one as
well.
The problem seems to be a bug on the NTT side which keeps stale routes in
the routing table for more specifics ( at random ).
If you have more routes affected please notify us of the routes and I will
ask them to clear the routing table for these routes.
NTT is working on this with their vendor to get this resolved as soon as
possible.


I had spoken to a sales rep for NTT a few weeks prior, and they assured me
that the NOC was top notch, and that all routes were redundant, and they
guaranteed less than 50ms in the US, and all kinds of marketing. However,
it looks like it's all marketing - for this entire day this router has been
causing tons of issues for our clients.

No-exporting it to NTT does not even solve the problem, as NTT's router in
Seattle apparently just decides to keep random small prefixes in it,
causing traffic to go there.

At a loss as to what to do now, since their NOC isn't receptive. Anyone
have someone I can contact off-list to get this issue resolved? It's
especially frustrating because the problem absolutely cannot be resolved on
our end, even with a no-export since NTT is keeping the routes in their
router.