Re: Verizon Business Contact

2024-02-26 Thread Richard Laager 
To close the loop on this, Verizon Wireless reported to me that they 
fixed the issue (whatever it was).


They further said that 63.56.37.4 was a typo; all IPs should have been 
in 63.59.x.x.


I am able to reach the 63.59.0.0/16 IPs in question: 63.59.39.232 & 
63.59.67.68.


Justin: Thanks for the detail that this was reproducible from Cogent's 
looking glass. I think there's a good chance that contributed to them 
being able to find it (i.e. having an easy way for them to test).


--
Richard

Re: Verizon Business Contact

2024-02-19 Thread sronan 
Based on the ASName of both AS, including CELLCO which is the actual name of 
the corporate entity known as Verizon Wireless, I would agree that both are in 
fact Verizon Wireless. The contacts are just corporate standard entities.

Shane

> On Feb 19, 2024, at 9:01 PM, Richard Laager  wrote:
> 
> I see the route originated by two different ASNs. I agree that when I use 
> the AS6167 path, it is broken (for the destinations where it is broken; 
> 63.59.166.100 was working despite using the AS6167 path).
> 
> BGP routing table entry for 63.59.0.0/16
> Paths: 2 available
>  6939 701 22394
>184.105.34.254 from 184.105.34.254 (216.218.253.228)
>  Origin IGP, metric 0, localpref 60, IGP metric 0, weight 0, tag 0
>  Received 21d19h ago, valid, external, best
>  Rx SAFI: Unicast
>  6461 701 6167
>69.89.205.202 from 69.89.205.202 (69.89.205.202)
>  Origin IGP, metric 887, localpref 60, IGP metric 40, weight 0, tag 0
>  Received 4d03h ago, valid, internal
>  Community: 6461:5997
>  Rx SAFI: Unicast
> 
> Based on the names in WHOIS, I would say that both AS6167 and AS22394 are 
> Verizon Wireless.
> 
> --
> Richard
>

Re: Verizon Business Contact

2024-02-19 Thread Richard Laager 
I see the route originated by two different ASNs. I agree that when I 
use the AS6167 path, it is broken (for the destinations where it is 
broken; 63.59.166.100 was working despite using the AS6167 path).


BGP routing table entry for 63.59.0.0/16
 Paths: 2 available
  6939 701 22394
184.105.34.254 from 184.105.34.254 (216.218.253.228)
  Origin IGP, metric 0, localpref 60, IGP metric 0, weight 0, tag 0
  Received 21d19h ago, valid, external, best
  Rx SAFI: Unicast
  6461 701 6167
69.89.205.202 from 69.89.205.202 (69.89.205.202)
  Origin IGP, metric 887, localpref 60, IGP metric 40, weight 0, tag 0
  Received 4d03h ago, valid, internal
  Community: 6461:5997
  Rx SAFI: Unicast

Based on the names in WHOIS, I would say that both AS6167 and AS22394 
are Verizon Wireless.


--
Richard

Re: Verizon Business Contact

2024-02-19 Thread sronan 
No, Verizon Wireless has their own AS # and doesn’t actually use Verizon Business as their primary provider.ShaneOn Feb 19, 2024, at 2:58 PM, Mike Hammett  wrote:But then MCI is the one running fiber to all of the Verizon Wireless sites, so that doesn't help in de-muddying the waters.-Mike HammettIntelligent Computing Solutionshttp://www.ics-il.comMidwest-IXhttp://www.midwest-ix.comFrom: sro...@ronan-online.comTo: "Justin Krejci" Cc: nanog@nanog.orgSent: Monday, February 19, 2024 1:54:43 PMSubject: Re: Verizon Business ContactVerizon Business is the fixed line business focused entity, formerly MCI and UUNET. Verizon Wireless is the wireless business entity.ShaneOn Feb 19, 2024, at 2:44 PM, Justin Krejci  wrote:




For me it is some AS 6167 destinations.
WHOIS for that ASN says this is Verizon Business.


AS Number:  6167
Org Name:   Verizon Business



I am not sure how I am supposed to accurately or authoritatively discern the differences in specific IP prefixes (or ASNs) as to whether they are are used in the Verizon Wireless, Verizon Business, Verizon XYZ, etc.
I am also not sure what the value would be understanding the difference as I have zero contacts at any Verizon entity: Wireless, Business, or any other.


I imagine at some level, there is a parent Verizon umbrella organization that is ultimately responsible for all underling organizations/divisions but I am not particularly interested in trying to pick apart the business silos of Verizon and then from there
 trying to chase down specific Verizon entity contacts to try and figure out who, might be the right contact to look into this. I have made efforts, prior to this NANOG thread even starting, to get this issue rectified but I have had zero luck so far getting
 any appropriate person at Verizon to take notice.


It kind of feels like trying to reach out to some company regarding a geolocation or IP-reputation type issue... just a lot of "Sorry, I don't know. try this other group that you already talked to" or simply "piss off" type responses. Both of which I have
 received in sizable quantities. Now that my brain is on that tangent, my favorite geolocation response was when I was told "your ISP needs to set the correct bits in the IP packets to designate the traffic as coming from the correct geography." I laughed and
 I cried at that one.






-Original Message-
From: Richard Laager <rlaa...@wiktel.com>
To: Justin Krejci <jkre...@usinternet.com>
Cc: nanog@nanog.org <nanog@nanog.org>
Subject: Re: Verizon Business Contact
Date: Fri, 16 Feb 2024 20:41:04 -0600



On 2024-02-09 18:10, Justin Krejci wrote:



For a good long while (months) we have had similar issues with various Verizon destinations.

Only Verizon Wireless destinations, or other Verizon Business things?

As of today, I'm told (via an upstream provider) that Verizon Business says this is a Verizon Wireless issue.

Re: Verizon Business Contact

2024-02-19 Thread Mike Hammett 
But then MCI is the one running fiber to all of the Verizon Wireless sites, so 
that doesn't help in de-muddying the waters. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: sro...@ronan-online.com 
To: "Justin Krejci"  
Cc: nanog@nanog.org 
Sent: Monday, February 19, 2024 1:54:43 PM 
Subject: Re: Verizon Business Contact 



Verizon Business is the fixed line business focused entity, formerly MCI and 
UUNET. Verizon Wireless is the wireless business entity. 


Shane 



On Feb 19, 2024, at 2:44 PM, Justin Krejci  wrote: 







For me it is some AS 6167 destinations. 
WHOIS for that ASN says this is Verizon Business. 

AS Number:  6167 Org Name:   Verizon Business 


I am not sure how I am supposed to accurately or authoritatively discern the 
differences in specific IP prefixes (or ASNs) as to whether they are are used 
in the Verizon Wireless, Verizon Business, Verizon XYZ, etc. 
I am also not sure what the value would be understanding the difference as I 
have zero contacts at any Verizon entity: Wireless, Business, or any other. 


I imagine at some level, there is a parent Verizon umbrella organization that 
is ultimately responsible for all underling organizations/divisions but I am 
not particularly interested in trying to pick apart the business silos of 
Verizon and then from there trying to chase down specific Verizon entity 
contacts to try and figure out who, might be the right contact to look into 
this. I have made efforts, prior to this NANOG thread even starting, to get 
this issue rectified but I have had zero luck so far getting any appropriate 
person at Verizon to take notice. 


It kind of feels like trying to reach out to some company regarding a 
geolocation or IP-reputation type issue... just a lot of "Sorry, I don't know. 
try this other group that you already talked to" or simply "piss off" type 
responses. Both of which I have received in sizable quantities. Now that my 
brain is on that tangent, my favorite geolocation response was when I was told 
"your ISP needs to set the correct bits in the IP packets to designate the 
traffic as coming from the correct geography." I laughed and I cried at that 
one. 






-Original Message- 
>From : Richard Laager < rlaa...@wiktel.com > 
To : Justin Krejci < jkre...@usinternet.com > 
Cc : nanog@nanog.org < nanog@nanog.org > 
Subject : Re: Verizon Business Contact 
Date : Fri, 16 Feb 2024 20:41:04 -0600 


On 2024-02-09 18:10, Justin Krejci wrote: 




For a good long while (months) we have had similar issues with various Verizon 
destinations. 


Only Verizon Wireless destinations, or other Verizon Business things? 

As of today, I'm told (via an upstream provider) that Verizon Business says 
this is a Verizon Wireless issue.

Re: Verizon Business Contact

2024-02-19 Thread sronan 
Verizon Business is the fixed line business focused entity, formerly MCI and UUNET. Verizon Wireless is the wireless business entity.ShaneOn Feb 19, 2024, at 2:44 PM, Justin Krejci  wrote:




For me it is some AS 6167 destinations.
WHOIS for that ASN says this is Verizon Business.


AS Number:  6167
Org Name:   Verizon Business



I am not sure how I am supposed to accurately or authoritatively discern the differences in specific IP prefixes (or ASNs) as to whether they are are used in the Verizon Wireless, Verizon Business, Verizon XYZ, etc.
I am also not sure what the value would be understanding the difference as I have zero contacts at any Verizon entity: Wireless, Business, or any other.


I imagine at some level, there is a parent Verizon umbrella organization that is ultimately responsible for all underling organizations/divisions but I am not particularly interested in trying to pick apart the business silos of Verizon and then from there
 trying to chase down specific Verizon entity contacts to try and figure out who, might be the right contact to look into this. I have made efforts, prior to this NANOG thread even starting, to get this issue rectified but I have had zero luck so far getting
 any appropriate person at Verizon to take notice.


It kind of feels like trying to reach out to some company regarding a geolocation or IP-reputation type issue... just a lot of "Sorry, I don't know. try this other group that you already talked to" or simply "piss off" type responses. Both of which I have
 received in sizable quantities. Now that my brain is on that tangent, my favorite geolocation response was when I was told "your ISP needs to set the correct bits in the IP packets to designate the traffic as coming from the correct geography." I laughed and
 I cried at that one.






-Original Message-
From: Richard Laager <rlaa...@wiktel.com>
To: Justin Krejci <jkre...@usinternet.com>
Cc: nanog@nanog.org <nanog@nanog.org>
Subject: Re: Verizon Business Contact
Date: Fri, 16 Feb 2024 20:41:04 -0600



On 2024-02-09 18:10, Justin Krejci wrote:



For a good long while (months) we have had similar issues with various Verizon destinations.

Only Verizon Wireless destinations, or other Verizon Business things?

As of today, I'm told (via an upstream provider) that Verizon Business says this is a Verizon Wireless issue.

Re: Verizon Business Contact

2024-02-19 Thread Justin Krejci 
For me it is some AS 6167 destinations.
WHOIS for that ASN says this is Verizon Business.


AS Number:  6167

Org Name:   Verizon Business


I am not sure how I am supposed to accurately or authoritatively discern the 
differences in specific IP prefixes (or ASNs) as to whether they are are used 
in the Verizon Wireless, Verizon Business, Verizon XYZ, etc.
I am also not sure what the value would be understanding the difference as I 
have zero contacts at any Verizon entity: Wireless, Business, or any other.

I imagine at some level, there is a parent Verizon umbrella organization that 
is ultimately responsible for all underling organizations/divisions but I am 
not particularly interested in trying to pick apart the business silos of 
Verizon and then from there trying to chase down specific Verizon entity 
contacts to try and figure out who, might be the right contact to look into 
this. I have made efforts, prior to this NANOG thread even starting, to get 
this issue rectified but I have had zero luck so far getting any appropriate 
person at Verizon to take notice.

It kind of feels like trying to reach out to some company regarding a 
geolocation or IP-reputation type issue... just a lot of "Sorry, I don't know. 
try this other group that you already talked to" or simply "piss off" type 
responses. Both of which I have received in sizable quantities. Now that my 
brain is on that tangent, my favorite geolocation response was when I was told 
"your ISP needs to set the correct bits in the IP packets to designate the 
traffic as coming from the correct geography." I laughed and I cried at that 
one.



-Original Message-
From: Richard Laager 
mailto:richard%20laager%20%3crlaa...@wiktel.com%3e>>
To: Justin Krejci 
mailto:justin%20krejci%20%3cjkre...@usinternet.com%3e>>
Cc: nanog@nanog.org 
mailto:%22na...@nanog.org%22%20%3cna...@nanog.org%3e>>
Subject: Re: Verizon Business Contact
Date: Fri, 16 Feb 2024 20:41:04 -0600

On 2024-02-09 18:10, Justin Krejci wrote:

For a good long while (months) we have had similar issues with various Verizon 
destinations.

Only Verizon Wireless destinations, or other Verizon Business things?

As of today, I'm told (via an upstream provider) that Verizon Business says 
this is a Verizon Wireless issue.

Re: Verizon Business Contact

2024-02-16 Thread Richard Laager 

On 2024-02-09 18:10, Justin Krejci wrote:
For a good long while (months) we have had similar issues with various 
Verizon destinations.


Only Verizon *Wireless* destinations, or other Verizon *Business* things?

As of today, I'm told (via an upstream provider) that Verizon Business 
says this is a Verizon Wireless issue.


--
Richard

Re: Verizon Business Contact

2024-02-09 Thread Justin Krejci 
For a good long while (months) we have had similar issues with various Verizon 
destinations.
I observed it only happens when passing through certain geographic regions of 
the US. Other regions make it through without issue.

This is directly observable and repeatable using Cogent's Looking glass website.
Do an IPv4 Trace to 63.59.67.68 using their US-Minneapollis router. It dies.
Do an IPv4 Trace to 63.59.67.68 using their US-Los Angeles router. It reaches 
the destination.

I went through a handful of Cogent's looking glass locations and found some 
that work and some that don't and concluded there must be one or more Verizon 
routers in a certain set geographic area that are having the problems.

Ultimately the issue is not resolved for me but I was able to BGP TE the 
traffic around the problem areas to facilitate reachability to the impacted 
destinations. This is obviously a tenuous band-aid.


Long story short: please, please, please, someone at Verizon or someone who has 
the ear of someone at Verizon, please, please, please, look into this.




-Original Message-
From: Richard Laager 
mailto:richard%20laager%20%3crlaa...@wiktel.com%3e>>
To: nanog@nanog.org
Subject: Verizon Business Contact
Date: Thu, 08 Feb 2024 13:01:14 -0600

Can someone from Verizon Business please contact me?

It appears that your network is losing traffic from Verizon Wireless
(e.g. 63.59.39.232, 63.56.37.4, or 63.59.67.68) to me (AS33362, e.g. to
69.89.207.16). Note that 63.59.166.100 -> 69.89.207.16 was successfully
(around 2023-11-27).

This breaks email between us and it's been MONTHS of VZW getting nowhere.

Based on some traceroutes (on 2023-11-27 and again just now), the
working ones go through 140.222.234.223 (0.ae10.GW7.CHI13.ALTER.NET)
while the broken ones stop at 140.222.234.221 (0.ae9.GW7.CHI13.ALTER.NET).