Re: [outages] Fiber cut in the austin area affecting multiple carriers

[Matthew Blackstone via Outages]

At this time crews are successfully excavated one end of the fiber cable
and this side is being prepped for testing, with splicing to follow. The
other end of the cable is still located underground and crews are working
to excavate. There is no ETA on the splicing for this end of the cable. We
currently do not have an estimated time to restoration (ETR).


On Aug 9, 2017 10:29 PM, "Matthew Blackstone" <blackst...@indeed.com> wrote:

Latest update for anyone else still affected: he Fiber cut has been
isolated and crews are working to clear the damaged area. Splicing crews
are on standby for repairs to commence. Level 3 has migrated their circuits
to spare circuits. Level 3 services should be restored at this time. We
currently do not have an estimated time to restoration (ETR).


On Aug 9, 2017 5:34 PM, "Matthew Blackstone" <blackst...@indeed.com> wrote:

Good to hear your service has been restored, ours has not.

Latest update: We continue to receive updates from the providers affected
by this fiber cut affecting the Austin, Texas region. At this time
providers are engaging with their outside plant teams in an effort to
isolate the damage. We currently do not have an estimated time to
restoration (ETR); however we hope to receive one soon, as the carriers
continue to address the issue.


Matt Blackstone
Network Engineer


On Wed, Aug 9, 2017 at 5:16 PM, John Knutson <john.m.knut...@gmail.com>
wrote:

> CenturyLink is back in Minneapolis area. Both business and residential
> were affected and are back up.
>
> On Wed, Aug 9, 2017 at 5:09 PM John Knutson <john.m.knut...@gmail.com>
> wrote:
>
>> intermittent issues with CenturyLink in Minneapolis area, Comcast is fine
>>
>> On Wed, Aug 9, 2017, 4:49 PM Matthew Blackstone via Outages <
>> outages@outages.org> wrote:
>>
>>> I just received a notification from our provider about this but they
>>> were light on details.
>>>
>>> Matt Blackstone
>>> Network Engineer
>>>
>>> ___
>>> Outages mailing list
>>> Outages@outages.org
>>> https://puck.nether.net/mailman/listinfo/outages
>>>
>>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Fiber cut in the austin area affecting multiple carriers

[Matthew Blackstone via Outages]

Latest update for anyone else still affected: he Fiber cut has been
isolated and crews are working to clear the damaged area. Splicing crews
are on standby for repairs to commence. Level 3 has migrated their circuits
to spare circuits. Level 3 services should be restored at this time. We
currently do not have an estimated time to restoration (ETR).

On Aug 9, 2017 5:34 PM, "Matthew Blackstone" <blackst...@indeed.com> wrote:

Good to hear your service has been restored, ours has not.

Latest update: We continue to receive updates from the providers affected
by this fiber cut affecting the Austin, Texas region. At this time
providers are engaging with their outside plant teams in an effort to
isolate the damage. We currently do not have an estimated time to
restoration (ETR); however we hope to receive one soon, as the carriers
continue to address the issue.


Matt Blackstone
Network Engineer


On Wed, Aug 9, 2017 at 5:16 PM, John Knutson <john.m.knut...@gmail.com>
wrote:

> CenturyLink is back in Minneapolis area. Both business and residential
> were affected and are back up.
>
> On Wed, Aug 9, 2017 at 5:09 PM John Knutson <john.m.knut...@gmail.com>
> wrote:
>
>> intermittent issues with CenturyLink in Minneapolis area, Comcast is fine
>>
>> On Wed, Aug 9, 2017, 4:49 PM Matthew Blackstone via Outages <
>> outages@outages.org> wrote:
>>
>>> I just received a notification from our provider about this but they
>>> were light on details.
>>>
>>> Matt Blackstone
>>> Network Engineer
>>>
>>> ___
>>> Outages mailing list
>>> Outages@outages.org
>>> https://puck.nether.net/mailman/listinfo/outages
>>>
>>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Fiber cut in the austin area affecting multiple carriers

[Matthew Blackstone via Outages]

Good to hear your service has been restored, ours has not.

Latest update: We continue to receive updates from the providers affected
by this fiber cut affecting the Austin, Texas region. At this time
providers are engaging with their outside plant teams in an effort to
isolate the damage. We currently do not have an estimated time to
restoration (ETR); however we hope to receive one soon, as the carriers
continue to address the issue.


Matt Blackstone
Network Engineer


On Wed, Aug 9, 2017 at 5:16 PM, John Knutson <john.m.knut...@gmail.com>
wrote:

> CenturyLink is back in Minneapolis area. Both business and residential
> were affected and are back up.
>
> On Wed, Aug 9, 2017 at 5:09 PM John Knutson <john.m.knut...@gmail.com>
> wrote:
>
>> intermittent issues with CenturyLink in Minneapolis area, Comcast is fine
>>
>> On Wed, Aug 9, 2017, 4:49 PM Matthew Blackstone via Outages <
>> outages@outages.org> wrote:
>>
>>> I just received a notification from our provider about this but they
>>> were light on details.
>>>
>>> Matt Blackstone
>>> Network Engineer
>>>
>>> ___
>>> Outages mailing list
>>> Outages@outages.org
>>> https://puck.nether.net/mailman/listinfo/outages
>>>
>>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

[outages] Fiber cut in the austin area affecting multiple carriers

[Matthew Blackstone via Outages]

I just received a notification from our provider about this but they were
light on details.

Matt Blackstone
Network Engineer
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

[outages] Google DNS issues in Brazil

[Matthew Blackstone via Outages]

We are noticing that we no longer receive DNS responses to requests sent to
8.8.8.8 and 8.8.4.4 on circuits in Sao Paulo.

Matt Blackstone
Network Engineer

Indeed - We help people get jobs.
Indeed.com 
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

[outages] CenturyLink(old Qwest) issues in Seattle

[Matthew Blackstone via Outages]

Getting reports from CenturyLink that an outage affecting our site is due
to the loss of a core router in Seattle.  Circuit down since Sunday.  If
anyone else is having issues in the area this may explain it.


Matt Blackstone
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Fiber cut between Houston and Austin

[Matthew Blackstone via Outages]

11:10pm
ALPH fiber vender informed us
Splicing in East Pit is complete. They have verified the west side
damage does not go beyond the west side pit. They are prepping the west
side now and splicing should start in 15 mins. ETTR is expected around
12:30 AM.

Matt Blackstone
On Jul 1, 2016 9:50 PM, "Matthew Blackstone" <blackst...@indeed.com> wrote:

> 9:46pm
> ALPH fiber vender informed us they have removed the water from West Side
> splice pit. They are started to go through the DUCTS trying to find
> their cable. Once they fine their cable they will be able to complete
> their assessment on the west side and will be able to prep that cable.
> They did inform us they will start splicing on the east side shortly.
> They stated once west side is prepped they will be able to provide ETTR.
>
> Matt Blackstone
> On Jul 1, 2016 9:06 PM, "Matthew Blackstone" <blackst...@indeed.com>
> wrote:
>
>> Updates so far from Alpheus:
>>
>> 5:55pm
>> The fiber vendor informed they have located the damage. It is 3.6 miles
>> from intersection of Highway 99 and Highway 1093. They are currently
>> waiting for the 144 count real of fiber to be picked up and delivered
>> to site. No ETR at this time.
>>
>> 5:57pm
>> The fiber vendor informed us they are waiting on the construction crew
>> to arrive to start excavation of damaged cable. They are still waiting
>> for a 144 count fiber reel to arrive as well. No ETR at this time.
>>
>> 7:19pm
>> ALPH fiber vender informed us their Aerial team has arrived and
>> starting to put up temp poles and steel cable to connect cable to when
>> it arrives. They have completed digging of splice pit on west side and
>> still working on east side splice pit. They are still not able to
>> provide ETTR at this time.
>> On Jul 1, 2016 3:18 PM, "Chris Boyd via Outages" <outages@outages.org>
>> wrote:
>>
>> I have DIA from TWT/L3/4323 and Alpheus in Austin and I’m not seeing any
>> issues.
>>
>> —Chris
>>
>> > On Jul 1, 2016, at 1:11 PM, Matthew Blackstone via Outages <
>> outages@outages.org> wrote:
>> >
>> > Our provider is reporting a cut somewhere between between Houston and
>> Austin that is affecting one of our metro circuits.  Anyone else
>> experiencing issues in the area?
>> >
>> > Matt Blackstone
>> >
>> >
>> > ___
>> > Outages mailing list
>> > Outages@outages.org
>> > https://puck.nether.net/mailman/listinfo/outages
>>
>> ___
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

[outages] Fiber cut between Houston and Austin

[Matthew Blackstone via Outages]

Our provider is reporting a cut somewhere between between Houston and
Austin that is affecting one of our metro circuits.  Anyone else
experiencing issues in the area?

Matt Blackstone
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] East coast Telia issues?

[Matthew Blackstone via Outages]

RFO from Telia:
Incorrect ISIS metric and multiple commits while turning up new Telia
Carrier backbone links in Dallas caused a loop of reconverging BGP and ISIS
protocols. This put a very high CPU load on our U.S. routers and caused
some trans-Atlantic congestion.




On Thu, Jun 9, 2016 at 7:23 PM, David Thompson via Outages <
outages@outages.org> wrote:

> So is there an official RFO from Telia and a time of repair? We were
> seeing major call quality and loss of SIP registration across the board.
> Does anyone know if Cox cable was affected by this? I’m imagining so since
> Cox only owns it’s backbone in certain areas and relies upon other carriers
> for content delivery.
>
>
>
> David Thompson
> Network Services Support Technician
> (O) 858.357.8794
> (F) 858-225-1882
> (E) dthomp...@esi-estech.com
> (W) www.esi-estech.com
>
>
>
> *From:* Outages [mailto:outages-boun...@outages.org] *On Behalf Of *Wolf,
> Geoffrey via Outages
> *Sent:* Thursday, June 09, 2016 1:55 PM
> *To:* outages@outages.org
> *Subject:* [outages] East coast Telia issues?
>
>
>
> We are seeing heavy packet loss between Europe and locations in NA.
> Evidence is pointing towards an issue with TeliaSonera’s network border
> with a number of other large networks (AT and Cogent) on the East Coast.
>
>
>
> Anyone else seeing problems?
>
>
>
> Geoffrey Wolf
>
>
>
> The information contained in this message is confidential and may be
> privileged and/or protected under law. If you received this message in
> error, please notify us immediately by forwarding a copy to
> complia...@intermedix.com and delete the original message and any
> attachments.
>
>
>
>
>
>
>
> The information contained in this message is confidential and may be 
> privileged and/or protected under law. If you receive this message in error, 
> please notify us immediately by forwarding a copy to 
> complia...@intermedix.com and delete the original message and any attachments.
>
>
>
>
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] East coast Telia issues?

[Matthew Blackstone via Outages]

Telia has been flaky for us as well lately.   Looks like they routed around
the issue and through NYC. My traces have been clean for several minutes
now.



On Thu, Jun 9, 2016 at 2:25 PM, Wolf, Geoffrey  wrote:

> Seems like they have a flapping route somewhere. The loss went down to
> nothing for a little while then very quickly started back up again. In the
> process of writing this the loss disappeared yet again. Hopefully it holds.
> Over the past two months we’ve been had a lot of reports of issues that
> very heavily suggest and point to Telia’s NA ßà EU backbone being very
> flaky.
>
>
>
> Geoffrey Wolf
>
>
>
> The information contained in this message is confidential and may be
> privileged and/or protected under law. If you received this message in
> error, please notify us immediately by forwarding a copy to
> complia...@intermedix.com and delete the original message and any
> attachments.
>
>
>
> *From:* Matthew Blackstone [mailto:blackst...@indeed.com]
> *Sent:* Thursday, June 09, 2016 3:04 PM
> *To:* Mike Davis 
> *Cc:* Wolf, Geoffrey ; outages@outages.org
> *Subject:* Re: [outages] East coast Telia issues?
>
>
>
> Seeing high loss from Austin as well.
>
>
>
> Host Loss%
>   Snt   Last   Avg  Best  Wrst StDev
>
>  2. 198-58-79-18.cyrusone.com
>  0.0%130.5   0.6   0.4   1.1   0.2
>
>  3. dls-b22-link.telia.net
> 0.0%130.6   0.7   0.6   0.8   0.0
>
>  4. dls-b22-link.telia.net
> 7.7%138.9   9.1   8.9  11.2   0.7
>
>  5. 62.115.120.110
>  75.0%13   60.4  60.5  60.4  60.8   0.2
>
>  6. ???
>
>  7. ldn-b1-link.telia.net
> 46.2%13  174.0 147.0 137.6 174.0  13.2
>
> ldn-b1-link.telia.net
>
> ldn-b1-link.telia.net
>
> ldn-b1-link.telia.net
>
> ldn-b1-link.telia.net
>
> ldn-b1-link.telia.net
>
> ldn-b1-link.telia.net
>
>
>
>
>
>
>
>
> On Thu, Jun 9, 2016 at 1:57 PM, Mike Davis via Outages <
> outages@outages.org> wrote:
>
> We are seeing these problems as well:
>
>
>
>  1.|-- 192.168.5.20.0% 5
>  1.2   1.4   1.2   2.1   0.0
>
>  2.|-- ge-0-1-0--ar1.sof.itdnet.net (http://ge-0-1-0--ar1.sof.itdnet.net/)
>   0.0% 54.0   4.2   4.0   5.1   0.0
>
>  3.|-- br1.sof.itdnet.net (http://br1.sof.itdnet.net/)
>   0.0% 54.2   4.2   4.1   4.3   0.0
>
>  4.|-- 212.116.133.21 0.0% 5
>  4.4   4.3   4.3   4.4   0.0
>
>  5.|-- te0-0-2-2.nr11.b023063-0.sof02.atlas.cogentco.com (
> http://te0-0-2-2.nr11.b023063-0.sof02.atlas.cogentco.com/)  0.0% 5
> 41.9  42.3  41.7  44.0   0.9
>
>  6.|-- te0-0-1-1.agr12.sof02.atlas.cogentco.com (
> http://te0-0-1-1.agr12.sof02.atlas.cogentco.com/)   0.0% 5
> 41.9  41.9  41.8  42.0   0.0
>
>  7.|-- be2230.ccr21.sof02.atlas.cogentco.com (
> http://be2230.ccr21.sof02.atlas.cogentco.com/)  0.0% 5
> 41.8  42.6  41.8  45.5   1.5
>
>  8.|-- be2046.ccr21.vie01.atlas.cogentco.com (
> http://be2046.ccr21.vie01.atlas.cogentco.com/)  0.0% 5
> 42.0  43.0  41.9  47.0   2.2
>
>  9.|-- be2974.ccr21.muc03.atlas.cogentco.com (
> http://be2974.ccr21.muc03.atlas.cogentco.com/)  0.0% 5
> 33.9  33.9  33.8  34.2   0.0
>
> 10.|-- be2959.ccr41.fra03.atlas.cogentco.com (
> http://be2959.ccr41.fra03.atlas.cogentco.com/)  0.0% 5
> 33.5  34.4  33.5  37.1   1.4
>
> 11.|-- be2184.agr21.fra03.atlas.cogentco.com (
> http://be2184.agr21.fra03.atlas.cogentco.com/)  0.0% 5
> 44.5  36.7  33.9  44.5   4.6
>
> 12.|-- telia.fra03.atlas.cogentco.com (
> http://telia.fra03.atlas.cogentco.com/) 0.0% 5
> 33.5  34.0  33.5  35.2   0.5
>
> 13.|-- ffm-bb3-link.telia.net (http://ffm-bb3-link.telia.net/)
>   0.0% 5   33.9  34.0  33.9  34.2   0.0
>
> 14.|-- ash-bb4-link.telia.net (http://ash-bb4-link.telia.net/)
>  60.0% 5  131.6 131.5 131.5 131.6   0.0
>
> 15.|-- 62.115.120.11180.0% 5
>  923.9 923.9 923.9 923.9   0.0
>
> 16.|-- ???   100.0 5
>  0.0   0.0   0.0   0.0   0.0
>
> 17.|-- ???   100.0 5
>  0.0   0.0   0.0   0.0   0.0
>
> 18.|-- ???   100.0 5
>  0.0   0.0   0.0   0.0   0.0
>
>
>
> On Thu, Jun 9, 2016 at 11:55 AM, Wolf, Geoffrey via Outages <
> outages@outages.org> wrote:
>
> We are seeing heavy packet loss between Europe and locations in NA.
> Evidence is pointing towards an issue with TeliaSonera’s network border
> with a number of other large networks (AT and Cogent) on the East Coast.
>
>
>
> Anyone else seeing problems?
>
>
>
> Geoffrey Wolf
>
>
>
> The information contained in this message is confidential and may be
> privileged and/or protected under law. 

Re: [outages] CA\Toronto - Allstream - Outage?

[Matthew Blackstone via Outages]

We are taking IP packet loss over allstream to our site in toronto. 30-40%



Matt Blackstone
Network Engineer
Indeed - How the World Works.™
www.indeed.com

On Tue, Apr 19, 2016 at 11:20 AM, Ivan Kovacevic via Outages <
outages@outages.org> wrote:

> What service? Tdm, sip, network?
> On 19 Apr 2016 12:19 p.m., "Nick Pron via Outages" 
> wrote:
>
>> Anyone seeing/heard about issues with them? Heard a few reports from
>> friends in the downtown core. Might be wide spread as well?
>>
>>
>>
>> Thanks,
>>
>>
>>
>> [image: Description: Description: Description: Description:
>> XBase-Logo-ITSolutions]
>>
>>
>>
>> [image: Description: Description: Description:
>> cid:image002.png@01CD94EC.19B66930]  [image: Description: Description:
>> Description: cid:image003.png@01CD94EC.19B66930]
>>
>>
>>
>> *nick.p...@xbase.com *
>>
>> *Support Specialist *(416) 340-1020 Option 1
>> (800) XBASE-03
>>
>> www.xbase.com
>> [image: Description: Description: Description:
>> cid:image004.png@01CD94EC.19B66930]
>>   
>> [image:
>> Description: Description: Description: cid:image005.png@01CD94EC.19B66930]
>> 
>>
>> The information transmitted is intended only for the person or entity to
>> which it is addressed and may contain confidential and/or privileged
>> material.   Any review, re-transmission, dissemination or other use of, or
>> taking of any action in reliance upon this information by persons or
>> entities other than the intended recipient is prohibited.  If you received
>> this email in error, please contact the sender immediately by return
>> electronic transmission and then immediately delete this transmission,
>> including all attachments, without copying, distributing or disclosing
>> same.  No member of the XBASE Technologies Corp. is liable for any errors
>> or omissions in the content or transmission of this email or accepts any
>> responsibility or liability for loss or damage arising from the receipt or
>> use of this transmission.  XBASE Technologies Corp. may monitor, retain
>> and/or review email.  Unless indicated in writing, opinions contained in
>> this email are those of the author and are not endorsed by any member of
>> the XBASE Technologies Corp.
>>
>>
>>
>> ___
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages