Re: [AusNOG] Urgent: Telstra NOC

2020-10-03 Thread Jim Woodward
sNOG] Urgent: Telstra NOC > Job didn’t ask for a RCA, he asked what action was taken at a very specific > time that resolved the impact. This is precisely why helpful people stop helping. I see this list as a gentle[man|woman]'s agreement that support structures "kinda suck"

Re: [AusNOG] Urgent: Telstra NOC

2020-10-03 Thread Jason Leschnik
> Job didn’t ask for a RCA, he asked what action was taken at a very specific time that resolved the impact. This is precisely why helpful people stop helping. I see this list as a gentle[man|woman]'s agreement that support structures "kinda suck" and are not always great at getting a contact

Re: [AusNOG] Urgent: Telstra NOC

2020-10-02 Thread Terry Manderson
Thanks for the timely transparency Russell!! Efforts like this set a nice trajectory for the operational standards of networks operators in Australia. Look we get it, sometimes s..t happens. Owning it and fixing it, Speaks volumes. Cheers, Terry > On 2 Oct 2020, at 6:22 pm, Russell Langton

Re: [AusNOG] Urgent: Telstra NOC

2020-10-02 Thread Russell Langton
Hi all, Telstra blog post on issue is now live - http://exchange.telstra.com.au/an-update-on-our-september-30-bgp-issue/ On Thu, 1 Oct 2020, 8:32 am James Gray, wrote: > On 30 Sep 2020, at 8:24 pm, Paul Gear wrote: > > > That escalated quickly... :-) > > > > >

Re: [AusNOG] Urgent: Telstra NOC

2020-09-30 Thread James Gray
On 30 Sep 2020, at 8:24 pm, Paul Gear wrote: > That escalated quickly... :-) > > https://honestnetworker.net/2020/09/29/telstra-engineer-managing-prefix-lists/ CRIKEY!! Gold :) I mean, Telstra is the perpetual “whipping boy” but this is kinda funny in a harmless way, and I don’t mind that.

Re: [AusNOG] Urgent: Telstra NOC

2020-09-30 Thread Paul Gear
That escalated quickly... :-) https://honestnetworker.net/2020/09/29/telstra-engineer-managing-prefix-lists/ On 30/9/20 7:38 am, DaZZa wrote: > What part of "Root cause investigations are underway" is difficult to > understand? > > As much as I have an intense dislike for all things Telstra,

Re: [AusNOG] Urgent: Telstra NOC

2020-09-29 Thread Mark Andrews
Yes there is an assumption there. I may be wrong on this occasion but the industry has a whole and repeated episodes on people reporting that the NOCs are unreachable and then coming here, nanog, etc. to go in via the back door. For NOCs that are known to be responsive other members usually say

Re: [AusNOG] Urgent: Telstra NOC

2020-09-29 Thread Nick Stallman
I completely understand people wanting answers, since Telstra hijacked our IP block (and others) back in November 2018 and blackholed all traffic. It's a massively disruptive thing to occur to you, and there's absolutely nothing you can do to prevent it (while providers don't enforce RPKI and

Re: [AusNOG] Urgent: Telstra NOC

2020-09-29 Thread David Hughes
That's an assumption Mark. The 'formal channel' for a network operator chasing a route leak would be to look up PeeringDB and try that contact info. We don't k now if the OP tried that so we don't know if that failed. The fact that Telstra has staff (and I assume it's more than just Russell)

Re: [AusNOG] Urgent: Telstra NOC

2020-09-29 Thread Noel Butler
Story on Itnews this morning, its seem the problem was far bigger than just a few ip ranges :) https://www.itnews.com.au/news/telstra-routing-flub-affects-hundreds-of-networks-worldwide-554097 On 30/09/2020 09:42, Mark Andrews wrote: > That however does mean that Telstra's formal channels

Re: [AusNOG] Urgent: Telstra NOC

2020-09-29 Thread Mark Andrews
That however does mean that Telstra’s formal channels aren’t working properly. If you can’t reach the NOC via formal channels then there is a problem. No one should have to come here and ask for bad announcements to be withdrawn. Russell shouldn’t have to do anything but enjoy his time off when

Re: [AusNOG] Urgent: Telstra NOC

2020-09-29 Thread David Hughes
I'll second this. Having Russell and is peers on the list and in a position to help resolve problems in a timely manner is fantastic for our industry and our clients. Telstra is a big beast with lots of moving parts, yet I've lost count of the number of times issues have been addressed

Re: [AusNOG] Urgent: Telstra NOC

2020-09-29 Thread Nikolas Geyer
Job didn’t ask for a RCA, he asked what action was taken at a very specific time that resolved the impact. This had major effects on the global Internet routing system, and a lot of networks have applied some level of traffic engineering against AS1222 originated routes. I don’t think it

Re: [AusNOG] Urgent: Telstra NOC

2020-09-29 Thread Joseph Goldman
Perhaps it is best you take it through official channels then, rather than pushing for information from a staffer who is  a) on leave  b) has no obligation to provide support through this channel - but does so for the good of the community  It is amazing that it got fixed this way at all.

Re: [AusNOG] Urgent: Telstra NOC

2020-09-29 Thread Job Snijders
Dear 'DaZZa', On Wed, Sep 30, 2020 at 07:38:24AM +1000, DaZZa wrote: > What part of "Root cause investigations are underway" is difficult to > understand? I didn't ask for a root cause analysis. > As much as I have an intense dislike for all things Telstra, this got > fixed at 5:48 am AEST,

Re: [AusNOG] Urgent: Telstra NOC

2020-09-29 Thread DaZZa
What part of "Root cause investigations are underway" is difficult to understand? As much as I have an intense dislike for all things Telstra, this got fixed at 5:48 am AEST, according to Russell - it's now barely 07:40 am AEST - demanding a root cause analysis in less than 2 hours from an

Re: [AusNOG] Urgent: Telstra NOC

2020-09-29 Thread Job Snijders
Dear Russell, At 20:32 UTC something transpired that remedied the issue, what was that action? What assurances do we have that this doesn't happen again in the next 24 hours? Kind regards, Job On Wed, Sep 30, 2020 at 07:06:38AM +1000, Russell Langton wrote: > Hi job, > Root cause

Re: [AusNOG] Urgent: Telstra NOC

2020-09-29 Thread Russell Langton
Hi job, Root cause investigations are underway with the teams. Thanks. On Wed, 30 Sep 2020, 6:49 am Job Snijders, wrote: > Dear Russell, > > Thank you for this update. > > Do you have any information on what transpired? > > Kind regards, > > Job > > On Wed, Sep 30, 2020 at 06:47:00AM +1000,

Re: [AusNOG] Urgent: Telstra NOC

2020-09-29 Thread Job Snijders
Dear Russell, Thank you for this update. Do you have any information on what transpired? Kind regards, Job On Wed, Sep 30, 2020 at 06:47:00AM +1000, Russell Langton wrote: > Hi, > This should be resolved now. > Please advise if any further issues. > > > On Wed, 30 Sep 2020, 5:48 am Russell

Re: [AusNOG] Urgent: Telstra NOC

2020-09-29 Thread Sadiq Saif
Looks good to me now, thank you again for your assistance! On Tue, 29 Sep 2020, at 16:47, Russell Langton wrote: > Hi, > This should be resolved now. > Please advise if any further issues. > > > On Wed, 30 Sep 2020, 5:48 am Russell Langton, wrote: > > Hi, > > I'm on leave but getting the

Re: [AusNOG] Urgent: Telstra NOC

2020-09-29 Thread Russell Langton
Hi, This should be resolved now. Please advise if any further issues. On Wed, 30 Sep 2020, 5:48 am Russell Langton, wrote: > Hi, > I'm on leave but getting the right teams to review asap. > > > On Wed, 30 Sep 2020, 4:44 am Sadiq Saif, wrote: > >> Can somebody from Telstra's NOC contact me off

Re: [AusNOG] Urgent: Telstra NOC

2020-09-29 Thread Russell Langton
Hi, I'm on leave but getting the right teams to review asap. On Wed, 30 Sep 2020, 4:44 am Sadiq Saif, wrote: > Can somebody from Telstra's NOC contact me off list please? > > Y'all are leaking the following prefixes: > 208.100.53.0/24, 67.202.70.0/24, 69.162.134.0/24, 208.117.38.0/24, >

[AusNOG] Urgent: Telstra NOC

2020-09-29 Thread Sadiq Saif
Can somebody from Telstra's NOC contact me off list please? Y'all are leaking the following prefixes: 208.100.53.0/24, 67.202.70.0/24, 69.162.134.0/24, 208.117.38.0/24, 50.31.98.0/24 Correct origin is AS32748, not AS1221. I am contacting on behalf of my employer who is a Steadfast customer.