Re: [VoiceOps] VoIP Innovations reliability (NeuStar DDoS

[Tim Linn]

Kraig,

That's a great video showing the routing changes that we have made over the 
course of the issue, thank you for sharing.  We have made a lot of changes over 
the course of these issues in hopes to combat and mitigate the problems being 
caused by these attacks. I hope to have an extensive, detailed description of 
the attacks and what we did at some point in the near future for Ryan's email 
thread about DDoS attacks. In the meantime, I can share a PR release that we 
made yesterday describing this mitigation addition:

http://www.prweb.com/releases/2016/03/prweb13286689.htm


Timothy Linn
Lead Systems Engineer
Voip Innovations

--

Message: 4
Date: Mon, 21 Mar 2016 21:29:15 -0400
From: Kraig Beahn <kr...@enguity.com>
To: "voiceops@voiceops.org" <voiceops@voiceops.org>
Subject: Re: [VoiceOps] VoIP Innovations reliability (NeuStar DDoS
Mitigation Addition)
Message-ID:

Re: [VoiceOps] VoIP Innovations reliability

[Robert Johnson]

You're right. Was this mentioned anywhere other than buried in that
maintenance notification?

On Thu, Mar 17, 2016 at 6:45 PM, Nathan Anderson <nath...@fsr.com> wrote:

> Not true re: DR IP.  Details about it are not being broadcast publicly,
> though, to prevent the attackers from getting their hands on it.  You are
> probably thinking of their primary and secondary SBCs within the main data
> center.  For details about the DR SBC you need to contact VI (or read your
> e-mails).
>
>
>
> -- Nathan
>
>
>
> *From:* VoiceOps [mailto:voiceops-boun...@voiceops.org] *On Behalf Of *Robert
> Johnson
> *Sent:* Thursday, March 17, 2016 3:43 PM
>
> *To:* voiceops@voiceops.org
> *Subject:* Re: [VoiceOps] VoIP Innovations reliability
>
>
>
> Their primary SBC and DR IP are both in the same IP netblock, so whenever
> the DDOS hits, both IPs are affected. The past few outages have involved
> 80% packet loss or so to both hosts, so some calls do make their way
> through, and plenty of wierdness ensues when an INVITE makes its way
> through but not the OK on the way back.
>
>
>
> Can't wait to get our numbers ported out.
>
>
>
> On Thu, Mar 17, 2016 at 6:27 PM, Nate Burke <n...@blastcomm.com> wrote:
>
> Annnd they're down again.
>
>
>
> On 3/17/2016 5:14 PM, Nate Burke wrote:
>
> 6 calls from 4 different CID numbers.  All within 3 minutes.
>
> On 3/17/2016 4:58 PM, Nathan Anderson wrote:
>
> Yesterday I definitely saw calls coming from the DR IP in my logs, but I
> had not yet added that IP as a peerin our SBC.  I'll have to comb through
> logs today to see if we got any.
>
>
>
> Are you saying that the multiple calls you saw coming to your desk were
> all from the same number?  If I had to guess, their side probably sent
> continuous INVITEs to you when it failed to get back an OK for any of them
> (not that you weren't sending back OK, but that their either didn't reach
> their SBC or did not reach it in a timely manner).
>
>
>
> -- Nathan
>
>
>
> *From:* VoiceOps [mailto:voiceops-boun...@voiceops.org
> <voiceops-boun...@voiceops.org>] *On Behalf Of *Jeff Waddell
> *Sent:* Thursday, March 17, 2016 2:53 PM
> *To:* Nate Burke
> *Cc:* voiceops@voiceops.org
> *Subject:* Re: [VoiceOps] VoIP Innovations reliability
>
>
>
> That is the issue a lot of our customers are reporting - where multiple
> calls are sent
>
>
>
> On Thu, Mar 17, 2016 at 5:49 PM, Nate Burke <n...@blastcomm.com> wrote:
>
> I didn't see any traffic increment on the DR IP Address in my firewall
> rules, but this was odd.  During the 15 minute period, I had probably 5 or
> 6 simultaneous calls ring into my desk.  I normally only take a handfull of
> calls a day.
>
>
>
> On 3/17/2016 4:39 PM, Jeff Waddell wrote:
>
> We implemented it too - I haven't checked to see if any traffic was sent
> across it
>
>
>
>
>
>
>
> On Thu, Mar 17, 2016 at 5:20 PM, Nate Burke <n...@blastcomm.com> wrote:
>
> Only 15 Minutes this time though.  I had implemented the Disaster Recover
> Trunk as mentioned previously, but I didn't seem to be getting any calls
> completed through it.
>
>
>
> On 3/17/2016 4:16 PM, Shripal Daphtary wrote:
>
> Down again!
>
> Thanks,
>
>
>
> Shripal
>
>
> On Mar 16, 2016, at 9:50 AM, Nate Burke <n...@blastcomm.com> wrote:
>
> Looks like it just came back up for me.  Just over 30 min.
>
> Nate
>
> On 3/16/2016 8:45 AM, Shripal Daphtary wrote:
>
> We are experiencing an outage as well.
>
> Thanks,
>
>
>
> Shripal
>
>
> On Mar 16, 2016, at 9:36 AM, Nate Burke <n...@blastcomm.com> wrote:
>
> Problems again this morning?  Looks to be acting the same as it has been.
>
> On 3/11/2016 6:00 PM, Alexander Lopez wrote:
>
> I added them to our monitoring platform, stated getting alarms this past
> hour or so.
>
> Up and down.
>
>
>
>  Original message 
> From: Nathan Anderson <nath...@fsr.com> <nath...@fsr.com>
> Date: 3/11/2016 6:31 PM (GMT-05:00)
> To: 'Nate Burke' <n...@blastcomm.com> <n...@blastcomm.com>,
> voiceops@voiceops.org
> Subject: Re: [VoiceOps] VoIP Innovations reliability
>
> ...aand we're back.
>
> -- Nathan
>
> -Original Message-
> From: VoiceOps [mailto:voiceops-boun...@voiceops.org
> <voiceops-boun...@voiceops.org>] On Behalf Of Nathan Anderson
> Sent: Friday, March 11, 2016 3:30 PM
> To: 'Nate Burke'; voiceops@voiceops.org
> Subject: Re: [VoiceOps] VoIP Innovations reliability
>
> It *feels* like they are under attack again, since I get a response to a
&

Re: [VoiceOps] VoIP Innovations reliability

[Paul Timmins]

Sadly, assassins win if i stop breathing, so if someone starts to choke 
me, I fight it the best I can and run away at my earliest opportunity.


On 03/17/2016 07:38 PM, Anthony Orlando via VoiceOps wrote:
It's a shame we can't support them. This could be anyone of us. 
Hackers win if you port away.


Sent from my iPhone

On Mar 17, 2016, at 5:43 PM, Robert Johnson <fasterfour...@gmail.com 
<mailto:fasterfour...@gmail.com>> wrote:


Their primary SBC and DR IP are both in the same IP netblock, so 
whenever the DDOS hits, both IPs are affected. The past few outages 
have involved 80% packet loss or so to both hosts, so some calls do 
make their way through, and plenty of wierdness ensues when an INVITE 
makes its way through but not the OK on the way back.


Can't wait to get our numbers ported out.

On Thu, Mar 17, 2016 at 6:27 PM, Nate Burke <n...@blastcomm.com 
<mailto:n...@blastcomm.com>> wrote:


Annnd they're down again.


On 3/17/2016 5:14 PM, Nate Burke wrote:

6 calls from 4 different CID numbers.  All within 3 minutes.

On 3/17/2016 4:58 PM, Nathan Anderson wrote:


Yesterday I definitely saw calls coming from the DR IP in my
logs, but I had not yet added that IP as a peerin our SBC. 
I'll have to comb through logs today to see if we got any.


Are you saying that the multiple calls you saw coming to your
desk were all from the same number?  If I had to guess, their
side probably sent continuous INVITEs to you when it failed to
get back an OK for any of them (not that you weren't sending
back OK, but that their either didn't reach their SBC or did
not reach it in a timely manner).

-- Nathan

*From:*VoiceOps [mailto:voiceops-boun...@voiceops.org] *On
Behalf Of *Jeff Waddell
*Sent:* Thursday, March 17, 2016 2:53 PM
*To:* Nate Burke
*Cc:* voiceops@voiceops.org <mailto:voiceops@voiceops.org>
*Subject:* Re: [VoiceOps] VoIP Innovations reliability

That is the issue a lot of our customers are reporting - where
multiple calls are sent

On Thu, Mar 17, 2016 at 5:49 PM, Nate Burke <n...@blastcomm.com
<mailto:n...@blastcomm.com>> wrote:

I didn't see any traffic increment on the DR IP Address in my
firewall rules, but this was odd.  During the 15 minute period,
I had probably 5 or 6 simultaneous calls ring into my desk.  I
normally only take a handfull of calls a day.

On 3/17/2016 4:39 PM, Jeff Waddell wrote:

We implemented it too - I haven't checked to see if any
traffic was sent across it

On Thu, Mar 17, 2016 at 5:20 PM, Nate Burke
<n...@blastcomm.com <mailto:n...@blastcomm.com>> wrote:

Only 15 Minutes this time though.  I had implemented the
Disaster Recover Trunk as mentioned previously, but I
didn't seem to be getting any calls completed through it.

On 3/17/2016 4:16 PM, Shripal Daphtary wrote:

Down again!

Thanks,

Shripal


On Mar 16, 2016, at 9:50 AM, Nate Burke
<n...@blastcomm.com <mailto:n...@blastcomm.com>> wrote:

Looks like it just came back up for me. Just over
30 min.

Nate

On 3/16/2016 8:45 AM, Shripal Daphtary wrote:

We are experiencing an outage as well.

Thanks,

Shripal


On Mar 16, 2016, at 9:36 AM, Nate Burke
<n...@blastcomm.com
<mailto:n...@blastcomm.com>> wrote:

Problems again this morning?  Looks to be
acting the same as it has been.

On 3/11/2016 6:00 PM, Alexander Lopez wrote:

I added them to our monitoring
platform, stated getting alarms this
past hour or so.

Up and down.



 Original message 
From: Nathan Anderson <nath...@fsr.com>
<mailto:nath...@fsr.com>
Date: 3/11/2016 6:31 PM (GMT-05:00)
To: 'Nate Burke' <n...@blastcomm.com>
<mailto:n...@blastcomm.com>,
voiceops@voiceops.org
        <mailto:voiceops@voiceops.org>
Subject: Re: [VoiceOps] VoIP
Innovations reliability

...aand we're back.

-- Nathan

-Original Message-
From: VoiceOps
[mailto:voiceops-boun...@voiceops.org]
On Behalf Of Nathan And

Re: [VoiceOps] VoIP Innovations reliability

[Nathan Anderson]

Down again.   me.

-- Nathan

From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of Nate Burke
Sent: Wednesday, March 16, 2016 6:51 AM
To: voiceops@voiceops.org
Subject: Re: [VoiceOps] VoIP Innovations reliability

Looks like it just came back up for me.  Just over 30 min.

Nate
On 3/16/2016 8:45 AM, Shripal Daphtary wrote:
We are experiencing an outage as well.
Thanks,

Shripal

On Mar 16, 2016, at 9:36 AM, Nate Burke 
<n...@blastcomm.com<mailto:n...@blastcomm.com>> wrote:
Problems again this morning?  Looks to be acting the same as it has been.
On 3/11/2016 6:00 PM, Alexander Lopez wrote:
I added them to our monitoring platform, stated getting alarms this past hour 
or so.
Up and down.


 Original message 
From: Nathan Anderson <nath...@fsr.com><mailto:nath...@fsr.com>
Date: 3/11/2016 6:31 PM (GMT-05:00)
To: 'Nate Burke' <n...@blastcomm.com><mailto:n...@blastcomm.com>, 
voiceops@voiceops.org<mailto:voiceops@voiceops.org>
Subject: Re: [VoiceOps] VoIP Innovations reliability
...aand we're back.

-- Nathan

-Original Message-
From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of Nathan 
Anderson
Sent: Friday, March 11, 2016 3:30 PM
To: 'Nate Burke'; voiceops@voiceops.org<mailto:voiceops@voiceops.org>
Subject: Re: [VoiceOps] VoIP Innovations reliability

It *feels* like they are under attack again, since I get a response to a ping 
once every 20 or so.

-- Nathan

-Original Message-
From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of Nathan 
Anderson
Sent: Friday, March 11, 2016 3:28 PM
To: 'Nate Burke'; voiceops@voiceops.org<mailto:voiceops@voiceops.org>
Subject: Re: [VoiceOps] VoIP Innovations reliability

Confirmed.

-- Nathan

-Original Message-
From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of Nate Burke
Sent: Friday, March 11, 2016 3:26 PM
To: voiceops@voiceops.org<mailto:voiceops@voiceops.org>
Subject: Re: [VoiceOps] VoIP Innovations reliability

Anyone else show them down again right now?  My traceroutes aren't even
leaving Chicago.  Dying at a Chicago hop on Level3.

Nate

On 3/6/2016 6:50 PM, Nathan Anderson wrote:
> Did anybody else just suffer another 45-minute-ish long outage from about 
> 4:00p PST to 4:45p PST (ending about 5 minutes ago)?
>
> -- Nathan
>
> -Original Message-
> From: frnk...@iname.com<mailto:frnk...@iname.com> [mailto:frnk...@iname.com]
> Sent: Sunday, March 06, 2016 4:43 PM
> To: Nathan Anderson; voiceops@voiceops.org<mailto:voiceops@voiceops.org>
> Subject: RE: VoIP Innovations reliability
>
> More here: http://blog.voipinnovations.com/blog
>
> Frank
>
> -Original Message-
> From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of Nathan
> Anderson
> Sent: Tuesday, March 01, 2016 8:27 PM
> To: voiceops@voiceops.org<mailto:voiceops@voiceops.org>
> Subject: [VoiceOps] VoIP Innovations reliability
>
> Holy schlamoly.  Anybody else use them here and being handed outage after
> outage over the last 2 days?  Seriously thinking at this point about doing
> something else.  This is ridiculous.
>
> I desperately need sleep and if my cell goes off one more time...
>
> -- Nathan
> ___
> VoiceOps mailing list
> VoiceOps@voiceops.org<mailto:VoiceOps@voiceops.org>
> https://puck.nether.net/mailman/listinfo/voiceops
>
>
>
> ___
> VoiceOps mailing list
> VoiceOps@voiceops.org<mailto:VoiceOps@voiceops.org>
> https://puck.nether.net/mailman/listinfo/voiceops

___
VoiceOps mailing list
VoiceOps@voiceops.org<mailto:VoiceOps@voiceops.org>
https://puck.nether.net/mailman/listinfo/voiceops

___
VoiceOps mailing list
VoiceOps@voiceops.org<mailto:VoiceOps@voiceops.org>
https://puck.nether.net/mailman/listinfo/voiceops

___
VoiceOps mailing list
VoiceOps@voiceops.org<mailto:VoiceOps@voiceops.org>
https://puck.nether.net/mailman/listinfo/voiceops

___
VoiceOps mailing list
VoiceOps@voiceops.org<mailto:VoiceOps@voiceops.org>
https://puck.nether.net/mailman/listinfo/voiceops

___
VoiceOps mailing list
VoiceOps@voiceops.org<mailto:VoiceOps@voiceops.org>
https://puck.nether.net/mailman/listinfo/voiceops

___
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops

Re: [VoiceOps] VoIP Innovations reliability

[Nate Burke]

Problems again this morning?  Looks to be acting the same as it has been.

On 3/11/2016 6:00 PM, Alexander Lopez wrote:
I added them to our monitoring platform, stated getting alarms this 
past hour or so.

Up and down.


 Original message 
From: Nathan Anderson <nath...@fsr.com>
Date: 3/11/2016 6:31 PM (GMT-05:00)
To: 'Nate Burke' <n...@blastcomm.com>, voiceops@voiceops.org
Subject: Re: [VoiceOps] VoIP Innovations reliability

...aand we're back.

-- Nathan

-Original Message-
From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of 
Nathan Anderson

Sent: Friday, March 11, 2016 3:30 PM
To: 'Nate Burke'; voiceops@voiceops.org
Subject: Re: [VoiceOps] VoIP Innovations reliability

It *feels* like they are under attack again, since I get a response to 
a ping once every 20 or so.


-- Nathan

-Original Message-
From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of 
Nathan Anderson

Sent: Friday, March 11, 2016 3:28 PM
To: 'Nate Burke'; voiceops@voiceops.org
Subject: Re: [VoiceOps] VoIP Innovations reliability

Confirmed.

-- Nathan

-Original Message-
From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of 
Nate Burke

Sent: Friday, March 11, 2016 3:26 PM
To: voiceops@voiceops.org
Subject: Re: [VoiceOps] VoIP Innovations reliability

Anyone else show them down again right now?  My traceroutes aren't even
leaving Chicago.  Dying at a Chicago hop on Level3.

Nate

On 3/6/2016 6:50 PM, Nathan Anderson wrote:
> Did anybody else just suffer another 45-minute-ish long outage from 
about 4:00p PST to 4:45p PST (ending about 5 minutes ago)?

>
> -- Nathan
>
> -Original Message-
> From: frnk...@iname.com [mailto:frnk...@iname.com]
> Sent: Sunday, March 06, 2016 4:43 PM
> To: Nathan Anderson; voiceops@voiceops.org
> Subject: RE: VoIP Innovations reliability
>
> More here: http://blog.voipinnovations.com/blog
>
> Frank
>
> -Original Message-
> From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of 
Nathan

> Anderson
> Sent: Tuesday, March 01, 2016 8:27 PM
> To: voiceops@voiceops.org
> Subject: [VoiceOps] VoIP Innovations reliability
>
> Holy schlamoly.  Anybody else use them here and being handed outage 
after
> outage over the last 2 days?  Seriously thinking at this point about 
doing

> something else.  This is ridiculous.
>
> I desperately need sleep and if my cell goes off one more time...
>
> -- Nathan
> ___
> VoiceOps mailing list
> VoiceOps@voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
>
>
>
> ___
> VoiceOps mailing list
> VoiceOps@voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops

___
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops

___
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops

___
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops

___
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops


___
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops

Re: [VoiceOps] VoIP Innovations reliability

[Ivan Kovacevic]

Peter,



We do connect privately to most providers and *larger clients*. This is
usually a Fibre cross connect or an MPLS extension, private IP address
range. And yes we do work with some of those Tier1’s below using their
“internet peer” model (although at least some of them can also get you on
their IPVPN service, which should be private and separate from their
Dedicated Internet Access technology, but our volume with these providers
hasn’t really warranted us digging too deep).



Where we use the Internet we are able to CIC away this traffic on short
notice. We connect via public internet for testing, smaller clients (or
providers) or for clients who do not want to put in a private connection…
but we make sure the client understands we may not be able to help them if
something goes sideways, although we still always try – at least to
identify the hop that’s causing packet loss (or whatever)… and we have
multiple (more than two) sites with different multi-homed (BGP) internet
connections so that we can (hopefully) avoid being taken out by DDOS and
work around the inherent Internet **funkiness**.



I am not saying it is bullet proof (jinx), but these are things that we’ve
found work for us.



Best Regards,



Ivan Kovacevic

Vice President, Client Services

Star Telecom | www.startelecom.ca | SIP Based Services for Contact Centers
| LinkedIn
<http://www.linkedin.com/company/star-telecom-www.startelecom.ca-?trk=top_nav_home>



*From:* Pete Eisengrein [mailto:peeip...@gmail.com]
*Sent:* Friday, March 18, 2016 11:54 AM
*To:* Ivan Kovacevic <ivan.kovace...@startelecom.ca>
*Cc:* voiceops@voiceops.org
*Subject:* Re: [VoiceOps] VoIP Innovations reliability



> Although this will probably draw a well worded and highly sarcastic
> retort, I have to disagree :).

Not by me. I'm not that smart.


> Firstly, on my scale "impacted in some way" is way better than being
> completely down.



Agree.



> And secondly, we have had instances where the public side of our network
> was hammered by DDOS, while the private part of our network (including
> most connections to carriers and large clients) was not affected. So it
> does make a difference, at least in our environment.



So, are your customers connecting to a truly private side of your network?
I'm talking from my experience with the likes of Verizon, Level3,
CenturyLink, etc. We're directly connected, but that only means we are,
essentially, an internet peer. Unless they've been lying to me all these
years, they don't offer a truly private access to their gateways. Also,
I've connected to Star in the past and they were public IP's  ;-)







On Fri, Mar 18, 2016 at 10:38 AM, Ivan Kovacevic <
ivan.kovace...@startelecom.ca> wrote:

Although this will probably draw a well worded and highly sarcastic
retort, I have to disagree :).

Firstly, on my scale "impacted in some way" is way better than being
completely down.

And secondly, we have had instances where the public side of our network
was hammered by DDOS, while the private part of our network (including
most connections to carriers and large clients) was not affected. So it
does make a difference, at least in our environment.

Best Regards,

Ivan Kovacevic
Vice President, Client Services
Star Telecom | www.startelecom.ca | SIP Based Services for Contact Centers
| LinkedIn


-Original Message-
From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of Alex
Balashov
Sent: Thursday, March 17, 2016 10:53 PM
To: voiceops@voiceops.org
Subject: Re: [VoiceOps] VoIP Innovations reliability

On 03/17/2016 10:51 PM, Peter E wrote:

> If you use a carrier and they are attacked, chances are you're going
> to be impacted in some way, regardless if you are directly connected.

+1.

Direct connection vs. public Internet is irrelevant here.

--
Alex Balashov | Principal | Evariste Systems LLC
1447 Peachtree Street NE, Suite 700
Atlanta, GA 30309
United States

Tel: +1-800-250-5920 (toll-free) / +1-678-954-0671 (direct)
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
___
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops
___
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops
___
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops

Re: [VoiceOps] VoIP Innovations reliability

[Nate Burke]

I didn't see any traffic increment on the DR IP Address in my firewall 
rules, but this was odd.  During the 15 minute period, I had probably 5 
or 6 simultaneous calls ring into my desk.  I normally only take a 
handfull of calls a day.


On 3/17/2016 4:39 PM, Jeff Waddell wrote:
We implemented it too - I haven't checked to see if any traffic was 
sent across it




On Thu, Mar 17, 2016 at 5:20 PM, Nate Burke <n...@blastcomm.com 
<mailto:n...@blastcomm.com>> wrote:


Only 15 Minutes this time though.  I had implemented the Disaster
Recover Trunk as mentioned previously, but I didn't seem to be
getting any calls completed through it.


On 3/17/2016 4:16 PM, Shripal Daphtary wrote:

Down again!

Thanks,

Shripal

On Mar 16, 2016, at 9:50 AM, Nate Burke <n...@blastcomm.com
<mailto:n...@blastcomm.com>> wrote:


Looks like it just came back up for me. Just over 30 min.

Nate

On 3/16/2016 8:45 AM, Shripal Daphtary wrote:

We are experiencing an outage as well.

Thanks,

Shripal

On Mar 16, 2016, at 9:36 AM, Nate Burke <n...@blastcomm.com
<mailto:n...@blastcomm.com>> wrote:


Problems again this morning?  Looks to be acting the same as
it has been.

On 3/11/2016 6:00 PM, Alexander Lopez wrote:

I added them to our monitoring platform, stated getting
alarms this past hour or so.
Up and down.


 Original message 
From: Nathan Anderson <nath...@fsr.com> <mailto:nath...@fsr.com>
Date: 3/11/2016 6:31 PM (GMT-05:00)
To: 'Nate Burke' <n...@blastcomm.com>
<mailto:n...@blastcomm.com>, voiceops@voiceops.org
    <mailto:voiceops@voiceops.org>
Subject: Re: [VoiceOps] VoIP Innovations reliability

...aand we're back.

-- Nathan

-Original Message-
From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On
Behalf Of Nathan Anderson
Sent: Friday, March 11, 2016 3:30 PM
To: 'Nate Burke'; voiceops@voiceops.org
    <mailto:voiceops@voiceops.org>
Subject: Re: [VoiceOps] VoIP Innovations reliability

It *feels* like they are under attack again, since I get a
response to a ping once every 20 or so.

-- Nathan

-Original Message-
From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On
Behalf Of Nathan Anderson
Sent: Friday, March 11, 2016 3:28 PM
To: 'Nate Burke'; voiceops@voiceops.org
<mailto:voiceops@voiceops.org>
Subject: Re: [VoiceOps] VoIP Innovations reliability

Confirmed.

-- Nathan

-Original Message-
From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On
Behalf Of Nate Burke
Sent: Friday, March 11, 2016 3:26 PM
    To: voiceops@voiceops.org <mailto:voiceops@voiceops.org>
Subject: Re: [VoiceOps] VoIP Innovations reliability

Anyone else show them down again right now?  My traceroutes
aren't even
leaving Chicago.  Dying at a Chicago hop on Level3.

Nate

On 3/6/2016 6:50 PM, Nathan Anderson wrote:
> Did anybody else just suffer another 45-minute-ish long
outage from about 4:00p PST to 4:45p PST (ending about 5
minutes ago)?
>
> -- Nathan
>
> -Original Message-
> From: frnk...@iname.com <mailto:frnk...@iname.com>
[mailto:frnk...@iname.com]
> Sent: Sunday, March 06, 2016 4:43 PM
> To: Nathan Anderson; voiceops@voiceops.org
<mailto:voiceops@voiceops.org>
> Subject: RE: VoIP Innovations reliability
>
> More here: http://blog.voipinnovations.com/blog
>
> Frank
>
> -Original Message-
> From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On
Behalf Of Nathan
> Anderson
> Sent: Tuesday, March 01, 2016 8:27 PM
> To: voiceops@voiceops.org <mailto:voiceops@voiceops.org>
> Subject: [VoiceOps] VoIP Innovations reliability
>
> Holy schlamoly.  Anybody else use them here and being
handed outage after
> outage over the last 2 days? Seriously thinking at this
point about doing
> something else.  This is ridiculous.
>
> I desperately need sleep and if my cell goes off one more
time...
>
> -- Nathan
> ___
> VoiceOps mailing list
> VoiceOps@voiceops.org <mailto:VoiceOps@voiceops.org>
> https://puck.nether.net/mailman/listinfo/voiceops
>
>
>
> ___
> VoiceOps mailing list
> VoiceOps@voiceops.org <mailto:VoiceOps@voiceops.org>
> https://puck.nether.net/mailman/listinfo/voiceops

___
VoiceOps mailing list
VoiceOps@voiceops.org <mailto:VoiceOps@voiceops.org>
ht

Re: [VoiceOps] VoIP Innovations reliability

[Jeff Waddell]

That is the issue a lot of our customers are reporting - where multiple
calls are sent

On Thu, Mar 17, 2016 at 5:49 PM, Nate Burke <n...@blastcomm.com> wrote:

> I didn't see any traffic increment on the DR IP Address in my firewall
> rules, but this was odd.  During the 15 minute period, I had probably 5 or
> 6 simultaneous calls ring into my desk.  I normally only take a handfull of
> calls a day.
>
> On 3/17/2016 4:39 PM, Jeff Waddell wrote:
>
> We implemented it too - I haven't checked to see if any traffic was sent
> across it
>
>
>
> On Thu, Mar 17, 2016 at 5:20 PM, Nate Burke <n...@blastcomm.com> wrote:
>
>> Only 15 Minutes this time though.  I had implemented the Disaster Recover
>> Trunk as mentioned previously, but I didn't seem to be getting any calls
>> completed through it.
>>
>>
>> On 3/17/2016 4:16 PM, Shripal Daphtary wrote:
>>
>> Down again!
>>
>> Thanks,
>>
>> Shripal
>>
>> On Mar 16, 2016, at 9:50 AM, Nate Burke < <n...@blastcomm.com>
>> n...@blastcomm.com> wrote:
>>
>> Looks like it just came back up for me.  Just over 30 min.
>>
>> Nate
>>
>> On 3/16/2016 8:45 AM, Shripal Daphtary wrote:
>>
>> We are experiencing an outage as well.
>>
>> Thanks,
>>
>> Shripal
>>
>> On Mar 16, 2016, at 9:36 AM, Nate Burke < <n...@blastcomm.com>
>> n...@blastcomm.com> wrote:
>>
>> Problems again this morning?  Looks to be acting the same as it has
>> been.
>>
>> On 3/11/2016 6:00 PM, Alexander Lopez wrote:
>>
>> I added them to our monitoring platform, stated getting alarms this past
>> hour or so.
>> Up and down.
>>
>>
>>  Original message 
>> From: Nathan Anderson <nath...@fsr.com><nath...@fsr.com>
>> <nath...@fsr.com>
>> Date: 3/11/2016 6:31 PM (GMT-05:00)
>> To: 'Nate Burke' <n...@blastcomm.com><n...@blastcomm.com>
>> <n...@blastcomm.com>, voiceops@voiceops.org
>> Subject: Re: [VoiceOps] VoIP Innovations reliability
>>
>> ...aand we're back.
>>
>> -- Nathan
>>
>> -Original Message-
>> From: VoiceOps [ <voiceops-boun...@voiceops.org>
>> mailto:voiceops-boun...@voiceops.org <voiceops-boun...@voiceops.org>] On
>> Behalf Of Nathan Anderson
>> Sent: Friday, March 11, 2016 3:30 PM
>> To: 'Nate Burke'; <voiceops@voiceops.org>voiceops@voiceops.org
>> Subject: Re: [VoiceOps] VoIP Innovations reliability
>>
>> It *feels* like they are under attack again, since I get a response to a
>> ping once every 20 or so.
>>
>> -- Nathan
>>
>> -Original Message-
>> From: VoiceOps [ <voiceops-boun...@voiceops.org>
>> mailto:voiceops-boun...@voiceops.org <voiceops-boun...@voiceops.org>] On
>> Behalf Of Nathan Anderson
>> Sent: Friday, March 11, 2016 3:28 PM
>> To: 'Nate Burke'; <voiceops@voiceops.org>voiceops@voiceops.org
>> Subject: Re: [VoiceOps] VoIP Innovations reliability
>>
>> Confirmed.
>>
>> -- Nathan
>>
>> -Original Message-
>> From: VoiceOps [ <voiceops-boun...@voiceops.org>
>> mailto:voiceops-boun...@voiceops.org <voiceops-boun...@voiceops.org>] On
>> Behalf Of Nate Burke
>> Sent: Friday, March 11, 2016 3:26 PM
>> To: voiceops@voiceops.org
>> Subject: Re: [VoiceOps] VoIP Innovations reliability
>>
>> Anyone else show them down again right now?  My traceroutes aren't even
>> leaving Chicago.  Dying at a Chicago hop on Level3.
>>
>> Nate
>>
>> On 3/6/2016 6:50 PM, Nathan Anderson wrote:
>> > Did anybody else just suffer another 45-minute-ish long outage from
>> about 4:00p PST to 4:45p PST (ending about 5 minutes ago)?
>> >
>> > -- Nathan
>> >
>> > -Original Message-
>> > From: <frnk...@iname.com>frnk...@iname.com [mailto:frnk...@iname.com
>> <frnk...@iname.com>]
>> > Sent: Sunday, March 06, 2016 4:43 PM
>> > To: Nathan Anderson; <voiceops@voiceops.org>voiceops@voiceops.org
>> > Subject: RE: VoIP Innovations reliability
>> >
>> > More here: <http://blog.voipinnovations.com/blog>
>> http://blog.voipinnovations.com/blog
>> >
>> > Frank
>> >
>> > -Original Message-
>> > From: VoiceOps [ <voiceops-boun...@voiceops.org>
>> mailto:voiceops-boun...@voiceops.org <voiceops-boun...@voiceops.org>] On
>> Behalf Of Nathan
>> > Anderson
>> >

Re: [VoiceOps] VoIP Innovations reliability

[Peter E]

> larger clients

Touché my friend.



On Mar 18, 2016, at 12:44, Ivan Kovacevic <ivan.kovace...@startelecom.ca> wrote:

Peter,
 
We do connect privately to most providers and larger clients. This is usually a 
Fibre cross connect or an MPLS extension, private IP address range. And yes we 
do work with some of those Tier1’s below using their “internet peer” model 
(although at least some of them can also get you on their IPVPN service, which 
should be private and separate from their Dedicated Internet Access technology, 
but our volume with these providers hasn’t really warranted us digging too 
deep).
 
Where we use the Internet we are able to CIC away this traffic on short notice. 
We connect via public internet for testing, smaller clients (or providers) or 
for clients who do not want to put in a private connection… but we make sure 
the client understands we may not be able to help them if something goes 
sideways, although we still always try – at least to identify the hop that’s 
causing packet loss (or whatever)… and we have multiple (more than two) sites 
with different multi-homed (BGP) internet connections so that we can 
(hopefully) avoid being taken out by DDOS and work around the inherent Internet 
*funkiness*.
 
I am not saying it is bullet proof (jinx), but these are things that we’ve 
found work for us.  
 
Best Regards,
 
Ivan Kovacevic
Vice President, Client Services
Star Telecom | www.startelecom.ca | SIP Based Services for Contact Centers | 
LinkedIn
 
From: Pete Eisengrein [mailto:peeip...@gmail.com] 
Sent: Friday, March 18, 2016 11:54 AM
To: Ivan Kovacevic <ivan.kovace...@startelecom.ca>
Cc: voiceops@voiceops.org
Subject: Re: [VoiceOps] VoIP Innovations reliability
 
> Although this will probably draw a well worded and highly sarcastic
> retort, I have to disagree :).

Not by me. I'm not that smart.

> Firstly, on my scale "impacted in some way" is way better than being
> completely down.
 
Agree.
 
> And secondly, we have had instances where the public side of our network
> was hammered by DDOS, while the private part of our network (including
> most connections to carriers and large clients) was not affected. So it
> does make a difference, at least in our environment.
 
So, are your customers connecting to a truly private side of your network? I'm 
talking from my experience with the likes of Verizon, Level3, CenturyLink, etc. 
We're directly connected, but that only means we are, essentially, an internet 
peer. Unless they've been lying to me all these years, they don't offer a truly 
private access to their gateways. Also, I've connected to Star in the past and 
they were public IP's  ;-)
 
 
 
On Fri, Mar 18, 2016 at 10:38 AM, Ivan Kovacevic 
<ivan.kovace...@startelecom.ca> wrote:
Although this will probably draw a well worded and highly sarcastic
retort, I have to disagree :).

Firstly, on my scale "impacted in some way" is way better than being
completely down.

And secondly, we have had instances where the public side of our network
was hammered by DDOS, while the private part of our network (including
most connections to carriers and large clients) was not affected. So it
does make a difference, at least in our environment.

Best Regards,

Ivan Kovacevic
Vice President, Client Services
Star Telecom | www.startelecom.ca | SIP Based Services for Contact Centers
| LinkedIn


-Original Message-
From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of Alex
Balashov
Sent: Thursday, March 17, 2016 10:53 PM
To: voiceops@voiceops.org
Subject: Re: [VoiceOps] VoIP Innovations reliability

On 03/17/2016 10:51 PM, Peter E wrote:

> If you use a carrier and they are attacked, chances are you're going
> to be impacted in some way, regardless if you are directly connected.

+1.

Direct connection vs. public Internet is irrelevant here.

--
Alex Balashov | Principal | Evariste Systems LLC
1447 Peachtree Street NE, Suite 700
Atlanta, GA 30309
United States

Tel: +1-800-250-5920 (toll-free) / +1-678-954-0671 (direct)
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
___
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops
___
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops
 ___
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops

Re: [VoiceOps] VoIP Innovations reliability

[Ivan Kovacevic]

Although this will probably draw a well worded and highly sarcastic
retort, I have to disagree :).

Firstly, on my scale "impacted in some way" is way better than being
completely down.

And secondly, we have had instances where the public side of our network
was hammered by DDOS, while the private part of our network (including
most connections to carriers and large clients) was not affected. So it
does make a difference, at least in our environment.

Best Regards,

Ivan Kovacevic
Vice President, Client Services
Star Telecom | www.startelecom.ca | SIP Based Services for Contact Centers
| LinkedIn


-Original Message-
From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of Alex
Balashov
Sent: Thursday, March 17, 2016 10:53 PM
To: voiceops@voiceops.org
Subject: Re: [VoiceOps] VoIP Innovations reliability

On 03/17/2016 10:51 PM, Peter E wrote:

> If you use a carrier and they are attacked, chances are you're going
> to be impacted in some way, regardless if you are directly connected.

+1.

Direct connection vs. public Internet is irrelevant here.

--
Alex Balashov | Principal | Evariste Systems LLC
1447 Peachtree Street NE, Suite 700
Atlanta, GA 30309
United States

Tel: +1-800-250-5920 (toll-free) / +1-678-954-0671 (direct)
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
___
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops
___
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops

Re: [VoiceOps] VoIP Innovations reliability

[Peter E]

Being directly connected is indeed better -- faster with fewer points of 
failure -- but is not infallible. It just means you're fewer AS hops away. If 
you use a carrier and they are attacked, chances are you're going to be 
impacted in some way, regardless if you are directly connected. Even the big 
guys have been victims of attack. Just a few short years ago (2011?) Verizon 
was attacked. AT was attacked (2012). UltraDNS/NeuStar was attacked (2014). 
It happens.

How you respond, both technically and customer-service-wise, makes all the 
difference. I think that's a hard lesson VI is learning from this. And for some 
on the list (I presume) are learning to diversify their access.

I agree with Alex that it's time the PSTN grows up and (we all) begin to 
architect the next generation PSTN to eliminate single route points of failure.

-Pete


On Mar 17, 2016, at 21:17, Alexander Lopez <alex.lo...@opsys.com> wrote:

I think that most on this list understand the inherent risks in using the 
Internet As A Cross Connect  (IaaCC, I want credit for that term BTW :-) ). 

However, we cannot just accept that this will be common place in the future. 
Dedicated links are becoming less and less prevalent as most move to the 
Internet to provide that last mile as well as back haul.

The Internet was built with resilience in its design, it is time that the PSTN 
steps up to the plate and breaks free from the single route architecture and 
provide multiple paths to connect a call. 

When my customers complain about a call not being perfect, I remind them that 
they are paying cents for what used to cost dollars. Sure it's cheaper, but 
99.9 percent of the time it's just fine.

 Since VI has most of the small to medium ITSP market, most of our competition 
is having the same issues, we are better off than most because we have multiple 
ingress points and don't have all eggs in the VI basket.

I wholeheartedly agree with what you said, you get what you pay for.




 Original message 
From: Ivan Kovacevic <ivan.kovace...@startelecom.ca> 
Date: 3/17/2016 8:55 PM (GMT-05:00) 
To: voiceops@voiceops.org 
Subject: Re: [VoiceOps] VoIP Innovations reliability 

Ahem… 
 
At a risk of this not being a popular point of view… I view issues with VoIP 
Innovations as being largely self-inflicted.
 
Not only on part of VoIP Innovations (no relationship) who, as it appears, 
could have designed and communicated the DR better; but also on part of people 
complaining about the outage. You are using a non-guaranteed service. No SLAs, 
no promises, no commitments. If stuff like this happens, it is only because it 
is to be expected. No one can guarantee the quality of Internet between your IP 
and most other IPs on the Internet. That’s the reality. And if you are relying 
on a non-guaranteed service, you should at least have contingencies to move 
traffic quickly (CIC TFNs) or reroute (outbound).
 
And where that’s not possible (DIDs), having a dedicated data link covered by 
SLAs may be more appropriate.
 
And if you are porting your services to another provider with whom you will 
also interconnect over the Internet, if I was your client, I would be looking 
elsewhere. Unless your clients are not looking for a guaranteed 100% (or 99% or 
whatever) uptime. In which case, staying with VoIP Innovations is just fine…
 
And I know the response will likely be that it isn’t economically feasible to 
get dedicated connectivity, but by making that argument you are just saying 
that your business (and your clients’ business) doesn’t require a guaranteed 
level of uptime.
 
And either your clients have made this decision knowingly, in exchange for cost 
savings (in which case you are still delivering what you promised) or you made  
this decision on behalf of your clients unbeknownst to them, in which case you 
are up a creek… because this is either going to happen again and you will have 
to justify yet again why their services are down, or you’ll have to pony up the 
cost of a dedicated connection without being to recover it from your’ clients’ 
service fees. Either way, it wasn’t a good idea in the first place and it isn’t 
VoIP Innovation’s fault, unless they mislead you by saying Internet 
communication is infallible, and you believed it… I have a bridge to sell you.
 

 
Best Regards,
 
Ivan Kovacevic
Vice President, Client Services
Star Telecom | www.startelecom.ca | SIP Based Services for Contact Centers
 
From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of Anthony 
Orlando via VoiceOps
Sent: Thursday, March 17, 2016 8:09 PM
To: Paul Timmins <p...@timmins.net>
Cc: voiceops@voiceops.org
Subject: Re: [VoiceOps] VoIP Innovations reliability
 
I get it Paul. It's just a shame the terrorists win. Too bad we can't put a 
bounty on their sorry asses. 

Sent from my iPhone

On Mar 17, 2016, at 6:55 PM, Paul Timmins <p...@timmins.net> wrote:

Sadly, assassins win if i stop breathi

Re: [VoiceOps] VoIP Innovations reliability

[Nate Burke]

Only 15 Minutes this time though.  I had implemented the Disaster 
Recover Trunk as mentioned previously, but I didn't seem to be getting 
any calls completed through it.


On 3/17/2016 4:16 PM, Shripal Daphtary wrote:

Down again!

Thanks,

Shripal

On Mar 16, 2016, at 9:50 AM, Nate Burke <n...@blastcomm.com 
<mailto:n...@blastcomm.com>> wrote:



Looks like it just came back up for me.  Just over 30 min.

Nate

On 3/16/2016 8:45 AM, Shripal Daphtary wrote:

We are experiencing an outage as well.

Thanks,

Shripal

On Mar 16, 2016, at 9:36 AM, Nate Burke <n...@blastcomm.com 
<mailto:n...@blastcomm.com>> wrote:


Problems again this morning?  Looks to be acting the same as it has 
been.


On 3/11/2016 6:00 PM, Alexander Lopez wrote:
I added them to our monitoring platform, stated getting alarms 
this past hour or so.

Up and down.


 Original message 
From: Nathan Anderson <nath...@fsr.com>
Date: 3/11/2016 6:31 PM (GMT-05:00)
To: 'Nate Burke' <n...@blastcomm.com>, voiceops@voiceops.org
Subject: Re: [VoiceOps] VoIP Innovations reliability

...aand we're back.

-- Nathan

-Original Message-
From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of 
Nathan Anderson

Sent: Friday, March 11, 2016 3:30 PM
To: 'Nate Burke'; voiceops@voiceops.org
Subject: Re: [VoiceOps] VoIP Innovations reliability

It *feels* like they are under attack again, since I get a 
response to a ping once every 20 or so.


-- Nathan

-Original Message-
From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of 
Nathan Anderson

Sent: Friday, March 11, 2016 3:28 PM
To: 'Nate Burke'; voiceops@voiceops.org
Subject: Re: [VoiceOps] VoIP Innovations reliability

Confirmed.

-- Nathan

-Original Message-
From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of 
Nate Burke

Sent: Friday, March 11, 2016 3:26 PM
To: voiceops@voiceops.org
Subject: Re: [VoiceOps] VoIP Innovations reliability

Anyone else show them down again right now?  My traceroutes aren't 
even

leaving Chicago.  Dying at a Chicago hop on Level3.

Nate

On 3/6/2016 6:50 PM, Nathan Anderson wrote:
> Did anybody else just suffer another 45-minute-ish long outage 
from about 4:00p PST to 4:45p PST (ending about 5 minutes ago)?

>
> -- Nathan
>
> -Original Message-
> From: frnk...@iname.com [mailto:frnk...@iname.com]
> Sent: Sunday, March 06, 2016 4:43 PM
> To: Nathan Anderson; voiceops@voiceops.org
> Subject: RE: VoIP Innovations reliability
>
> More here: http://blog.voipinnovations.com/blog
>
> Frank
>
> -Original Message-
> From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf 
Of Nathan

> Anderson
> Sent: Tuesday, March 01, 2016 8:27 PM
> To: voiceops@voiceops.org
> Subject: [VoiceOps] VoIP Innovations reliability
>
> Holy schlamoly.  Anybody else use them here and being handed 
outage after
> outage over the last 2 days?  Seriously thinking at this point 
about doing

> something else.  This is ridiculous.
>
> I desperately need sleep and if my cell goes off one more time...
>
> -- Nathan
> ___
> VoiceOps mailing list
> VoiceOps@voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
>
>
>
> ___
> VoiceOps mailing list
> VoiceOps@voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops

___
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops

___
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops

___
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops

___
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops


___
VoiceOps mailing list
VoiceOps@voiceops.org <mailto:VoiceOps@voiceops.org>
https://puck.nether.net/mailman/listinfo/voiceops


___
VoiceOps mailing list
VoiceOps@voiceops.org <mailto:VoiceOps@voiceops.org>
https://puck.nether.net/mailman/listinfo/voiceops


___
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops

Re: [VoiceOps] VoIP Innovations reliability

[Jeff Waddell]

Not that it helps a lot - this was posted on the VI facebook page

"[Update] As of this morning, we regret to inform you that we are again
currently experiencing issues that are related to the DDoS attack being
committed by unknown parties against our network. Our Network Engineers are
currently working diligently with our upstream carriers as well as a
network security firm to combat the attack and will continue to do so until
all of the issues are resolved. Our Senior Leadership has also been in
contact with the FBI. The FBI is considering these events a national
security issue due to the number of firms impacted, the magnitude of the
attacks, and the persistence of these attacks. The Social Media Team will
post here the very moment we have further information."

On Wed, Mar 16, 2016 at 10:29 AM, Nathan Anderson <nath...@fsr.com> wrote:

> Down again.   me.
>
>
>
> -- Nathan
>
>
>
> *From:* VoiceOps [mailto:voiceops-boun...@voiceops.org] *On Behalf Of *Nate
> Burke
> *Sent:* Wednesday, March 16, 2016 6:51 AM
>
> *To:* voiceops@voiceops.org
> *Subject:* Re: [VoiceOps] VoIP Innovations reliability
>
>
>
> Looks like it just came back up for me.  Just over 30 min.
>
> Nate
>
> On 3/16/2016 8:45 AM, Shripal Daphtary wrote:
>
> We are experiencing an outage as well.
>
> Thanks,
>
>
>
> Shripal
>
>
> On Mar 16, 2016, at 9:36 AM, Nate Burke <n...@blastcomm.com> wrote:
>
> Problems again this morning?  Looks to be acting the same as it has been.
>
> On 3/11/2016 6:00 PM, Alexander Lopez wrote:
>
> I added them to our monitoring platform, stated getting alarms this past
> hour or so.
>
> Up and down.
>
>
>
>  Original message 
> From: Nathan Anderson <nath...@fsr.com> <nath...@fsr.com>
> Date: 3/11/2016 6:31 PM (GMT-05:00)
> To: 'Nate Burke' <n...@blastcomm.com> <n...@blastcomm.com>,
> voiceops@voiceops.org
> Subject: Re: [VoiceOps] VoIP Innovations reliability
>
> ...aand we're back.
>
> -- Nathan
>
> -Original Message-----
> From: VoiceOps [mailto:voiceops-boun...@voiceops.org
> <voiceops-boun...@voiceops.org>] On Behalf Of Nathan Anderson
> Sent: Friday, March 11, 2016 3:30 PM
> To: 'Nate Burke'; voiceops@voiceops.org
> Subject: Re: [VoiceOps] VoIP Innovations reliability
>
> It *feels* like they are under attack again, since I get a response to a
> ping once every 20 or so.
>
> -- Nathan
>
> -Original Message-
> From: VoiceOps [mailto:voiceops-boun...@voiceops.org
> <voiceops-boun...@voiceops.org>] On Behalf Of Nathan Anderson
> Sent: Friday, March 11, 2016 3:28 PM
> To: 'Nate Burke'; voiceops@voiceops.org
> Subject: Re: [VoiceOps] VoIP Innovations reliability
>
> Confirmed.
>
> -- Nathan
>
> -Original Message-
> From: VoiceOps [mailto:voiceops-boun...@voiceops.org
> <voiceops-boun...@voiceops.org>] On Behalf Of Nate Burke
> Sent: Friday, March 11, 2016 3:26 PM
> To: voiceops@voiceops.org
> Subject: Re: [VoiceOps] VoIP Innovations reliability
>
> Anyone else show them down again right now?  My traceroutes aren't even
> leaving Chicago.  Dying at a Chicago hop on Level3.
>
> Nate
>
> On 3/6/2016 6:50 PM, Nathan Anderson wrote:
> > Did anybody else just suffer another 45-minute-ish long outage from
> about 4:00p PST to 4:45p PST (ending about 5 minutes ago)?
> >
> > -- Nathan
> >
> > -Original Message-
> > From: frnk...@iname.com [mailto:frnk...@iname.com <frnk...@iname.com>]
> > Sent: Sunday, March 06, 2016 4:43 PM
> > To: Nathan Anderson; voiceops@voiceops.org
> > Subject: RE: VoIP Innovations reliability
> >
> > More here: http://blog.voipinnovations.com/blog
> >
> > Frank
> >
> > -Original Message-
> > From: VoiceOps [mailto:voiceops-boun...@voiceops.org
> <voiceops-boun...@voiceops.org>] On Behalf Of Nathan
> > Anderson
> > Sent: Tuesday, March 01, 2016 8:27 PM
> > To: voiceops@voiceops.org
> > Subject: [VoiceOps] VoIP Innovations reliability
> >
> > Holy schlamoly.  Anybody else use them here and being handed outage after
> > outage over the last 2 days?  Seriously thinking at this point about
> doing
> > something else.  This is ridiculous.
> >
> > I desperately need sleep and if my cell goes off one more time...
> >
> > -- Nathan
> > ___
> > VoiceOps mailing list
> > VoiceOps@voiceops.org
> > https://puck.nether.net/mailman/listinfo/voiceops
> >
> >
> >
> > 

Re: [VoiceOps] VoIP Innovations reliability

[Robert Johnson]

Their primary SBC and DR IP are both in the same IP netblock, so whenever
the DDOS hits, both IPs are affected. The past few outages have involved
80% packet loss or so to both hosts, so some calls do make their way
through, and plenty of wierdness ensues when an INVITE makes its way
through but not the OK on the way back.

Can't wait to get our numbers ported out.

On Thu, Mar 17, 2016 at 6:27 PM, Nate Burke <n...@blastcomm.com> wrote:

> Annnd they're down again.
>
>
> On 3/17/2016 5:14 PM, Nate Burke wrote:
>
> 6 calls from 4 different CID numbers.  All within 3 minutes.
>
> On 3/17/2016 4:58 PM, Nathan Anderson wrote:
>
> Yesterday I definitely saw calls coming from the DR IP in my logs, but I
> had not yet added that IP as a peerin our SBC.  I'll have to comb through
> logs today to see if we got any.
>
>
>
> Are you saying that the multiple calls you saw coming to your desk were
> all from the same number?  If I had to guess, their side probably sent
> continuous INVITEs to you when it failed to get back an OK for any of them
> (not that you weren't sending back OK, but that their either didn't reach
> their SBC or did not reach it in a timely manner).
>
>
>
> -- Nathan
>
>
>
> *From:* VoiceOps [mailto:voiceops-boun...@voiceops.org
> <voiceops-boun...@voiceops.org>] *On Behalf Of *Jeff Waddell
> *Sent:* Thursday, March 17, 2016 2:53 PM
> *To:* Nate Burke
> *Cc:* voiceops@voiceops.org
> *Subject:* Re: [VoiceOps] VoIP Innovations reliability
>
>
>
> That is the issue a lot of our customers are reporting - where multiple
> calls are sent
>
>
>
> On Thu, Mar 17, 2016 at 5:49 PM, Nate Burke <n...@blastcomm.com> wrote:
>
> I didn't see any traffic increment on the DR IP Address in my firewall
> rules, but this was odd.  During the 15 minute period, I had probably 5 or
> 6 simultaneous calls ring into my desk.  I normally only take a handfull of
> calls a day.
>
>
>
> On 3/17/2016 4:39 PM, Jeff Waddell wrote:
>
> We implemented it too - I haven't checked to see if any traffic was sent
> across it
>
>
>
>
>
>
>
> On Thu, Mar 17, 2016 at 5:20 PM, Nate Burke < <n...@blastcomm.com>
> n...@blastcomm.com> wrote:
>
> Only 15 Minutes this time though.  I had implemented the Disaster Recover
> Trunk as mentioned previously, but I didn't seem to be getting any calls
> completed through it.
>
>
>
> On 3/17/2016 4:16 PM, Shripal Daphtary wrote:
>
> Down again!
>
> Thanks,
>
>
>
> Shripal
>
>
> On Mar 16, 2016, at 9:50 AM, Nate Burke < <n...@blastcomm.com>
> n...@blastcomm.com> wrote:
>
> Looks like it just came back up for me.  Just over 30 min.
>
> Nate
>
> On 3/16/2016 8:45 AM, Shripal Daphtary wrote:
>
> We are experiencing an outage as well.
>
> Thanks,
>
>
>
> Shripal
>
>
> On Mar 16, 2016, at 9:36 AM, Nate Burke < <n...@blastcomm.com>
> n...@blastcomm.com> wrote:
>
> Problems again this morning?  Looks to be acting the same as it has been.
>
> On 3/11/2016 6:00 PM, Alexander Lopez wrote:
>
> I added them to our monitoring platform, stated getting alarms this past
> hour or so.
>
> Up and down.
>
>
>
>  Original message 
> From: Nathan Anderson <nath...@fsr.com><nath...@fsr.com> <nath...@fsr.com>
> Date: 3/11/2016 6:31 PM (GMT-05:00)
> To: 'Nate Burke' <n...@blastcomm.com><n...@blastcomm.com>
> <n...@blastcomm.com>, <voiceops@voiceops.org>voiceops@voiceops.org
> Subject: Re: [VoiceOps] VoIP Innovations reliability
>
> ...aand we're back.
>
> -- Nathan
>
> -Original Message-
> From: VoiceOps [ <voiceops-boun...@voiceops.org>
> mailto:voiceops-boun...@voiceops.org <voiceops-boun...@voiceops.org>] On
> Behalf Of Nathan Anderson
> Sent: Friday, March 11, 2016 3:30 PM
> To: 'Nate Burke'; <voiceops@voiceops.org>voiceops@voiceops.org
> Subject: Re: [VoiceOps] VoIP Innovations reliability
>
> It *feels* like they are under attack again, since I get a response to a
> ping once every 20 or so.
>
> -- Nathan
>
> -Original Message-
> From: VoiceOps [ <voiceops-boun...@voiceops.org>
> mailto:voiceops-boun...@voiceops.org <voiceops-boun...@voiceops.org>] On
> Behalf Of Nathan Anderson
> Sent: Friday, March 11, 2016 3:28 PM
> To: 'Nate Burke'; <voiceops@voiceops.org>voiceops@voiceops.org
> Subject: Re: [VoiceOps] VoIP Innovations reliability
>
> Confirmed.
>
> -- Nathan
>
> -Original Message-
> From: VoiceOps [ <voiceops-boun...@voiceops.org>
> mailto:voiceops-boun...@voiceops.org <voiceops-bou

Re: [VoiceOps] VoIP Innovations reliability

[Shripal Daphtary]

We're good too. But long pdd and no ringback. 


Thanks, 

Shripal

> On Mar 16, 2016, at 9:50 AM, Nate Burke <n...@blastcomm.com> wrote:
> 
> Looks like it just came back up for me.  Just over 30 min.
> 
> Nate
> 
>> On 3/16/2016 8:45 AM, Shripal Daphtary wrote:
>> We are experiencing an outage as well. 
>> 
>> Thanks, 
>> 
>> Shripal
>> 
>> On Mar 16, 2016, at 9:36 AM, Nate Burke <n...@blastcomm.com> wrote:
>> 
>>> Problems again this morning?  Looks to be acting the same as it has been.  
>>> 
>>>> On 3/11/2016 6:00 PM, Alexander Lopez wrote:
>>>> I added them to our monitoring platform, stated getting alarms this past 
>>>> hour or so.
>>>> Up and down.
>>>> 
>>>> 
>>>>  Original message ----
>>>> From: Nathan Anderson <nath...@fsr.com> 
>>>> Date: 3/11/2016 6:31 PM (GMT-05:00) 
>>>> To: 'Nate Burke' <n...@blastcomm.com>, voiceops@voiceops.org 
>>>> Subject: Re: [VoiceOps] VoIP Innovations reliability 
>>>> 
>>>> ...aand we're back.
>>>> 
>>>> -- Nathan
>>>> 
>>>> -Original Message-
>>>> From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of Nathan 
>>>> Anderson
>>>> Sent: Friday, March 11, 2016 3:30 PM
>>>> To: 'Nate Burke'; voiceops@voiceops.org
>>>> Subject: Re: [VoiceOps] VoIP Innovations reliability
>>>> 
>>>> It *feels* like they are under attack again, since I get a response to a 
>>>> ping once every 20 or so.
>>>> 
>>>> -- Nathan
>>>> 
>>>> -Original Message-
>>>> From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of Nathan 
>>>> Anderson
>>>> Sent: Friday, March 11, 2016 3:28 PM
>>>> To: 'Nate Burke'; voiceops@voiceops.org
>>>> Subject: Re: [VoiceOps] VoIP Innovations reliability
>>>> 
>>>> Confirmed.
>>>> 
>>>> -- Nathan
>>>> 
>>>> -Original Message-
>>>> From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of Nate 
>>>> Burke
>>>> Sent: Friday, March 11, 2016 3:26 PM
>>>> To: voiceops@voiceops.org
>>>> Subject: Re: [VoiceOps] VoIP Innovations reliability
>>>> 
>>>> Anyone else show them down again right now?  My traceroutes aren't even 
>>>> leaving Chicago.  Dying at a Chicago hop on Level3.
>>>> 
>>>> Nate
>>>> 
>>>> On 3/6/2016 6:50 PM, Nathan Anderson wrote:
>>>> > Did anybody else just suffer another 45-minute-ish long outage from 
>>>> > about 4:00p PST to 4:45p PST (ending about 5 minutes ago)?
>>>> >
>>>> > -- Nathan
>>>> >
>>>> > -Original Message-
>>>> > From: frnk...@iname.com [mailto:frnk...@iname.com]
>>>> > Sent: Sunday, March 06, 2016 4:43 PM
>>>> > To: Nathan Anderson; voiceops@voiceops.org
>>>> > Subject: RE: VoIP Innovations reliability
>>>> >
>>>> > More here: http://blog.voipinnovations.com/blog
>>>> >
>>>> > Frank
>>>> >
>>>> > -Original Message-
>>>> > From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of Nathan
>>>> > Anderson
>>>> > Sent: Tuesday, March 01, 2016 8:27 PM
>>>> > To: voiceops@voiceops.org
>>>> > Subject: [VoiceOps] VoIP Innovations reliability
>>>> >
>>>> > Holy schlamoly.  Anybody else use them here and being handed outage after
>>>> > outage over the last 2 days?  Seriously thinking at this point about 
>>>> > doing
>>>> > something else.  This is ridiculous.
>>>> >
>>>> > I desperately need sleep and if my cell goes off one more time...
>>>> >
>>>> > -- Nathan
>>>> > ___
>>>> > VoiceOps mailing list
>>>> > VoiceOps@voiceops.org
>>>> > https://puck.nether.net/mailman/listinfo/voiceops
>>>> >
>>>> >
>>>> >
>>>> > ___
>>>> > VoiceOps mailing list
>>>> > VoiceOps@voiceops.org
>>>> > https://puck.nether.net/mailman/listinfo/voiceops
>>>> 
>>>> ___
>>>> VoiceOps mailing list
>>>> VoiceOps@voiceops.org
>>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>> 
>>>> ___
>>>> VoiceOps mailing list
>>>> VoiceOps@voiceops.org
>>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>> 
>>>> ___
>>>> VoiceOps mailing list
>>>> VoiceOps@voiceops.org
>>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>> 
>>>> ___
>>>> VoiceOps mailing list
>>>> VoiceOps@voiceops.org
>>>> https://puck.nether.net/mailman/listinfo/voiceops
>>> ___
>>> VoiceOps mailing list
>>> VoiceOps@voiceops.org
>>> https://puck.nether.net/mailman/listinfo/voiceops
> 
> ___
> VoiceOps mailing list
> VoiceOps@voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
___
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops

Re: [VoiceOps] VoIP Innovations reliability

[Tim Linn]

All,



I can confirm that we did have 2 outages this morning spanning little over an 
hour due to another DDoS attack.



We have been careful of the information that we have been giving out in public 
updates due to the possibility of the attacker reading that information and 
using it against us in some way.



The biggest change that we've made to combat this is a DDoS protection service, 
which has caught a handful of attacks, but was largely useless in this latest 
attack. We've made mitigating changes to our existing setup which appears to 
have stopped today's attack so far.



We do have a disaster recovery site and have, honestly, appear to have done a 
poor job communicating that to our customers. If you are a Voip Innovations 
customer, please contact me for the IP address of the disaster recovery site. 
During these outages, the site was taking origination calls from most of our 
DID vendors. We're currently working on figuring out why all of them did not 
switch over.



I know that words aren't much right now, but I am truly sorry for the issues 
that you have been having. I understand the importance of reliability to you 
and your customers, and I know that those expectations have not been met over 
the last few weeks. We are working diligently with several different companies 
to continue to make changes to our network to mitigate these attacks. The 
changes have been successful for a number of the attacks that we have seen, but 
it's clear that we are still vulnerable and have work to do.



Please contact me if you have any questions, and I will do my best to answer.




Timothy Linn
Lead Systems Engineer
Voip Innovations





--



Message: 1

Date: Wed, 16 Mar 2016 11:24:04 -0400

From: Jeff Waddell 
<jeff+voice...@waddellsolutions.com<mailto:jeff+voice...@waddellsolutions.com>>

To: Nathan Anderson <nath...@fsr.com<mailto:nath...@fsr.com>>

Cc: Nate Burke <n...@blastcomm.com<mailto:n...@blastcomm.com>>, 
"voiceops@voiceops.org<mailto:voiceops@voiceops.org>"

<voiceops@voiceops.org<mailto:voiceops@voiceops.org>>

Subject: Re: [VoiceOps] VoIP Innovations reliability

Message-ID:


<CAO+x49LG2+RQyRU6a0zmgdgaAKd0yYyxASHm6+=9z97idxj...@mail.gmail.com<mailto:CAO+x49LG2+RQyRU6a0zmgdgaAKd0yYyxASHm6+=9z97idxj...@mail.gmail.com>>

Content-Type: text/plain; charset="utf-8"



Not that it helps a lot - this was posted on the VI facebook page



"[Update] As of this morning, we regret to inform you that we are again 
currently experiencing issues that are related to the DDoS attack being 
committed by unknown parties against our network. Our Network Engineers are 
currently working diligently with our upstream carriers as well as a network 
security firm to combat the attack and will continue to do so until all of the 
issues are resolved. Our Senior Leadership has also been in contact with the 
FBI. The FBI is considering these events a national security issue due to the 
number of firms impacted, the magnitude of the attacks, and the persistence of 
these attacks. The Social Media Team will post here the very moment we have 
further information."



On Wed, Mar 16, 2016 at 10:29 AM, Nathan Anderson 
<nath...@fsr.com<mailto:nath...@fsr.com>> wrote:



> Down again.   me.

>

>

>

> -- Nathan

>

>

>

> *From:* VoiceOps [mailto:voiceops-boun...@voiceops.org] *On Behalf Of

> *Nate Burke

> *Sent:* Wednesday, March 16, 2016 6:51 AM

>

> *To:* voiceops@voiceops.org<mailto:voiceops@voiceops.org>

> *Subject:* Re: [VoiceOps] VoIP Innovations reliability

>

>

>

> Looks like it just came back up for me.  Just over 30 min.

>

> Nate

>

> On 3/16/2016 8:45 AM, Shripal Daphtary wrote:

>

> We are experiencing an outage as well.

>

> Thanks,

>

>

>

> Shripal

>

>

> On Mar 16, 2016, at 9:36 AM, Nate Burke 
> <n...@blastcomm.com<mailto:n...@blastcomm.com>> wrote:

>

> Problems again this morning?  Looks to be acting the same as it has been.

>

> On 3/11/2016 6:00 PM, Alexander Lopez wrote:

>

> I added them to our monitoring platform, stated getting alarms this

> past hour or so.

>

> Up and down.

>

>

>

>  Original message 

> From: Nathan Anderson <nath...@fsr.com<mailto:nath...@fsr.com>> 
> <nath...@fsr.com<mailto:nath...@fsr.com>>

> Date: 3/11/2016 6:31 PM (GMT-05:00)

> To: 'Nate Burke' <n...@blastcomm.com<mailto:n...@blastcomm.com>> 
> <n...@blastcomm.com<mailto:n...@blastcomm.com>>,

> voiceops@voiceops.org<mailto:voiceops@voiceops.org>

> Subject: Re: [VoiceOps] VoIP Innovations reliability

>

> ...aand we

Re: [VoiceOps] VoIP Innovations reliability

[Nate Burke]

6 calls from 4 different CID numbers.  All within 3 minutes.

On 3/17/2016 4:58 PM, Nathan Anderson wrote:


Yesterday I definitely saw calls coming from the DR IP in my logs, but 
I had not yet added that IP as a peerin our SBC.  I'll have to comb 
through logs today to see if we got any.


Are you saying that the multiple calls you saw coming to your desk 
were all from the same number?  If I had to guess, their side probably 
sent continuous INVITEs to you when it failed to get back an OK for 
any of them (not that you weren't sending back OK, but that their 
either didn't reach their SBC or did not reach it in a timely manner).


-- Nathan

*From:*VoiceOps [mailto:voiceops-boun...@voiceops.org] *On Behalf Of 
*Jeff Waddell

*Sent:* Thursday, March 17, 2016 2:53 PM
*To:* Nate Burke
*Cc:* voiceops@voiceops.org
*Subject:* Re: [VoiceOps] VoIP Innovations reliability

That is the issue a lot of our customers are reporting - where 
multiple calls are sent


On Thu, Mar 17, 2016 at 5:49 PM, Nate Burke <n...@blastcomm.com 
<mailto:n...@blastcomm.com>> wrote:


I didn't see any traffic increment on the DR IP Address in my firewall 
rules, but this was odd.  During the 15 minute period, I had probably 
5 or 6 simultaneous calls ring into my desk.  I normally only take a 
handfull of calls a day.


On 3/17/2016 4:39 PM, Jeff Waddell wrote:

We implemented it too - I haven't checked to see if any traffic
was sent across it

On Thu, Mar 17, 2016 at 5:20 PM, Nate Burke <n...@blastcomm.com
<mailto:n...@blastcomm.com>> wrote:

Only 15 Minutes this time though.  I had implemented the Disaster
Recover Trunk as mentioned previously, but I didn't seem to be
getting any calls completed through it.

On 3/17/2016 4:16 PM, Shripal Daphtary wrote:

Down again!

Thanks,

Shripal


On Mar 16, 2016, at 9:50 AM, Nate Burke <n...@blastcomm.com
<mailto:n...@blastcomm.com>> wrote:

Looks like it just came back up for me. Just over 30 min.

Nate

On 3/16/2016 8:45 AM, Shripal Daphtary wrote:

We are experiencing an outage as well.

Thanks,

Shripal


On Mar 16, 2016, at 9:36 AM, Nate Burke
<n...@blastcomm.com <mailto:n...@blastcomm.com>> wrote:

Problems again this morning?  Looks to be acting
the same as it has been.

On 3/11/2016 6:00 PM, Alexander Lopez wrote:

I added them to our monitoring platform,
stated getting alarms this past hour or so.

Up and down.



 Original message 
From: Nathan Anderson <nath...@fsr.com>
<mailto:nath...@fsr.com>
Date: 3/11/2016 6:31 PM (GMT-05:00)
To: 'Nate Burke' <n...@blastcomm.com>
<mailto:n...@blastcomm.com>,
voiceops@voiceops.org
        <mailto:voiceops@voiceops.org>
        Subject: Re: [VoiceOps] VoIP Innovations
reliability

...aand we're back.

-- Nathan

-Original Message-
From: VoiceOps
[mailto:voiceops-boun...@voiceops.org] On
Behalf Of Nathan Anderson
Sent: Friday, March 11, 2016 3:30 PM
To: 'Nate Burke'; voiceops@voiceops.org
        <mailto:voiceops@voiceops.org>
    Subject: Re: [VoiceOps] VoIP Innovations
reliability

It *feels* like they are under attack again,
since I get a response to a ping once every 20
or so.

-- Nathan

-Original Message-
From: VoiceOps
[mailto:voiceops-boun...@voiceops.org] On
Behalf Of Nathan Anderson
Sent: Friday, March 11, 2016 3:28 PM
To: 'Nate Burke'; voiceops@voiceops.org
        <mailto:voiceops@voiceops.org>
    Subject: Re: [VoiceOps] VoIP Innovations
reliability

Confirmed.

-- Nathan

-Original Message-
From: VoiceOps
[mailto:voiceops-boun...@voiceops.org] On
Behalf Of Nate Burke
Sent: Friday, March 11, 2016 3:26 PM
To: voiceops@voiceops.org

Re: [VoiceOps] VoIP Innovations reliability

[Peter E]

I've seen this behavior before with another carrier  where they send an invite 
from the original gateway and when they don't get a response they overflow to 
another gate way before the first call is killed and therefore The phone rings 
twice but only the second call can get answered.

On Mar 17, 2016, at 17:52, Jeff Waddell <jeff+voice...@waddellsolutions.com> 
wrote:

That is the issue a lot of our customers are reporting - where multiple calls 
are sent 

> On Thu, Mar 17, 2016 at 5:49 PM, Nate Burke <n...@blastcomm.com> wrote:
> I didn't see any traffic increment on the DR IP Address in my firewall rules, 
> but this was odd.  During the 15 minute period, I had probably 5 or 6 
> simultaneous calls ring into my desk.  I normally only take a handfull of 
> calls a day. 
> 
>> On 3/17/2016 4:39 PM, Jeff Waddell wrote:
>> We implemented it too - I haven't checked to see if any traffic was sent 
>> across it
>> 
>> 
>> 
>> On Thu, Mar 17, 2016 at 5:20 PM, Nate Burke <n...@blastcomm.com> wrote:
>>> Only 15 Minutes this time though.  I had implemented the Disaster Recover 
>>> Trunk as mentioned previously, but I didn't seem to be getting any calls 
>>> completed through it.
>>> 
>>> 
>>>> On 3/17/2016 4:16 PM, Shripal Daphtary wrote:
>>>> Down again! 
>>>> 
>>>> Thanks, 
>>>> 
>>>> Shripal
>>>> 
>>>> On Mar 16, 2016, at 9:50 AM, Nate Burke <n...@blastcomm.com> wrote:
>>>> 
>>>>> Looks like it just came back up for me.  Just over 30 min.
>>>>> 
>>>>> Nate
>>>>> 
>>>>>> On 3/16/2016 8:45 AM, Shripal Daphtary wrote:
>>>>>> We are experiencing an outage as well. 
>>>>>> 
>>>>>> Thanks, 
>>>>>> 
>>>>>> Shripal
>>>>>> 
>>>>>> On Mar 16, 2016, at 9:36 AM, Nate Burke <n...@blastcomm.com> wrote:
>>>>>> 
>>>>>>> Problems again this morning?  Looks to be acting the same as it has 
>>>>>>> been.  
>>>>>>> 
>>>>>>>> On 3/11/2016 6:00 PM, Alexander Lopez wrote:
>>>>>>>> I added them to our monitoring platform, stated getting alarms this 
>>>>>>>> past hour or so.
>>>>>>>> Up and down.
>>>>>>>> 
>>>>>>>> 
>>>>>>>>  Original message 
>>>>>>>> From: Nathan Anderson <nath...@fsr.com> 
>>>>>>>> Date: 3/11/2016 6:31 PM (GMT-05:00) 
>>>>>>>> To: 'Nate Burke' <n...@blastcomm.com>, voiceops@voiceops.org 
>>>>>>>> Subject: Re: [VoiceOps] VoIP Innovations reliability 
>>>>>>>> 
>>>>>>>> ...aand we're back.
>>>>>>>> 
>>>>>>>> -- Nathan
>>>>>>>> 
>>>>>>>> -Original Message-
>>>>>>>> From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of 
>>>>>>>> Nathan Anderson
>>>>>>>> Sent: Friday, March 11, 2016 3:30 PM
>>>>>>>> To: 'Nate Burke'; voiceops@voiceops.org
>>>>>>>> Subject: Re: [VoiceOps] VoIP Innovations reliability
>>>>>>>> 
>>>>>>>> It *feels* like they are under attack again, since I get a response to 
>>>>>>>> a ping once every 20 or so.
>>>>>>>> 
>>>>>>>> -- Nathan
>>>>>>>> 
>>>>>>>> -Original Message-
>>>>>>>> From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of 
>>>>>>>> Nathan Anderson
>>>>>>>> Sent: Friday, March 11, 2016 3:28 PM
>>>>>>>> To: 'Nate Burke'; voiceops@voiceops.org
>>>>>>>> Subject: Re: [VoiceOps] VoIP Innovations reliability
>>>>>>>> 
>>>>>>>> Confirmed.
>>>>>>>> 
>>>>>>>> -- Nathan
>>>>>>>> 
>>>>>>>> -Original Message-
>>>>>>>> From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of 
>>>>>>>> Nate Burke
>>>>>>>> Sent: Friday, March 11, 2016 3:26 PM
>>>>>>>>

Re: [VoiceOps] VoIP Innovations reliability

[Ivan Kovacevic]

Ahem… 



At a risk of this not being a popular point of view… I view issues with
VoIP Innovations as being largely self-inflicted.



Not only on part of VoIP Innovations (no relationship) who, as it appears,
could have designed and communicated the DR better; but also on part of
people complaining about the outage. You are using a non-guaranteed
service. No SLAs, no promises, no commitments. If stuff like this happens,
it is only because it is to be expected. No one can guarantee the quality
of Internet between your IP and most other IPs on the Internet. That’s the
reality. And if you are relying on a non-guaranteed service, you should at
least have contingencies to move traffic quickly (CIC TFNs) or reroute
(outbound).



And where that’s not possible (DIDs), having a dedicated data link covered
by SLAs may be more appropriate.



And if you are porting your services to another provider with whom you will
also interconnect over the Internet, if I was your client, I would be
looking elsewhere. Unless your clients are not looking for a guaranteed
100% (or 99% or whatever) uptime. In which case, staying with VoIP
Innovations is just fine…



And I know the response will likely be that it isn’t economically feasible
to get dedicated connectivity, but by making that argument you are just
saying that your business (and your clients’ business) doesn’t require a
guaranteed level of uptime.



And either your clients have made this decision knowingly, in exchange for
cost savings (in which case you are still delivering what you promised) or
you made this decision on behalf of your clients unbeknownst to them, in
which case you are up a creek… because this is either going to happen again
and you will have to justify yet again why their services are down, or
you’ll have to pony up the cost of a dedicated connection without being to
recover it from your’ clients’ service fees. Either way, it wasn’t a good
idea in the first place and it isn’t VoIP Innovation’s fault, unless they
mislead you by saying Internet communication is infallible, and you
believed it… I have a bridge to sell you.







Best Regards,



Ivan Kovacevic

Vice President, Client Services

Star Telecom | www.startelecom.ca | SIP Based Services for Contact Centers



*From:* VoiceOps [mailto:voiceops-boun...@voiceops.org] *On Behalf Of *Anthony
Orlando via VoiceOps
*Sent:* Thursday, March 17, 2016 8:09 PM
*To:* Paul Timmins <p...@timmins.net>
*Cc:* voiceops@voiceops.org
*Subject:* Re: [VoiceOps] VoIP Innovations reliability



I get it Paul. It's just a shame the terrorists win. Too bad we can't put a
bounty on their sorry asses.

Sent from my iPhone


On Mar 17, 2016, at 6:55 PM, Paul Timmins <p...@timmins.net> wrote:

Sadly, assassins win if i stop breathing, so if someone starts to choke me,
I fight it the best I can and run away at my earliest opportunity.

On 03/17/2016 07:38 PM, Anthony Orlando via VoiceOps wrote:

It's a shame we can't support them. This could be anyone of us. Hackers win
if you port away.

Sent from my iPhone


On Mar 17, 2016, at 5:43 PM, Robert Johnson <fasterfour...@gmail.com> wrote:

Their primary SBC and DR IP are both in the same IP netblock, so whenever
the DDOS hits, both IPs are affected. The past few outages have involved
80% packet loss or so to both hosts, so some calls do make their way
through, and plenty of wierdness ensues when an INVITE makes its way
through but not the OK on the way back.



Can't wait to get our numbers ported out.



On Thu, Mar 17, 2016 at 6:27 PM, Nate Burke <n...@blastcomm.com> wrote:

Annnd they're down again.



On 3/17/2016 5:14 PM, Nate Burke wrote:

6 calls from 4 different CID numbers.  All within 3 minutes.

On 3/17/2016 4:58 PM, Nathan Anderson wrote:

Yesterday I definitely saw calls coming from the DR IP in my logs, but I
had not yet added that IP as a peerin our SBC.  I'll have to comb through
logs today to see if we got any.



Are you saying that the multiple calls you saw coming to your desk were all
from the same number?  If I had to guess, their side probably sent
continuous INVITEs to you when it failed to get back an OK for any of them
(not that you weren't sending back OK, but that their either didn't reach
their SBC or did not reach it in a timely manner).



-- Nathan



*From:* VoiceOps [mailto:voiceops-boun...@voiceops.org
<voiceops-boun...@voiceops.org>] *On Behalf Of *Jeff Waddell
*Sent:* Thursday, March 17, 2016 2:53 PM
*To:* Nate Burke
*Cc:* voiceops@voiceops.org
*Subject:* Re: [VoiceOps] VoIP Innovations reliability



That is the issue a lot of our customers are reporting - where multiple
calls are sent



On Thu, Mar 17, 2016 at 5:49 PM, Nate Burke <n...@blastcomm.com> wrote:

I didn't see any traffic increment on the DR IP Address in my firewall
rules, but this was odd.  During the 15 minute period, I had probably 5 or
6 simultaneous calls ring into my desk.  I normally only take a handfull of
calls a day.



On 3

Re: [VoiceOps] VoIP Innovations reliability

[Alex Balashov]

On 03/17/2016 10:51 PM, Peter E wrote:


If you use a carrier and they are attacked, chances are you're going to
be impacted in some way, regardless if you are directly connected.


+1.

Direct connection vs. public Internet is irrelevant here.

--
Alex Balashov | Principal | Evariste Systems LLC
1447 Peachtree Street NE, Suite 700
Atlanta, GA 30309
United States

Tel: +1-800-250-5920 (toll-free) / +1-678-954-0671 (direct)
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
___
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops

Re: [VoiceOps] VoIP Innovations reliability

[Nathan Anderson]

I did see some traffic come in from DR during that last outage, but when I 
placed test calls in from my cell, one call never completed ("the number you 
are dialing is not available") and the INVITE for that call definitely did not 
come in from the DR.

-- Nathan

From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of Nate Burke
Sent: Thursday, March 17, 2016 3:27 PM
To: voiceops@voiceops.org
Subject: Re: [VoiceOps] VoIP Innovations reliability

Annnd they're down again.
On 3/17/2016 5:14 PM, Nate Burke wrote:
6 calls from 4 different CID numbers.  All within 3 minutes.
On 3/17/2016 4:58 PM, Nathan Anderson wrote:
Yesterday I definitely saw calls coming from the DR IP in my logs, but I had 
not yet added that IP as a peerin our SBC.  I'll have to comb through logs 
today to see if we got any.

Are you saying that the multiple calls you saw coming to your desk were all 
from the same number?  If I had to guess, their side probably sent continuous 
INVITEs to you when it failed to get back an OK for any of them (not that you 
weren't sending back OK, but that their either didn't reach their SBC or did 
not reach it in a timely manner).

-- Nathan

From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of Jeff Waddell
Sent: Thursday, March 17, 2016 2:53 PM
To: Nate Burke
Cc: voiceops@voiceops.org<mailto:voiceops@voiceops.org>
Subject: Re: [VoiceOps] VoIP Innovations reliability

That is the issue a lot of our customers are reporting - where multiple calls 
are sent

On Thu, Mar 17, 2016 at 5:49 PM, Nate Burke 
<n...@blastcomm.com<mailto:n...@blastcomm.com>> wrote:
I didn't see any traffic increment on the DR IP Address in my firewall rules, 
but this was odd.  During the 15 minute period, I had probably 5 or 6 
simultaneous calls ring into my desk.  I normally only take a handfull of calls 
a day.

On 3/17/2016 4:39 PM, Jeff Waddell wrote:
We implemented it too - I haven't checked to see if any traffic was sent across 
it



On Thu, Mar 17, 2016 at 5:20 PM, Nate Burke 
<n...@blastcomm.com<mailto:n...@blastcomm.com>> wrote:
Only 15 Minutes this time though.  I had implemented the Disaster Recover Trunk 
as mentioned previously, but I didn't seem to be getting any calls completed 
through it.

On 3/17/2016 4:16 PM, Shripal Daphtary wrote:
Down again!
Thanks,

Shripal

On Mar 16, 2016, at 9:50 AM, Nate Burke 
<n...@blastcomm.com<mailto:n...@blastcomm.com>> wrote:
Looks like it just came back up for me.  Just over 30 min.

Nate
On 3/16/2016 8:45 AM, Shripal Daphtary wrote:
We are experiencing an outage as well.
Thanks,

Shripal

On Mar 16, 2016, at 9:36 AM, Nate Burke 
<n...@blastcomm.com<mailto:n...@blastcomm.com>> wrote:
Problems again this morning?  Looks to be acting the same as it has been.
On 3/11/2016 6:00 PM, Alexander Lopez wrote:
I added them to our monitoring platform, stated getting alarms this past hour 
or so.
Up and down.


 Original message 
From: Nathan Anderson <nath...@fsr.com><mailto:nath...@fsr.com>
Date: 3/11/2016 6:31 PM (GMT-05:00)
To: 'Nate Burke' <n...@blastcomm.com><mailto:n...@blastcomm.com>, 
voiceops@voiceops.org<mailto:voiceops@voiceops.org>
Subject: Re: [VoiceOps] VoIP Innovations reliability
...aand we're back.

-- Nathan

-Original Message-
From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of Nathan 
Anderson
Sent: Friday, March 11, 2016 3:30 PM
To: 'Nate Burke'; voiceops@voiceops.org<mailto:voiceops@voiceops.org>
Subject: Re: [VoiceOps] VoIP Innovations reliability

It *feels* like they are under attack again, since I get a response to a ping 
once every 20 or so.

-- Nathan

-Original Message-
From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of Nathan 
Anderson
Sent: Friday, March 11, 2016 3:28 PM
To: 'Nate Burke'; voiceops@voiceops.org<mailto:voiceops@voiceops.org>
Subject: Re: [VoiceOps] VoIP Innovations reliability

Confirmed.

-- Nathan

-Original Message-
From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of Nate Burke
Sent: Friday, March 11, 2016 3:26 PM
To: voiceops@voiceops.org<mailto:voiceops@voiceops.org>
Subject: Re: [VoiceOps] VoIP Innovations reliability

Anyone else show them down again right now?  My traceroutes aren't even
leaving Chicago.  Dying at a Chicago hop on Level3.

Nate

On 3/6/2016 6:50 PM, Nathan Anderson wrote:
> Did anybody else just suffer another 45-minute-ish long outage from about 
> 4:00p PST to 4:45p PST (ending about 5 minutes ago)?
>
> -- Nathan
>
> -Original Message-
> From: frnk...@iname.com<mailto:frnk...@iname.com> [mailto:frnk...@iname.com]
> Sent: Sunday, March 06, 2016 4:43 PM
> To: Nathan Anderson; voiceops@voiceops.org<mailto:voiceops@voiceops.org>
> Subject: RE: VoIP Innovations reliability
>
> More here: http://blog.voipinnovations.com/blog
>
&g

Re: [VoiceOps] VoIP Innovations reliability

[Shripal Daphtary]

Down again! 

Thanks, 

Shripal

> On Mar 16, 2016, at 9:50 AM, Nate Burke <n...@blastcomm.com> wrote:
> 
> Looks like it just came back up for me.  Just over 30 min.
> 
> Nate
> 
>> On 3/16/2016 8:45 AM, Shripal Daphtary wrote:
>> We are experiencing an outage as well. 
>> 
>> Thanks, 
>> 
>> Shripal
>> 
>> On Mar 16, 2016, at 9:36 AM, Nate Burke <n...@blastcomm.com> wrote:
>> 
>>> Problems again this morning?  Looks to be acting the same as it has been.  
>>> 
>>>> On 3/11/2016 6:00 PM, Alexander Lopez wrote:
>>>> I added them to our monitoring platform, stated getting alarms this past 
>>>> hour or so.
>>>> Up and down.
>>>> 
>>>> 
>>>>  Original message ----
>>>> From: Nathan Anderson <nath...@fsr.com> 
>>>> Date: 3/11/2016 6:31 PM (GMT-05:00) 
>>>> To: 'Nate Burke' <n...@blastcomm.com>, voiceops@voiceops.org 
>>>> Subject: Re: [VoiceOps] VoIP Innovations reliability 
>>>> 
>>>> ...aand we're back.
>>>> 
>>>> -- Nathan
>>>> 
>>>> -Original Message-
>>>> From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of Nathan 
>>>> Anderson
>>>> Sent: Friday, March 11, 2016 3:30 PM
>>>> To: 'Nate Burke'; voiceops@voiceops.org
>>>> Subject: Re: [VoiceOps] VoIP Innovations reliability
>>>> 
>>>> It *feels* like they are under attack again, since I get a response to a 
>>>> ping once every 20 or so.
>>>> 
>>>> -- Nathan
>>>> 
>>>> -Original Message-
>>>> From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of Nathan 
>>>> Anderson
>>>> Sent: Friday, March 11, 2016 3:28 PM
>>>> To: 'Nate Burke'; voiceops@voiceops.org
>>>> Subject: Re: [VoiceOps] VoIP Innovations reliability
>>>> 
>>>> Confirmed.
>>>> 
>>>> -- Nathan
>>>> 
>>>> -Original Message-
>>>> From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of Nate 
>>>> Burke
>>>> Sent: Friday, March 11, 2016 3:26 PM
>>>> To: voiceops@voiceops.org
>>>> Subject: Re: [VoiceOps] VoIP Innovations reliability
>>>> 
>>>> Anyone else show them down again right now?  My traceroutes aren't even 
>>>> leaving Chicago.  Dying at a Chicago hop on Level3.
>>>> 
>>>> Nate
>>>> 
>>>> On 3/6/2016 6:50 PM, Nathan Anderson wrote:
>>>> > Did anybody else just suffer another 45-minute-ish long outage from 
>>>> > about 4:00p PST to 4:45p PST (ending about 5 minutes ago)?
>>>> >
>>>> > -- Nathan
>>>> >
>>>> > -Original Message-
>>>> > From: frnk...@iname.com [mailto:frnk...@iname.com]
>>>> > Sent: Sunday, March 06, 2016 4:43 PM
>>>> > To: Nathan Anderson; voiceops@voiceops.org
>>>> > Subject: RE: VoIP Innovations reliability
>>>> >
>>>> > More here: http://blog.voipinnovations.com/blog
>>>> >
>>>> > Frank
>>>> >
>>>> > -Original Message-
>>>> > From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of Nathan
>>>> > Anderson
>>>> > Sent: Tuesday, March 01, 2016 8:27 PM
>>>> > To: voiceops@voiceops.org
>>>> > Subject: [VoiceOps] VoIP Innovations reliability
>>>> >
>>>> > Holy schlamoly.  Anybody else use them here and being handed outage after
>>>> > outage over the last 2 days?  Seriously thinking at this point about 
>>>> > doing
>>>> > something else.  This is ridiculous.
>>>> >
>>>> > I desperately need sleep and if my cell goes off one more time...
>>>> >
>>>> > -- Nathan
>>>> > ___
>>>> > VoiceOps mailing list
>>>> > VoiceOps@voiceops.org
>>>> > https://puck.nether.net/mailman/listinfo/voiceops
>>>> >
>>>> >
>>>> >
>>>> > ___
>>>> > VoiceOps mailing list
>>>> > VoiceOps@voiceops.org
>>>> > https://puck.nether.net/mailman/listinfo/voiceops
>>>> 
>>>> ___
>>>> VoiceOps mailing list
>>>> VoiceOps@voiceops.org
>>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>> 
>>>> ___
>>>> VoiceOps mailing list
>>>> VoiceOps@voiceops.org
>>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>> 
>>>> ___
>>>> VoiceOps mailing list
>>>> VoiceOps@voiceops.org
>>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>> 
>>>> ___
>>>> VoiceOps mailing list
>>>> VoiceOps@voiceops.org
>>>> https://puck.nether.net/mailman/listinfo/voiceops
>>> ___
>>> VoiceOps mailing list
>>> VoiceOps@voiceops.org
>>> https://puck.nether.net/mailman/listinfo/voiceops
> 
> ___
> VoiceOps mailing list
> VoiceOps@voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
___
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops

Re: [VoiceOps] VoIP Innovations reliability

[Alexander Lopez]

I think that most on this list understand the inherent risks in using the 
Internet As A Cross Connect  (IaaCC, I want credit for that term BTW :-) ).

However, we cannot just accept that this will be common place in the future. 
Dedicated links are becoming less and less prevalent as most move to the 
Internet to provide that last mile as well as back haul.

The Internet was built with resilience in its design, it is time that the PSTN 
steps up to the plate and breaks free from the single route architecture and 
provide multiple paths to connect a call.

When my customers complain about a call not being perfect, I remind them that 
they are paying cents for what used to cost dollars. Sure it's cheaper, but 
99.9 percent of the time it's just fine.

 Since VI has most of the small to medium ITSP market, most of our competition 
is having the same issues, we are better off than most because we have multiple 
ingress points and don't have all eggs in the VI basket.

I wholeheartedly agree with what you said, you get what you pay for.




 Original message 
From: Ivan Kovacevic <ivan.kovace...@startelecom.ca>
Date: 3/17/2016 8:55 PM (GMT-05:00)
To: voiceops@voiceops.org
Subject: Re: [VoiceOps] VoIP Innovations reliability

Ahem… 

At a risk of this not being a popular point of view… I view issues with VoIP 
Innovations as being largely self-inflicted.

Not only on part of VoIP Innovations (no relationship) who, as it appears, 
could have designed and communicated the DR better; but also on part of people 
complaining about the outage. You are using a non-guaranteed service. No SLAs, 
no promises, no commitments. If stuff like this happens, it is only because it 
is to be expected. No one can guarantee the quality of Internet between your IP 
and most other IPs on the Internet. That’s the reality. And if you are relying 
on a non-guaranteed service, you should at least have contingencies to move 
traffic quickly (CIC TFNs) or reroute (outbound).

And where that’s not possible (DIDs), having a dedicated data link covered by 
SLAs may be more appropriate.

And if you are porting your services to another provider with whom you will 
also interconnect over the Internet, if I was your client, I would be looking 
elsewhere. Unless your clients are not looking for a guaranteed 100% (or 99% or 
whatever) uptime. In which case, staying with VoIP Innovations is just fine…

And I know the response will likely be that it isn’t economically feasible to 
get dedicated connectivity, but by making that argument you are just saying 
that your business (and your clients’ business) doesn’t require a guaranteed 
level of uptime.

And either your clients have made this decision knowingly, in exchange for cost 
savings (in which case you are still delivering what you promised) or you made 
this decision on behalf of your clients unbeknownst to them, in which case you 
are up a creek… because this is either going to happen again and you will have 
to justify yet again why their services are down, or you’ll have to pony up the 
cost of a dedicated connection without being to recover it from your’ clients’ 
service fees. Either way, it wasn’t a good idea in the first place and it isn’t 
VoIP Innovation’s fault, unless they mislead you by saying Internet 
communication is infallible, and you believed it… I have a bridge to sell you.



Best Regards,

Ivan Kovacevic
Vice President, Client Services
Star Telecom | www.startelecom.ca<http://www.startelecom.ca/> | SIP Based 
Services for Contact Centers

From: VoiceOps 
[mailto:voiceops-boun...@voiceops.org<mailto:voiceops-boun...@voiceops.org>] On 
Behalf Of Anthony Orlando via VoiceOps
Sent: Thursday, March 17, 2016 8:09 PM
To: Paul Timmins <p...@timmins.net<mailto:p...@timmins.net>>
Cc: voiceops@voiceops.org<mailto:voiceops@voiceops.org>
Subject: Re: [VoiceOps] VoIP Innovations reliability

I get it Paul. It's just a shame the terrorists win. Too bad we can't put a 
bounty on their sorry asses.

Sent from my iPhone

On Mar 17, 2016, at 6:55 PM, Paul Timmins 
<p...@timmins.net<mailto:p...@timmins.net>> wrote:
Sadly, assassins win if i stop breathing, so if someone starts to choke me, I 
fight it the best I can and run away at my earliest opportunity.

On 03/17/2016 07:38 PM, Anthony Orlando via VoiceOps wrote:
It's a shame we can't support them. This could be anyone of us. Hackers win if 
you port away.

Sent from my iPhone

On Mar 17, 2016, at 5:43 PM, Robert Johnson 
<fasterfour...@gmail.com<mailto:fasterfour...@gmail.com>> wrote:
Their primary SBC and DR IP are both in the same IP netblock, so whenever the 
DDOS hits, both IPs are affected. The past few outages have involved 80% packet 
loss or so to both hosts, so some calls do make their way through, and plenty 
of wierdness ensues when an INVITE makes its way through but not the OK on the 
way back.

Can't wait to get our numbers ported out.

On T

Re: [VoiceOps] VoIP Innovations reliability

[Alexander Lopez]

I added them to our monitoring platform, stated getting alarms this past hour 
or so.
Up and down.


 Original message 
From: Nathan Anderson <nath...@fsr.com>
Date: 3/11/2016 6:31 PM (GMT-05:00)
To: 'Nate Burke' <n...@blastcomm.com>, voiceops@voiceops.org
Subject: Re: [VoiceOps] VoIP Innovations reliability

...aand we're back.

-- Nathan

-Original Message-
From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of Nathan 
Anderson
Sent: Friday, March 11, 2016 3:30 PM
To: 'Nate Burke'; voiceops@voiceops.org
Subject: Re: [VoiceOps] VoIP Innovations reliability

It *feels* like they are under attack again, since I get a response to a ping 
once every 20 or so.

-- Nathan

-Original Message-
From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of Nathan 
Anderson
Sent: Friday, March 11, 2016 3:28 PM
To: 'Nate Burke'; voiceops@voiceops.org
Subject: Re: [VoiceOps] VoIP Innovations reliability

Confirmed.

-- Nathan

-Original Message-
From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of Nate Burke
Sent: Friday, March 11, 2016 3:26 PM
To: voiceops@voiceops.org
Subject: Re: [VoiceOps] VoIP Innovations reliability

Anyone else show them down again right now?  My traceroutes aren't even
leaving Chicago.  Dying at a Chicago hop on Level3.

Nate

On 3/6/2016 6:50 PM, Nathan Anderson wrote:
> Did anybody else just suffer another 45-minute-ish long outage from about 
> 4:00p PST to 4:45p PST (ending about 5 minutes ago)?
>
> -- Nathan
>
> -Original Message-
> From: frnk...@iname.com [mailto:frnk...@iname.com]
> Sent: Sunday, March 06, 2016 4:43 PM
> To: Nathan Anderson; voiceops@voiceops.org
> Subject: RE: VoIP Innovations reliability
>
> More here: http://blog.voipinnovations.com/blog
>
> Frank
>
> -Original Message-
> From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of Nathan
> Anderson
> Sent: Tuesday, March 01, 2016 8:27 PM
> To: voiceops@voiceops.org
> Subject: [VoiceOps] VoIP Innovations reliability
>
> Holy schlamoly.  Anybody else use them here and being handed outage after
> outage over the last 2 days?  Seriously thinking at this point about doing
> something else.  This is ridiculous.
>
> I desperately need sleep and if my cell goes off one more time...
>
> -- Nathan
> ___
> VoiceOps mailing list
> VoiceOps@voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
>
>
>
> ___
> VoiceOps mailing list
> VoiceOps@voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops

___
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops

___
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops

___
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops

___
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops
___
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops

Re: [VoiceOps] VoIP Innovations reliability

[Peter E]

Bouncing up and down over the last half hour or so…

On Mar 11, 2016, at 18:31, Nathan Anderson  wrote:

This message has no content.___
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops

Re: [VoiceOps] VoIP Innovations reliability

[Nathan Anderson]

It *feels* like they are under attack again, since I get a response to a ping 
once every 20 or so.

-- Nathan

-Original Message-
From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of Nathan 
Anderson
Sent: Friday, March 11, 2016 3:28 PM
To: 'Nate Burke'; voiceops@voiceops.org
Subject: Re: [VoiceOps] VoIP Innovations reliability

Confirmed.

-- Nathan

-Original Message-
From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of Nate Burke
Sent: Friday, March 11, 2016 3:26 PM
To: voiceops@voiceops.org
Subject: Re: [VoiceOps] VoIP Innovations reliability

Anyone else show them down again right now?  My traceroutes aren't even 
leaving Chicago.  Dying at a Chicago hop on Level3.

Nate

On 3/6/2016 6:50 PM, Nathan Anderson wrote:
> Did anybody else just suffer another 45-minute-ish long outage from about 
> 4:00p PST to 4:45p PST (ending about 5 minutes ago)?
>
> -- Nathan
>
> -Original Message-
> From: frnk...@iname.com [mailto:frnk...@iname.com]
> Sent: Sunday, March 06, 2016 4:43 PM
> To: Nathan Anderson; voiceops@voiceops.org
> Subject: RE: VoIP Innovations reliability
>
> More here: http://blog.voipinnovations.com/blog
>
> Frank
>
> -Original Message-
> From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of Nathan
> Anderson
> Sent: Tuesday, March 01, 2016 8:27 PM
> To: voiceops@voiceops.org
> Subject: [VoiceOps] VoIP Innovations reliability
>
> Holy schlamoly.  Anybody else use them here and being handed outage after
> outage over the last 2 days?  Seriously thinking at this point about doing
> something else.  This is ridiculous.
>
> I desperately need sleep and if my cell goes off one more time...
>
> -- Nathan
> ___
> VoiceOps mailing list
> VoiceOps@voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
>
>
>
> ___
> VoiceOps mailing list
> VoiceOps@voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops

___
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops

___
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops

___
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops

Re: [VoiceOps] VoIP Innovations reliability

[frnkblk]

More here: http://blog.voipinnovations.com/blog

Frank

-Original Message-
From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of Nathan
Anderson
Sent: Tuesday, March 01, 2016 8:27 PM
To: voiceops@voiceops.org
Subject: [VoiceOps] VoIP Innovations reliability

Holy schlamoly.  Anybody else use them here and being handed outage after
outage over the last 2 days?  Seriously thinking at this point about doing
something else.  This is ridiculous.

I desperately need sleep and if my cell goes off one more time...

-- Nathan
___
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops


___
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops

Re: [VoiceOps] VoIP Innovations reliability

[Nathan Anderson]

There has been a lot of good discussion off-list, but the quick newsflash 
update is that it appears they are being hit with DDoS.  This sucks, but at 
least grants me some degree of empathy.

My customers, on the other hand, could not care less what the underlying cause 
is.  All they know is that they have had hit-and-miss service today.

-- Nathan

-Original Message-
From: VoiceOps [mailto:voiceops-boun...@voiceops.org] On Behalf Of Nathan 
Anderson
Sent: Tuesday, March 01, 2016 6:27 PM
To: voiceops@voiceops.org
Subject: [VoiceOps] VoIP Innovations reliability

Holy schlamoly.  Anybody else use them here and being handed outage after 
outage over the last 2 days?  Seriously thinking at this point about doing 
something else.  This is ridiculous.

I desperately need sleep and if my cell goes off one more time...

-- Nathan
___
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops

___
VoiceOps mailing list
VoiceOps@voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops