Wow. I hope there's another side to this story because reading this gives everyone who touches VoIP a bad name. This is really bad.
On 3/25/16, 5:15 PM, "VoiceOps on behalf of Peter E" <[email protected] on behalf of [email protected]> wrote: >I'm not telling you anything you don't already know, but if the web interfaces >are exposed, especially with default passwords, they are going to get hacked, >likely this weekend, so you might want to prepare the CEO now for the >"savings" he's going to see on his first bill. > > > >On Mar 25, 2016, at 20:10, Peter E <[email protected]> wrote: > >Ouch, that was painful to read. I can't imagine having lived it. > >When in doubt, dig for Cisco documents because CxO's all know and trust the >brand. I'll forward whatever I can find. > >On Mar 25, 2016, at 19:55, Aaron C. de Bruyn <[email protected]> wrote: > >I'll try to make this short: I am an IT contractor for "Company X" that has >~26 offices around the western US. We are paid a flat fee to manage every >office, keep things secure, train and assist users, etc... > >About two weeks ago, offices suddenly started going offline. After 15-20 >minutes of frantically digging, we got a call from a VoIP provider who was >apparently told by Company X "we want to use your VoIP service, go get it >installed at all our offices". > >This VoIP provider walked in to several off the offices and just started >yanking out switches that had various VLANs running on them and replacing them >with their own Netgear PoE switches with no config and default passwords. >They took down tons of virtual servers and SANs. > >We spent the better part of a week ripping their stuff out and putting the >network back the way it was. > >We had a brief meeting with the VoIP provider and told them things need to be >planned in advance and we would be happy to work with them to get things going. > >We set up a VLAN for VoIP traffic and told them how to cross-connect their >switch to ours, what IP ranges to use, and even set up a VPN connection at >each office so they could access the equipment remotely. > >They they scheduled 6 installs in 3 days and with no testing or communication >they came in and hooked things up. I received repeated phone calls in the >evenings, mornings, and weekends that they needed huge swaths of ports >forwarded so they could remotely program phones and the phone server. > >And of course it was all an emergency. As in "we ported the numbers over >already and the office is opening in 15 minutes, just forward the damn ports". > >We were even told the 6 installs could not be stopped or re-scheduled because >the VoIP provider 'went out' on the equipment and really needs to finish the >install so they can recoup their money. > >The disaster should be coming to a close this weekend, and I'm trying to clean >up and gather information for a report to the CEO. My main concerns are: > >* We set up VPN connections. The VoIP guys aren't using them. They don't >have time to test and/or troubleshoot any issues they are complaining about. > >* The devices all have static IPs instead of using DHCP. The phones appear to >get a DHCP address off VLAN 100 properly, but when it's time for a renewal >they drop the VLAN tag, get switched to the wrong network, and lose >communication. > >* We were told to set up port forwards to every phone's *HTTP* interface as >well as a forward to the phone server HTTPS interface. > >* Most passwords appear to be factory defaults > >* The CEO of Company X was told the port forward must remain in place and they >can not be disabled for 'security reasons' because VoIP phone are not a >computer and therefore can't be hacked. (Seriously?!?!! Who cares about >hacking when your equipment has default passwords...) > >* They skimped on proper wiring in a lot of places and have computers jumpered >through the phones > >* Because of that, the phones are self-tagging packets with VLAN 100 and the >jumpered workstations are un-tagged which required us to accept un-tagged >packets on to the network containing patient data. > >* If the phones or phone server gets compromised, it seems like it would be >real easy to simply drop the VLAN tag and have access to a network containing >patient data. > >* A quick sniff at our WAN interface shows all the calls and communication are >happening with a server over HTTP. I was able to capture voice data in the >clear containing patient information, credit card details, etc... > >I have worked with professional VoIP companies before. When they do it right, >the networks are isolated, phones have their own network drops, no ports are >exposed to the internet, etc... > >The CEO of Company X appears to only have been informed that the offices were >'switching to VoIP to save costs' and nothing more. He is a very >data-oriented guy and loves technical documentation. When we make our case to >him, I'd like to back it up with as much 'best practice references' as >possible. > >Are there any best-practice documents out there I can provide to the CEO? > >I know what this provider is doing is horribly wrong and insecure, but the CEO >is the kind of person who wants documentation from lots of sources to back it >up. > >Basically the phone guys are blaming us for all the problems, and we are >blaming them for causing several thousand dollars in after-hours emergency >site visits and remote work because of poor planning, scheduling, and simply >ripping out equipment they know nothing about. (In addition to making the >network insecure as hell and not doing their due diligence.) > >Thanks for listening. :) > >-A >_______________________________________________ >VoiceOps mailing list >[email protected] >https://puck.nether.net/mailman/listinfo/voiceops >_______________________________________________ >VoiceOps mailing list >[email protected] >https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list [email protected] https://puck.nether.net/mailman/listinfo/voiceops
