I learned this the hard way, but now I only continue to service customers if they actually understand the value brought by a secure and reliable service. As soon as they want to cut corners, for one reason or another, and that becomes a huge issue (like in your case), I make use of a "termination for convenience" clause that I have in all my contracts and within 90 days I no longer have to service this customer, they can find some other provider to make their lives hell.
No more servicing every customer with insane expectations of cost vs benefit. *Rafael Possamai* Founder & CEO at E2W Solutions *office:* (414) 269-6000 *e-mail:* [email protected] On Fri, Mar 25, 2016 at 6:55 PM, Aaron C. de Bruyn <[email protected]> wrote: > I'll try to make this short: I am an IT contractor for "Company X" that > has ~26 offices around the western US. We are paid a flat fee to manage > every office, keep things secure, train and assist users, etc... > > About two weeks ago, offices suddenly started going offline. After 15-20 > minutes of frantically digging, we got a call from a VoIP provider who was > apparently told by Company X "we want to use your VoIP service, go get it > installed at all our offices". > > This VoIP provider walked in to several off the offices and just started > yanking out switches that had various VLANs running on them and replacing > them with their own Netgear PoE switches with no config and default > passwords. They took down tons of virtual servers and SANs. > > We spent the better part of a week ripping their stuff out and putting the > network back the way it was. > > We had a brief meeting with the VoIP provider and told them things need to > be planned in advance and we would be happy to work with them to get things > going. > > We set up a VLAN for VoIP traffic and told them how to cross-connect their > switch to ours, what IP ranges to use, and even set up a VPN connection at > each office so they could access the equipment remotely. > > They they scheduled 6 installs in 3 days and with no testing or > communication they came in and hooked things up. I received repeated phone > calls in the evenings, mornings, and weekends that they needed huge swaths > of ports forwarded so they could remotely program phones and the phone > server. > > And of course it was all an emergency. As in "we ported the numbers over > already and the office is opening in 15 minutes, just forward the damn > ports". > > We were even told the 6 installs could not be stopped or re-scheduled > because the VoIP provider 'went out' on the equipment and really needs to > finish the install so they can recoup their money. > > The disaster should be coming to a close this weekend, and I'm trying to > clean up and gather information for a report to the CEO. My main concerns > are: > > * We set up VPN connections. The VoIP guys aren't using them. They don't > have time to test and/or troubleshoot any issues they are complaining about. > > * The devices all have static IPs instead of using DHCP. The phones > appear to get a DHCP address off VLAN 100 properly, but when it's time for > a renewal they drop the VLAN tag, get switched to the wrong network, and > lose communication. > > * We were told to set up port forwards to every phone's *HTTP* interface > as well as a forward to the phone server HTTPS interface. > > * Most passwords appear to be factory defaults > > * The CEO of Company X was told the port forward must remain in place and > they can not be disabled for 'security reasons' because VoIP phone are not > a computer and therefore can't be hacked. (Seriously?!?!! Who cares about > hacking when your equipment has default passwords...) > > * They skimped on proper wiring in a lot of places and have computers > jumpered through the phones > > * Because of that, the phones are self-tagging packets with VLAN 100 and > the jumpered workstations are un-tagged which required us to accept > un-tagged packets on to the network containing patient data. > > * If the phones or phone server gets compromised, it seems like it would > be real easy to simply drop the VLAN tag and have access to a network > containing patient data. > > * A quick sniff at our WAN interface shows all the calls and communication > are happening with a server over HTTP. I was able to capture voice data in > the clear containing patient information, credit card details, etc... > > I have worked with professional VoIP companies before. When they do it > right, the networks are isolated, phones have their own network drops, no > ports are exposed to the internet, etc... > > The CEO of Company X appears to only have been informed that the offices > were 'switching to VoIP to save costs' and nothing more. He is a very > data-oriented guy and loves technical documentation. When we make our case > to him, I'd like to back it up with as much 'best practice references' as > possible. > > Are there any best-practice documents out there I can provide to the CEO? > > I know what this provider is doing is horribly wrong and insecure, but the > CEO is the kind of person who wants documentation from lots of sources to > back it up. > > Basically the phone guys are blaming us for all the problems, and we are > blaming them for causing several thousand dollars in after-hours emergency > site visits and remote work because of poor planning, scheduling, and > simply ripping out equipment they know nothing about. (In addition to > making the network insecure as hell and not doing their due diligence.) > > Thanks for listening. :) > > -A > > _______________________________________________ > VoiceOps mailing list > [email protected] > https://puck.nether.net/mailman/listinfo/voiceops > >
_______________________________________________ VoiceOps mailing list [email protected] https://puck.nether.net/mailman/listinfo/voiceops
