To be fair, I haven't ruled out the possibility that we did something stupid, and left a security door wide open somewhere. But I'm having a hard time figuring out what that would have been.
I also went through and checked a handfulof 8000s in the field, and haven't found any others that look like they either have an FTP server running, or that look like they have served as a dumping ground for these sorts of worms. We program each CPE identically when they get installed, and the configuration is stamped out of a template (we configured one CPE how we wanted it, then backed up the config and restore that config file to a new CPE during installation). This is a weird one, to be sure. -- Nathan -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Robert Sent: Thursday, February 09, 2017 3:48 AM To: Adam Moffett; [email protected] Subject: Re: [Telrad] UE upgrade failure rate Sometimes, you feel like reaching out and finding a particular software engineer and slapping them upside the back of the head... Over and over and over and over.... WTF were they thinking... On 2/9/17 2:56 AM, Adam Moffett wrote: > WOW. > > ------ Original Message ------ > From: "Nathan Anderson" <[email protected] <mailto:[email protected]>> > To: "Telrad List" <[email protected] <mailto:[email protected]>> > Sent: 2/9/2017 5:54:27 AM > Subject: Re: [Telrad] UE upgrade failure rate > >> After much trial and error, I managed to learn that the white port is >> a TTL-level serial interface. And there was much rejoicing. >> >> >> >> ALSO, I FIGURED OUT WHAT HAS BEEN KILLING (at least our) CPE8000s. >> >> >> >> Remember that problem that the EPC firmware had back when it was first >> released? Back when root access was still available on the EPC >> firmware, there was an FTP server running on it that accepted >> connections via the PDN IP address, and if you didn't change the root >> password from the default insecure one (which was ironically named), >> then infected machines trying to spread that stupid Photo.scr worm >> would successfully log into the EPC via FTP and, thinking that it had >> managed to log into a public web server somewhere, upload a bajillion >> copies of the virus to it in various directories, filling up the disk. >> >> >> >> The exact same thing is happening here, believe it or not. It hadn't >> ever occurred to me to test for this, but it turns out that under >> certain circumstances that I haven't yet managed to nail down, the >> CPE8000 firmware actually starts running an FTP server. Even worse, >> this FTP server, once enabled, does not ask for any credentials. You >> can literally type in any username when prompted, and you are in. I >> see no config option on the web interface for the CPE that allows you >> to turn this on and off...but whatever is triggering it ends up >> creating a ready and completely unsecured backdoor to the CPE. >> >> >> >> *headdesk* >> >> >> >> If you guys give out routable IPs to your LTE users, or if you have >> somebody on your network that has a PC infected with this particular >> virus, then it might be that this could also explain your CPE8000 >> firmware upgrade problems. >> >> >> >> After figuring out the serial port bit and examining the "dead" CPEs >> more in-depth, I found the filesystems littered with files named >> things like Photo.scr, IMG001.scr, Info.zip, etc. Once the writable >> partition with the CPE configuration is completely full, if at that >> point you issue either a reset-to-defaults, or upload a configuration >> backup, or initiate a firmware upgrade (which has to migrate your >> configuration from the old firmware version during the process), your >> CPE gets bricked because there isn't enough disk space left for it to >> properly finish writing the config changes to disk. So it gets only >> half-done, and the configuration is left in an inconsistent state. >> >> >> >> I've managed to fix my dead units, and also found the mechanism for >> disabling the FTP server. Still not sure how it is getting toggled on >> in the first place (perhaps there is some other vulnerability that is >> getting exploited first?), but I'll keep looking. >> >> >> >> I'll write up some instructions for y'all and post them here soon. >> >> >> >> -- Nathan >> >> >> >> *From:*[email protected] <mailto:[email protected]> >> [mailto:[email protected] <mailto:[email protected]>] >> *On Behalf Of *Nathan Anderson >> *Sent:* Wednesday, February 08, 2017 1:49 PM >> *To:* Telrad List; Adam Moffett >> *Subject:* Re: [Telrad] UE upgrade failure rate >> >> >> >> Does anybody happen to know if the 6-pin white connector on the 8000's >> board is either a serial port or a JTAG interface? >> >> >> >> -- Nathan >> >> >> >> *From:*[email protected] <mailto:[email protected]> >> [mailto:[email protected]] *On Behalf Of *Nathan Anderson >> *Sent:* Wednesday, February 08, 2017 1:40 PM >> *To:* Telrad List; Adam Moffett >> *Subject:* Re: [Telrad] UE upgrade failure rate >> >> >> >> This thread is interesting because I was just complaining last night >> to our vendor about how fragile the software on the CPE8000s seems to be. >> >> >> >> We have not had specific issues with flashing CPEs "over the air" from >> the web interface, but sometimes ACS-initiated updates don't complete >> correctly. On 7000s it usually takes the form of the upgrade not >> completing and the UE falling off of the ACS, but the radio stays up >> and attached to the network. We go in via the web interface OTA and >> reboot it and it comes back with the same version of firmware it was >> already running. Second time is usually the charm, and I'm thinking >> that perhaps if the UE had been freshly-rebooted before attempting the >> update, we might have a higher success rate. (We have also seen 7000s >> just stop talking to the ACS without us touching the firmware, and >> even though they are otherwise working fine. Again, rebooting the CPE >> fixes this. Although it is rare, we have seen this even on the latest >> .116) >> >> >> >> We once had a 7000 that did drop off the network after pushing the >> update via ACS. We never checked what state it was in from the >> ethernet side, but we had the customer powercycle it themselves and it >> came back...again running the same firmware. So the upgrade did not >> take, but it didn't brick it either and resetting config to defaults >> on the UE was not (and at least for us never has been) necessary. >> >> >> >> So we have never had to truck-roll to a 7000 as a result of a failed >> firmware upgrade. The 8000s, however, seem to be another story. I am >> so scared to touch the ones we have in the field anymore. We have had >> a couple that seem to get their configs corrupted after a firmware >> change, and get into very funky states. >> >> >> >> One of them had these symptoms: defaulted to a 192.168.0.1 IP on the >> ethernet (!), no web server running, no DHCP server running, had >> telnet access that didn't prompt for a password (!!). Fixed it by >> resetting to defaults (found a shell script that performs this >> function on the CPE's filesystem). I got lucky with this one. >> >> >> >> One that I have sitting on my desk now is one that we tried to >> rollback the firmware on (customer was experiencing random network >> detaches, and the latest 8000 firmware doesn't reattach for 15 minutes >> on-the-dot, so customer was -- I think justifiably -- getting a bit >> pissy). Current symptoms are: NO IPv4 on the ethernet, IPv6 >> link-local responds, no web server running, no DHCP server running, >> telnet responds (calls itself "KZTECH") but default root/root123 >> doesn't work, so I have NO way to get in and reset the damn thing, and >> the 8000s don't seem to have a reset button. Thus it seems that it is >> possible for a scrambled config to completely brick an 8000. >> >> >> >> If anybody has reliable information on how to get the 8000 to wipe its >> config during bootup even though it seemingly lacks a reset button, I >> would be eternally grateful... >> >> >> >> -- Nathan >> >> >> >> *From:*[email protected] <mailto:[email protected]> >> [mailto:[email protected]] *On Behalf Of *Matthew Carpenter >> *Sent:* Wednesday, February 08, 2017 6:38 AM >> *To:* Adam Moffett; Telrad List >> *Subject:* Re: [Telrad] UE upgrade failure rate >> >> >> >> Hi, >> >> >> >> So far only 1 CPE8000 UE that did not come back after a firmware >> update. Normally a hard reboot would fix it, but in this case we had >> to replace it. >> >> I have that CPE8000 on my desk and need to see what the status is from >> the LAN side. Thanks for the info on defaulting it, will try it. >> >> >> >> Matt C. >> >> >> >> >> >> >> >> On Wed, Feb 8, 2017 at 8:23 AM, Adam Moffett <[email protected] >> <mailto:[email protected]>> wrote: >> >> We've had a helluva time upgrading UE firmware over the air. It was >> worse with Wimax. On Wimax it was more like 75% of the time we would >> lose the channel scan table and have to go on site to add it back in. >> It became SOP to leave the operator password at default so we had the >> option of having the customer log in and fix the scan table for us. >> >> >> >> I think we've had more success since going to LTE. However, failed >> firmware updates was one of the incentives to set up a dedicated >> management bearer. I was hoping it would help with these things. We >> haven't pushed out an update recently enough to say whether it helped. >> >> >> >> -Adam >> >> >> >> >> >> >> >> ------ Original Message ------ >> >> From: "Shayne Lebrun" <[email protected] >> <mailto:[email protected]>> >> >> To: [email protected] <mailto:[email protected]> >> >> Sent: 2/8/2017 9:14:36 AM >> >> Subject: [Telrad] UE upgrade failure rate >> >> >> >> Does anybody else experience a ten to fifteen percent failure rate >> when upgrading UEs? The behavior is, you upgrade the firmware, >> reboot, and the device doesn't come back. Logging into the UE's >> management from LAN, you'll see it's stuck in 'device init.' >> Defaulting the unit and rebooting allows it to boot and attach. >> >> >> >> We're not using the residential gateway device or anything, and >> the only config we put in is device name, SNMP and ACS settings. >> Sometimes we hardcode the client's device in the DHCP server, to >> turn on DMZ to allow port forwarding, but that doesn't seem to be >> a causal factor. >> >> >> _______________________________________________ >> Telrad mailing list >> [email protected] <mailto:[email protected]> >> http://lists.wispa.org/mailman/listinfo/telrad >> >> >> >> >> >> -- >> >> *Matthew Carpenter* >> >> *806-316-5071 office* >> >> *806-236-9558 cell* >> >> >> > > > _______________________________________________ > Telrad mailing list > [email protected] > http://lists.wispa.org/mailman/listinfo/telrad > _______________________________________________ Telrad mailing list [email protected] http://lists.wispa.org/mailman/listinfo/telrad _______________________________________________ Telrad mailing list [email protected] http://lists.wispa.org/mailman/listinfo/telrad
