Hey All, Firmware status update is now posted on the forum. This is the best place to have constructive dialogue and I encourage all Telrad customers to follow each topic. This is where I intend to provide all progress updates going forward. Any questions posted on this forum will be addressed by Telrad and open for community discussion.
https://telradsupport.zendesk.com/hc/en-us/community/posts/115005879348-Firmware-Update Many of the issues discussed here on wispa list are related to the beta process and are addressed in my forum update Skywerx (Justin) has deployed the first of the LTE in a box (embedded EPC Compact) as of recently. The network to date is exactly the same product deployed throughout North America Regards, Nick -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of [email protected] Sent: March 14, 2017 10:35 PM To: [email protected] Subject: Telrad Digest, Vol 29, Issue 14 Send Telrad mailing list submissions to [email protected] To subscribe or unsubscribe via the World Wide Web, visit http://lists.wispa.org/mailman/listinfo/telrad or, via email, send a message with subject or body 'help' to [email protected] You can reach the person managing the list at [email protected] When replying, please edit your Subject line so it is more specific than "Re: Contents of Telrad digest..." Today's Topics: 1. Re: Uplink throughput again (Adam Moffett) ---------------------------------------------------------------------- Message: 1 Date: Wed, 15 Mar 2017 03:34:24 +0000 From: "Adam Moffett" <[email protected]> Subject: Re: [Telrad] Uplink throughput again To: [email protected], "[email protected]" <[email protected]> Message-ID: <em8679b033-89ac-40f8-bcbc-4ea296edaf0c@techlaptop1> Content-Type: text/plain; charset="utf-8" We've had EPC crashes, but I haven't gotten to the bottom of any of them. I rebooted and moved on with the day because I have a full time job outside of troubleshooting Telrad gear.....I made sure there's a serial cable connected now so I can hopefully get some info out of it next time. >Data corruption for user traffic going through the EPC > I have not noticed this one....or I didn't know what I was seeing if I did. What are the symptoms? In addition to Nathan's list: * 10 second cycle to traffic through the EPC. This is reproducible even with a single UE in a lab setup, and it's clearly tied to a process in the Breezeway2020. There's a dip in throughput every 10 seconds, and it gets worse with more UE. When it gets extreme you can see air link utilization flatten out at 65-75% as your eNB cycles between full throughput and zero throughput. * UE getting stuck at MCS4....apparently until an S1 reset. This may or may not be the same throughput issue that you guys were talking about earlier in the thread. These two are both known by Telrad and being worked on. * Saturating the upload on an eNB can cause CQI timeouts which makes UE disconnect from the eNB. The customer is down for 1-2 minutes. This is the main reason not to use configuration 2. This one (I was told) is not a bug, but simply part of how LTE functions. I'm skeptical of that explanation, but nevertheless if you're having better performance & reliability on configuration 1 this might be why. The point being that we're not having problems because we're dumb, we're having problems because the product has a few problems. Justin, if you're not seeing these issues, maybe you're using different equipment (embedded core as someone said?), or maybe you'd see them if you go looking for them. After migrating about 1/3 of our Wimax to LTE we now have about 400 LTE UE in service. Now we're in a holding pattern. I know we'll move forward with the rest of the migration, but I think I'm waiting for 6.6M first. ------ Original Message ------ From: "Nathan Anderson" <[email protected]> To: "[email protected]" <[email protected]> Sent: 3/14/2017 5:43:50 PM Subject: Re: [Telrad] Uplink throughput again >There really isn't much for me to add to Jeremy's excellent response. > > > >As far as what could be different with regard to the specific issue we >were talking about here? So many things. You could be running a >different software release...only people who have been wrestling with >(Telrad's acknowledged) issue with uplink performance likely are using >bleeding-edge eNB code to deal with it. You could be running a >different configuration that is not affected, or not affected as >noticeably. We use 15MHz channels. I'd guess not very many others do, >and until recently, different channel widths required some special >tweaks elsewhere in the config in order to get acceptable performance, >which tells you something. We also currently use subframe profile 1, >but before the latest CPE7000 firmware version, we *couldn't*, because >of bugs in the CPE firmware that just absolutely killed performance >with that configuration under some circumstances. > > > >Just because your particular configuration does not trigger some of >these bugs does not mean that they don't exist, and I don't think it is >fair or accurate for you to compare the issues being discussed here to >the "OMG whye cain't I do 100 megawattbits on me neg85 >Ubiquitay!!!!!111!!one!!" crowd. > > > >In fact, over the last 3 months or so, the majority of our problems >have been traced back to the EPC and have absolutely zero to do with >RF. Here are some of the issues we have dealt with there, some of >which are finally fixed, and some of which are not...granted, some of >these (especially towards the bottom) are on account of us running >prerelease code, but at this point, because the fixes for the crashers >we found on 6.6 haven't been backported to 6.6, we are stuck between a >rock and a hard place and forced to play beta tester on our production >network (!!!!): > > > >* The RADIUS/AAA (external HSS) client in the EPC crashing (causing the >whole EPC to reboot itself) > > > >* The switch FPGA dying/resetting for several seconds at a time >(causing the RADIUS client to crash, causing reboot...beautiful domino >effect!) > > > >* The switch FPGA dying/resetting for several MINUTES at a time (first >release we got as a fix for this made the problem worse instead of >better) > > > >* Dedicated bearers getting carte blanche on the uplink (100% >reproducible within 5 minutes of trying to use the feature...how did >this manage to ship?) > > > >* SGW process on the EPC crashing in a similar manner to RADIUS client > > > >* Broken eNB tracking (gets confused about which eNB has which IP >address; may be related to SGW issue: I suspect a linked list is >getting clobbered in memory) > > > >* Data corruption for user traffic going through the EPC (it seemingly >throws out and recomputes the TCP payload checksum, making it silent >but deadly!) > > > >Also, if I may (since I'm on the subject of frustrations, and I don't >know where else to put this): why does every little config change >(e.g., TFTP server? Really??) on either the Compact or BreezeWay >require a full system reboot to be applied? I mean, what year is this? > > > >-- Nathan > > > >From:[email protected] [mailto:[email protected]] On >Behalf Of Jeremy Austin >Sent: Thursday, March 09, 2017 11:56 PM To:[email protected] >Subject: Re: [Telrad] Uplink throughput again > > > >Justin, > > > >"What's Different" is an excellent question; I can't speak for others >on this list. > > > >What's different (in my case) is that we start from the assumption that >our experience should be as positive as yours. Success stories are >great. How a company handles success stories... they print a presser. > > > >What's different is that we start from the assumption that if there >were a problem with performance, it was *our* oversight or failure, not >Telrad's. We jump through prescribed troubleshooting hoops, repeatedly. >We invent new tests when necessary. > > > >What's different? Telrad's initial assumptions about our network and RF >conditions prove consistently false. We schedule many complete system >outages, and (working hand in hand with Telrad) discover Telrad >hardware failure modes that (guess what) nobody has ever seen before. > > > >What's different is that we have had a non-zero percentage of failed >Telrad equipment, and Telrad has RMAed, and we have not complained. >That can happen to anyone. > > > >What's different is that Telrad has eventually and repeatedly confirmed >that we have done everything right: that our network is intelligently >designed and built. ("His parents were poor but honest.") > >That our customers could not be better installed with attention to >signal and conditions. > >That poor performance is not, I repeat not, due to our incompetence, >our environment, our customers, or our network -- in fact, not due to >one single thing *under our control*. > > > >What's different is that we have repeatedly exhausted the abilities of >the North American support crew -- terrific as they are -- not because >we were asking dumb questions, but because the answers have not yet >been found. > > > >What's different is that Ubiquiti won't send you an engineer to check >your stuff out. Telrad's commitment to support is real, if spotty. We >had a truly enjoyable time spending three days with Guy Shahak from >Israel. We learned a great deal. He came ready to help, and earnestly >believed that he would fix all problems. And guess what -- we stumped >him too. > > > >What's different is that if there were any thing or things that I -- or >the entire engineering resources of Telrad that have been brought to >bear on this -- could do to find, fix, and permanently stop the >whack-a-mole process -- *it would be done by now*. > > > >What's different is that I do not assume that *my* experience >invalidates yours. I would hope you could extend the same courtesy to >me. Telrad knows, at least by now, that "Works for me!" or "Works in >the lab!" does not guarantee 100% success. We too added upwards of 40 >UEs a month, until we were forced to stop. > > > >How a company handles failure: that is telling. > > > >I apologize for the length of this, and I appreciate your feedback. The >list needs more success stories. I hope to be one of them soon. > > > >On Thu, Mar 9, 2017 at 6:36 PM, Skywerx Support ><[email protected]> wrote: > >The Telrad Wispa forum is starting to remind me of the Ubiquiti forum. >The same exact people posting with the same exact issues. But only >like three people!!! Issues like why can't I put 80 people on a Rocket >M5, or how do I configure a Rocket M5?Why does service suck on my >access points? Oh wait, they all have -81 signals. > >We have deployed over 700 UE's on 16 eNB's. Adding around 40 new UE's >per month and at least 1 new eNB per month. It's our fastest growing >product with the least amount of tech support calls offering speeds >anywhere from 3 Mbps to 30 Mbps on the LTE network. > >What's Different? > >-- >Justin Davis >COO >SkyWerx Industries, LLC > > > > On Mar 9, 2017, at 5:01 PM, Steve Cole <[email protected]> wrote: > > > >> On 3/9/2017 4:53 PM, Jeremy Austin wrote: > >> I have been fairly quiet on list about our outstanding issues, > >> thinking that they would be better solved by superior >troubleshooting > >> and Telrad engineering than by social engineering. > > > > You are certainly not alone. > > _______________________________________________ > > Telrad mailing list > > [email protected] > > http://lists.wispa.org/mailman/listinfo/telrad >_______________________________________________ >Telrad mailing list >[email protected] >http://lists.wispa.org/mailman/listinfo/telrad > > > > > > >-- > >Jeremy Austin > > > >(907) 895-2311 > >(907) 803-5422 > >[email protected] > > > >Heritage NetWorks > >Whitestone Power & Communications > >Vertical Broadband, LLC > > > >Schedule a meeting: http://doodle.com/jermudgeon ><http://doodle.com/jermudgeon> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.wispa.org/mailman/private/telrad/attachments/20170315/a8498b04/attachment.html ------------------------------ _______________________________________________ Telrad mailing list [email protected] http://lists.wispa.org/mailman/listinfo/telrad End of Telrad Digest, Vol 29, Issue 14 ************************************** ************************************************************************************ This footnote confirms that this email message has been scanned by PineApp Mail-SeCure for the presence of malicious code, vandals & computer viruses. ************************************************************************************ ************************************************************************************ This footnote confirms that this email message has been scanned by PineApp Mail-SeCure for the presence of malicious code, vandals & computer viruses. ************************************************************************************ _______________________________________________ Telrad mailing list [email protected] http://lists.wispa.org/mailman/listinfo/telrad
