As some of you may know, BT FTTC Huawei modems are being updated to new firmware which has a serious bug. This has been going on for months.
The bug is that the modem appears to cache some sessions passing through it, notably UDP where the cache is of source/target IP and port. My guess is that this is related to some sort of header compression type logic rather than conventional session tracking. If the PPPoE connection restarts, any packets on these cached sessions, which appear to be around 253 of recent sessions, no longer work. The packets do not appear at the ISP end. Obviously, to have any impact, this needs to be a restart without change of assigned IPs, which is one reason this is not so widely noticed. This has minor disruption to all sorts of UDP based protocols. It is most noticeable with VPNs based on UDP, but has also been seen affecting SIP registrations and other types of traffic. The "fix" is to disconnect the ethernet port from the modem and reconnect - this flushes the cache and all works from then. For customers making extensive use of VPNs, this bug is devastating as any blip causing a PPPoE restart leaves the VPN down until some manual intervention. It is causing a lot of problems for some customers. FYI, this has been demonstrated on BT Retail "infinity" services. I hope that is helpful if you are seeing issues on your customers FTTC services. However, BT/Huawei have identified the cause of the problem and know how to fix it. Huawei engineers came to our offices to diagnose the problem last year (more than two months ago). To date I have not had confirmation that BT have even stopped the firmware update, and they have not confirmed when they will fix this. At this stage I have advised my BT account managers that if not fixed soon, I will be reporting faults on each and every affected line giving them 40 clock hours to fix or pay out on service level guarantees. I'd appreciate it if other ISPs using BT that are seeing this issue explain the same to their account managers. Maybe this will kick them in to some sort of action. Do let me know if you need any more details of the issue. Regards. -- Adrian.
