Miroslaw thanks for tracking this down. Do you have an account on gitlab.rtems.org?
I think this needs a work item (formerly known as an issue) for main. And a merge request. @gedare This seems like it should be backported to 6. Thoughts? --joel On Wed, Oct 14, 2026 at 5:13 PM Miroslaw Dach <[email protected]> wrote: > Hi Gedare, > > The issue with RTEMS on RPi 3B+ was solved. > The problem was in the Raspberry Pi BSP interrupt dispatcher that makes > the whole system stop under interrupt load. We hit it on a Raspberry Pi 3 > Model B+ running an EPICS 7.0.10 IOC with rtems-libbsd (USB Ethernet) on > RTEMS 6.2; > > Symptom > ------- > With network traffic on the LAN7800 USB Ethernet the board froze after > 30 s to 3 min, idle after a few minutes: console dead, no task runs, an > externally visible "heartbeat" stops. It looked like a bus or CPU lockup > and we spent months on USB controller theories. Earlier reports of > "RTEMS hangs on Raspberry Pi with USB/network" on the lists may be the > same issue. > > Root cause > ---------- > bsp_interrupt_dispatch() in bsps/arm/raspberrypi/irq/irq.c reads > BCM2835_IRQ_BASIC and, when any "speed-up" bit is set, calls ffs() on > the UNMASKED value: > > pend = BCM2835_REG(BCM2835_IRQ_BASIC); > if ( pend & BCM2835_IRQ_BASIC_SPEEDUP_USED_BITS ) { > pend_bit = ffs(pend) - 1; > vector = bcm2835_irq_speedup_table[pend_bit]; > > Bit 8 of IRQ_BASIC ("one or more bits set in pending register 1") is set > whenever a bank-1 interrupt without a speed-up bit is pending, for > example the system timer compare 3 used for the clock tick (GPU IRQ 3). > If such an interrupt is pending at the same time as a speed-up one, for > example USB (GPU IRQ 9, speed-up bit 11), the register reads 0x900. > ffs(0x900) - 1 = 8, bcm2835_irq_speedup_table[8] is -1, vector becomes > 0xffffffff, the check "vector < 255" fails, no handler is called, the > interrupt stays asserted and the CPU re-enters the dispatcher forever. > > We confirmed this with a RAM trace that survives a watchdog reset: the > last 600 entries of every frozen run were the IRQ_BASIC read returning > 0x900, about 1.3 million times per second, with no handler call in > between. The probability per interrupt is small, so the failure rate > scales with the interrupt rate; with ~40000 USB interrupts/s and a 2 kHz > tick it took about 30 s. > > Fix > --- > Mask the value before the bit scan. With 0x900 the USB interrupt is then > dispatched first; the immediate re-entry sees 0x100, takes the > IRQ_PENDING1 path and dispatches the tick. > > Tested on Raspberry Pi 3B+ (raspberrypi2 BSP, RTEMS 6.2 be446b02d0, > rtems-libbsd 6-freebsd-14, EPICS 7.0.10): 15.5 hours of continuous > operation with 450 million USB transfers and 1.8 billion interrupts, > versus a freeze within a minute before the change. > The patch could be found in the attachment. > > Best Regards > Mirek > > > > czw., 21 maj 2026 o 14:41 Miroslaw Dach <[email protected]> > napisał(a): > >> Hi Gedare, >> >> Thanks for the good news about the RPi4. Please let me know when you have >> some results and working RPi4 with RTEMS. >> Did anybody try the lwIP network stack for RTEMS? >> >> Best Regards >> Mirek >> >> wt., 19 maj 2026 o 15:18 Gedare Bloom <[email protected]> napisał(a): >> >>> Just a quick note that the preferred venue for community engagement is >>> shifting toward https://users.rtems.org >>> >>> I think the RPi4 is maturing quite rapidly. We have a GSoC project >>> focused on it as well. >>> >>> lwIP might also be an option to consider. >>> >>> On Fri, May 8, 2026 at 4:00 PM Miroslaw Dach <[email protected]> >>> wrote: >>> > >>> > Hi John, >>> > >>> > Many Thanks for your reply and all of the hints. It looks like that at >>> present there is no good solution to use RTEMS on RPi 3 b+ due to the issue >>> with the >>> > usb-Network bridge. The most promising approach would be to use the >>> PRi v4 or v5 since the network chip is directly connected to the system >>> but the BSP is not yet complete for RTEMS so maybe in the future. >>> > >>> > Best Regards >>> > Mirek >>> > >>> > sob., 25 kwi 2026 o 10:37 John Howard <[email protected]> >>> napisał(a): >>> >> >>> >> Hi Mirek, >>> >> >>> >> I don't know what to tell you about this issue. You know more about >>> the details than I do. >>> >> >>> >> (I don't know what EPICS IOC does.) >>> >> >>> >> I knew RTEMS didn't use FIQ, but I assumed a 32/64-bit FreeBSD for >>> Raspberry Pi didn't either. Why not a continuing FIQ issue on their mailing >>> list after all this time? I haven't used FreeBSD for RPi (too many security >>> holes to me). I use Debian Bullseye for my 3B+ and I am not happy with >>> Linux. >>> >> >>> >> I don't expect this to occur on RPi Zero 2 W because it lacks the >>> Ethernet subsystem and has a newer revision of the 3B+ subsystem. (I will >>> deal with it later during my app testing phase.) >>> >> >>> >> I am not the CODEOWNER at RTEMS.org for that Dynaware/Synopsis driver. >>> >> >>> >> I do know that Synopsis released it unlicensed AS IS but is willing >>> to license it under obligation for their customers. >>> >> >>> >> I recently looked at an alternative kernel memory map document where >>> FIQ stacks are present for RPi 3B+ when Aarch32, but I can't find them when >>> Aarch64 which standardized calling conventions. >>> >> >>> >> Sorry I couldn't be of much help to you on this issue. Dealing with >>> FIQ handling is as idiosyncratic as it gets on Raspberry Pi and is a >>> deal-breaker to me. >>> >> >>> >> -- John >>> >> >>> >> On Apr 22, 2026, at 10:28 PM, Miroslaw Dach <[email protected]> >>> wrote: >>> >> >>> >> >>> >> Hi John, >>> >> >>> >> Thanks for your e-mail and your suggestion. >>> >> >>> >> Follow-up to my earlier report about the DWC OTG USB hang on RPi 3B+ >>> >> with RTEMS 6. I've done extensive instrumentation and applied BCM2835- >>> >> specific errata workarounds from the Linux dwc2 driver. Here are the >>> >> complete findings: >>> >> >>> >> I added diagnostic counters (#ifdef __rtems__) to the FreeBSD dwc_otg >>> >> driver, printed every 1 second from the 10ms timer callback: >>> >> >>> >> - yld: times dwc_otg_interrupt_poll_locked() hit its 16-iteration >>> cap >>> >> - afail: times dwc_otg_host_channel_alloc() failed (no free channels) >>> >> - hok: successful halt completions processed >>> >> - rxd: RX data discarded (no active endpoint listening) >>> >> - stuck: channels with wait_halted=1 but allocated=0 (leaked channels) >>> >> - filt: filter ISR entry/exit count (to detect stuck ISR) >>> >> - thr: thread handler entry/exit count >>> >> - gi: last GINTSTS register value >>> >> >>> >> Results: channel exhaustion theory DISPROVED >>> >> Across all tests, every counter remained perfectly healthy right up to >>> >> the instant of the freeze: >>> >> >>> >> DWC_OTG t=17500: yld=0 afail=0 hok=180038 rxd=0 ch=8/16 >>> >> alloc=0 wh=0 stuck=0 filt=738178/738178 thr=7280/7280 >>> >> gi=0x04000031 >>> >> >>> >> - yld=0: the 16-iteration poll loop cap was NEVER hit >>> >> - afail=0: channel allocation NEVER failed >>> >> - stuck=0: NO channels ever leaked (wait_halted was always cleared) >>> >> - filt balanced: filter ISR entered and exited the same number of >>> times >>> >> - thr balanced: thread handler entered and exited the same number of >>> times >>> >> - gi=0x04000031: benign (host mode + TX FIFOs empty + RX FIFO >>> non-empty) >>> >> >>> >> The ~1054 halt completions/second were steady with no degradation. >>> >> >>> >> Results: total CPU freeze, not software deadlock >>> >> >>> >> I also tried an independent RTEMS timer (rtems_timer_fire_after) as a >>> >> heartbeat, firing outside the USB bus lock. Both the USB timer and the >>> >> independent heartbeat stopped simultaneously, confirming the entire >>> ARM >>> >> core freezes — not just the USB subsystem. >>> >> >>> >> No RTEMS fatal error handler was triggered (I installed one via >>> >> CONFIGURE_INITIAL_EXTENSIONS). This means it is NOT a standard ARM >>> >> data abort — the CPU simply stops executing, likely due to an AHB bus >>> >> lockup caused by the DWC OTG controller hardware. >>> >> >>> >> BCM2835 errata workarounds applied >>> >> >>> >> Based on the Linux dwc2 driver (params.c, core.c) and the Ultibo >>> >> project documentation, I applied three workarounds: >>> >> >>> >> 1. FIFO size cap: BCM2835 hardware reports 4096 words but only 4080 >>> >> exist. Cap sc_fifo_size to 4080*4 bytes to prevent FIFO overrun. >>> >> (In practice, the hardware on my board reported <= 4080, so this >>> >> cap was NOT triggered.) >>> >> >>> >> 2. GAHBCFG AHB burst configuration: Broadcom redefined bits [4:1] of >>> >> GAHBCFG for AXI burst control. Linux dwc2 sets ahbcfg=0x10 for >>> >> BCM2835. Changed from GAHBCFG_GLBLINTRMSK (0x01) to 0x11. >>> >> >>> >> 3. AHB idle wait after core reset: Added a loop waiting for >>> >> GRSTCTL_AHBIDLE after GRSTCTL_CSFTRST, plus 250ms settling delay >>> >> (Linux dwc2 uses 100+ ms). >>> >> >>> >> Results with workarounds: >>> >> - Before: hangs after ~86 seconds (2 kHz tick) >>> >> - After: hangs after ~175 seconds (2 kHz tick) >>> >> >>> >> The workarounds approximately doubled the time-to-hang but did NOT >>> >> eliminate it. All software counters remained healthy throughout. >>> >> >>> >> The hang is caused most probably by a combination of factors: >>> >> >>> >> 1. The DWC OTG controller on BCM2835/BCM2837 requires sub-125µs >>> >> interrupt response times for USB split transaction phases (Start >>> >> Split → Complete Split through the USB hub). The RPi 3B+ Ethernet >>> >> goes through a USB hub (LAN7515), making every packet a split >>> >> transaction. >>> >> >>> >> 2. The Linux kernel addresses this with a dedicated FIQ (Fast >>> Interrupt >>> >> Request) handler (dwc_otg_fiq_fsm.c) that executes complete split >>> >> transactions in FIQ context, bypassing the normal interrupt stack. >>> >> Without FIQ, "certain USB devices become completely unusable." >>> >> >>> >> 3 The FreeBSD dwc_otg driver used by RTEMS handles all split >>> >> transactions in normal interrupt context. On RTEMS, the interrupt >>> >> filter and thread handler run back-to-back in the interrupt server >>> >> task (nexus_intr_with_filter in rtems-kernel-nexus.c), with no >>> >> preemption point between them. >>> >> >>> >> 4 When the controller's split transaction timing is violated, it >>> >> enters an unrecoverable state that locks the AHB bus, freezing the >>> >> entire ARM core including UART and system timers. >>> >> >>> >> The timing correlation with tick rate confirms this: higher tick rate >>> >> = more frequent scheduling = more interrupt latency jitter = faster >>> >> timing violation. >>> >> >>> >> A proper fix requires most probably implementing FIQ-based split >>> transaction handling >>> >> in the RTEMS BSP for BCM2835/BCM2837, similar to what Linux does in >>> >> dwc_otg_fiq_fsm.c. This is a significant undertaking but is essential >>> >> for reliable USB operation on RPi 3B+ (and RPi Zero 2 W, which uses >>> >> the same SoC). >>> >> >>> >> The GAHBCFG and reset sequence workarounds should also be applied as >>> >> they improve stability. >>> >> >>> >> Please give me your thoughts on that. Maybe it is easier to finalise >>> the BSP for RPi 4 and 5? >>> >> >>> >> Best Regards >>> >> Mirek >>> >> >>> >> śr., 22 kwi 2026 o 13:17 John Howard <[email protected]> >>> napisał(a): >>> >>> >>> >>> Fascinating. Thanks for that detailed report. >>> >>> >>> >>> You indicated USB enumerating continues running in the background. >>> >>> >>> >>> I am educated-guessing that a counter maximum is reached, and then >>> mistakenly breached. I would look for a test of that counter and correct it >>> from allowing greater-than comparing. >>> >>> >>> >>> Let us know how it turns out. >>> >>> >>> >>> I am developing an app for Raspberry Pi Zero 2 W (stripped-down >>> 3B+). I wasn't expecting any potential problem like this. >>> >>> >>> >>> -- John >>> >>> >>> >>> > On Apr 22, 2026, at 1:25 PM, Miroslaw Dach < >>> [email protected]> wrote: >>> >>> > >>> >>> > >>> >>> > Hi All, >>> >>> > >>> >>> > I'm running an EPICS ioc server (EPICS 7.0.10) with RTEMS 6.2 on >>> a Raspberry Pi 3B+ with rtems-libbsd (6-freebsd-14) and >>> >>> > encountering a reproducible system hang after several minutes of >>> operation. >>> >>> > Through systematic elimination testing I've narrowed the root >>> cause to the >>> >>> > DWC OTG USB controller driver. I'd appreciate any recommendations >>> on how >>> >>> > to address this. I can of course use the EPICS ioc server under >>> linux on RPi but just tried to have the RTEMS - Hard real time system which >>> is much more deterministic. >>> >>> > The boot time for the RPi with EPICS/RTEMS is around 7 sec which >>> is extremely fast! >>> >>> > >>> >>> > Environment >>> >>> > ----------- >>> >>> > - Board: Raspberry Pi 3B+ (BCM2837, boardrev a020d3) >>> >>> > - RTEMS: rtems-6.2 (ARM/ARMv4/raspberrypi2) >>> >>> > - Network stack: rtems-libbsd (RTEMS_BSD_CONFIG_BSP_CONFIG + >>> RTEMS_BSD_CONFIG_INIT) >>> >>> > - Ethernet: LAN7515 USB Ethernet (muge driver, via DWC OTG) >>> >>> > - Application: EPICS IOC (but hang occurs with minimal/empty IOC >>> as well) >>> >>> > - Console: UART serial (/dev/ttyS0) >>> >>> > >>> >>> > Symptom >>> >>> > ------- >>> >>> > The system boots and runs normally, then the entire system freezes >>> — >>> >>> > including the UART serial console (which is not USB-dependent). No >>> crash >>> >>> > message, no stack dump — a hard hang requiring power cycle. >>> >>> > >>> >>> > The time-to-hang depends on the system tick rate: >>> >>> > - CONFIGURE_MICROSECONDS_PER_TICK=500 (2 kHz): hangs after ~2 >>> minutes >>> >>> > - CONFIGURE_MICROSECONDS_PER_TICK=10000 (100 Hz): hangs after ~5-6 >>> minutes >>> >>> > >>> >>> > During the hang, the UART becomes completely unresponsive, >>> suggesting a >>> >>> > kernel-level deadlock or interrupt handler issue rather than an >>> >>> > application-level problem. >>> >>> > >>> >>> > Elimination testing performed >>> >>> > ----------------------------- >>> >>> > I systematically disabled components to isolate the cause. All >>> tests below >>> >>> > used CONFIGURE_MICROSECONDS_PER_TICK=10000 (100 Hz): >>> >>> > >>> >>> > 1. Disabled all EPICS database records (no record processing) -> >>> still hangs >>> >>> > 2. Disabled periodic NTP sync (no socket operations) -> still hangs >>> >>> > 3. Disabled IP configuration (no ifconfig, no route, no network >>> traffic, >>> >>> > but USB/DWC OTG still initialised via >>> RTEMS_BSD_CONFIG_BSP_CONFIG) >>> >>> > -> still hangs (~5-6 min), USB hub enumeration continues in >>> background: >>> >>> > ugen1.2: <vendor 0x0424 product 0x2514> at usbus1 >>> >>> > uhub1 on uhub0 >>> >>> > ... >>> >>> > Console output is garbled by concurrent USB enumeration >>> messages, >>> >>> > suggesting interrupt contention. >>> >>> > 4. Commented out RTEMS_BSD_CONFIG_BSP_CONFIG and the >>> >>> > #include <bsp/nexus-devices.h> to prevent DWC OTG initialisation >>> >>> > -> STABLE, ran for 14+ minutes with no hang (test stopped >>> manually) >>> >>> > >>> >>> > The libbsd software stack (loopback, sockets, telnetd) continues to >>> >>> > function in test 4 — only the hardware BSP devices (DWC OTG, muge, >>> >>> > uhub, ukphy) are excluded. >>> >>> > >>> >>> > Minimal reproduction >>> >>> > -------------------- >>> >>> > Build an RTEMS 6.2 application for raspberrypi2 BSP with: >>> >>> > >>> >>> > #define RTEMS_BSD_CONFIG_BSP_CONFIG >>> >>> > #define RTEMS_BSD_CONFIG_INIT >>> >>> > #include <machine/rtems-bsd-config.h> >>> >>> > #include <bsp/nexus-devices.h> /* in one translation unit */ >>> >>> > >>> >>> > #define CONFIGURE_MICROSECONDS_PER_TICK 10000 >>> >>> > >>> >>> > The application does not need to configure any network interface or >>> >>> > perform any USB transfers — the DWC OTG hub polling alone triggers >>> >>> > the hang after ~5-6 minutes. >>> >>> > >>> >>> > Commenting out RTEMS_BSD_CONFIG_BSP_CONFIG and the nexus-devices.h >>> >>> > include eliminates the hang. >>> >>> > >>> >>> > Boot log (abbreviated, from hanging configuration) >>> >>> > -------------------------------------------------- >>> >>> > RTEMS RPi 3B+ 1.3 (1GB) [00a020d3] >>> >>> > nexus0: <RTEMS Nexus device> >>> >>> > dwcotg0: <DWC OTG 2.0 integrated USB controller> on nexus0 >>> >>> > usbus1 on dwcotg0 >>> >>> > usbus1: 480Mbps High Speed USB v2.0 >>> >>> > ugen1.1: <DWCOTG OTG Root HUB> at usbus1 >>> >>> > uhub0 on usbus1 >>> >>> > uhub0: <DWCOTG OTG Root HUB, class 9/0, rev 2.00/1.00, addr 1> on >>> usbus1 >>> >>> > uhub0: 1 port with 1 removable, self powered >>> >>> > ugen1.2: <vendor 0x0424 product 0x2514> at usbus1 >>> >>> > uhub1 on uhub0 >>> >>> > uhub1: <vendor 0x0424 product 0x2514, class 9/0, rev 2.00/b.b3, >>> addr 2> on usbus1 >>> >>> > uhub1: 4 ports with 3 removable, self powered >>> >>> > ugen1.3: <vendor 0x0424 product 0x2514> at usbus1 >>> >>> > uhub2 on uhub1 >>> >>> > uhub2: <vendor 0x0424 product 0x2514, class 9/0, rev 2.00/b.b3, >>> addr 3> on usbus1 >>> >>> > uhub2: 3 ports with 2 removable, self powered >>> >>> > ugen1.4: <vendor 0x0424 product 0x7800> at usbus1 >>> >>> > muge0: <vendor 0x0424 product 0x7800, rev 2.10/3.00, addr 4> on >>> usbus1 >>> >>> > muge0: Chip ID 0x7800 rev 0002 >>> >>> > miibus0: <MII bus> on muge0 >>> >>> > ukphy0: <Generic IEEE 802.3u media interface> PHY 1 on miibus0 >>> >>> > info: ue0: <USB Ethernet> on muge0 >>> >>> > [system hangs after ~5-6 minutes, UART unresponsive] >>> >>> > >>> >>> > Questions >>> >>> > --------- >>> >>> > 1. Is this a known issue with the DWC OTG driver on RPi 3B+? >>> >>> > 2. Are there any configuration options (hub polling interval, >>> interrupt >>> >>> > coalescing, DMA settings) that might work around the problem? >>> >>> > 3. Would a newer version of rtems-libbsd contain fixes for this? >>> >>> > 4. Is there an alternative Ethernet driver approach for RPi 3B+ >>> that >>> >>> > avoids the DWC OTG USB path? >>> >>> > 5. Is there any known project which uses RPi with RTEMS? >>> >>> > (it looks like that the option to run RTEMS on RPi4 or RPi 5 can >>> not be considered since the BSP in RTEMS kernel is not yet finalised. >>> >>> > The RPi4 or RPi 5 would be much better candidates vs RPi 3 B+ >>> since they use direct connection to Ethernet instead of the USB-Ethernet) >>> >>> > >>> >>> > Thank you for any guidance. >>> >>> > >>> >>> > Mirek >>> >>> > >>> >>> > >>> >>> > _______________________________________________ >>> >>> > users mailing list >>> >>> > [email protected] >>> >>> > http://lists.rtems.org/mailman/listinfo/users >>> > >>> > _______________________________________________ >>> > users mailing list >>> > [email protected] >>> > http://lists.rtems.org/mailman/listinfo/users >>> >> _______________________________________________ > users mailing list > [email protected] > http://lists.rtems.org/mailman/listinfo/users
_______________________________________________ users mailing list [email protected] http://lists.rtems.org/mailman/listinfo/users
