On Mon, Aug 6, 2018 at 8:13 AM, Amaan Cheval <amaan.che...@gmail.com> wrote:
> Thanks for all the help! I have a simple test using the RTEMS > interrupt manager working successfully (tested by calling > rtems_interrupt_handler_install for vector 0, and then triggering a > divide-by-0 exception). > Yeah! > > Could someone shed any light on why the i386 only hooks the first 17 > vectors as "RTEMS interrupts"? > You are making me feel very old especially since I have the real IBM manual in my office which corresponds to the answer. It is dated Sept 1985. In fairness, I saved it from the garbage heap years later when someone was cleaning out their office. :) The x86 architecture is really vectored and the original i386 port actually used simple direct vectoring since the first BSP wasn't a PC. Imagine that! Another board using an i386 which didn't look like a PC at all. For better or worse, the PC/AT (286) and later used two i8259 PICs in a master and slave configuration. The slave PIC cascaded off the master PIC. This all fed into one CPU IRQ so many of the direct vectors were unused. The PIC arrangement is described here: https://en.wikipedia.org/wiki/Interrupt_request_(PC_architecture) > > Here's what I'm aiming to get done before the GSoC deadline: > > - Remap PIC (masking/disabling the PIC doesn't stop it from generating > spurious interrupts (IRQ7), which would look like exceptions to us) > - Disable PIC > - Enable APIC (done already, but confirm it plays well with the recent > changes to the IDT) > - Enable the PIT timer and use it to calibrate the APIC timer > - Clock driver using the APIC timer - (1) generate interrupts on ticks > and (2) tc_get_timecount function which calculates total time passed > through calculating (number of IRQs occured * time_per_irq + > time_passed_since_last_irq (through tick counter)) > > This does seem a bit ambitious given how short we are on time - I'll > finish this up even after the deadline if need be. > > What should our minimum deliverable be for this period? Should we try > to upstream the interrupt support before I finish the clock driver? (I > think we can have this discussion on Wednesday or so, since by then > I'll likely know how much progress on the clock driver remains.) > We could but do you think it is likely to have major changes based on getting the tick working? Try to see what gets done and post what you can by the end of GSoC. Then we will all wait patiently for you to get it working if it isn't then. You do need to do a final report. > > Sorry about the rush near the deadline - getting the APIC functioning > had a bunch more yak-shaving style issues than I anticipated. > The x86 architecture is years of hack upon hack. > > On Wed, Aug 1, 2018 at 10:51 PM, Joel Sherrill <j...@rtems.org> wrote: > > I have started to reply twice but you jumped in ahead. :) > > > > On Wed, Aug 1, 2018 at 12:12 PM, Amaan Cheval <amaan.che...@gmail.com> > > wrote: > >> > >> If my previous email _is_ in fact correct, could someone confirm? > >> Because this excerpt in the documentation here seems to contradict it > >> (which was what lead to the confusion in the first place): > >> > >> > >> https://docs.rtems.org/branches/master/c-user/interrupt_manager.html# > establishing-an-isr > >> > >> With my emphasis: > >> > >> > The rtems_interrupt_catch directive establishes an ISR for the system. > >> > The address of the ISR and its associated CPU vector number are > specified to > >> > this directive. This directive installs the **RTEMS interrupt wrapper > in the > >> > processor’s Interrupt Vector Table and the address of the user’s ISR > in the > >> > RTEMS’ Vector Table**. This directive returns the previous contents > of the > >> > specified vector in the RTEMS’ Vector Table. > > > > > > Almost but Gedare and I left out a detail. rtems_interrupt_catch is ONLY > > used on pure simple vectored architectures which do not use the > > bsp_interrupt_* > > or <rtems/irq.h> interfaces. Some embedded MCUs are so simple and have > > plenty of vectors so you don't need the complexity of supporting a PIC. > For > > example, the m68k family had 256 direct vectors and I don't recall ever > > seeing > > a PIC.[1] > > > > You should assume that you can ignore rtems_interrupt_catch and simple > > vectored support for x86_64. See cpukit/rtems/intrcatch.c and I hope you > > see an ifdef that results in the code disappearing on your port. Simple > > vectored is FACE is your cpu.h. :) > > > > [1] Disclaimer: The support for the <rtems/irq.h> interfaces is critical > to > > the > > libbsd stack. We haven't discussed it but any architecture that is > > sufficient > > to run the new stack will have to support this interface. If someone > wants > > the > > new stack on a 68040 VME board or a Coldfire board, we will have to find > > the simplest, non-bloated way to support this. When doing the MIPS Malta, > > we just converted the MIPS architecture away from simple vectored. > > > > So support the bsp_interrupt_* infrastructure. :) > > > > --joel > > > >> > >> > >> On Wed, Aug 1, 2018 at 10:39 PM, Amaan Cheval <amaan.che...@gmail.com> > >> wrote: > >> > Okay, I think I understand finally. Sorry about the rambling! > >> > > >> > When rtems_interrupt_catch is called, that's installing a "raw" ISR by > >> > modifying the processor specific table itself, so _ISR_Handler is > >> > never called, but the user ISR is. > >> > > >> > When rtems_interrupt_handler_install is called, that's an "RTEMS > >> > interrupt", and we go through the _ISR_Handler -> dispatch route I > >> > laid out earlier, leading to eventually the user's ISR. > >> > > >> > Thank you for letting me rubber-duck with you, everyone (let me know > >> > if anything above sounds off, though!) :P > >> > > >> > On Wed, Aug 1, 2018 at 10:20 PM, Amaan Cheval <amaan.che...@gmail.com > > > >> > wrote: > >> >> Thanks for the background! > >> >> > >> >> Let's use the gen5200 as the ongoing example - my confusion arises > >> >> here (correct me if any of the following points is incorrect!). In > >> >> overly simplified call chains: > >> >> > >> >> Register interrupt handler: > >> >> - bsp_interrupt_facility_initialize() -> ppc_exc_set_handler(vec, > >> >> C_dispatch_irq_handler) -> ppc_exc_handler_table[vec] = > >> >> C_dispatch_irq_handler > >> >> > >> >> Interrupt handler called: > >> >> - C_dispatch_irq_handler -> dispatch -> > bsp_interrupt_handler_dispatch > >> >> (irq-generic.h) -> bsp_interrupt_handler_table[index].handler() > >> >> > >> >> What I'm confused about is how the bsp_interrupt_handler_table is > >> >> updated at all - I just haven't found the link between how the > entries > >> >> in the two tables are synchronized, where the tables are: > >> >> > >> >> 1) the ppc_exc_handler_table (the processor IDT) and > >> >> 2) the bsp_interrupt_handler_table (the RTEMS interrupt table) > >> >> > >> >> ------------------------------------------------------------ > >> >> Another similar chain of confusion for i386 is: > >> >> - rtems_interrupt_catch (intrcatch.c) -> _ISR_Install_vector (isr.h) > >> >> -> _CPU_ISR_install_vector (i386/idt.c) -> idt_entry_tbl[vector] > >> >> updated > >> >> > >> >> But the i386 dispatch code chain is: > >> >> - _ISR_Handler (i386/irq_asm.S) -> BSP_dispatch_isr (i386/irq.c) -> > >> >> bsp_interrupt_handler_dispatch (irq-generic.h) -> > >> >> bsp_interrupt_handler_table[index].handler() > >> >> > >> >> But I don't see any updates to bsp_interrupt_handler_table that would > >> >> let this work. > >> >> ------------------------------------------------------------- > >> >> > >> >> Would you happen to know what that "missing link" is? > >> >> > >> >> On Wed, Aug 1, 2018 at 9:07 PM, Joel Sherrill <j...@rtems.org> > wrote: > >> >>> > >> >>> > >> >>> On Wed, Aug 1, 2018 at 10:11 AM, Gedare Bloom <ged...@rtems.org> > >> >>> wrote: > >> >>>> > >> >>>> On Wed, Aug 1, 2018 at 9:15 AM, Amaan Cheval < > amaan.che...@gmail.com> > >> >>>> wrote: > >> >>>> > That's definitely very illuminating, thank you so much for all > the > >> >>>> > details! > >> >>>> > > >> >>>> > A few more questions that have arisen for me. Feel free to skip > >> >>>> > over > >> >>>> > them (I'll likely figure them out given enough time, so I'm only > >> >>>> > asking in case any of them are obvious to anyone): > >> >>>> > > >> >>>> > - The i386 doesn't use CPU_Interrupt_frame at all. It seems like > it > >> >>>> > stores some of the data onto the stack? > >> >>>> > > >> >>>> the interrupt frame structure was introduced during 4.11 > development. > >> >>>> probably i386 never got updated to use a struct to encapsulate the > >> >>>> interrupt frame. the interrupt frame should contain the registers > >> >>>> that > >> >>>> are preserved by the interrupt entry code I believe. > >> >>> > >> >>> > >> >>> +1 > >> >>> > >> >>> Historically, there was no structure to represent the set of > >> >>> registers and information saved on interrupt entry. Over time > >> >>> this has been added. > >> >>> > >> >>> i386 also is missing the SMP synchronization check in the > >> >>> middle of the context which ensures it is safe for a thread to > >> >>> be migrated. > >> >>> > >> >>>> > >> >>>> > - There used to be defines in cpu.h regarding hardware/software > >> >>>> > based > >> >>>> > interrupt stacks, and how they'd be setup, which were made > >> >>>> > superfluous[1] - I'm not quite sure how these are meant to work > - I > >> >>>> > see references to "stack high" and "stack low" and I'm not quite > >> >>>> > sure > >> >>>> > what the code is referencing when using those. > >> >>>> > > >> >>>> > >> >>>> a hardware interrupt stack is one that the hardware switches to > >> >>>> during > >> >>>> an interrupt. i think m68k has such. > >> >>>> > >> >>>> most interrupt stacks in RTEMS are software-managed, meaning that > >> >>>> RTEMS explicitly switches the stack region off the task stack and > to > >> >>>> an interrupt stack region. > >> >>>> > >> >>>> some stacks start high and grow down, and some stacks start low and > >> >>>> grow up. maybe this is what the "stack high" and "stack low" you > >> >>>> mention are in relation to? > >> >>> > >> >>> > >> >>> They are used to denote the top and bottom of the memory reserved > >> >>> for the interrupt stack. One important use is in > >> >>> cpukit/libmisc/stackchk/check.c > >> >>> to report on usage. > >> >>> > >> >>>> > >> >>>> > >> >>>> > - c/src/lib/libbsp/no_cpu/no_bsp/Makefile.am doesn't include > >> >>>> > irq-sources.am, by the way (this is part of why I used to think > a > >> >>>> > lot > >> >>>> > of what your email mentioned was unnecessary, until you...ahem, > >> >>>> > pre-empted that line of thought and helped clarify it :P). > Should I > >> >>>> > add a ticket to update the no_bsp code to be more in line with > >> >>>> > current > >> >>>> > use? > >> >>>> > > >> >>>> Sure. I don't know that anyone is in particular maintaining > >> >>>> no_cpu/no_bsp since we can't compile it, it is basically best > effort > >> >>>> stuff that sometimes we miss updating. > >> >>> > >> >>> > >> >>> +1 > >> >>> > >> >>> Also there are variations based on simple vectored and PIC vectored > >> >>> architectures. > >> >>> > >> >>> The architecture is responsible for the managing the minimal actions > >> >>> based on what the CPU does for an interrupt/exception. Logically > >> >>> a PIC is part of the BSP even if it comes integrated with the CPU as > >> >>> is common on x86 and SoCs. The PIC could fairly easily change > >> >>> and the architecture is harder to change. > >> >>> > >> >>>> > >> >>>> > >> >>>> > - My understanding of _ISR_Handler is that it'll be the handler > for > >> >>>> > _all_ interrupt vectors by default - it'll then dispatch > interrupts > >> >>>> > to > >> >>>> > user-handlers (or internal handlers, for the timer, for eg.). Is > >> >>>> > that > >> >>>> > right? (I don't quite understand its interaction with the RTEMS > >> >>>> > interrupt manager yet, but irq-generic's > >> >>>> > "bsp_interrupt_handler_table" > >> >>>> > seems to be the RTEMS equivalent to the processor-specific vector > >> >>>> > table, and "bsp_interrupt_handler_dispatch" seems to call the > >> >>>> > actual > >> >>>> > handler within that table as appropriate. Accurate? (I just > haven't > >> >>>> > found how that table actually gets its handlers setup besides > >> >>>> > during > >> >>>> > initialization, since rtems_interrupt_catch just calls > >> >>>> > _CPU_install_vector, which updates the processor vector table, > not > >> >>>> > the > >> >>>> > RTEMS interrupt manager vector table.) > >> >>>> > > >> >>>> > >> >>>> You have discovered a couple of different but related interrupt > >> >>>> processing code bases. I can see why you get confused. > >> >>>> > >> >>>> Basically, a CPU port should support two kinds of interrupts that > may > >> >>>> be installed, "RTEMS" and "Raw" interrupts. The "Raw" interrupts > are > >> >>>> installed directly in the processor's vector table. For processors > >> >>>> that use simple vectored interrupts, the "RTEMS" interrupts > install a > >> >>>> call to the _ISR_Handler() function in the processor's vector > table, > >> >>>> and will put the user's isr function pointer in the > >> >>>> _ISR_Vector_table(), which is the RTEMS Interrupt Manager's vector > >> >>>> table. > >> >>>> > >> >>>> I'm not really familiar with the processors that use a different > >> >>>> model > >> >>>> for interrupt handling besides simple vectored. Probably, you will > >> >>>> have to study one of them. > >> >>>> > >> >>>> This irq-generic bsp_interrupt_* code is used by the "IRQ Server" > >> >>>> that > >> >>>> builds from the CPU port capabilities to allow for some advanced > >> >>>> features like chaining multiple isrs from the same source. I don't > >> >>>> think you want to be focusing on those right now, but I could be > >> >>>> mistaken. I haven't spent much time looking at the IRQ Server > >> >>>> codebase. > >> >>> > >> >>> > >> >>> For the most part, the bsp_interrupt_* code is hung off of one > >> >>> (or more) of the simple vectored sources. On many architectures, > >> >>> you have very few "CPU vectors" and the bsp_interrupt_* is > >> >>> the secondary layer managing the PIC. Notice that it is all > >> >>> in C and thus isn't first level processing. And it doesn't deal > >> >>> with RTEMS critical section, interrupt entry/exit, and dispatching.. > >> >>> > >> >>> For example, in its original form, the PowerPC didn't have many > >> >>> "exceptions" (e.g. CPU vectors) at all. In fact, there was a single > >> >>> External Exception which was always tied to a board dependent > >> >>> PIC. The first level processing of PPC exceptions is in assembly > >> >>> and deals with what I pointed you to in various _ISR_Handler's. > >> >>> The External Exception handler now process bsp_interrupt_*. > >> >>> The gen5200 BSP irq.c file has a bsp_interrupt_* implementation > >> >>> which registers for two "exceptions" which both go to > >> >>> C_dispatch_irq_handler() which reads the PIC, determines > >> >>> the source(s), and dispatches (via bsp_interrupt_handler_dispatch() > >> >>> the registered IRQ. > >> >>> > >> >>> If you think of this as two levels of processing, it should help. > >> >>> > >> >>> The two levels of processing is for RTEMS interrupts which can > >> >>> unblock threads, etc. The first level ensures that locking and > >> >>> dispatching is handled. > >> >>> > >> >>> As Gedare stated, there are also Raw interrupts which ignore > >> >>> RTEMS and can't use RTEMS services. These should never > >> >>> be anything but the highest priority interrupt in a system. The > >> >>> RTEMS first level processing can switch to a dedicated > >> >>> interrupt stack and mixing RTEMS and Raw IRQs can > >> >>> screw up the stack handling as well as the interrupt nesting > >> >>> count. > >> >>> > >> >>> --joel > >> >>> > >> >>>> > >> >>>> > >> >>>> Gedare > >> >>>> > >> >>>> > - My understanding of the interaction between RTEMS' interrupt > >> >>>> > manager > >> >>>> > (i.e. support for nested interrupts and thread dispatch once an > >> >>>> > interrupt ends) and the BSP's processor-specific interrupt > manager > >> >>>> > (code to use the APIC and IDT in my case) is that they're tied > >> >>>> > together through the use of irq-generic.c's > >> >>>> > "bsp_interrupt_initialize" > >> >>>> > - is that right? (m68k never seems to call it, though, so perhaps > >> >>>> > not?) > >> >>>> > > >> >>>> > Sorry about the rambling! To reiterate, I'll likely figure it out > >> >>>> > given enough time, so if the answers aren't at the top of your > >> >>>> > head, I > >> >>>> > can figure it out without wasting your time :) > >> >>>> > > >> >>>> > [1] https://devel.rtems.org/ticket/3459#comment:11 > >> >>>> > > >> >>>> > On Wed, Aug 1, 2018 at 3:18 AM, Joel Sherrill <j...@rtems.org> > >> >>>> > wrote: > >> >>>> >> > >> >>>> >> > >> >>>> >> On Tue, Jul 31, 2018 at 3:05 PM, Amaan Cheval > >> >>>> >> <amaan.che...@gmail.com> > >> >>>> >> wrote: > >> >>>> >>> > >> >>>> >>> Hm, I'm not sure what to look for in the other ports > >> >>>> >>> specifically, > >> >>>> >>> really. > >> >>>> >>> The BSP porting documentation doesn't have a section on > >> >>>> >>> interrupts, so > >> >>>> >>> I'm > >> >>>> >>> doing this on more of an "as it comes up" basis. > >> >>>> >>> > >> >>>> >>> What I've got right now (the interrupt handlers in C) are what > I > >> >>>> >>> need > >> >>>> >>> for > >> >>>> >>> calibrating the APIC timer (through the PIT) - so simply > hooking > >> >>>> >>> IRQ0 > >> >>>> >>> (for > >> >>>> >>> the timer) and IRQ7 (spurious vector), since those are needed > for > >> >>>> >>> the > >> >>>> >>> timer > >> >>>> >>> work to continue. > >> >>>> >>> > >> >>>> >>> What constitutes as a requirement for basic interrupt support? > >> >>>> >> > >> >>>> >> > >> >>>> >> There used to be a generic porting guide. I can see that this > >> >>>> >> particular > >> >>>> >> section > >> >>>> >> has bit rotted some but the interrupt dispatching section. Some > of > >> >>>> >> this > >> >>>> >> will have evolved to support SMP and fine grained locking but > the > >> >>>> >> pseudo-code > >> >>>> >> here will give you a push toward the right line of thinking: > >> >>>> >> > >> >>>> >> > >> >>>> >> > >> >>>> >> https://docs.rtems.org/releases/rtemsdocs-4.10.2/ > share/rtems/html/porting/porting00034.html > >> >>>> >> > >> >>>> >> The idea is that you need to ensure RTEMS knows it is inside an > >> >>>> >> interrupt > >> >>>> >> and the current locking scheme (old was dispatching, new is ...) > >> >>>> >> is > >> >>>> >> honored. > >> >>>> >> > >> >>>> >> The ARM and PowerPC (plus RISCV) are good ports to look at for > how > >> >>>> >> SMP > >> >>>> >> plays into this. But the CPU supplement is thin for their > >> >>>> >> interrupt > >> >>>> >> processing. > >> >>>> >> > >> >>>> >> > >> >>>> >> This is the CPU Architecture supplement section for the m68k. > This > >> >>>> >> is a > >> >>>> >> relatively simple > >> >>>> >> architecture to describe. There is also a section for the i386 > >> >>>> >> which > >> >>>> >> reads > >> >>>> >> similarly. > >> >>>> >> > >> >>>> >> > >> >>>> >> > >> >>>> >> https://docs.rtems.org/branches/master/cpu- > supplement/m68xxx_and_coldfire.html#interrupt-processing > >> >>>> >> > >> >>>> >> Personally, I find the m68k a fairly easy processor to read > >> >>>> >> assembly > >> >>>> >> in. > >> >>>> >> Look at cpukit/score/cpu/m68k/cpu_asm.S and _ISR_Handler to see > >> >>>> >> what > >> >>>> >> is done there w/o SMP. On the m68k _ISR_Handler is directly put > >> >>>> >> into > >> >>>> >> the > >> >>>> >> vector table. But this isn't the most similar example for you. > >> >>>> >> > >> >>>> >> For the i386 (better example), it is in > >> >>>> >> bsps/i386/shared/irq/irq_asm.S > >> >>>> >> with > >> >>>> >> the > >> >>>> >> same name. There _ISR_Handler is installed via the > >> >>>> >> DISTINCT_INTERRUPT_ENTRY > >> >>>> >> macros at the bottom of the file where some prologue jumps to > the > >> >>>> >> common > >> >>>> >> _ISR_Handler and then the actions are similar. Usually > >> >>>> >> _ISR_Handler > >> >>>> >> type of > >> >>>> >> code ends up invoking a PIC decode method in normal C without an > >> >>>> >> interrupt attribute. > >> >>>> >> > >> >>>> >> Long and multi-architecture answer but maybe that makes sense. > The > >> >>>> >> goal > >> >>>> >> in ticker.exe is to take a number of tick interrupts which don't > >> >>>> >> schedule > >> >>>> >> and > >> >>>> >> then take one that does -- it schedules a preemption of the idle > >> >>>> >> thread. > >> >>>> >> > >> >>>> >> Hope this helps. > >> >>>> >> > >> >>>> >> --joel > >> >>>> >> > >> >>>> >> > >> >>>> >> > >> >>>> >>> > >> >>>> >>> > >> >>>> >>> On Wed, Aug 1, 2018, 1:29 AM Joel Sherrill <j...@rtems.org> > >> >>>> >>> wrote: > >> >>>> >>>> > >> >>>> >>>> > >> >>>> >>>> > >> >>>> >>>> On Tue, Jul 31, 2018 at 2:52 PM, Amaan Cheval > >> >>>> >>>> <amaan.che...@gmail.com> > >> >>>> >>>> wrote: > >> >>>> >>>>> > >> >>>> >>>>> Hi Chris! > >> >>>> >>>>> > >> >>>> >>>>> I currently have code like this in > >> >>>> >>>>> c/src/lib/libbsp/x86_64/amd64/Makefile.am: > >> >>>> >>>>> > >> >>>> >>>>> librtemsbsp_a_SOURCES += > >> >>>> >>>>> ../../../../../../bsps/x86_64/amd64/interrupts/handlers.c > >> >>>> >>>>> # XXX: Needed to use GCC "interrupt" attribute > directives - > >> >>>> >>>>> can > >> >>>> >>>>> we > >> >>>> >>>>> pass these > >> >>>> >>>>> # flags only for the handlers.c source file (compile to > an > >> >>>> >>>>> object > >> >>>> >>>>> file first and > >> >>>> >>>>> # then link with the rest for librtemsbsp.a?) > >> >>>> >>>>> librtemsbsp_a_CFLAGS = -mgeneral-regs-only > >> >>>> >>>>> > >> >>>> >>>>> The CFLAGS arg is required to allow us to use > >> >>>> >>>>> "__attribute__((interrupt))" to setup interrupt handlers in > C. > >> >>>> >>>>> (See > >> >>>> >>>>> [1] and ctrl+f "interrupt" for more.) > >> >>>> >>>>> > >> >>>> >>>>> Is there a way to not force the CFLAGS for _all_ of > >> >>>> >>>>> librtemsbsp, but > >> >>>> >>>>> to limit it only to handlers.c? > >> >>>> >>>>> > >> >>>> >>>>> If not, is the above code something that would be acceptable > to > >> >>>> >>>>> have > >> >>>> >>>>> upstream? > >> >>>> >>>>> > >> >>>> >>>>> [1] > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> https://gcc.gnu.org/onlinedocs/gcc/x86-Function- > Attributes.html#x86-Function-Attributes > >> >>>> >>>> > >> >>>> >>>> > >> >>>> >>>> Are we basically talking about the outermost layer of your > >> >>>> >>>> interrupt > >> >>>> >>>> dispatching? > >> >>>> >>>> > >> >>>> >>>> > >> >>>> >>>> Have you looked at the basic approach taken by the other > ports? > >> >>>> >>>> They > >> >>>> >>>> end > >> >>>> >>>> up switching the stack pointer to a dedicated stack on the > >> >>>> >>>> outermost > >> >>>> >>>> interrupt > >> >>>> >>>> and, if a context switch/dispatch is needed, arrange for the > >> >>>> >>>> interrupted > >> >>>> >>>> task to call _Thread_Dispatch.But tinker with its stack so > some > >> >>>> >>>> registers > >> >>>> >>>> are saved and it looks like it made the call itself. > >> >>>> >>>> > >> >>>> >>>> If you can do it in C, I am ok with an attribute. I just don't > >> >>>> >>>> think > >> >>>> >>>> you > >> >>>> >>>> can pull off all the stack and return to dispatch magic that > >> >>>> >>>> way. > >> >>>> >>>> > >> >>>> >>>> --joel > >> >>>> >>>> > >> >>>> >> > >> >>> > >> >>> > > > > >
_______________________________________________ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
