Re: [Xenomai] Kernel Warning : I-pipe: Detected illicit call from head domain 'Xenomai'
On 07/11/2017 06:09 PM, Nitin Kulkarni wrote: > Hi Philippe, > Thanks for the answer. Can you refer me to any DMA driver that you integrated > with ipipe ? SDMA often found on the i.MX6 series can be adapted this way. The code supporting TI's EDMA too, although this is more tricky. > I found out that the DMA driver running is that idam64 and as you mentioned > it hooks to the generic DMA engine frame work > with the handlers implemented in idma64.c . > > I still have one question though, Do we adda real-time irq handler or keep > the existing non-realtime interrupt (through request_irq) for the DMA ? We > need the callback to be in real-time, so how is that possible without adding > a realtime irq ? > This is not possible doing sane things, hence the trickiness of dealing with that issue. > I think a reference will help me understand how to port the dma driver. > I'm not the copyright holder of that particular customer code, only the author. Besides, each DMA backend requires a specific implementation for that purpose, I don't think you could generalize the SDMA/EDMA hooks I mentioned in order to use them with another backend. To summarize the idea, the "only" issue is the requirement for sharing the DMA chip with regular Linux operations. This option I mentioned involves: 1- allocating Rx/Tx DMA channels for your rt SPI transfers the usual way, e.g. calling dma_request_slave_channel(). That operation would be done from a non-rt context, so you would get them once at init. 2- providing a rt capable dmaengine handler for kicking the DMA transfer on a channel; rt capable meaning which may be called from primary mode safely. You should see this as a basic, stateless "pulse" mode operation, i.e. when called, the handler routine triggers the DMA transfer immediately on the proper channel. That code needs to go to the idma64 backend driver. Since setting up the DMA for the next pulse has to be a very basic operation, you would have to work with a fixed DMA buffer of coherent memory attached to the channel, that should never change. Or maybe you could just run your channel strictly in continuous cyclic mode, programming your DMA channels to reset themselves after each transfer for the next run, in that case start and stop would be the only operations required. Instead of waiting for a completion IRQ, working asynchronously, maybe you could inspect the DMA channel status (on the chip) directly at some point in time later, to make sure the last transfer has completed right before starting the next one. Assuming your Linux-based SoC is the SPI master and the SPI bus is dedicated to rt traffic, a single software controller is likely to pace the overall transfer anyway. If you really have to work with an async notification of transfer completions, then you could look for a way to have the SPI controller issue the interrupt after a burst is received/sent, not the DMA chip. Maybe playing with the Rx/Tx FIFO thresholds if your SPI controller supports this. If you manage to do that, then grabbing the IRQ source of a SPI controller dedicated to rt processing is fine, you would not have to share this with the regular kernel. Again, it all depends on your requirements, provided you can work with fixed size transfers over a dedicated SPI bus, etc. YMMV. Ok, this is pretty much all I have on that topic. In any case, the key to success is to guarantee that such DMA "fast path" remains transparent with respect to the regular Linux operations using the same DMA engine. -- Philippe. ___ Xenomai mailing list Xenomai@xenomai.org https://xenomai.org/mailman/listinfo/xenomai
Re: [Xenomai] Kernel Warning : I-pipe: Detected illicit call from head domain 'Xenomai'
Hi Philippe, Thanks for the answer. Can you refer me to any DMA driver that you integrated with ipipe ? I found out that the DMA driver running is that idam64 and as you mentioned it hooks to the generic DMA engine frame work with the handlers implemented in idma64.c . I still have one question though, Do we adda real-time irq handler or keep the existing non-realtime interrupt (through request_irq) for the DMA ? We need the callback to be in real-time, so how is that possible without adding a realtime irq ? I think a reference will help me understand how to port the dma driver. Thanks a lot for the guidance. - Nitin From: Philippe Gerum <r...@xenomai.org> Sent: Monday, July 10, 2017 8:25 PM To: Nitin Kulkarni; xenomai@xenomai.org Subject: Re: [Xenomai] Kernel Warning : I-pipe: Detected illicit call from head domain 'Xenomai' On 07/10/2017 10:32 AM, Nitin Kulkarni wrote: > Hi, > > I am getting this message from I-Pipe trace when I enabled the I-pipe debug > option. > > Does this affect the performance of my driver in terms of execution time ? > and does this mean it switches mode from primary to > > secondary mode ? This message basically means that your system is no more in a reliable state, you just can't ignore it. > > Background: > > I am trying to port the linux spi-driver to rtdm driver and instead of going > all the way from scratch > > I thought of just trying to remove all the linux related scheduling and try > to initiate the transfer directly in > > real-time domain. I think I have done that partially (dma completion > interrupt is still in non-real time domain) but I get this message the first > time when I send some data over spi. I know that it is because of some memory > allocation for setting the dma buffers > > but the transmission is successful and quite quick now because I removed the > mechanism used by linux which used tasklets > > and work queues to schedule the transfer. I just want the transfer to start > as soon as possible triggered by an interrupt. > Sharing the DMA engine between the regular Linux kernel and the Cobalt co-kernel would require significant adjustments to the DMA engine code, you can't do that by simply calling the DMA API from primary mode, this would be fundamentally unsafe. Bottom line: both kernels run asynchronously for the most part, with Cobalt preempting the Linux kernel in sections that are otherwise deemed atomic, from Linux's standpoint. The interrupt pipeline machinery makes this possible. The price to pay for this is that you may not re-enter the regular Linux kernel code from a context which might have preempted it in the middle of nowhere, so to speak. Some hints about a working approach I've used a couple of times already for sharing a DMA engine. Assuming the DMA chip on this board is driven by the generic dmaengine framework, it is possible to extend the latter with a couple of (per-chip) handlers allowing to prepare and monitor transfers from a real-time context (i.e. Cobalt primary mode). Those handlers are defined in the dma_device structure, and implemented in the DMA backend code which is DMA engine-specific. Cobalt would then use a dedicated set of DMA channels obtained via dma_request_slave_channel() as usual, but controlled exclusively from these new handlers, for running real-time SPI transfers. Typically, you would have a handler to prepare a transfer, maybe another one to wait for an ongoing transfer to finish, all from/to a real-time Cobalt context, instead of a plain Linux context. You would have to make sure that your implementation never attempts to synchronize/serialize on regular Linux spinlocks/mutexes/sema4 etc. Interfacing with the IRQ subsystem for asynchronous handling of the DMA completion may be the main issue here if you need to, because the normal linux activities would still expect to receive end-of-transfer IRQs in the "root" domain as usual (request_irq interface) in parallel to the real-time activities, which would then be incompatible with assigning the DMA IRQ to the head/Cobalt domain (rtdm_irq_request()) exclusively. If you can live with receiving end-of-transfer IRQs without rt guarantees in the Linux domain though, then you don't have to deal with this problem. -- Philippe. ___ Xenomai mailing list Xenomai@xenomai.org https://xenomai.org/mailman/listinfo/xenomai
Re: [Xenomai] Kernel Warning : I-pipe: Detected illicit call from head domain 'Xenomai'
On 07/10/2017 10:32 AM, Nitin Kulkarni wrote: > Hi, > > I am getting this message from I-Pipe trace when I enabled the I-pipe debug > option. > > Does this affect the performance of my driver in terms of execution time ? > and does this mean it switches mode from primary to > > secondary mode ? This message basically means that your system is no more in a reliable state, you just can't ignore it. > > Background: > > I am trying to port the linux spi-driver to rtdm driver and instead of going > all the way from scratch > > I thought of just trying to remove all the linux related scheduling and try > to initiate the transfer directly in > > real-time domain. I think I have done that partially (dma completion > interrupt is still in non-real time domain) but I get this message the first > time when I send some data over spi. I know that it is because of some memory > allocation for setting the dma buffers > > but the transmission is successful and quite quick now because I removed the > mechanism used by linux which used tasklets > > and work queues to schedule the transfer. I just want the transfer to start > as soon as possible triggered by an interrupt. > Sharing the DMA engine between the regular Linux kernel and the Cobalt co-kernel would require significant adjustments to the DMA engine code, you can't do that by simply calling the DMA API from primary mode, this would be fundamentally unsafe. Bottom line: both kernels run asynchronously for the most part, with Cobalt preempting the Linux kernel in sections that are otherwise deemed atomic, from Linux's standpoint. The interrupt pipeline machinery makes this possible. The price to pay for this is that you may not re-enter the regular Linux kernel code from a context which might have preempted it in the middle of nowhere, so to speak. Some hints about a working approach I've used a couple of times already for sharing a DMA engine. Assuming the DMA chip on this board is driven by the generic dmaengine framework, it is possible to extend the latter with a couple of (per-chip) handlers allowing to prepare and monitor transfers from a real-time context (i.e. Cobalt primary mode). Those handlers are defined in the dma_device structure, and implemented in the DMA backend code which is DMA engine-specific. Cobalt would then use a dedicated set of DMA channels obtained via dma_request_slave_channel() as usual, but controlled exclusively from these new handlers, for running real-time SPI transfers. Typically, you would have a handler to prepare a transfer, maybe another one to wait for an ongoing transfer to finish, all from/to a real-time Cobalt context, instead of a plain Linux context. You would have to make sure that your implementation never attempts to synchronize/serialize on regular Linux spinlocks/mutexes/sema4 etc. Interfacing with the IRQ subsystem for asynchronous handling of the DMA completion may be the main issue here if you need to, because the normal linux activities would still expect to receive end-of-transfer IRQs in the "root" domain as usual (request_irq interface) in parallel to the real-time activities, which would then be incompatible with assigning the DMA IRQ to the head/Cobalt domain (rtdm_irq_request()) exclusively. If you can live with receiving end-of-transfer IRQs without rt guarantees in the Linux domain though, then you don't have to deal with this problem. -- Philippe. ___ Xenomai mailing list Xenomai@xenomai.org https://xenomai.org/mailman/listinfo/xenomai
