>>>>> "MAG" == Mark A Greer <mgreer at mvista.com> writes:
MAG> Hi Phil, MAG> [Note: I'm cc'ing linuxppc-embedded for others to reference and to MAG> add their thoughts.] OK, I've just subscribed... MAG> On Tue, Dec 20, 2005 at 10:49:58AM +1030, Phil Nitschke wrote: >> Hi Mark, >> >> I'm developing a device driver to run in the 2.6.10 kernel. I want to MAG> That's a pretty old kernel. Do you have the option of using a more MAG> recent one like 2.6.14? That might be possible if I reverse-engineer a patch file by comparing the Artesyn reference kernel (2.6.10) with the kernel.org version, then trying to apply that patch to the latest kernel. I'll try this later... >> get large amounts of data from a custom peripheral on the PCI bus. The >> software is running on an Artesyn PmPPC7448, which includes a Discovery >> III bridge. MAG> Can you share exact platform you're using? I'm using a PMC processor on a custom carrier card (not made by Avalon). Here are the respective links: Carrier: http://www.tenix.com.au/Main.asp?ID=938 Processor: http://www.artesyncp.com/products/PmPPC7448.html MAG> The bridge supports bursting on the PCI bus as long as the bridge MAG> is configured correctly and the PCI device is making an MAG> appropriate request. Note, however, that there are many errata MAG> for the Marvell parts including some with cache coherency. If MAG> your system is running with coherency on, you may have to limit MAG> your bursts to 32 bytes (i.e., the size of one cache line). MAG> You can see how the bursting is set up on the bridge by looking MAG> at the platform file for your board (e.g., MAG> <file:arch/ppc/platforms/katana.c> in the latest linux MAG> kernel)--search for 'BURST'. As far as I can tell, there is no platform file for this board in the mainstream kernel. In the reference kernel provided by Artesyn, there is a file named arch/ppc/configs/pmppc7447_defconfig, where CONFIG_NOT_COHERENT_CACHE=y Therefore in arch/ppc/platforms/pmppc7447.c, there is some code which does this: #if defined(CONFIG_NOT_COHERENT_CACHE) mv64x60_write(&bh, MV64360_SRAM_CONFIG, 0x00160000); #else mv64x60_write(&bh, MV64360_SRAM_CONFIG, 0x001600b2); #endif ... and later ... for (i = 0; i < MV64x60_CPU2MEM_WINDOWS; i++) { #if defined(CONFIG_NOT_COHERENT_CACHE) si.cpu_prot_options[i] = 0; si.enet_options[i] = MV64360_ENET2MEM_SNOOP_NONE; si.mpsc_options[i] = MV64360_MPSC2MEM_SNOOP_NONE; si.idma_options[i] = MV64360_IDMA2MEM_SNOOP_NONE; si.pci_0.acc_cntl_options[i] = MV64360_PCI_ACC_CNTL_SNOOP_NONE | MV64360_PCI_ACC_CNTL_SWAP_NONE | MV64360_PCI_ACC_CNTL_MBURST_128_BYTES | MV64360_PCI_ACC_CNTL_RDSIZE_256_BYTES; #else si.cpu_prot_options[i] = 0; si.enet_options[i] = MV64360_ENET2MEM_SNOOP_NONE; /* errata */ si.mpsc_options[i] = MV64360_MPSC2MEM_SNOOP_NONE; /* errata */ si.idma_options[i] = MV64360_IDMA2MEM_SNOOP_NONE; /* errata */ si.pci_0.acc_cntl_options[i] = MV64360_PCI_ACC_CNTL_SNOOP_WB | MV64360_PCI_ACC_CNTL_SWAP_NONE | MV64360_PCI_ACC_CNTL_MBURST_32_BYTES | MV64360_PCI_ACC_CNTL_RDSIZE_32_BYTES; #endif } But I'm yet to learn what all this means... >> Is there a summary of what is possible and/or not possible with the 4 >> IDMA channels on the mv64460? MAG> The only real documentation is the bridge's user manual from Marvell. MAG> Unfortunately, you must sign an NDA to get access to it so I can't share MAG> mine with you. You will need access to that info to get very far so I MAG> recommend you contact the people in your company that can make that MAG> happen, ASAP. I talked with a person from Marvell's only Australian distributor, who told me that they'd not be too keen to give us an NDA, since we're not developing a project specifically for the Marvell, rather we're using a Marvell which has already been integrated in the Artesyn card. Therefore, he argued, Marvell would tell me to go to Artesyn for the info, as they already have the NDA. So for now, assume no NDA, no errata. >> For example, if the device that I'm trying to get data from supported a >> DMA engine capable of initiating bursts on the PCI bus (it currently >> can't do this), does the current kernel code support that? MAG> That's a hardware feature so its not really an issue of kernel support MAG> other than ensuring that the firmware and/or kernel configures the bridge MAG> correctly. IOW, it can be supported by software but its an issue of MAG> whether your hardware supports it (and it actually works). I'm not sure here whether you're talking about the hardware in the CPU/bridge, or the hardware in the device. Since the device interfaces to the PCI bus using firmware inside an FPGA, this is configurable (to a certain extent). Currently there is a 2M aperture on the device, but it is not being seen as "prefetchable", so when I try to get data from the device using repetitive reads, they are very slow. Hence my efforts to get DMA happening. Presumably the CPU/bridge discovers PCI device memory regions during bus enumeration. What characteristic of a device determines whether the memory region is going to be marked as "prefetchable"? Does this attribute also affect whether DMA will work? >> Or if I wanted to suck the data into main memory using the mv64460 IDMA >> controller (assuming the device couldn't initiate its own burst writes), >> is there a standard kernel interface to allow me to do this? MAG> Yes. You would make a "dma ctlr driver" for the dma ctlr(s). I MAG> don't know what the best example would be but hopefully someone MAG> else has a suggestion. OK, I'll look into this. I've been using the O'Reilly book "Linux Device Drivers, Third Edition" by Jonathan Corbet, Alessandro Rubini, and Greg Kroah-Hartman. They say "The kernel developers recommend the use of streaming mappings over coherent mappings whenever possible." I'm not sure how the H/W vs S/W coherency discussion has anything to do with their assertion. I had previously thought that allocating a huge buffer (for example at boot time) would be the way to go, but perhaps getting the CPU to collect the data in smaller amounts into cache coherent memory will give me the best performance? MAG> You may want to pick up "PCI System Architecture" from Mindshare, MAG> Inc. There are ones for PCI-X and PCI-Express too, I think. MAG> Well worth the money. Sounds like a good idea. I'd hoped not to have to become a PCI expert, but it seems that there is a lot for me to learn just to determine how best to design my driver. Thanks for your input. -- Phil
