though i haven't dealt with 8260 specific cache consistency protocol and all , but feel using pci_alloc_consistent may be cause. also snooping should be clearly understood whether it is presenting a consistent view between cpu and devices or diffeent cpu in a smp setup.
David Ashley wrote: > > I'm running into repeated problems related to cache problems on > ppc 8260 based hardware. The kernel is based on Montevista's 2.4.2. > The same problem keeps coming up again and again. Here it is: > > If some external device, such as the pci usb device, writes into system > ram, the 8260 doesn't get the changes--instead when the 8260 reads the > ram it uses the data in the cache, defeating the whole purpose. We're > using the usb-ohci.c driver. In order to get it to work I've had to hack > the driver to instead of using pci_alloc_consistent, allocate memory > out of a section of local bus ram, which is non-cacheable. The burden > of doing this for every driver is just too great. > > The original driver itself assumes that the dma works properly, meaning it > does nothing to invalidate the cache or cause a refresh/flush/update or > whatever. The driver initiates a transfer, the usb writes into ram, an > interrupt occurs, the driver examines the buffer--and crashes and generally > doesn't work because the data it sees isn't in ram, it is what was in the > cache. > > My belief is there is something misconfigured on the hardware level. Meaning > the cache should work properly. In studying the 8260 docs I read hints at > snooping, but no details. My understanding is that snooping is when the > 8260 watches the bus to see what activity there is, and if there is a write > to something the 8260 holds in a cache, the cache is updated accordingly. > Perhaps this snooping is turned off? > > Also there is talk of C + R bits in the page tables (C = changed, R = > referenced). Possibly is the external dma mechanism supposed to update the > C + R bits in the page tables itself, causing the 8260 to realize something > has been done to the page and so it needs to be updated? > > The driver doesn't ask for any special non-cacheable memory, just dma > capable. Following this through in the code that means in the case of > the ppc, all of dram because there is no limit on the dma range. On the > i386 tree dma appears limited to the first 16 megabytes. But nothing appears > to be demanding non-cacheable memory. > > I guess I have several questions: > 1) How is the system supposed to work so that the cpu and other pci devices > have the same concept of the contents of memory? > 2) What is missing in our system that is causing this not to work? > 3) What is snooping, and can it be turned on or off by the 8260, or is it > fixed in the hardware? > 4) Am I misunderstanding how the linux drivers where, such that when they > allocate memory for dma transfers, it is automatically marked as > non-cacheable? > 5) While I'm at it, I'm getting lock ups related to the external pci device > writing into dram. I've heard mention of cache deadlocks, but I don't know > details. Any pointers to documentation will be helpful. > > Thanks very much for any ideas/advice. > > -Dave > dash at xdr.com > ** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/
