Re: SCSI breakage on non-cache coherent architectures

2007-11-20 Thread James Bottomley
On Wed, 2007-11-21 at 09:39 +1100, Benjamin Herrenschmidt wrote: > On Tue, 2007-11-20 at 15:10 -0600, James Bottomley wrote: > > We're talking about trying to fix this for 2.4; which is already at > > -rc3 ... Is an entire arch change for dma alignment really a merge > > candidate at this stage?

Re: SCSI breakage on non-cache coherent architectures

2007-11-20 Thread Benjamin Herrenschmidt
On Tue, 2007-11-20 at 15:10 -0600, James Bottomley wrote: > We're talking about trying to fix this for 2.4; which is already at > -rc3 ... Is an entire arch change for dma alignment really a merge > candidate at this stage? Well, as I said before... it's a matter of what seems to be the less

Re: SCSI breakage on non-cache coherent architectures

2007-11-20 Thread James Bottomley
On Tue, 2007-11-20 at 12:05 -0800, Roland Dreier wrote: > > Actually, we already established on IRC that the lasi700 driver doesn't > > need this, principally because the parisc architecture doesn't do an > > invalidate for DMA_FROM_DEVICE but a flush and invalidate > > (architecturally, if

Re: SCSI breakage on non-cache coherent architectures

2007-11-20 Thread Roland Dreier
> Actually, we already established on IRC that the lasi700 driver doesn't > need this, principally because the parisc architecture doesn't do an > invalidate for DMA_FROM_DEVICE but a flush and invalidate > (architecturally, if you read our manuals, even pdc is entitled to write > back dirty

Re: SCSI breakage on non-cache coherent architectures

2007-11-20 Thread James Bottomley
On Tue, 2007-11-20 at 14:14 +1100, Benjamin Herrenschmidt wrote: > FYI, Here's what I have for the SCSI change. I haven't updated drivers > to care for the new return code though, help appreciated with that as I > don't know much about these drivers. It looks to me like the return problem could

Re: SCSI breakage on non-cache coherent architectures

2007-11-20 Thread James Bottomley
On Tue, 2007-11-20 at 09:29 +0100, Thomas Bogendoerfer wrote: > On Tue, Nov 20, 2007 at 06:51:14AM +1100, Benjamin Herrenschmidt wrote: > > > > On Mon, 2007-11-19 at 00:38 -0800, David Miller wrote: > > > From: Benjamin Herrenschmidt <[EMAIL PROTECTED]> > > > Date: Mon, 19 Nov 2007 16:35:23

Re: SCSI breakage on non-cache coherent architectures

2007-11-20 Thread Thomas Bogendoerfer
On Tue, Nov 20, 2007 at 06:51:14AM +1100, Benjamin Herrenschmidt wrote: > > On Mon, 2007-11-19 at 00:38 -0800, David Miller wrote: > > From: Benjamin Herrenschmidt <[EMAIL PROTECTED]> > > Date: Mon, 19 Nov 2007 16:35:23 +1100 > > > > > I'm not sure what is the best way to fix that. Internally,

Re: SCSI breakage on non-cache coherent architectures

2007-11-20 Thread Thomas Bogendoerfer
On Tue, Nov 20, 2007 at 06:51:14AM +1100, Benjamin Herrenschmidt wrote: On Mon, 2007-11-19 at 00:38 -0800, David Miller wrote: From: Benjamin Herrenschmidt [EMAIL PROTECTED] Date: Mon, 19 Nov 2007 16:35:23 +1100 I'm not sure what is the best way to fix that. Internally, I've done

Re: SCSI breakage on non-cache coherent architectures

2007-11-20 Thread James Bottomley
On Tue, 2007-11-20 at 09:29 +0100, Thomas Bogendoerfer wrote: On Tue, Nov 20, 2007 at 06:51:14AM +1100, Benjamin Herrenschmidt wrote: On Mon, 2007-11-19 at 00:38 -0800, David Miller wrote: From: Benjamin Herrenschmidt [EMAIL PROTECTED] Date: Mon, 19 Nov 2007 16:35:23 +1100 I'm

Re: SCSI breakage on non-cache coherent architectures

2007-11-20 Thread James Bottomley
On Tue, 2007-11-20 at 14:14 +1100, Benjamin Herrenschmidt wrote: FYI, Here's what I have for the SCSI change. I haven't updated drivers to care for the new return code though, help appreciated with that as I don't know much about these drivers. It looks to me like the return problem could be

Re: SCSI breakage on non-cache coherent architectures

2007-11-20 Thread Roland Dreier
Actually, we already established on IRC that the lasi700 driver doesn't need this, principally because the parisc architecture doesn't do an invalidate for DMA_FROM_DEVICE but a flush and invalidate (architecturally, if you read our manuals, even pdc is entitled to write back dirty

Re: SCSI breakage on non-cache coherent architectures

2007-11-20 Thread James Bottomley
On Tue, 2007-11-20 at 12:05 -0800, Roland Dreier wrote: Actually, we already established on IRC that the lasi700 driver doesn't need this, principally because the parisc architecture doesn't do an invalidate for DMA_FROM_DEVICE but a flush and invalidate (architecturally, if you read

Re: SCSI breakage on non-cache coherent architectures

2007-11-20 Thread Benjamin Herrenschmidt
On Tue, 2007-11-20 at 15:10 -0600, James Bottomley wrote: We're talking about trying to fix this for 2.4; which is already at -rc3 ... Is an entire arch change for dma alignment really a merge candidate at this stage? Well, as I said before... it's a matter of what seems to be the less likely

Re: SCSI breakage on non-cache coherent architectures

2007-11-20 Thread James Bottomley
On Wed, 2007-11-21 at 09:39 +1100, Benjamin Herrenschmidt wrote: On Tue, 2007-11-20 at 15:10 -0600, James Bottomley wrote: We're talking about trying to fix this for 2.4; which is already at -rc3 ... Is an entire arch change for dma alignment really a merge candidate at this stage?

Re: SCSI breakage on non-cache coherent architectures

2007-11-19 Thread Benjamin Herrenschmidt
FYI, Here's what I have for the SCSI change. I haven't updated drivers to care for the new return code though, help appreciated with that as I don't know much about these drivers. Index: linux-work/drivers/scsi/scsi_error.c === ---

Re: SCSI breakage on non-cache coherent architectures

2007-11-19 Thread Benjamin Herrenschmidt
On Mon, 2007-11-19 at 18:10 -0800, Roland Dreier wrote: > > I wrapped this ugliness up inside the macro back in what I posted in > 2002 (http://lkml.org/lkml/2002/6/12/234): > > #define __dma_buffer __dma_buffer_line(__LINE__) > #define __dma_buffer_line(line) __dma_buffer_expand_line(line) >

Re: SCSI breakage on non-cache coherent architectures

2007-11-19 Thread Roland Dreier
> 2) Add the __dma_cacheline_aligned tag. > > But note that with #2 it could get quite ugly because the > alignment and size both have a minimum that needs to be > enforced, not just the alignment alone. So either: > > struct foo { > unsigned int other_unrelated_stuff; > >

Re: SCSI breakage on non-cache coherent architectures

2007-11-19 Thread David Miller
From: Benjamin Herrenschmidt <[EMAIL PROTECTED]> Date: Tue, 20 Nov 2007 11:55:01 +1100 > BTW. What is the status nowadays with skb's ? Good question. Some drivers are problematic (or were) because they put DMA descriptor chaining information at the head of the buffer, but those have been fixed

Re: SCSI breakage on non-cache coherent architectures

2007-11-19 Thread Benjamin Herrenschmidt
On Mon, 2007-11-19 at 16:46 -0800, David Miller wrote: > > 1) Require that entire buffers are commited by call sites, >and thus "embedding" DMA'd within non-DMA stuff isn't allowed > > 2) Add the __dma_cacheline_aligned tag. > > But note that with #2 it could get quite ugly because the >

Re: SCSI breakage on non-cache coherent architectures

2007-11-19 Thread David Miller
From: Benjamin Herrenschmidt <[EMAIL PROTECTED]> Date: Tue, 20 Nov 2007 11:34:24 +1100 > Do you still think we should introduce this __dma_cacheline_aligned ? Do > you see other cases of drivers where it would be useful ? It tend to > agree with your earlier statement that drivers doing that are

Re: SCSI breakage on non-cache coherent architectures

2007-11-19 Thread Benjamin Herrenschmidt
On Mon, 2007-11-19 at 14:31 -0800, David Miller wrote: > From: Benjamin Herrenschmidt <[EMAIL PROTECTED]> > Date: Tue, 20 Nov 2007 06:51:14 +1100 > > > On Mon, 2007-11-19 at 00:38 -0800, David Miller wrote: > > > From: Benjamin Herrenschmidt <[EMAIL PROTECTED]> > > > Date: Mon, 19 Nov 2007

Re: SCSI breakage on non-cache coherent architectures

2007-11-19 Thread David Miller
From: Benjamin Herrenschmidt <[EMAIL PROTECTED]> Date: Tue, 20 Nov 2007 06:51:14 +1100 > On Mon, 2007-11-19 at 00:38 -0800, David Miller wrote: > > From: Benjamin Herrenschmidt <[EMAIL PROTECTED]> > > Date: Mon, 19 Nov 2007 16:35:23 +1100 > > > > You could make a dma_cacheline_aligned and

Re: SCSI breakage on non-cache coherent architectures

2007-11-19 Thread Benjamin Herrenschmidt
On Mon, 2007-11-19 at 13:43 -0800, Roland Dreier wrote: > > I've been debugging various issues on the PowerPC 44x embedded > > architecture which happens to have non-coherent PCI DMA. > > > > One of the problem I'm hitting is that one really need to enforce > > kmalloc alignement to cache

Re: SCSI breakage on non-cache coherent architectures

2007-11-19 Thread Roland Dreier
> I've been debugging various issues on the PowerPC 44x embedded > architecture which happens to have non-coherent PCI DMA. > > One of the problem I'm hitting is that one really need to enforce > kmalloc alignement to cache lines or bad things will happen (among > others with USB), for some

Re: SCSI breakage on non-cache coherent architectures

2007-11-19 Thread Benjamin Herrenschmidt
> I'd like to be rid of it inside the command for various reasons: every > command has one of these, and they're expensive in the allocation (at 96 > bytes). There's no reason we have to allocate and free that amount of > space with every command. In theory, the number of these is bounded at >

Re: SCSI breakage on non-cache coherent architectures

2007-11-19 Thread Benjamin Herrenschmidt
On Mon, 2007-11-19 at 09:09 -0600, James Bottomley wrote: > > What other drivers do is DMA to their own allocation and then memcpy to > > the sense buffer. > > > > There is a movement to allocate the sense data as its own sg list, but > > I don't think that patch has even been posted yet. > >

Re: SCSI breakage on non-cache coherent architectures

2007-11-19 Thread Benjamin Herrenschmidt
On Mon, 2007-11-19 at 05:32 -0700, Matthew Wilcox wrote: > On Mon, Nov 19, 2007 at 04:35:23PM +1100, Benjamin Herrenschmidt wrote: > > The other one I'm hitting now is that the SCSI layer nowadays embeds the > > 'nowadays'? It has always been so. Wasn't it kmalloc'ed at one point ? > >

Re: SCSI breakage on non-cache coherent architectures

2007-11-19 Thread Benjamin Herrenschmidt
On Mon, 2007-11-19 at 00:38 -0800, David Miller wrote: > From: Benjamin Herrenschmidt <[EMAIL PROTECTED]> > Date: Mon, 19 Nov 2007 16:35:23 +1100 > > > I'm not sure what is the best way to fix that. Internally, I've done > > some test whacking some cacheline_aligned in the scsi_cmnd data > >

Re: SCSI breakage on non-cache coherent architectures

2007-11-19 Thread James Bottomley
On Mon, 2007-11-19 at 05:32 -0700, Matthew Wilcox wrote: > On Mon, Nov 19, 2007 at 04:35:23PM +1100, Benjamin Herrenschmidt wrote: > > The other one I'm hitting now is that the SCSI layer nowadays embeds the > > 'nowadays'? It has always been so. > > > sense_buffer inside the scsi_cmnd

Re: SCSI breakage on non-cache coherent architectures

2007-11-19 Thread Matthew Wilcox
On Mon, Nov 19, 2007 at 04:35:23PM +1100, Benjamin Herrenschmidt wrote: > The other one I'm hitting now is that the SCSI layer nowadays embeds the 'nowadays'? It has always been so. > sense_buffer inside the scsi_cmnd structure without any kind of > alignment whatsoever. I've been hitting

Re: SCSI breakage on non-cache coherent architectures

2007-11-19 Thread David Miller
From: Benjamin Herrenschmidt <[EMAIL PROTECTED]> Date: Mon, 19 Nov 2007 16:35:23 +1100 > I'm not sure what is the best way to fix that. Internally, I've done > some test whacking some cacheline_aligned in the scsi_cmnd data > structure to verify I no longer get random SLAB corruption when

Re: SCSI breakage on non-cache coherent architectures

2007-11-19 Thread David Miller
From: Benjamin Herrenschmidt [EMAIL PROTECTED] Date: Mon, 19 Nov 2007 16:35:23 +1100 I'm not sure what is the best way to fix that. Internally, I've done some test whacking some cacheline_aligned in the scsi_cmnd data structure to verify I no longer get random SLAB corruption when using my

Re: SCSI breakage on non-cache coherent architectures

2007-11-19 Thread Matthew Wilcox
On Mon, Nov 19, 2007 at 04:35:23PM +1100, Benjamin Herrenschmidt wrote: The other one I'm hitting now is that the SCSI layer nowadays embeds the 'nowadays'? It has always been so. sense_buffer inside the scsi_cmnd structure without any kind of alignment whatsoever. I've been hitting

Re: SCSI breakage on non-cache coherent architectures

2007-11-19 Thread James Bottomley
On Mon, 2007-11-19 at 05:32 -0700, Matthew Wilcox wrote: On Mon, Nov 19, 2007 at 04:35:23PM +1100, Benjamin Herrenschmidt wrote: The other one I'm hitting now is that the SCSI layer nowadays embeds the 'nowadays'? It has always been so. sense_buffer inside the scsi_cmnd structure

Re: SCSI breakage on non-cache coherent architectures

2007-11-19 Thread Benjamin Herrenschmidt
On Mon, 2007-11-19 at 00:38 -0800, David Miller wrote: From: Benjamin Herrenschmidt [EMAIL PROTECTED] Date: Mon, 19 Nov 2007 16:35:23 +1100 I'm not sure what is the best way to fix that. Internally, I've done some test whacking some cacheline_aligned in the scsi_cmnd data structure

Re: SCSI breakage on non-cache coherent architectures

2007-11-19 Thread Benjamin Herrenschmidt
On Mon, 2007-11-19 at 05:32 -0700, Matthew Wilcox wrote: On Mon, Nov 19, 2007 at 04:35:23PM +1100, Benjamin Herrenschmidt wrote: The other one I'm hitting now is that the SCSI layer nowadays embeds the 'nowadays'? It has always been so. Wasn't it kmalloc'ed at one point ? sense_buffer

Re: SCSI breakage on non-cache coherent architectures

2007-11-19 Thread Benjamin Herrenschmidt
On Mon, 2007-11-19 at 09:09 -0600, James Bottomley wrote: What other drivers do is DMA to their own allocation and then memcpy to the sense buffer. There is a movement to allocate the sense data as its own sg list, but I don't think that patch has even been posted yet. I'd like to

Re: SCSI breakage on non-cache coherent architectures

2007-11-19 Thread Benjamin Herrenschmidt
I'd like to be rid of it inside the command for various reasons: every command has one of these, and they're expensive in the allocation (at 96 bytes). There's no reason we have to allocate and free that amount of space with every command. In theory, the number of these is bounded at the

Re: SCSI breakage on non-cache coherent architectures

2007-11-19 Thread Roland Dreier
I've been debugging various issues on the PowerPC 44x embedded architecture which happens to have non-coherent PCI DMA. One of the problem I'm hitting is that one really need to enforce kmalloc alignement to cache lines or bad things will happen (among others with USB), for some

Re: SCSI breakage on non-cache coherent architectures

2007-11-19 Thread Benjamin Herrenschmidt
On Mon, 2007-11-19 at 13:43 -0800, Roland Dreier wrote: I've been debugging various issues on the PowerPC 44x embedded architecture which happens to have non-coherent PCI DMA. One of the problem I'm hitting is that one really need to enforce kmalloc alignement to cache lines or bad

Re: SCSI breakage on non-cache coherent architectures

2007-11-19 Thread David Miller
From: Benjamin Herrenschmidt [EMAIL PROTECTED] Date: Tue, 20 Nov 2007 06:51:14 +1100 On Mon, 2007-11-19 at 00:38 -0800, David Miller wrote: From: Benjamin Herrenschmidt [EMAIL PROTECTED] Date: Mon, 19 Nov 2007 16:35:23 +1100 You could make a dma_cacheline_aligned and use that. It

Re: SCSI breakage on non-cache coherent architectures

2007-11-19 Thread Benjamin Herrenschmidt
On Mon, 2007-11-19 at 14:31 -0800, David Miller wrote: From: Benjamin Herrenschmidt [EMAIL PROTECTED] Date: Tue, 20 Nov 2007 06:51:14 +1100 On Mon, 2007-11-19 at 00:38 -0800, David Miller wrote: From: Benjamin Herrenschmidt [EMAIL PROTECTED] Date: Mon, 19 Nov 2007 16:35:23 +1100

Re: SCSI breakage on non-cache coherent architectures

2007-11-19 Thread David Miller
From: Benjamin Herrenschmidt [EMAIL PROTECTED] Date: Tue, 20 Nov 2007 11:34:24 +1100 Do you still think we should introduce this __dma_cacheline_aligned ? Do you see other cases of drivers where it would be useful ? It tend to agree with your earlier statement that drivers doing that are

Re: SCSI breakage on non-cache coherent architectures

2007-11-19 Thread Benjamin Herrenschmidt
On Mon, 2007-11-19 at 16:46 -0800, David Miller wrote: 1) Require that entire buffers are commited by call sites, and thus embedding DMA'd within non-DMA stuff isn't allowed 2) Add the __dma_cacheline_aligned tag. But note that with #2 it could get quite ugly because the alignment

Re: SCSI breakage on non-cache coherent architectures

2007-11-19 Thread David Miller
From: Benjamin Herrenschmidt [EMAIL PROTECTED] Date: Tue, 20 Nov 2007 11:55:01 +1100 BTW. What is the status nowadays with skb's ? Good question. Some drivers are problematic (or were) because they put DMA descriptor chaining information at the head of the buffer, but those have been fixed

Re: SCSI breakage on non-cache coherent architectures

2007-11-19 Thread Roland Dreier
2) Add the __dma_cacheline_aligned tag. But note that with #2 it could get quite ugly because the alignment and size both have a minimum that needs to be enforced, not just the alignment alone. So either: struct foo { unsigned int other_unrelated_stuff; struct

Re: SCSI breakage on non-cache coherent architectures

2007-11-19 Thread Benjamin Herrenschmidt
On Mon, 2007-11-19 at 18:10 -0800, Roland Dreier wrote: I wrapped this ugliness up inside the macro back in what I posted in 2002 (http://lkml.org/lkml/2002/6/12/234): #define __dma_buffer __dma_buffer_line(__LINE__) #define __dma_buffer_line(line) __dma_buffer_expand_line(line) #define

Re: SCSI breakage on non-cache coherent architectures

2007-11-19 Thread Benjamin Herrenschmidt
FYI, Here's what I have for the SCSI change. I haven't updated drivers to care for the new return code though, help appreciated with that as I don't know much about these drivers. Index: linux-work/drivers/scsi/scsi_error.c === ---

SCSI breakage on non-cache coherent architectures

2007-11-18 Thread Benjamin Herrenschmidt
Hi James ! (Please CC me on replies as I'm not subscribed to linux-scsi) I've been debugging various issues on the PowerPC 44x embedded architecture which happens to have non-coherent PCI DMA. One of the problem I'm hitting is that one really need to enforce kmalloc alignement to cache lines or

SCSI breakage on non-cache coherent architectures

2007-11-18 Thread Benjamin Herrenschmidt
Hi James ! (Please CC me on replies as I'm not subscribed to linux-scsi) I've been debugging various issues on the PowerPC 44x embedded architecture which happens to have non-coherent PCI DMA. One of the problem I'm hitting is that one really need to enforce kmalloc alignement to cache lines or