Re: [PATCH rdma-next 00/10] Enable relaxed ordering for ULPs

2021-04-14 Thread Jason Gunthorpe
On Wed, Apr 14, 2021 at 02:41:52PM +, David Laight wrote: > So whatever driver initialises the target needs to configure whatever > target-specific register enables the RO transfers themselves. RDMA in general, and mlx5 in particular, is a layered design: mlx5_core <- owns the PCI function,

Re: [PATCH rdma-next 00/10] Enable relaxed ordering for ULPs

2021-04-14 Thread Jason Gunthorpe
On Wed, Apr 14, 2021 at 10:16:28AM -0400, Tom Talpey wrote: > On 4/12/2021 6:48 PM, Jason Gunthorpe wrote: > > On Mon, Apr 12, 2021 at 04:20:47PM -0400, Tom Talpey wrote: > > > > > So the issue is only in testing all the providers and platforms, > > > to be sure this new behavior isn't tickling

RE: [PATCH rdma-next 00/10] Enable relaxed ordering for ULPs

2021-04-14 Thread David Laight
From: Tom Talpey > Sent: 14 April 2021 15:16 > > On 4/12/2021 6:48 PM, Jason Gunthorpe wrote: > > On Mon, Apr 12, 2021 at 04:20:47PM -0400, Tom Talpey wrote: > > > >> So the issue is only in testing all the providers and platforms, > >> to be sure this new behavior isn't tickling anything that

Re: [PATCH rdma-next 00/10] Enable relaxed ordering for ULPs

2021-04-14 Thread Tom Talpey
On 4/12/2021 6:48 PM, Jason Gunthorpe wrote: On Mon, Apr 12, 2021 at 04:20:47PM -0400, Tom Talpey wrote: So the issue is only in testing all the providers and platforms, to be sure this new behavior isn't tickling anything that went unnoticed all along, because no RDMA provider ever issued RO.

Re: [PATCH rdma-next 00/10] Enable relaxed ordering for ULPs

2021-04-12 Thread Jason Gunthorpe
On Mon, Apr 12, 2021 at 04:20:47PM -0400, Tom Talpey wrote: > So the issue is only in testing all the providers and platforms, > to be sure this new behavior isn't tickling anything that went > unnoticed all along, because no RDMA provider ever issued RO. The mlx5 ethernet driver has run in RO

Re: [PATCH rdma-next 00/10] Enable relaxed ordering for ULPs

2021-04-12 Thread Tom Talpey
On 4/12/2021 2:32 PM, Haakon Bugge wrote: On 10 Apr 2021, at 15:30, David Laight wrote: From: Tom Talpey Sent: 09 April 2021 18:49 On 4/9/2021 12:27 PM, Haakon Bugge wrote: On 9 Apr 2021, at 17:32, Tom Talpey wrote: On 4/9/2021 10:45 AM, Chuck Lever III wrote: On Apr 9, 2021, at

Re: [PATCH rdma-next 00/10] Enable relaxed ordering for ULPs

2021-04-12 Thread Haakon Bugge
> On 10 Apr 2021, at 15:30, David Laight wrote: > > From: Tom Talpey >> Sent: 09 April 2021 18:49 >> On 4/9/2021 12:27 PM, Haakon Bugge wrote: >>> >>> On 9 Apr 2021, at 17:32, Tom Talpey wrote: On 4/9/2021 10:45 AM, Chuck Lever III wrote: >> On Apr 9, 2021, at 10:26 AM,

Re: [PATCH rdma-next 00/10] Enable relaxed ordering for ULPs

2021-04-11 Thread Max Gurtovoy
On 4/6/2021 2:53 PM, Jason Gunthorpe wrote: On Tue, Apr 06, 2021 at 08:09:43AM +0300, Leon Romanovsky wrote: On Tue, Apr 06, 2021 at 10:37:38AM +0800, Honggang LI wrote: On Mon, Apr 05, 2021 at 08:23:54AM +0300, Leon Romanovsky wrote: From: Leon Romanovsky From Avihai, Relaxed Ordering

RE: [PATCH rdma-next 00/10] Enable relaxed ordering for ULPs

2021-04-10 Thread David Laight
From: Tom Talpey > Sent: 09 April 2021 18:49 > On 4/9/2021 12:27 PM, Haakon Bugge wrote: > > > > > >> On 9 Apr 2021, at 17:32, Tom Talpey wrote: > >> > >> On 4/9/2021 10:45 AM, Chuck Lever III wrote: > On Apr 9, 2021, at 10:26 AM, Tom Talpey wrote: > > On 4/6/2021 7:49 AM, Jason

Re: [PATCH rdma-next 00/10] Enable relaxed ordering for ULPs

2021-04-09 Thread Tom Talpey
On 4/9/2021 12:27 PM, Haakon Bugge wrote: On 9 Apr 2021, at 17:32, Tom Talpey wrote: On 4/9/2021 10:45 AM, Chuck Lever III wrote: On Apr 9, 2021, at 10:26 AM, Tom Talpey wrote: On 4/6/2021 7:49 AM, Jason Gunthorpe wrote: On Mon, Apr 05, 2021 at 11:42:31PM +, Chuck Lever III wrote:

Re: [PATCH rdma-next 00/10] Enable relaxed ordering for ULPs

2021-04-09 Thread Tom Talpey
On 4/9/2021 12:40 PM, Jason Gunthorpe wrote: On Fri, Apr 09, 2021 at 10:26:21AM -0400, Tom Talpey wrote: My belief is that the biggest risk is from situations where completions are batched, and therefore polling is used to detect them without interrupts (which explicitly). We don't do this

Re: [PATCH rdma-next 00/10] Enable relaxed ordering for ULPs

2021-04-09 Thread Jason Gunthorpe
On Fri, Apr 09, 2021 at 10:26:21AM -0400, Tom Talpey wrote: > My belief is that the biggest risk is from situations where completions > are batched, and therefore polling is used to detect them without > interrupts (which explicitly). We don't do this in the kernel. All kernel ULPs only read

Re: [PATCH rdma-next 00/10] Enable relaxed ordering for ULPs

2021-04-09 Thread Haakon Bugge
> On 9 Apr 2021, at 17:32, Tom Talpey wrote: > > On 4/9/2021 10:45 AM, Chuck Lever III wrote: >>> On Apr 9, 2021, at 10:26 AM, Tom Talpey wrote: >>> >>> On 4/6/2021 7:49 AM, Jason Gunthorpe wrote: On Mon, Apr 05, 2021 at 11:42:31PM +, Chuck Lever III wrote: > We need to

Re: [PATCH rdma-next 00/10] Enable relaxed ordering for ULPs

2021-04-09 Thread Tom Talpey
On 4/9/2021 10:45 AM, Chuck Lever III wrote: On Apr 9, 2021, at 10:26 AM, Tom Talpey wrote: On 4/6/2021 7:49 AM, Jason Gunthorpe wrote: On Mon, Apr 05, 2021 at 11:42:31PM +, Chuck Lever III wrote: We need to get a better idea what correctness testing has been done, and whether

Re: [PATCH rdma-next 00/10] Enable relaxed ordering for ULPs

2021-04-09 Thread Chuck Lever III
> On Apr 9, 2021, at 10:26 AM, Tom Talpey wrote: > > On 4/6/2021 7:49 AM, Jason Gunthorpe wrote: >> On Mon, Apr 05, 2021 at 11:42:31PM +, Chuck Lever III wrote: >> >>> We need to get a better idea what correctness testing has been done, >>> and whether positive correctness testing

Re: [PATCH rdma-next 00/10] Enable relaxed ordering for ULPs

2021-04-09 Thread Tom Talpey
On 4/6/2021 7:49 AM, Jason Gunthorpe wrote: On Mon, Apr 05, 2021 at 11:42:31PM +, Chuck Lever III wrote: We need to get a better idea what correctness testing has been done, and whether positive correctness testing results can be replicated on a variety of platforms. RO has been

Re: [PATCH rdma-next 00/10] Enable relaxed ordering for ULPs

2021-04-06 Thread Jason Gunthorpe
On Tue, Apr 06, 2021 at 08:09:43AM +0300, Leon Romanovsky wrote: > On Tue, Apr 06, 2021 at 10:37:38AM +0800, Honggang LI wrote: > > On Mon, Apr 05, 2021 at 08:23:54AM +0300, Leon Romanovsky wrote: > > > From: Leon Romanovsky > > > > > > From Avihai, > > > > > > Relaxed Ordering is a PCIe

Re: [PATCH rdma-next 00/10] Enable relaxed ordering for ULPs

2021-04-06 Thread Jason Gunthorpe
On Mon, Apr 05, 2021 at 11:42:31PM +, Chuck Lever III wrote: > We need to get a better idea what correctness testing has been done, > and whether positive correctness testing results can be replicated > on a variety of platforms. RO has been rolling out slowly on mlx5 over a few years and

Re: [PATCH rdma-next 00/10] Enable relaxed ordering for ULPs

2021-04-05 Thread Leon Romanovsky
On Mon, Apr 05, 2021 at 11:42:31PM +, Chuck Lever III wrote: > > > > On Apr 5, 2021, at 4:07 PM, Jason Gunthorpe wrote: > > > > On Mon, Apr 05, 2021 at 03:41:15PM +0200, Christoph Hellwig wrote: > >> On Mon, Apr 05, 2021 at 08:23:54AM +0300, Leon Romanovsky wrote: > >>> From: Leon

Re: [PATCH rdma-next 00/10] Enable relaxed ordering for ULPs

2021-04-05 Thread Leon Romanovsky
On Tue, Apr 06, 2021 at 10:37:38AM +0800, Honggang LI wrote: > On Mon, Apr 05, 2021 at 08:23:54AM +0300, Leon Romanovsky wrote: > > From: Leon Romanovsky > > > > From Avihai, > > > > Relaxed Ordering is a PCIe mechanism that relaxes the strict ordering > > imposed on PCI transactions, and thus,

Re: [PATCH rdma-next 00/10] Enable relaxed ordering for ULPs

2021-04-05 Thread Honggang LI
On Mon, Apr 05, 2021 at 08:23:54AM +0300, Leon Romanovsky wrote: > From: Leon Romanovsky > > From Avihai, > > Relaxed Ordering is a PCIe mechanism that relaxes the strict ordering > imposed on PCI transactions, and thus, can improve performance. > > Until now, relaxed ordering could be set

Re: [PATCH rdma-next 00/10] Enable relaxed ordering for ULPs

2021-04-05 Thread Keith Busch
On Mon, Apr 05, 2021 at 11:42:31PM +, Chuck Lever III wrote: > > On Apr 5, 2021, at 4:07 PM, Jason Gunthorpe wrote: > > On Mon, Apr 05, 2021 at 03:41:15PM +0200, Christoph Hellwig wrote: > >> On Mon, Apr 05, 2021 at 08:23:54AM +0300, Leon Romanovsky wrote: > >>> From: Leon Romanovsky > >>>

Re: [PATCH rdma-next 00/10] Enable relaxed ordering for ULPs

2021-04-05 Thread Chuck Lever III
> On Apr 5, 2021, at 4:07 PM, Jason Gunthorpe wrote: > > On Mon, Apr 05, 2021 at 03:41:15PM +0200, Christoph Hellwig wrote: >> On Mon, Apr 05, 2021 at 08:23:54AM +0300, Leon Romanovsky wrote: >>> From: Leon Romanovsky >>> From Avihai, >>> >>> Relaxed Ordering is a PCIe mechanism that

Re: [PATCH rdma-next 00/10] Enable relaxed ordering for ULPs

2021-04-05 Thread Jason Gunthorpe
On Mon, Apr 05, 2021 at 03:41:15PM +0200, Christoph Hellwig wrote: > On Mon, Apr 05, 2021 at 08:23:54AM +0300, Leon Romanovsky wrote: > > From: Leon Romanovsky > > > > >From Avihai, > > > > Relaxed Ordering is a PCIe mechanism that relaxes the strict ordering > > imposed on PCI transactions,

Re: [PATCH rdma-next 00/10] Enable relaxed ordering for ULPs

2021-04-05 Thread Tom Talpey
On 4/5/2021 10:08 AM, Leon Romanovsky wrote: On Mon, Apr 05, 2021 at 03:41:15PM +0200, Christoph Hellwig wrote: On Mon, Apr 05, 2021 at 08:23:54AM +0300, Leon Romanovsky wrote: From: Leon Romanovsky >From Avihai, Relaxed Ordering is a PCIe mechanism that relaxes the strict ordering imposed

Re: [PATCH rdma-next 00/10] Enable relaxed ordering for ULPs

2021-04-05 Thread Santosh Shilimkar
> On Apr 5, 2021, at 7:08 AM, Leon Romanovsky wrote: > > On Mon, Apr 05, 2021 at 03:41:15PM +0200, Christoph Hellwig wrote: >> On Mon, Apr 05, 2021 at 08:23:54AM +0300, Leon Romanovsky wrote: >>> From: Leon Romanovsky >>> From Avihai, >>> >>> Relaxed Ordering is a PCIe mechanism that

Re: [PATCH rdma-next 00/10] Enable relaxed ordering for ULPs

2021-04-05 Thread Leon Romanovsky
On Mon, Apr 05, 2021 at 03:41:15PM +0200, Christoph Hellwig wrote: > On Mon, Apr 05, 2021 at 08:23:54AM +0300, Leon Romanovsky wrote: > > From: Leon Romanovsky > > > > >From Avihai, > > > > Relaxed Ordering is a PCIe mechanism that relaxes the strict ordering > > imposed on PCI transactions,

Re: [PATCH rdma-next 00/10] Enable relaxed ordering for ULPs

2021-04-05 Thread Christoph Hellwig
On Mon, Apr 05, 2021 at 08:23:54AM +0300, Leon Romanovsky wrote: > From: Leon Romanovsky > > >From Avihai, > > Relaxed Ordering is a PCIe mechanism that relaxes the strict ordering > imposed on PCI transactions, and thus, can improve performance. > > Until now, relaxed ordering could be set