Re: mellanox mlx4_core and SR-IOV

2012-08-10 Thread Lukas Hejtmanek
On Fri, Aug 10, 2012 at 12:51:53PM -0600, Chris Friesen wrote: > On 08/03/2012 02:33 AM, Lukas Hejtmanek wrote: > >I also tried OFED package from Mellanox which seems to have better SR-IOV > >support (at least mlx4_ib does not complain that SR-IOV is not supported). > >However, it does not work

Re: mellanox mlx4_core and SR-IOV

2012-08-10 Thread Chris Friesen
On 08/03/2012 02:33 AM, Lukas Hejtmanek wrote: I also tried OFED package from Mellanox which seems to have better SR-IOV support (at least mlx4_ib does not complain that SR-IOV is not supported). However, it does not work when SR-IOV enabled: Last I heard they were not officially providing

Re: mellanox mlx4_core and SR-IOV

2012-08-10 Thread Chris Friesen
On 08/03/2012 02:33 AM, Lukas Hejtmanek wrote: I also tried OFED package from Mellanox which seems to have better SR-IOV support (at least mlx4_ib does not complain that SR-IOV is not supported). However, it does not work when SR-IOV enabled: Last I heard they were not officially providing

Re: mellanox mlx4_core and SR-IOV

2012-08-10 Thread Lukas Hejtmanek
On Fri, Aug 10, 2012 at 12:51:53PM -0600, Chris Friesen wrote: On 08/03/2012 02:33 AM, Lukas Hejtmanek wrote: I also tried OFED package from Mellanox which seems to have better SR-IOV support (at least mlx4_ib does not complain that SR-IOV is not supported). However, it does not work when

Re: mellanox mlx4_core and SR-IOV

2012-08-06 Thread Lukas Hejtmanek
On Mon, Aug 06, 2012 at 10:07:06AM -0400, Konrad Rzeszutek Wilk wrote: > > good catch. I forgot to pass swiotl=force for DomU in Xen. So now, it seems > > that mlx4_core works, mlx4_en (ethernet part) works as well. Unfortunately, > > the IB part does not. IB layer complains that SR-IOV is

Re: mellanox mlx4_core and SR-IOV

2012-08-06 Thread Konrad Rzeszutek Wilk
On Sun, Aug 05, 2012 at 10:05:00AM +0200, Lukas Hejtmanek wrote: > Hi, > > On Fri, Aug 03, 2012 at 06:49:59AM -0700, Konrad Wilk wrote: > > This looks like you are using PV PCI passthrough? If so, did you > > remember to use 'iommu=soft' to enable the Xen-SWIOTLB in your guest? > > And are you

Re: mellanox mlx4_core and SR-IOV

2012-08-06 Thread Konrad Rzeszutek Wilk
On Sun, Aug 05, 2012 at 10:05:00AM +0200, Lukas Hejtmanek wrote: Hi, On Fri, Aug 03, 2012 at 06:49:59AM -0700, Konrad Wilk wrote: This looks like you are using PV PCI passthrough? If so, did you remember to use 'iommu=soft' to enable the Xen-SWIOTLB in your guest? And are you booting

Re: mellanox mlx4_core and SR-IOV

2012-08-06 Thread Lukas Hejtmanek
On Mon, Aug 06, 2012 at 10:07:06AM -0400, Konrad Rzeszutek Wilk wrote: good catch. I forgot to pass swiotl=force for DomU in Xen. So now, it seems that mlx4_core works, mlx4_en (ethernet part) works as well. Unfortunately, the IB part does not. IB layer complains that SR-IOV is currently

Re: mellanox mlx4_core and SR-IOV

2012-08-05 Thread Lukas Hejtmanek
Hi, On Fri, Aug 03, 2012 at 06:49:59AM -0700, Konrad Wilk wrote: > This looks like you are using PV PCI passthrough? If so, did you > remember to use 'iommu=soft' to enable the Xen-SWIOTLB in your guest? > And are you booting with more than 4GB? Or is less than 3GB (so that you have > a nice gap

Re: mellanox mlx4_core and SR-IOV

2012-08-05 Thread Lukas Hejtmanek
Hi, On Fri, Aug 03, 2012 at 06:49:59AM -0700, Konrad Wilk wrote: This looks like you are using PV PCI passthrough? If so, did you remember to use 'iommu=soft' to enable the Xen-SWIOTLB in your guest? And are you booting with more than 4GB? Or is less than 3GB (so that you have a nice gap in

Re: mellanox mlx4_core and SR-IOV

2012-08-03 Thread Yinghai Lu
On Fri, Aug 3, 2012 at 1:33 AM, Lukas Hejtmanek wrote: > On Wed, Aug 01, 2012 at 04:36:14PM -0700, Yinghai Lu wrote: >> > so it seems, that pic=nocsr is a must now. >> >> yes. Or you have bios provide SRIOV support or 64 bit resource in _CRS. > > Well, I can use PCI passthrough in Xen now,

Re: mellanox mlx4_core and SR-IOV

2012-08-03 Thread Konrad Wilk
). - Original Message - From: xhejt...@ics.muni.cz To: ying...@kernel.org Cc: rol...@kernel.org, linux-kernel@vger.kernel.org, linux-...@vger.kernel.org Sent: Friday, August 3, 2012 4:34:03 AM GMT -05:00 US/Canada Eastern Subject: Re: mellanox mlx4_core and SR-IOV On Wed, Aug 01, 2012 at 04:36:14PM -0700

Re: mellanox mlx4_core and SR-IOV

2012-08-03 Thread Lukas Hejtmanek
On Wed, Aug 01, 2012 at 04:36:14PM -0700, Yinghai Lu wrote: > > so it seems, that pic=nocsr is a must now. > > yes. Or you have bios provide SRIOV support or 64 bit resource in _CRS. Well, I can use PCI passthrough in Xen now, however, it seems SR-IOV does not work in case of Mellanox mlx4

Re: mellanox mlx4_core and SR-IOV

2012-08-03 Thread Lukas Hejtmanek
On Wed, Aug 01, 2012 at 04:36:14PM -0700, Yinghai Lu wrote: so it seems, that pic=nocsr is a must now. yes. Or you have bios provide SRIOV support or 64 bit resource in _CRS. Well, I can use PCI passthrough in Xen now, however, it seems SR-IOV does not work in case of Mellanox mlx4 driver.

Re: mellanox mlx4_core and SR-IOV

2012-08-03 Thread Konrad Wilk
). - Original Message - From: xhejt...@ics.muni.cz To: ying...@kernel.org Cc: rol...@kernel.org, linux-kernel@vger.kernel.org, linux-...@vger.kernel.org Sent: Friday, August 3, 2012 4:34:03 AM GMT -05:00 US/Canada Eastern Subject: Re: mellanox mlx4_core and SR-IOV On Wed, Aug 01, 2012 at 04:36:14PM -0700

Re: mellanox mlx4_core and SR-IOV

2012-08-03 Thread Yinghai Lu
On Fri, Aug 3, 2012 at 1:33 AM, Lukas Hejtmanek xhejt...@ics.muni.cz wrote: On Wed, Aug 01, 2012 at 04:36:14PM -0700, Yinghai Lu wrote: so it seems, that pic=nocsr is a must now. yes. Or you have bios provide SRIOV support or 64 bit resource in _CRS. Well, I can use PCI passthrough in Xen

Re: mellanox mlx4_core and SR-IOV

2012-08-01 Thread Yinghai Lu
On Wed, Aug 1, 2012 at 3:08 PM, Lukas Hejtmanek wrote: > On Wed, Aug 01, 2012 at 02:32:17PM -0700, Yinghai Lu wrote: >> yes, i knew that. >> >> one patch in my for-pci-next should address that. >> >>

Re: mellanox mlx4_core and SR-IOV

2012-08-01 Thread Lukas Hejtmanek
On Wed, Aug 01, 2012 at 02:32:17PM -0700, Yinghai Lu wrote: > yes, i knew that. > > one patch in my for-pci-next should address that. > > http://git.kernel.org/?p=linux/kernel/git/yinghai/linux-yinghai.git;a=patch;h=fcce563f868e296f46a2eeaa88d6959bcee26a2d this is probably only half-way. well

Re: mellanox mlx4_core and SR-IOV

2012-08-01 Thread Lukas Hejtmanek
On Wed, Aug 01, 2012 at 02:27:34PM -0700, Yinghai Lu wrote: > you may try to boot with pci=nocrs ok, pci=nocrs i got: 02:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0) 02:00.1 InfiniBand: Mellanox Technologies MT25400 Family [ConnectX-2

Re: mellanox mlx4_core and SR-IOV

2012-08-01 Thread Yinghai Lu
On Wed, Aug 1, 2012 at 1:56 PM, Lukas Hejtmanek wrote: > On Wed, Aug 01, 2012 at 11:29:02AM -0700, Yinghai Lu wrote: >> iov bar is not assigned by BIOS, and kernel can not find range for it too. >> >> Lukas, can you post whole boot log with PCI_DEBUG enabled? That will >> tell exact why kernel

Re: mellanox mlx4_core and SR-IOV

2012-08-01 Thread Yinghai Lu
On Wed, Aug 1, 2012 at 1:28 PM, Lukas Hejtmanek wrote: > On Wed, Aug 01, 2012 at 11:29:02AM -0700, Yinghai Lu wrote: >> On Wed, Aug 1, 2012 at 10:37 AM, Roland Dreier wrote: >> > On Wed, Aug 1, 2012 at 6:38 AM, Lukas Hejtmanek >> > wrote: >> >> [3.558296] mlx4_core :02:00.0: not enough

Re: mellanox mlx4_core and SR-IOV

2012-08-01 Thread Lukas Hejtmanek
On Wed, Aug 01, 2012 at 11:29:02AM -0700, Yinghai Lu wrote: > iov bar is not assigned by BIOS, and kernel can not find range for it too. > > Lukas, can you post whole boot log with PCI_DEBUG enabled? That will > tell exact why kernel does not assign them. > > Recent kernel from 3.4... should

Re: mellanox mlx4_core and SR-IOV

2012-08-01 Thread Lukas Hejtmanek
On Wed, Aug 01, 2012 at 11:29:02AM -0700, Yinghai Lu wrote: > On Wed, Aug 1, 2012 at 10:37 AM, Roland Dreier wrote: > > On Wed, Aug 1, 2012 at 6:38 AM, Lukas Hejtmanek > > wrote: > >> [3.558296] mlx4_core :02:00.0: not enough MMIO resources for > >> SR-IOV (nres: 0, iov->nres: 1) > > >

Re: mellanox mlx4_core and SR-IOV

2012-08-01 Thread Yinghai Lu
On Wed, Aug 1, 2012 at 10:37 AM, Roland Dreier wrote: > On Wed, Aug 1, 2012 at 6:38 AM, Lukas Hejtmanek wrote: >> [3.558296] mlx4_core :02:00.0: not enough MMIO resources for SR-IOV >> (nres: 0, iov->nres: 1) > > This comes from the core sriov_enable() function, not anything in mlx4. >

Re: mellanox mlx4_core and SR-IOV

2012-08-01 Thread Roland Dreier
On Wed, Aug 1, 2012 at 6:38 AM, Lukas Hejtmanek wrote: > [3.558296] mlx4_core :02:00.0: not enough MMIO resources for SR-IOV > (nres: 0, iov->nres: 1) This comes from the core sriov_enable() function, not anything in mlx4. (although my kernel doesn't have the print of nres in that

mellanox mlx4_core and SR-IOV

2012-08-01 Thread Lukas Hejtmanek
Hello, I tried to use SR-IOV virtualizaton for Mellanox ConnectX2 card with mlx4_core driver with kernel 3.5.0. I built firware for the IB card with sriov_en = true, lspci shows: 02:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0)

mellanox mlx4_core and SR-IOV

2012-08-01 Thread Lukas Hejtmanek
Hello, I tried to use SR-IOV virtualizaton for Mellanox ConnectX2 card with mlx4_core driver with kernel 3.5.0. I built firware for the IB card with sriov_en = true, lspci shows: 02:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0)

Re: mellanox mlx4_core and SR-IOV

2012-08-01 Thread Roland Dreier
On Wed, Aug 1, 2012 at 6:38 AM, Lukas Hejtmanek xhejt...@ics.muni.cz wrote: [3.558296] mlx4_core :02:00.0: not enough MMIO resources for SR-IOV (nres: 0, iov-nres: 1) This comes from the core sriov_enable() function, not anything in mlx4. (although my kernel doesn't have the print of

Re: mellanox mlx4_core and SR-IOV

2012-08-01 Thread Yinghai Lu
On Wed, Aug 1, 2012 at 10:37 AM, Roland Dreier rol...@kernel.org wrote: On Wed, Aug 1, 2012 at 6:38 AM, Lukas Hejtmanek xhejt...@ics.muni.cz wrote: [3.558296] mlx4_core :02:00.0: not enough MMIO resources for SR-IOV (nres: 0, iov-nres: 1) This comes from the core sriov_enable()

Re: mellanox mlx4_core and SR-IOV

2012-08-01 Thread Lukas Hejtmanek
On Wed, Aug 01, 2012 at 11:29:02AM -0700, Yinghai Lu wrote: On Wed, Aug 1, 2012 at 10:37 AM, Roland Dreier rol...@kernel.org wrote: On Wed, Aug 1, 2012 at 6:38 AM, Lukas Hejtmanek xhejt...@ics.muni.cz wrote: [3.558296] mlx4_core :02:00.0: not enough MMIO resources for SR-IOV

Re: mellanox mlx4_core and SR-IOV

2012-08-01 Thread Lukas Hejtmanek
On Wed, Aug 01, 2012 at 11:29:02AM -0700, Yinghai Lu wrote: iov bar is not assigned by BIOS, and kernel can not find range for it too. Lukas, can you post whole boot log with PCI_DEBUG enabled? That will tell exact why kernel does not assign them. Recent kernel from 3.4... should enable

Re: mellanox mlx4_core and SR-IOV

2012-08-01 Thread Yinghai Lu
On Wed, Aug 1, 2012 at 1:28 PM, Lukas Hejtmanek xhejt...@ics.muni.cz wrote: On Wed, Aug 01, 2012 at 11:29:02AM -0700, Yinghai Lu wrote: On Wed, Aug 1, 2012 at 10:37 AM, Roland Dreier rol...@kernel.org wrote: On Wed, Aug 1, 2012 at 6:38 AM, Lukas Hejtmanek xhejt...@ics.muni.cz wrote: [

Re: mellanox mlx4_core and SR-IOV

2012-08-01 Thread Yinghai Lu
On Wed, Aug 1, 2012 at 1:56 PM, Lukas Hejtmanek xhejt...@ics.muni.cz wrote: On Wed, Aug 01, 2012 at 11:29:02AM -0700, Yinghai Lu wrote: iov bar is not assigned by BIOS, and kernel can not find range for it too. Lukas, can you post whole boot log with PCI_DEBUG enabled? That will tell exact

Re: mellanox mlx4_core and SR-IOV

2012-08-01 Thread Lukas Hejtmanek
On Wed, Aug 01, 2012 at 02:27:34PM -0700, Yinghai Lu wrote: you may try to boot with pci=nocrs ok, pci=nocrs i got: 02:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0) 02:00.1 InfiniBand: Mellanox Technologies MT25400 Family [ConnectX-2

Re: mellanox mlx4_core and SR-IOV

2012-08-01 Thread Lukas Hejtmanek
On Wed, Aug 01, 2012 at 02:32:17PM -0700, Yinghai Lu wrote: yes, i knew that. one patch in my for-pci-next should address that. http://git.kernel.org/?p=linux/kernel/git/yinghai/linux-yinghai.git;a=patch;h=fcce563f868e296f46a2eeaa88d6959bcee26a2d this is probably only half-way. well mlx

Re: mellanox mlx4_core and SR-IOV

2012-08-01 Thread Yinghai Lu
On Wed, Aug 1, 2012 at 3:08 PM, Lukas Hejtmanek xhejt...@ics.muni.cz wrote: On Wed, Aug 01, 2012 at 02:32:17PM -0700, Yinghai Lu wrote: yes, i knew that. one patch in my for-pci-next should address that.