RE: [PATCH v3 net 2/2] net: ixgbe: Use new PCI_DEV_FLAGS_NO_RELAXED_ORDERING flag

[Tantilov, Emil S]

>-Original Message-
>From: linux-kernel-ow...@vger.kernel.org [mailto:linux-kernel-
>ow...@vger.kernel.org] On Behalf Of Ding Tianhong
>Sent: Thursday, August 17, 2017 11:21 PM
>To: da...@davemloft.net; Kirsher, Jeffrey T ;
>keesc...@chromium.org; linux-kernel@vger.kernel.org;
>sparcli...@vger.kernel.org; intel-wired-...@lists.osuosl.org;
>alexander.du...@gmail.com; net...@vger.kernel.org; linux...@huawei.com
>Cc: Ding Tianhong 
>Subject: [PATCH v3 net 2/2] net: ixgbe: Use new
>PCI_DEV_FLAGS_NO_RELAXED_ORDERING flag
>
>The ixgbe driver use the compile check to determine if it can
>send TLPs to Root Port with the Relaxed Ordering Attribute set,
>this is too inconvenient, now the new flag
>PCI_DEV_FLAGS_NO_RELAXED_ORDERING
>has been added to the kernel and we could check the bit4 in the PCIe
>Device Control register to determine whether we should use the Relaxed
>Ordering Attributes or not, so use this new way in the ixgbe driver.
>
>Signed-off-by: Ding Tianhong 
>---
> drivers/net/ethernet/intel/ixgbe/ixgbe_82598.c  | 22 -
>-
> drivers/net/ethernet/intel/ixgbe/ixgbe_common.c | 19 ---
> 2 files changed, 41 deletions(-)

This change looks good to me for ixgbe.

Acked-by: Emil Tantilov 

Thanks,
Emil 
 
>
>diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_82598.c
>b/drivers/net/ethernet/intel/ixgbe/ixgbe_82598.c
>index 523f9d0..8a32eb7 100644
>--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_82598.c
>+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_82598.c
>@@ -175,31 +175,9 @@ static s32 ixgbe_init_phy_ops_82598(struct ixgbe_hw
>*hw)
>  **/
> static s32 ixgbe_start_hw_82598(struct ixgbe_hw *hw)
> {
>-#ifndef CONFIG_SPARC
>-  u32 regval;
>-  u32 i;
>-#endif
>   s32 ret_val;
>
>   ret_val = ixgbe_start_hw_generic(hw);
>-
>-#ifndef CONFIG_SPARC
>-  /* Disable relaxed ordering */
>-  for (i = 0; ((i < hw->mac.max_tx_queues) &&
>-   (i < IXGBE_DCA_MAX_QUEUES_82598)); i++) {
>-  regval = IXGBE_READ_REG(hw, IXGBE_DCA_TXCTRL(i));
>-  regval &= ~IXGBE_DCA_TXCTRL_DESC_WRO_EN;
>-  IXGBE_WRITE_REG(hw, IXGBE_DCA_TXCTRL(i), regval);
>-  }
>-
>-  for (i = 0; ((i < hw->mac.max_rx_queues) &&
>-   (i < IXGBE_DCA_MAX_QUEUES_82598)); i++) {
>-  regval = IXGBE_READ_REG(hw, IXGBE_DCA_RXCTRL(i));
>-  regval &= ~(IXGBE_DCA_RXCTRL_DATA_WRO_EN |
>-  IXGBE_DCA_RXCTRL_HEAD_WRO_EN);
>-  IXGBE_WRITE_REG(hw, IXGBE_DCA_RXCTRL(i), regval);
>-  }
>-#endif
>   if (ret_val)
>   return ret_val;
>
>diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_common.c
>b/drivers/net/ethernet/intel/ixgbe/ixgbe_common.c
>index d4933d2..96c324f 100644
>--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_common.c
>+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_common.c
>@@ -350,25 +350,6 @@ s32 ixgbe_start_hw_gen2(struct ixgbe_hw *hw)
>   }
>   IXGBE_WRITE_FLUSH(hw);
>
>-#ifndef CONFIG_SPARC
>-  /* Disable relaxed ordering */
>-  for (i = 0; i < hw->mac.max_tx_queues; i++) {
>-  u32 regval;
>-
>-  regval = IXGBE_READ_REG(hw, IXGBE_DCA_TXCTRL_82599(i));
>-  regval &= ~IXGBE_DCA_TXCTRL_DESC_WRO_EN;
>-  IXGBE_WRITE_REG(hw, IXGBE_DCA_TXCTRL_82599(i), regval);
>-  }
>-
>-  for (i = 0; i < hw->mac.max_rx_queues; i++) {
>-  u32 regval;
>-
>-  regval = IXGBE_READ_REG(hw, IXGBE_DCA_RXCTRL(i));
>-  regval &= ~(IXGBE_DCA_RXCTRL_DATA_WRO_EN |
>-  IXGBE_DCA_RXCTRL_HEAD_WRO_EN);
>-  IXGBE_WRITE_REG(hw, IXGBE_DCA_RXCTRL(i), regval);
>-  }
>-#endif
>   return 0;
> }
>
>--
>1.8.3.1
>

RE: [PATCH v3 net 2/2] net: ixgbe: Use new PCI_DEV_FLAGS_NO_RELAXED_ORDERING flag

[Tantilov, Emil S]

>-Original Message-
>From: linux-kernel-ow...@vger.kernel.org [mailto:linux-kernel-
>ow...@vger.kernel.org] On Behalf Of Ding Tianhong
>Sent: Thursday, August 17, 2017 11:21 PM
>To: da...@davemloft.net; Kirsher, Jeffrey T ;
>keesc...@chromium.org; linux-kernel@vger.kernel.org;
>sparcli...@vger.kernel.org; intel-wired-...@lists.osuosl.org;
>alexander.du...@gmail.com; net...@vger.kernel.org; linux...@huawei.com
>Cc: Ding Tianhong 
>Subject: [PATCH v3 net 2/2] net: ixgbe: Use new
>PCI_DEV_FLAGS_NO_RELAXED_ORDERING flag
>
>The ixgbe driver use the compile check to determine if it can
>send TLPs to Root Port with the Relaxed Ordering Attribute set,
>this is too inconvenient, now the new flag
>PCI_DEV_FLAGS_NO_RELAXED_ORDERING
>has been added to the kernel and we could check the bit4 in the PCIe
>Device Control register to determine whether we should use the Relaxed
>Ordering Attributes or not, so use this new way in the ixgbe driver.
>
>Signed-off-by: Ding Tianhong 
>---
> drivers/net/ethernet/intel/ixgbe/ixgbe_82598.c  | 22 -
>-
> drivers/net/ethernet/intel/ixgbe/ixgbe_common.c | 19 ---
> 2 files changed, 41 deletions(-)

This change looks good to me for ixgbe.

Acked-by: Emil Tantilov 

Thanks,
Emil 
 
>
>diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_82598.c
>b/drivers/net/ethernet/intel/ixgbe/ixgbe_82598.c
>index 523f9d0..8a32eb7 100644
>--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_82598.c
>+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_82598.c
>@@ -175,31 +175,9 @@ static s32 ixgbe_init_phy_ops_82598(struct ixgbe_hw
>*hw)
>  **/
> static s32 ixgbe_start_hw_82598(struct ixgbe_hw *hw)
> {
>-#ifndef CONFIG_SPARC
>-  u32 regval;
>-  u32 i;
>-#endif
>   s32 ret_val;
>
>   ret_val = ixgbe_start_hw_generic(hw);
>-
>-#ifndef CONFIG_SPARC
>-  /* Disable relaxed ordering */
>-  for (i = 0; ((i < hw->mac.max_tx_queues) &&
>-   (i < IXGBE_DCA_MAX_QUEUES_82598)); i++) {
>-  regval = IXGBE_READ_REG(hw, IXGBE_DCA_TXCTRL(i));
>-  regval &= ~IXGBE_DCA_TXCTRL_DESC_WRO_EN;
>-  IXGBE_WRITE_REG(hw, IXGBE_DCA_TXCTRL(i), regval);
>-  }
>-
>-  for (i = 0; ((i < hw->mac.max_rx_queues) &&
>-   (i < IXGBE_DCA_MAX_QUEUES_82598)); i++) {
>-  regval = IXGBE_READ_REG(hw, IXGBE_DCA_RXCTRL(i));
>-  regval &= ~(IXGBE_DCA_RXCTRL_DATA_WRO_EN |
>-  IXGBE_DCA_RXCTRL_HEAD_WRO_EN);
>-  IXGBE_WRITE_REG(hw, IXGBE_DCA_RXCTRL(i), regval);
>-  }
>-#endif
>   if (ret_val)
>   return ret_val;
>
>diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_common.c
>b/drivers/net/ethernet/intel/ixgbe/ixgbe_common.c
>index d4933d2..96c324f 100644
>--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_common.c
>+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_common.c
>@@ -350,25 +350,6 @@ s32 ixgbe_start_hw_gen2(struct ixgbe_hw *hw)
>   }
>   IXGBE_WRITE_FLUSH(hw);
>
>-#ifndef CONFIG_SPARC
>-  /* Disable relaxed ordering */
>-  for (i = 0; i < hw->mac.max_tx_queues; i++) {
>-  u32 regval;
>-
>-  regval = IXGBE_READ_REG(hw, IXGBE_DCA_TXCTRL_82599(i));
>-  regval &= ~IXGBE_DCA_TXCTRL_DESC_WRO_EN;
>-  IXGBE_WRITE_REG(hw, IXGBE_DCA_TXCTRL_82599(i), regval);
>-  }
>-
>-  for (i = 0; i < hw->mac.max_rx_queues; i++) {
>-  u32 regval;
>-
>-  regval = IXGBE_READ_REG(hw, IXGBE_DCA_RXCTRL(i));
>-  regval &= ~(IXGBE_DCA_RXCTRL_DATA_WRO_EN |
>-  IXGBE_DCA_RXCTRL_HEAD_WRO_EN);
>-  IXGBE_WRITE_REG(hw, IXGBE_DCA_RXCTRL(i), regval);
>-  }
>-#endif
>   return 0;
> }
>
>--
>1.8.3.1
>

RE: [PATCH net v2 2/2] net: ixgbe: Use new PCI_DEV_FLAGS_NO_RELAXED_ORDERING flag

[Tantilov, Emil S]

>-Original Message-
>From: Ding Tianhong [mailto:dingtianh...@huawei.com]
>Sent: Thursday, August 17, 2017 5:39 PM
>To: Tantilov, Emil S <emil.s.tanti...@intel.com>; da...@davemloft.net;
>Kirsher, Jeffrey T <jeffrey.t.kirs...@intel.com>; keesc...@chromium.org;
>linux-kernel@vger.kernel.org; sparcli...@vger.kernel.org; intel-wired-
>l...@lists.osuosl.org; alexander.du...@gmail.com; net...@vger.kernel.org;
>linux...@huawei.com
>Subject: Re: [PATCH net v2 2/2] net: ixgbe: Use new
>PCI_DEV_FLAGS_NO_RELAXED_ORDERING flag
>
>
>
>On 2017/8/17 22:17, Tantilov, Emil S wrote:
>
>>> ret_val = ixgbe_start_hw_generic(hw);
>>>
>>> -#ifndef CONFIG_SPARC
>>> -   /* Disable relaxed ordering */
>>> -   for (i = 0; ((i < hw->mac.max_tx_queues) &&
>>> -(i < IXGBE_DCA_MAX_QUEUES_82598)); i++) {
>>> -   regval = IXGBE_READ_REG(hw, IXGBE_DCA_TXCTRL(i));
>>> -   regval &= ~IXGBE_DCA_TXCTRL_DESC_WRO_EN;
>>> -   IXGBE_WRITE_REG(hw, IXGBE_DCA_TXCTRL(i), regval);
>>> -   }
>>> +   if (!pcie_relaxed_ordering_enabled(adapter->pdev)) {
>>
>> As Alex mentioned there is no need for this check in any form.
>>
>> The HW defaults to Relaxed Ordering enabled unless it is disabled in
>> the PCIe Device Control Register. So the above logic is already done by
>HW.
>>
>> All you have to do is strip the code disabling relaxed ordering.
>>
>
>Hi Tantilov:
>
>I misunderstood Alex's suggestion, But I still couldn't find the logic
>where
>the HW disable the Relaxed Ordering when the PCIe Device Control Register
>disable it, can you point it out?

If you look at the datasheet (82599) - the description of CTRL_EXT.RO_DIS (bit 
17, 0b):

Relaxed Ordering Disable. When set to 1b, the device does not request any 
relaxed
ordering transactions. When this bit is cleared and the Enable Relaxed Ordering 
bit in
the Device Control register is set, the device requests relaxed ordering 
transactions per queues as configured in the DCA_RXCTRL[n] and DCA_TXCTRL[n] 
registers.

So if you remove the code that clears the bits in DCA_T/RXCTRL relaxed ordering 
should
be enabled by HW when the bus allows it.

Thanks,
Emil

RE: [PATCH net v2 2/2] net: ixgbe: Use new PCI_DEV_FLAGS_NO_RELAXED_ORDERING flag

[Tantilov, Emil S]

>-Original Message-
>From: Ding Tianhong [mailto:dingtianh...@huawei.com]
>Sent: Thursday, August 17, 2017 5:39 PM
>To: Tantilov, Emil S ; da...@davemloft.net;
>Kirsher, Jeffrey T ; keesc...@chromium.org;
>linux-kernel@vger.kernel.org; sparcli...@vger.kernel.org; intel-wired-
>l...@lists.osuosl.org; alexander.du...@gmail.com; net...@vger.kernel.org;
>linux...@huawei.com
>Subject: Re: [PATCH net v2 2/2] net: ixgbe: Use new
>PCI_DEV_FLAGS_NO_RELAXED_ORDERING flag
>
>
>
>On 2017/8/17 22:17, Tantilov, Emil S wrote:
>
>>> ret_val = ixgbe_start_hw_generic(hw);
>>>
>>> -#ifndef CONFIG_SPARC
>>> -   /* Disable relaxed ordering */
>>> -   for (i = 0; ((i < hw->mac.max_tx_queues) &&
>>> -(i < IXGBE_DCA_MAX_QUEUES_82598)); i++) {
>>> -   regval = IXGBE_READ_REG(hw, IXGBE_DCA_TXCTRL(i));
>>> -   regval &= ~IXGBE_DCA_TXCTRL_DESC_WRO_EN;
>>> -   IXGBE_WRITE_REG(hw, IXGBE_DCA_TXCTRL(i), regval);
>>> -   }
>>> +   if (!pcie_relaxed_ordering_enabled(adapter->pdev)) {
>>
>> As Alex mentioned there is no need for this check in any form.
>>
>> The HW defaults to Relaxed Ordering enabled unless it is disabled in
>> the PCIe Device Control Register. So the above logic is already done by
>HW.
>>
>> All you have to do is strip the code disabling relaxed ordering.
>>
>
>Hi Tantilov:
>
>I misunderstood Alex's suggestion, But I still couldn't find the logic
>where
>the HW disable the Relaxed Ordering when the PCIe Device Control Register
>disable it, can you point it out?

If you look at the datasheet (82599) - the description of CTRL_EXT.RO_DIS (bit 
17, 0b):

Relaxed Ordering Disable. When set to 1b, the device does not request any 
relaxed
ordering transactions. When this bit is cleared and the Enable Relaxed Ordering 
bit in
the Device Control register is set, the device requests relaxed ordering 
transactions per queues as configured in the DCA_RXCTRL[n] and DCA_TXCTRL[n] 
registers.

So if you remove the code that clears the bits in DCA_T/RXCTRL relaxed ordering 
should
be enabled by HW when the bus allows it.

Thanks,
Emil

RE: [PATCH net v2 2/2] net: ixgbe: Use new PCI_DEV_FLAGS_NO_RELAXED_ORDERING flag

[Tantilov, Emil S]

>-Original Message-
>From: linux-kernel-ow...@vger.kernel.org [mailto:linux-kernel-
>ow...@vger.kernel.org] On Behalf Of Ding Tianhong
>Sent: Wednesday, August 16, 2017 8:25 PM
>To: da...@davemloft.net; Kirsher, Jeffrey T ;
>keesc...@chromium.org; linux-kernel@vger.kernel.org;
>sparcli...@vger.kernel.org; intel-wired-...@lists.osuosl.org;
>alexander.du...@gmail.com; net...@vger.kernel.org; linux...@huawei.com
>Cc: Ding Tianhong 
>Subject: [PATCH net v2 2/2] net: ixgbe: Use new
>PCI_DEV_FLAGS_NO_RELAXED_ORDERING flag
>
>The ixgbe driver use the compile check to determine if it can
>send TLPs to Root Port with the Relaxed Ordering Attribute set,
>this is too inconvenient, now the new flag
>PCI_DEV_FLAGS_NO_RELAXED_ORDERING
>has been added to the kernel and we could check the bit4 in the PCIe
>Device Control register to determine whether we should use the Relaxed
>Ordering Attributes or not, so use this new way in the ixgbe driver.
>
>Signed-off-by: Ding Tianhong 
>---
> drivers/net/ethernet/intel/ixgbe/ixgbe_82598.c  | 37 -
>
> drivers/net/ethernet/intel/ixgbe/ixgbe_common.c | 32 +++--
> 2 files changed, 35 insertions(+), 34 deletions(-)
>
>diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_82598.c
>b/drivers/net/ethernet/intel/ixgbe/ixgbe_82598.c
>index 523f9d0..d1571e3 100644
>--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_82598.c
>+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_82598.c
>@@ -175,31 +175,30 @@ static s32 ixgbe_init_phy_ops_82598(struct ixgbe_hw
>*hw)
>  **/
> static s32 ixgbe_start_hw_82598(struct ixgbe_hw *hw)
> {
>-#ifndef CONFIG_SPARC
>-  u32 regval;
>-  u32 i;
>-#endif
>+  u32 regval, i;
>   s32 ret_val;
>+  struct ixgbe_adapter *adapter = hw->back;
>
>   ret_val = ixgbe_start_hw_generic(hw);
>
>-#ifndef CONFIG_SPARC
>-  /* Disable relaxed ordering */
>-  for (i = 0; ((i < hw->mac.max_tx_queues) &&
>-   (i < IXGBE_DCA_MAX_QUEUES_82598)); i++) {
>-  regval = IXGBE_READ_REG(hw, IXGBE_DCA_TXCTRL(i));
>-  regval &= ~IXGBE_DCA_TXCTRL_DESC_WRO_EN;
>-  IXGBE_WRITE_REG(hw, IXGBE_DCA_TXCTRL(i), regval);
>-  }
>+  if (!pcie_relaxed_ordering_enabled(adapter->pdev)) {

As Alex mentioned there is no need for this check in any form.

The HW defaults to Relaxed Ordering enabled unless it is disabled in 
the PCIe Device Control Register. So the above logic is already done by HW.

All you have to do is strip the code disabling relaxed ordering.

Thanks,
Emil

RE: [PATCH net v2 2/2] net: ixgbe: Use new PCI_DEV_FLAGS_NO_RELAXED_ORDERING flag

[Tantilov, Emil S]

>-Original Message-
>From: linux-kernel-ow...@vger.kernel.org [mailto:linux-kernel-
>ow...@vger.kernel.org] On Behalf Of Ding Tianhong
>Sent: Wednesday, August 16, 2017 8:25 PM
>To: da...@davemloft.net; Kirsher, Jeffrey T ;
>keesc...@chromium.org; linux-kernel@vger.kernel.org;
>sparcli...@vger.kernel.org; intel-wired-...@lists.osuosl.org;
>alexander.du...@gmail.com; net...@vger.kernel.org; linux...@huawei.com
>Cc: Ding Tianhong 
>Subject: [PATCH net v2 2/2] net: ixgbe: Use new
>PCI_DEV_FLAGS_NO_RELAXED_ORDERING flag
>
>The ixgbe driver use the compile check to determine if it can
>send TLPs to Root Port with the Relaxed Ordering Attribute set,
>this is too inconvenient, now the new flag
>PCI_DEV_FLAGS_NO_RELAXED_ORDERING
>has been added to the kernel and we could check the bit4 in the PCIe
>Device Control register to determine whether we should use the Relaxed
>Ordering Attributes or not, so use this new way in the ixgbe driver.
>
>Signed-off-by: Ding Tianhong 
>---
> drivers/net/ethernet/intel/ixgbe/ixgbe_82598.c  | 37 -
>
> drivers/net/ethernet/intel/ixgbe/ixgbe_common.c | 32 +++--
> 2 files changed, 35 insertions(+), 34 deletions(-)
>
>diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_82598.c
>b/drivers/net/ethernet/intel/ixgbe/ixgbe_82598.c
>index 523f9d0..d1571e3 100644
>--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_82598.c
>+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_82598.c
>@@ -175,31 +175,30 @@ static s32 ixgbe_init_phy_ops_82598(struct ixgbe_hw
>*hw)
>  **/
> static s32 ixgbe_start_hw_82598(struct ixgbe_hw *hw)
> {
>-#ifndef CONFIG_SPARC
>-  u32 regval;
>-  u32 i;
>-#endif
>+  u32 regval, i;
>   s32 ret_val;
>+  struct ixgbe_adapter *adapter = hw->back;
>
>   ret_val = ixgbe_start_hw_generic(hw);
>
>-#ifndef CONFIG_SPARC
>-  /* Disable relaxed ordering */
>-  for (i = 0; ((i < hw->mac.max_tx_queues) &&
>-   (i < IXGBE_DCA_MAX_QUEUES_82598)); i++) {
>-  regval = IXGBE_READ_REG(hw, IXGBE_DCA_TXCTRL(i));
>-  regval &= ~IXGBE_DCA_TXCTRL_DESC_WRO_EN;
>-  IXGBE_WRITE_REG(hw, IXGBE_DCA_TXCTRL(i), regval);
>-  }
>+  if (!pcie_relaxed_ordering_enabled(adapter->pdev)) {

As Alex mentioned there is no need for this check in any form.

The HW defaults to Relaxed Ordering enabled unless it is disabled in 
the PCIe Device Control Register. So the above logic is already done by HW.

All you have to do is strip the code disabling relaxed ordering.

Thanks,
Emil

RE: [Intel-wired-lan] [PATCH v2] PCI: lock each enable/disable num_vfs operation in sysfs

[Tantilov, Emil S]

>-Original Message-
>From: Intel-wired-lan [mailto:intel-wired-lan-boun...@lists.osuosl.org] On
>Behalf Of Greg
>Sent: Friday, January 06, 2017 3:18 PM
>To: Tantilov, Emil S <emil.s.tanti...@intel.com>
>Cc: linux-...@vger.kernel.org; intel-wired-...@lists.osuosl.org; linux-
>ker...@vger.kernel.org; net...@vger.kernel.org
>Subject: Re: [Intel-wired-lan] [PATCH v2] PCI: lock each enable/disable
>num_vfs operation in sysfs
>
>On Fri, 2017-01-06 at 13:59 -0800, Emil Tantilov wrote:
>> Enabling/disabling SRIOV via sysfs by echo-ing multiple values
>> simultaneously:
>>
>> echo 63 > /sys/class/net/ethX/device/sriov_numvfs&
>> echo 63 > /sys/class/net/ethX/device/sriov_numvfs
>>
>> sleep 5
>>
>> echo 0 > /sys/class/net/ethX/device/sriov_numvfs&
>> echo 0 > /sys/class/net/ethX/device/sriov_numvfs
>>
>> Results in the following bug:
>>
>> kernel BUG at drivers/pci/iov.c:495!
>> invalid opcode:  [#1] SMP
>> CPU: 1 PID: 8050 Comm: bash Tainted: G   W   4.9.0-rc7-net-next #2092
>> RIP: 0010:[]
>>[] pci_iov_release+0x57/0x60
>>
>> Call Trace:
>>  [] pci_release_dev+0x26/0x70
>>  [] device_release+0x3e/0xb0
>>  [] kobject_cleanup+0x67/0x180
>>  [] kobject_put+0x2d/0x60
>>  [] put_device+0x17/0x20
>>  [] pci_dev_put+0x1a/0x20
>>  [] pci_get_dev_by_id+0x5b/0x90
>>  [] pci_get_subsys+0x35/0x40
>>  [] pci_get_device+0x18/0x20
>>  [] pci_get_domain_bus_and_slot+0x2b/0x60
>>  [] pci_iov_remove_virtfn+0x57/0x180
>>  [] pci_disable_sriov+0x65/0x140
>>  [] ixgbe_disable_sriov+0xc7/0x1d0 [ixgbe]
>>  [] ixgbe_pci_sriov_configure+0x3d/0x170 [ixgbe]
>>  [] sriov_numvfs_store+0xdc/0x130
>> ...
>> RIP  [] pci_iov_release+0x57/0x60
>>
>> Use the existing mutex lock to protect each enable/disable operation.
>>
>> -v2: move the existing lock from protecting the config of the IOV bus
>> to protecting the writes to sriov_numvfs in sysfs without maintaining
>> a "locked" version of pci_iov_add/remove_virtfn().
>> As suggested by Gavin Shan <gws...@linux.vnet.ibm.com>
>>
>> CC: Alexander Duyck <alexander.h.du...@intel.com>
>> Signed-off-by: Emil Tantilov <emil.s.tanti...@intel.com>
>> ---
>>  drivers/pci/iov.c   |7 ---
>>  drivers/pci/pci-sysfs.c |   23 ---
>>  drivers/pci/pci.h   |2 +-
>>  3 files changed, 17 insertions(+), 15 deletions(-)
>>
>> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
>> index 4722782..2479ae8 100644
>> --- a/drivers/pci/iov.c
>> +++ b/drivers/pci/iov.c
>> @@ -124,7 +124,6 @@ int pci_iov_add_virtfn(struct pci_dev *dev, int id,
>int reset)
>>  struct pci_sriov *iov = dev->sriov;
>>  struct pci_bus *bus;
>>
>> -mutex_lock(>dev->sriov->lock);
>>  bus = virtfn_add_bus(dev->bus, pci_iov_virtfn_bus(dev, id));
>>  if (!bus)
>>  goto failed;
>> @@ -162,7 +161,6 @@ int pci_iov_add_virtfn(struct pci_dev *dev, int id,
>int reset)
>>  __pci_reset_function(virtfn);
>>
>>  pci_device_add(virtfn, virtfn->bus);
>> -mutex_unlock(>dev->sriov->lock);
>>
>>  pci_bus_add_device(virtfn);
>>  sprintf(buf, "virtfn%u", id);
>> @@ -181,12 +179,10 @@ int pci_iov_add_virtfn(struct pci_dev *dev, int id,
>int reset)
>>  sysfs_remove_link(>dev.kobj, buf);
>>  failed1:
>>  pci_dev_put(dev);
>> -mutex_lock(>dev->sriov->lock);
>>  pci_stop_and_remove_bus_device(virtfn);
>>  failed0:
>>  virtfn_remove_bus(dev->bus, bus);
>>  failed:
>> -mutex_unlock(>dev->sriov->lock);
>>
>>  return rc;
>>  }
>> @@ -195,7 +191,6 @@ void pci_iov_remove_virtfn(struct pci_dev *dev, int
>id, int reset)
>>  {
>>  char buf[VIRTFN_ID_LEN];
>>  struct pci_dev *virtfn;
>> -struct pci_sriov *iov = dev->sriov;
>>
>>  virtfn = pci_get_domain_bus_and_slot(pci_domain_nr(dev->bus),
>>   pci_iov_virtfn_bus(dev, id),
>> @@ -218,10 +213,8 @@ void pci_iov_remove_virtfn(struct pci_dev *dev, int
>id, int reset)
>>  if (virtfn->dev.kobj.sd)
>>  sysfs_remove_link(>dev.kobj, "physfn");
>>
>> -mutex_lock(>dev->sriov->lock);
>>  pci_stop_and_remove_bus_device(virtfn);
>>  virtfn_remove_bus(dev->bus, virtfn->bus);
>> -mutex_unlock(>dev->sriov->l

RE: [Intel-wired-lan] [PATCH v2] PCI: lock each enable/disable num_vfs operation in sysfs

[Tantilov, Emil S]

>-Original Message-
>From: Intel-wired-lan [mailto:intel-wired-lan-boun...@lists.osuosl.org] On
>Behalf Of Greg
>Sent: Friday, January 06, 2017 3:18 PM
>To: Tantilov, Emil S 
>Cc: linux-...@vger.kernel.org; intel-wired-...@lists.osuosl.org; linux-
>ker...@vger.kernel.org; net...@vger.kernel.org
>Subject: Re: [Intel-wired-lan] [PATCH v2] PCI: lock each enable/disable
>num_vfs operation in sysfs
>
>On Fri, 2017-01-06 at 13:59 -0800, Emil Tantilov wrote:
>> Enabling/disabling SRIOV via sysfs by echo-ing multiple values
>> simultaneously:
>>
>> echo 63 > /sys/class/net/ethX/device/sriov_numvfs&
>> echo 63 > /sys/class/net/ethX/device/sriov_numvfs
>>
>> sleep 5
>>
>> echo 0 > /sys/class/net/ethX/device/sriov_numvfs&
>> echo 0 > /sys/class/net/ethX/device/sriov_numvfs
>>
>> Results in the following bug:
>>
>> kernel BUG at drivers/pci/iov.c:495!
>> invalid opcode:  [#1] SMP
>> CPU: 1 PID: 8050 Comm: bash Tainted: G   W   4.9.0-rc7-net-next #2092
>> RIP: 0010:[]
>>[] pci_iov_release+0x57/0x60
>>
>> Call Trace:
>>  [] pci_release_dev+0x26/0x70
>>  [] device_release+0x3e/0xb0
>>  [] kobject_cleanup+0x67/0x180
>>  [] kobject_put+0x2d/0x60
>>  [] put_device+0x17/0x20
>>  [] pci_dev_put+0x1a/0x20
>>  [] pci_get_dev_by_id+0x5b/0x90
>>  [] pci_get_subsys+0x35/0x40
>>  [] pci_get_device+0x18/0x20
>>  [] pci_get_domain_bus_and_slot+0x2b/0x60
>>  [] pci_iov_remove_virtfn+0x57/0x180
>>  [] pci_disable_sriov+0x65/0x140
>>  [] ixgbe_disable_sriov+0xc7/0x1d0 [ixgbe]
>>  [] ixgbe_pci_sriov_configure+0x3d/0x170 [ixgbe]
>>  [] sriov_numvfs_store+0xdc/0x130
>> ...
>> RIP  [] pci_iov_release+0x57/0x60
>>
>> Use the existing mutex lock to protect each enable/disable operation.
>>
>> -v2: move the existing lock from protecting the config of the IOV bus
>> to protecting the writes to sriov_numvfs in sysfs without maintaining
>> a "locked" version of pci_iov_add/remove_virtfn().
>> As suggested by Gavin Shan 
>>
>> CC: Alexander Duyck 
>> Signed-off-by: Emil Tantilov 
>> ---
>>  drivers/pci/iov.c   |7 ---
>>  drivers/pci/pci-sysfs.c |   23 ---
>>  drivers/pci/pci.h   |2 +-
>>  3 files changed, 17 insertions(+), 15 deletions(-)
>>
>> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
>> index 4722782..2479ae8 100644
>> --- a/drivers/pci/iov.c
>> +++ b/drivers/pci/iov.c
>> @@ -124,7 +124,6 @@ int pci_iov_add_virtfn(struct pci_dev *dev, int id,
>int reset)
>>  struct pci_sriov *iov = dev->sriov;
>>  struct pci_bus *bus;
>>
>> -mutex_lock(>dev->sriov->lock);
>>  bus = virtfn_add_bus(dev->bus, pci_iov_virtfn_bus(dev, id));
>>  if (!bus)
>>  goto failed;
>> @@ -162,7 +161,6 @@ int pci_iov_add_virtfn(struct pci_dev *dev, int id,
>int reset)
>>  __pci_reset_function(virtfn);
>>
>>  pci_device_add(virtfn, virtfn->bus);
>> -mutex_unlock(>dev->sriov->lock);
>>
>>  pci_bus_add_device(virtfn);
>>  sprintf(buf, "virtfn%u", id);
>> @@ -181,12 +179,10 @@ int pci_iov_add_virtfn(struct pci_dev *dev, int id,
>int reset)
>>  sysfs_remove_link(>dev.kobj, buf);
>>  failed1:
>>  pci_dev_put(dev);
>> -mutex_lock(>dev->sriov->lock);
>>  pci_stop_and_remove_bus_device(virtfn);
>>  failed0:
>>  virtfn_remove_bus(dev->bus, bus);
>>  failed:
>> -mutex_unlock(>dev->sriov->lock);
>>
>>  return rc;
>>  }
>> @@ -195,7 +191,6 @@ void pci_iov_remove_virtfn(struct pci_dev *dev, int
>id, int reset)
>>  {
>>  char buf[VIRTFN_ID_LEN];
>>  struct pci_dev *virtfn;
>> -struct pci_sriov *iov = dev->sriov;
>>
>>  virtfn = pci_get_domain_bus_and_slot(pci_domain_nr(dev->bus),
>>   pci_iov_virtfn_bus(dev, id),
>> @@ -218,10 +213,8 @@ void pci_iov_remove_virtfn(struct pci_dev *dev, int
>id, int reset)
>>  if (virtfn->dev.kobj.sd)
>>  sysfs_remove_link(>dev.kobj, "physfn");
>>
>> -mutex_lock(>dev->sriov->lock);
>>  pci_stop_and_remove_bus_device(virtfn);
>>  virtfn_remove_bus(dev->bus, virtfn->bus);
>> -mutex_unlock(>dev->sriov->lock);
>>
>>  /* balance pci_get_domain_bus_and_slot() */
>>  pci_dev_put(virtfn);
>> diff -

RE: [PATCH 2/2] PCI: lock each enable/disable num_vfs operation in sysfs

[Tantilov, Emil S]

>-Original Message-
>From: Gavin Shan [mailto:gws...@linux.vnet.ibm.com]
>Sent: Wednesday, January 04, 2017 3:12 PM
>To: Tantilov, Emil S <emil.s.tanti...@intel.com>
>Cc: Gavin Shan <gws...@linux.vnet.ibm.com>; linux-...@vger.kernel.org;
>intel-wired-...@lists.osuosl.org; Duyck, Alexander H
><alexander.h.du...@intel.com>; net...@vger.kernel.org; linux-
>ker...@vger.kernel.org
>Subject: Re: [PATCH 2/2] PCI: lock each enable/disable num_vfs operation in
>sysfs
>
>On Wed, Jan 04, 2017 at 04:00:20PM +, Tantilov, Emil S wrote:
>>>On Tue, Jan 03, 2017 at 04:48:31PM -0800, Emil Tantilov wrote:
>>>>Enabling/disabling SRIOV via sysfs by echo-ing multiple values
>>>>simultaneously:
>>>>
>>>>echo 63 > /sys/class/net/ethX/device/sriov_numvfs&
>>>>echo 63 > /sys/class/net/ethX/device/sriov_numvfs
>>>>
>>>>sleep 5
>>>>
>>>>echo 0 > /sys/class/net/ethX/device/sriov_numvfs&
>>>>echo 0 > /sys/class/net/ethX/device/sriov_numvfs
>>>>
>>>>Results in the following bug:
>>>>
>>>>kernel BUG at drivers/pci/iov.c:495!
>>>>invalid opcode:  [#1] SMP
>>>>CPU: 1 PID: 8050 Comm: bash Tainted: G   W   4.9.0-rc7-net-next #2092
>>>>RIP: 0010:[]
>>>>  [] pci_iov_release+0x57/0x60
>>>>
>>>>Call Trace:
>>>> [] pci_release_dev+0x26/0x70
>>>> [] device_release+0x3e/0xb0
>>>> [] kobject_cleanup+0x67/0x180
>>>> [] kobject_put+0x2d/0x60
>>>> [] put_device+0x17/0x20
>>>> [] pci_dev_put+0x1a/0x20
>>>> [] pci_get_dev_by_id+0x5b/0x90
>>>> [] pci_get_subsys+0x35/0x40
>>>> [] pci_get_device+0x18/0x20
>>>> [] pci_get_domain_bus_and_slot+0x2b/0x60
>>>> [] pci_iov_remove_virtfn+0x57/0x180
>>>> [] pci_disable_sriov+0x65/0x140
>>>> [] ixgbe_disable_sriov+0xc7/0x1d0 [ixgbe]
>>>> [] ixgbe_pci_sriov_configure+0x3d/0x170 [ixgbe]
>>>> [] sriov_numvfs_store+0xdc/0x130
>>>>...
>>>>RIP  [] pci_iov_release+0x57/0x60
>>>>
>>>>Use the existing mutex lock to protect each enable/disable operation.
>>>>
>>>>CC: Alexander Duyck <alexander.h.du...@intel.com>
>>>>Signed-off-by: Emil Tantilov <emil.s.tanti...@intel.com>
>>>
>>>Emil, It's going to change semantics of pci_enable_sriov() and
>pci_disable_sriov().
>>>They can be invoked when writing to the sysfs entry, or loading PF's
>>>driver. With the change applied, the lock (pf->sriov->lock) isn't acquired 
>>>and released
>>>in the PF's driver loading path.
>>
>>The enablement of SRIOV on driver load is done via deprecated module 
>>parameter.
>>Perhaps we can just remove it, although there are probably still people that 
>>use it
>>and may not be happy if we get rid of it.
>>
>
>Yeah, some drivers are still using the interface. So we cannot affect it
>until it can be droped.
>
>>>I think the reasonable way would be adding a flag in "struct sriov", to
>>>indicate someone is accessing the IOV capability through sysfs file. With 
>>>this, the
>>>code returns with "-EBUSY" immediately for contenders. With it, nothing is 
>>>going
>>>to be changed in PF's driver loading path.
>>
>>Flag is what I initially had in mind, but did not want to add extra locking 
>>if we
>>can make use of the existing.
>>
>
>The problem is sriov->lock wasn't introduced to protect the whole IOV 
>capability.
>Instead, it protects the allocation of virtual bus (if needed). In your patch,
>it will be used to protect the whole IOV capability, ensure accessing the
>IOV capability exclusively. So the usage of this lock is changed.
>
> code extracted from pci.h:
>
> struct pci_sriov {
>:
>struct mutex lock;  /* lock for VF bus */
>:
> }
>
>The lock was introduced by commit d1b054da8 ("PCI: initialize and release
>SR-IOV capability"). If I'm correct enough, I don't think this lock is needed 
>when
>pci_enable_sriov() or pci_disable_sriov() are called in driver because of
>module
>parameters. I don't see the usage case calling pci_disable_sriov() while
>previous pci_enable_sriov() isn't finished yet. Also, it's not needed in EEH
>subsystem.
>So I think the lock can be dropped, then it can be used to protect sysfs path.

That's pretty much what this patch does, except I kept the locking for EEH 

RE: [PATCH 2/2] PCI: lock each enable/disable num_vfs operation in sysfs

[Tantilov, Emil S]

>-Original Message-
>From: Gavin Shan [mailto:gws...@linux.vnet.ibm.com]
>Sent: Wednesday, January 04, 2017 3:12 PM
>To: Tantilov, Emil S 
>Cc: Gavin Shan ; linux-...@vger.kernel.org;
>intel-wired-...@lists.osuosl.org; Duyck, Alexander H
>; net...@vger.kernel.org; linux-
>ker...@vger.kernel.org
>Subject: Re: [PATCH 2/2] PCI: lock each enable/disable num_vfs operation in
>sysfs
>
>On Wed, Jan 04, 2017 at 04:00:20PM +, Tantilov, Emil S wrote:
>>>On Tue, Jan 03, 2017 at 04:48:31PM -0800, Emil Tantilov wrote:
>>>>Enabling/disabling SRIOV via sysfs by echo-ing multiple values
>>>>simultaneously:
>>>>
>>>>echo 63 > /sys/class/net/ethX/device/sriov_numvfs&
>>>>echo 63 > /sys/class/net/ethX/device/sriov_numvfs
>>>>
>>>>sleep 5
>>>>
>>>>echo 0 > /sys/class/net/ethX/device/sriov_numvfs&
>>>>echo 0 > /sys/class/net/ethX/device/sriov_numvfs
>>>>
>>>>Results in the following bug:
>>>>
>>>>kernel BUG at drivers/pci/iov.c:495!
>>>>invalid opcode:  [#1] SMP
>>>>CPU: 1 PID: 8050 Comm: bash Tainted: G   W   4.9.0-rc7-net-next #2092
>>>>RIP: 0010:[]
>>>>  [] pci_iov_release+0x57/0x60
>>>>
>>>>Call Trace:
>>>> [] pci_release_dev+0x26/0x70
>>>> [] device_release+0x3e/0xb0
>>>> [] kobject_cleanup+0x67/0x180
>>>> [] kobject_put+0x2d/0x60
>>>> [] put_device+0x17/0x20
>>>> [] pci_dev_put+0x1a/0x20
>>>> [] pci_get_dev_by_id+0x5b/0x90
>>>> [] pci_get_subsys+0x35/0x40
>>>> [] pci_get_device+0x18/0x20
>>>> [] pci_get_domain_bus_and_slot+0x2b/0x60
>>>> [] pci_iov_remove_virtfn+0x57/0x180
>>>> [] pci_disable_sriov+0x65/0x140
>>>> [] ixgbe_disable_sriov+0xc7/0x1d0 [ixgbe]
>>>> [] ixgbe_pci_sriov_configure+0x3d/0x170 [ixgbe]
>>>> [] sriov_numvfs_store+0xdc/0x130
>>>>...
>>>>RIP  [] pci_iov_release+0x57/0x60
>>>>
>>>>Use the existing mutex lock to protect each enable/disable operation.
>>>>
>>>>CC: Alexander Duyck 
>>>>Signed-off-by: Emil Tantilov 
>>>
>>>Emil, It's going to change semantics of pci_enable_sriov() and
>pci_disable_sriov().
>>>They can be invoked when writing to the sysfs entry, or loading PF's
>>>driver. With the change applied, the lock (pf->sriov->lock) isn't acquired 
>>>and released
>>>in the PF's driver loading path.
>>
>>The enablement of SRIOV on driver load is done via deprecated module 
>>parameter.
>>Perhaps we can just remove it, although there are probably still people that 
>>use it
>>and may not be happy if we get rid of it.
>>
>
>Yeah, some drivers are still using the interface. So we cannot affect it
>until it can be droped.
>
>>>I think the reasonable way would be adding a flag in "struct sriov", to
>>>indicate someone is accessing the IOV capability through sysfs file. With 
>>>this, the
>>>code returns with "-EBUSY" immediately for contenders. With it, nothing is 
>>>going
>>>to be changed in PF's driver loading path.
>>
>>Flag is what I initially had in mind, but did not want to add extra locking 
>>if we
>>can make use of the existing.
>>
>
>The problem is sriov->lock wasn't introduced to protect the whole IOV 
>capability.
>Instead, it protects the allocation of virtual bus (if needed). In your patch,
>it will be used to protect the whole IOV capability, ensure accessing the
>IOV capability exclusively. So the usage of this lock is changed.
>
> code extracted from pci.h:
>
> struct pci_sriov {
>:
>struct mutex lock;  /* lock for VF bus */
>:
> }
>
>The lock was introduced by commit d1b054da8 ("PCI: initialize and release
>SR-IOV capability"). If I'm correct enough, I don't think this lock is needed 
>when
>pci_enable_sriov() or pci_disable_sriov() are called in driver because of
>module
>parameters. I don't see the usage case calling pci_disable_sriov() while
>previous pci_enable_sriov() isn't finished yet. Also, it's not needed in EEH
>subsystem.
>So I think the lock can be dropped, then it can be used to protect sysfs path.

That's pretty much what this patch does, except I kept the locking for EEH 
since 
it is the only driver that calls pci_iov_add/remove_virtfn() directly.

I'll write it up and run some tests, although I have no way to test EEH.
 
>>

RE: [PATCH 2/2] PCI: lock each enable/disable num_vfs operation in sysfs

[Tantilov, Emil S]

>-Original Message-
>From: Gavin Shan [mailto:gws...@linux.vnet.ibm.com]
>Sent: Tuesday, January 03, 2017 6:16 PM
>To: Tantilov, Emil S <emil.s.tanti...@intel.com>
>Cc: linux-...@vger.kernel.org; intel-wired-...@lists.osuosl.org; Duyck,
>Alexander H <alexander.h.du...@intel.com>; net...@vger.kernel.org; linux-
>ker...@vger.kernel.org
>Subject: Re: [PATCH 2/2] PCI: lock each enable/disable num_vfs operation in
>sysfs
>
>On Tue, Jan 03, 2017 at 04:48:31PM -0800, Emil Tantilov wrote:
>>Enabling/disabling SRIOV via sysfs by echo-ing multiple values
>>simultaneously:
>>
>>echo 63 > /sys/class/net/ethX/device/sriov_numvfs&
>>echo 63 > /sys/class/net/ethX/device/sriov_numvfs
>>
>>sleep 5
>>
>>echo 0 > /sys/class/net/ethX/device/sriov_numvfs&
>>echo 0 > /sys/class/net/ethX/device/sriov_numvfs
>>
>>Results in the following bug:
>>
>>kernel BUG at drivers/pci/iov.c:495!
>>invalid opcode:  [#1] SMP
>>CPU: 1 PID: 8050 Comm: bash Tainted: G   W   4.9.0-rc7-net-next #2092
>>RIP: 0010:[]
>>[] pci_iov_release+0x57/0x60
>>
>>Call Trace:
>> [] pci_release_dev+0x26/0x70
>> [] device_release+0x3e/0xb0
>> [] kobject_cleanup+0x67/0x180
>> [] kobject_put+0x2d/0x60
>> [] put_device+0x17/0x20
>> [] pci_dev_put+0x1a/0x20
>> [] pci_get_dev_by_id+0x5b/0x90
>> [] pci_get_subsys+0x35/0x40
>> [] pci_get_device+0x18/0x20
>> [] pci_get_domain_bus_and_slot+0x2b/0x60
>> [] pci_iov_remove_virtfn+0x57/0x180
>> [] pci_disable_sriov+0x65/0x140
>> [] ixgbe_disable_sriov+0xc7/0x1d0 [ixgbe]
>> [] ixgbe_pci_sriov_configure+0x3d/0x170 [ixgbe]
>> [] sriov_numvfs_store+0xdc/0x130
>>...
>>RIP  [] pci_iov_release+0x57/0x60
>>
>>Use the existing mutex lock to protect each enable/disable operation.
>>
>>CC: Alexander Duyck <alexander.h.du...@intel.com>
>>Signed-off-by: Emil Tantilov <emil.s.tanti...@intel.com>
>
>Emil, It's going to change semantics of pci_enable_sriov() and 
>pci_disable_sriov().
>They can be invoked when writing to the sysfs entry, or loading PF's
>driver. With the change applied, the lock (pf->sriov->lock) isn't acquired and 
>released
>in the PF's driver loading path.

The enablement of SRIOV on driver load is done via deprecated module parameter.
Perhaps we can just remove it, although there are probably still people that 
use it
and may not be happy if we get rid of it. 

>I think the reasonable way would be adding a flag in "struct sriov", to
>indicate someone is accessing the IOV capability through sysfs file. With 
>this, the
>code returns with "-EBUSY" immediately for contenders. With it, nothing is 
>going
>to be changed in PF's driver loading path.

Flag is what I initially had in mind, but did not want to add extra locking if 
we 
can make use of the existing.

>Also, there are some minor comments as below and I guess most of them won't
>be applied if you take my suggestion eventually. However, I'm trying to make
>the comments complete.

Thanks a lot for reviewing!

>
>>---
>> drivers/pci/pci-sysfs.c |   24 +---
>> 1 file changed, 17 insertions(+), 7 deletions(-)
>>
>>diff --git a/drivers/pci/pci-sysfs.c b/drivers/pci/pci-sysfs.c
>>index 0666287..5b54cf5 100644
>>--- a/drivers/pci/pci-sysfs.c
>>+++ b/drivers/pci/pci-sysfs.c
>>@@ -472,7 +472,9 @@ static ssize_t sriov_numvfs_store(struct device *dev,
>>const char *buf, size_t count)
>> {
>>  struct pci_dev *pdev = to_pci_dev(dev);
>>+ struct pci_sriov *iov = pdev->sriov;
>>  int ret;
>>+
>
>Unnecessary change.
>
>>  u16 num_vfs;
>>
>>  ret = kstrtou16(buf, 0, _vfs);
>>@@ -482,38 +484,46 @@ static ssize_t sriov_numvfs_store(struct device
>*dev,
>>  if (num_vfs > pci_sriov_get_totalvfs(pdev))
>>  return -ERANGE;
>>
>>+ mutex_lock(>dev->sriov->lock);
>>+
>>  if (num_vfs == pdev->sriov->num_VFs)
>>- return count;   /* no change */
>>+ goto exit;
>>
>>  /* is PF driver loaded w/callback */
>>  if (!pdev->driver || !pdev->driver->sriov_configure) {
>>  dev_info(>dev, "Driver doesn't support SRIOV
>configuration via sysfs\n");
>>- return -ENOSYS;
>>+ ret = -EINVAL;
>>+ goto exit;
>
>Why we need change the error code here?

checkpatch was complaining about the use of the ENOSYS error code being specific
and even though i

RE: [PATCH 2/2] PCI: lock each enable/disable num_vfs operation in sysfs

[Tantilov, Emil S]

>-Original Message-
>From: Gavin Shan [mailto:gws...@linux.vnet.ibm.com]
>Sent: Tuesday, January 03, 2017 6:16 PM
>To: Tantilov, Emil S 
>Cc: linux-...@vger.kernel.org; intel-wired-...@lists.osuosl.org; Duyck,
>Alexander H ; net...@vger.kernel.org; linux-
>ker...@vger.kernel.org
>Subject: Re: [PATCH 2/2] PCI: lock each enable/disable num_vfs operation in
>sysfs
>
>On Tue, Jan 03, 2017 at 04:48:31PM -0800, Emil Tantilov wrote:
>>Enabling/disabling SRIOV via sysfs by echo-ing multiple values
>>simultaneously:
>>
>>echo 63 > /sys/class/net/ethX/device/sriov_numvfs&
>>echo 63 > /sys/class/net/ethX/device/sriov_numvfs
>>
>>sleep 5
>>
>>echo 0 > /sys/class/net/ethX/device/sriov_numvfs&
>>echo 0 > /sys/class/net/ethX/device/sriov_numvfs
>>
>>Results in the following bug:
>>
>>kernel BUG at drivers/pci/iov.c:495!
>>invalid opcode:  [#1] SMP
>>CPU: 1 PID: 8050 Comm: bash Tainted: G   W   4.9.0-rc7-net-next #2092
>>RIP: 0010:[]
>>[] pci_iov_release+0x57/0x60
>>
>>Call Trace:
>> [] pci_release_dev+0x26/0x70
>> [] device_release+0x3e/0xb0
>> [] kobject_cleanup+0x67/0x180
>> [] kobject_put+0x2d/0x60
>> [] put_device+0x17/0x20
>> [] pci_dev_put+0x1a/0x20
>> [] pci_get_dev_by_id+0x5b/0x90
>> [] pci_get_subsys+0x35/0x40
>> [] pci_get_device+0x18/0x20
>> [] pci_get_domain_bus_and_slot+0x2b/0x60
>> [] pci_iov_remove_virtfn+0x57/0x180
>> [] pci_disable_sriov+0x65/0x140
>> [] ixgbe_disable_sriov+0xc7/0x1d0 [ixgbe]
>> [] ixgbe_pci_sriov_configure+0x3d/0x170 [ixgbe]
>> [] sriov_numvfs_store+0xdc/0x130
>>...
>>RIP  [] pci_iov_release+0x57/0x60
>>
>>Use the existing mutex lock to protect each enable/disable operation.
>>
>>CC: Alexander Duyck 
>>Signed-off-by: Emil Tantilov 
>
>Emil, It's going to change semantics of pci_enable_sriov() and 
>pci_disable_sriov().
>They can be invoked when writing to the sysfs entry, or loading PF's
>driver. With the change applied, the lock (pf->sriov->lock) isn't acquired and 
>released
>in the PF's driver loading path.

The enablement of SRIOV on driver load is done via deprecated module parameter.
Perhaps we can just remove it, although there are probably still people that 
use it
and may not be happy if we get rid of it. 

>I think the reasonable way would be adding a flag in "struct sriov", to
>indicate someone is accessing the IOV capability through sysfs file. With 
>this, the
>code returns with "-EBUSY" immediately for contenders. With it, nothing is 
>going
>to be changed in PF's driver loading path.

Flag is what I initially had in mind, but did not want to add extra locking if 
we 
can make use of the existing.

>Also, there are some minor comments as below and I guess most of them won't
>be applied if you take my suggestion eventually. However, I'm trying to make
>the comments complete.

Thanks a lot for reviewing!

>
>>---
>> drivers/pci/pci-sysfs.c |   24 +---
>> 1 file changed, 17 insertions(+), 7 deletions(-)
>>
>>diff --git a/drivers/pci/pci-sysfs.c b/drivers/pci/pci-sysfs.c
>>index 0666287..5b54cf5 100644
>>--- a/drivers/pci/pci-sysfs.c
>>+++ b/drivers/pci/pci-sysfs.c
>>@@ -472,7 +472,9 @@ static ssize_t sriov_numvfs_store(struct device *dev,
>>const char *buf, size_t count)
>> {
>>  struct pci_dev *pdev = to_pci_dev(dev);
>>+ struct pci_sriov *iov = pdev->sriov;
>>  int ret;
>>+
>
>Unnecessary change.
>
>>  u16 num_vfs;
>>
>>  ret = kstrtou16(buf, 0, _vfs);
>>@@ -482,38 +484,46 @@ static ssize_t sriov_numvfs_store(struct device
>*dev,
>>  if (num_vfs > pci_sriov_get_totalvfs(pdev))
>>  return -ERANGE;
>>
>>+ mutex_lock(>dev->sriov->lock);
>>+
>>  if (num_vfs == pdev->sriov->num_VFs)
>>- return count;   /* no change */
>>+ goto exit;
>>
>>  /* is PF driver loaded w/callback */
>>  if (!pdev->driver || !pdev->driver->sriov_configure) {
>>  dev_info(>dev, "Driver doesn't support SRIOV
>configuration via sysfs\n");
>>- return -ENOSYS;
>>+ ret = -EINVAL;
>>+ goto exit;
>
>Why we need change the error code here?

checkpatch was complaining about the use of the ENOSYS error code being specific
and even though it was not my patch introducing it I had to change it to shut 
it up.

>>  }
>>
>>  if (num_vfs == 0) {
>&g

RE: [PATCH] ixgbe: mark symbols static where possible

[Tantilov, Emil S]

>-Original Message-
>From: netdev-ow...@vger.kernel.org [mailto:netdev-ow...@vger.kernel.org] On
>Behalf Of Baoyou Xie
>Sent: Sunday, September 18, 2016 1:48 AM
>To: Kirsher, Jeffrey T ; intel-wired-
>l...@lists.osuosl.org
>Cc: net...@vger.kernel.org; linux-kernel@vger.kernel.org; a...@arndb.de;
>baoyou@linaro.org; xie.bao...@zte.com.cn
>Subject: [PATCH] ixgbe: mark symbols static where possible
>
>We get 2 warnings when building kernel with W=1:
>drivers/net/ethernet/intel/ixgbe/ixgbe_x550.c:2128:5: warning: no previous
>prototype for 'ixgbe_led_on_t_x550em' [-Wmissing-prototypes]
>drivers/net/ethernet/intel/ixgbe/ixgbe_x550.c:2150:5: warning: no previous
>prototype for 'ixgbe_led_off_t_x550em' [-Wmissing-prototypes]

Already taken care of:
http://patchwork.ozlabs.org/patch/661647/

Thanks,
Emil

RE: [PATCH] ixgbe: mark symbols static where possible

[Tantilov, Emil S]

>-Original Message-
>From: netdev-ow...@vger.kernel.org [mailto:netdev-ow...@vger.kernel.org] On
>Behalf Of Baoyou Xie
>Sent: Sunday, September 18, 2016 1:48 AM
>To: Kirsher, Jeffrey T ; intel-wired-
>l...@lists.osuosl.org
>Cc: net...@vger.kernel.org; linux-kernel@vger.kernel.org; a...@arndb.de;
>baoyou@linaro.org; xie.bao...@zte.com.cn
>Subject: [PATCH] ixgbe: mark symbols static where possible
>
>We get 2 warnings when building kernel with W=1:
>drivers/net/ethernet/intel/ixgbe/ixgbe_x550.c:2128:5: warning: no previous
>prototype for 'ixgbe_led_on_t_x550em' [-Wmissing-prototypes]
>drivers/net/ethernet/intel/ixgbe/ixgbe_x550.c:2150:5: warning: no previous
>prototype for 'ixgbe_led_off_t_x550em' [-Wmissing-prototypes]

Already taken care of:
http://patchwork.ozlabs.org/patch/661647/

Thanks,
Emil

RE: [E1000-devel] [PATCH v2 3/3] ixgbe: Add new ndo to allow VF multicast promiscuous mode

[Tantilov, Emil S]

>-Original Message-
>From: Hiroshi Shimamoto [mailto:h-shimam...@ct.jp.nec.com] 
>Sent: Thursday, February 19, 2015 5:01 PM
> Subject: [E1000-devel] [PATCH v2 3/3] ixgbe: Add new ndo to allow VF 
> multicast promiscuous mode
>
>From: Hiroshi Shimamoto 
>
>Implements the new netdev op to allow VF multicast promiscuous mode.
>
>The administrator can allow to VF multicast promiscuous mode for only
>trusted VM. After allowing multicast promiscuous mode from the host,
>we can use over 30 IPv6 addresses on VM.
> # ./ip link set dev eth0 vf 1 mc_promisc on
>
>When disallowing multicast promiscuous mode, we can only use 30 IPv6 addresses.
> # ./ip link set dev eth0 vf 1 mc_promisc off
>
>Signed-off-by: Hiroshi Shimamoto 
>Reviewed-by: Hayato Momma 
>CC: Choi, Sy Jong 



+int ixgbe_ndo_set_vf_mc_promisc(struct net_device *netdev, int vf, bool 
setting)
+{
+   struct ixgbe_adapter *adapter = netdev_priv(netdev);
+   struct ixgbe_hw *hw = >hw;
+   u32 vmolr;

vmolr is unused variable in this function.

+
+   if (vf >= adapter->num_vfs)
+   return -EINVAL;
+
+   /* nothing to do */
+   if (adapter->vfinfo[vf].mc_promisc_allowed == setting)
+   return 0;
+
+   adapter->vfinfo[vf].mc_promisc_allowed = setting;
+
+   /* if VF requests multicast promiscuous */
+   if (adapter->vfinfo[vf].mc_promisc) {
+   if (setting)
+   ixgbe_enable_vf_mc_promisc(adapter, vf);
+   else
+   ixgbe_disable_vf_mc_promisc(adapter, vf);
+   }
+
+   return 0;
+}

Thanks,
Emil

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

RE: [E1000-devel] [PATCH v2 3/3] ixgbe: Add new ndo to allow VF multicast promiscuous mode

[Tantilov, Emil S]

-Original Message-
From: Hiroshi Shimamoto [mailto:h-shimam...@ct.jp.nec.com] 
Sent: Thursday, February 19, 2015 5:01 PM
 Subject: [E1000-devel] [PATCH v2 3/3] ixgbe: Add new ndo to allow VF 
 multicast promiscuous mode

From: Hiroshi Shimamoto h-shimam...@ct.jp.nec.com

Implements the new netdev op to allow VF multicast promiscuous mode.

The administrator can allow to VF multicast promiscuous mode for only
trusted VM. After allowing multicast promiscuous mode from the host,
we can use over 30 IPv6 addresses on VM.
 # ./ip link set dev eth0 vf 1 mc_promisc on

When disallowing multicast promiscuous mode, we can only use 30 IPv6 addresses.
 # ./ip link set dev eth0 vf 1 mc_promisc off

Signed-off-by: Hiroshi Shimamoto h-shimam...@ct.jp.nec.com
Reviewed-by: Hayato Momma h-mo...@ce.jp.nec.com
CC: Choi, Sy Jong sy.jong.c...@intel.com

snip

+int ixgbe_ndo_set_vf_mc_promisc(struct net_device *netdev, int vf, bool 
setting)
+{
+   struct ixgbe_adapter *adapter = netdev_priv(netdev);
+   struct ixgbe_hw *hw = adapter-hw;
+   u32 vmolr;

vmolr is unused variable in this function.

+
+   if (vf = adapter-num_vfs)
+   return -EINVAL;
+
+   /* nothing to do */
+   if (adapter-vfinfo[vf].mc_promisc_allowed == setting)
+   return 0;
+
+   adapter-vfinfo[vf].mc_promisc_allowed = setting;
+
+   /* if VF requests multicast promiscuous */
+   if (adapter-vfinfo[vf].mc_promisc) {
+   if (setting)
+   ixgbe_enable_vf_mc_promisc(adapter, vf);
+   else
+   ixgbe_disable_vf_mc_promisc(adapter, vf);
+   }
+
+   return 0;
+}

Thanks,
Emil

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

RE: 3.19: ixgbe 0000:01:00.0 eth4: initiating reset due to tx timeout

[Tantilov, Emil S]

>-Original Message-
>From: linux-kernel-ow...@vger.kernel.org 
>[mailto:linux-kernel-ow...@vger.kernel.org] On Behalf Of Justin Piszcz
>Sent: Sunday, February 22, 2015 4:01 AM
>To: linux-kernel@vger.kernel.org
>Subject: 3.19: ixgbe :01:00.0 eth4: initiating reset due to tx timeout
>
>Hello,
>
>Kernel: 3.19.0
>Issue: When using robocopy to copy files (from Windows 8/8.1) to
>Linux/samba, the 10GbE NIC resets - dmesg [1] below.  To get it back working
>again, I have to down/up the interface.  Jumbo frames are being used (mtu of
>9014) on each side. The lspci output is listed below.  Are there any other
>recommended workarounds for this issue as LRO is already off for me as shown
>below.  When using Linux<->Linux with rsync or NFS, there are no errors with
>10GbE.  When using Samba<->Windows 8 over 10GbE, this issue occurs
>persistently as shown below when a copy is running.
>
># ethtool -k eth4|grep large
>large-receive-offload: off [fixed]

The issue is a Tx timeout, so LRO is unlikely to have an effect. Is the 
interface that hangs (eth4) mostly receiving or transmitting? Posting the stats 
(ethtool -S eth4) would help here.

>There is/was a similar issue as reported here:
>https://communities.intel.com/message/207408
>
> [1] dmesg
>
> [538576.098186] ixgbe :01:00.0 eth4: NIC Link is Up 10 Gbps, Flow 
> Control: RX/TX
> [541013.223961] [ cut here ]
> [541013.223970] WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:303 
> dev_watchdog+0x227/0x230()
> [541013.223971] NETDEV WATCHDOG: eth4 (ixgbe): transmit queue 0 timed out
> [541013.223972] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.19.0 #2
> [541013.223973] Hardware name: Supermicro X9SRL-F/X9SRL-F, BIOS 3.0a 
> 12/05/2013
> [541013.223974]  81d3a6ae 88107fc03da8 819d07d7 
> 81e34d98
> [541013.223976]  88107fc03df8 88107fc03de8 810dbdab 
> 
> [541013.223977]   881036304000  
> 0010
> [541013.223979] Call Trace:
> [541013.223979][] dump_stack+0x45/0x57
> [541013.223985]  [] warn_slowpath_common+0x7b/0xc0
> [541013.223987]  [] warn_slowpath_fmt+0x41/0x50
> [541013.223990]  [] ? __queue_work+0xfc/0x290
> [541013.223996]  [] dev_watchdog+0x227/0x230
> [541013.223997]  [] ? qdisc_rcu_free+0x40/0x40
> [541013.223998]  [] ? qdisc_rcu_free+0x40/0x40
> [541013.224001]  [] call_timer_fn.isra.29+0x17/0x80
> [541013.224002]  [] run_timer_softirq+0x1c9/0x280
> [541013.224004]  [] __do_softirq+0xff/0x200
> [541013.224005]  [] irq_exit+0x76/0xa0
> [541013.224007]  [] smp_apic_timer_interrupt+0x41/0x50
> [541013.224009]  [] apic_timer_interrupt+0x6a/0x70
> [541013.224009][] ? cpuidle_enter_state+0x48/0xc0
> [541013.224013]  [] ? cpuidle_enter_state+0x3d/0xc0
> [541013.224014]  [] cpuidle_enter+0x12/0x20
> [541013.224017]  [] cpu_startup_entry+0x272/0x2f0
> [541013.224018]  [] rest_init+0x6d/0x70
> [541013.224021]  [] start_kernel+0x353/0x360
> [541013.224022]  [] x86_64_start_reservations+0x2a/0x2c
> [541013.224023]  [] x86_64_start_kernel+0xc8/0xcc
> [541013.224024] ---[ end trace 59877113cf8b7358 ]---
> [541013.224026] ixgbe :01:00.0 eth4: initiating reset due to tx timeout
> [541013.224036] ixgbe :01:00.0 eth4: Reset adapter
> [541020.099402] ixgbe :01:00.0 eth4: NIC Link is Up 10 Gbps, Flow 
> Control: RX/TX
>
> ( .. it continue but without the trace later .. )
>
> [567457.771728] ixgbe :01:00.0 eth4: NIC Link is Down
> [567458.140112] ixgbe :01:00.0 eth4: NIC Link is Up 10 Gbps, Flow 
> Control: RX/TX
> [567561.611941] ixgbe :01:00.0 eth4: NIC Link is Down
> [567568.188422] ixgbe :01:00.0 eth4: NIC Link is Up 10 Gbps, Flow 
> Control: RX/TX
> [570130.483823] ixgbe :01:00.0 eth4: initiating reset due to tx timeout
> [570130.483924] ixgbe :01:00.0 eth4: Reset adapter

The reset is a side effect of the Tx hang - the driver is trying to recover 
from the hang by resetting the interface.

If you could open up a ticket at e1000.sf.net with details about your setup and 
how you configure the interfaces that would help us get a better idea of the 
issue. You can also upload the stats, kernel config and any other logs that may 
be relevant.

Thanks,
Emil


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

RE: 3.19: ixgbe 0000:01:00.0 eth4: initiating reset due to tx timeout

[Tantilov, Emil S]

-Original Message-
From: linux-kernel-ow...@vger.kernel.org 
[mailto:linux-kernel-ow...@vger.kernel.org] On Behalf Of Justin Piszcz
Sent: Sunday, February 22, 2015 4:01 AM
To: linux-kernel@vger.kernel.org
Subject: 3.19: ixgbe :01:00.0 eth4: initiating reset due to tx timeout

Hello,

Kernel: 3.19.0
Issue: When using robocopy to copy files (from Windows 8/8.1) to
Linux/samba, the 10GbE NIC resets - dmesg [1] below.  To get it back working
again, I have to down/up the interface.  Jumbo frames are being used (mtu of
9014) on each side. The lspci output is listed below.  Are there any other
recommended workarounds for this issue as LRO is already off for me as shown
below.  When using Linux-Linux with rsync or NFS, there are no errors with
10GbE.  When using Samba-Windows 8 over 10GbE, this issue occurs
persistently as shown below when a copy is running.

# ethtool -k eth4|grep large
large-receive-offload: off [fixed]

The issue is a Tx timeout, so LRO is unlikely to have an effect. Is the 
interface that hangs (eth4) mostly receiving or transmitting? Posting the stats 
(ethtool -S eth4) would help here.

There is/was a similar issue as reported here:
https://communities.intel.com/message/207408

 [1] dmesg

 [538576.098186] ixgbe :01:00.0 eth4: NIC Link is Up 10 Gbps, Flow 
 Control: RX/TX
 [541013.223961] [ cut here ]
 [541013.223970] WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:303 
 dev_watchdog+0x227/0x230()
 [541013.223971] NETDEV WATCHDOG: eth4 (ixgbe): transmit queue 0 timed out
 [541013.223972] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.19.0 #2
 [541013.223973] Hardware name: Supermicro X9SRL-F/X9SRL-F, BIOS 3.0a 
 12/05/2013
 [541013.223974]  81d3a6ae 88107fc03da8 819d07d7 
 81e34d98
 [541013.223976]  88107fc03df8 88107fc03de8 810dbdab 
 
 [541013.223977]   881036304000  
 0010
 [541013.223979] Call Trace:
 [541013.223979]  IRQ  [819d07d7] dump_stack+0x45/0x57
 [541013.223985]  [810dbdab] warn_slowpath_common+0x7b/0xc0
 [541013.223987]  [810dbe61] warn_slowpath_fmt+0x41/0x50
 [541013.223990]  [810eec4c] ? __queue_work+0xfc/0x290
 [541013.223996]  [818ef0a7] dev_watchdog+0x227/0x230
 [541013.223997]  [818eee80] ? qdisc_rcu_free+0x40/0x40
 [541013.223998]  [818eee80] ? qdisc_rcu_free+0x40/0x40
 [541013.224001]  [811251f7] call_timer_fn.isra.29+0x17/0x80
 [541013.224002]  [81125429] run_timer_softirq+0x1c9/0x280
 [541013.224004]  [810dec7f] __do_softirq+0xff/0x200
 [541013.224005]  [810deea6] irq_exit+0x76/0xa0
 [541013.224007]  [8106ac11] smp_apic_timer_interrupt+0x41/0x50
 [541013.224009]  [819da6aa] apic_timer_interrupt+0x6a/0x70
 [541013.224009]  EOI  [8184e8f8] ? cpuidle_enter_state+0x48/0xc0
 [541013.224013]  [8184e8ed] ? cpuidle_enter_state+0x3d/0xc0
 [541013.224014]  [8184ea42] cpuidle_enter+0x12/0x20
 [541013.224017]  [8110f222] cpu_startup_entry+0x272/0x2f0
 [541013.224018]  [819cdd5d] rest_init+0x6d/0x70
 [541013.224021]  [81ef0dbb] start_kernel+0x353/0x360
 [541013.224022]  [81ef0495] x86_64_start_reservations+0x2a/0x2c
 [541013.224023]  [81ef055f] x86_64_start_kernel+0xc8/0xcc
 [541013.224024] ---[ end trace 59877113cf8b7358 ]---
 [541013.224026] ixgbe :01:00.0 eth4: initiating reset due to tx timeout
 [541013.224036] ixgbe :01:00.0 eth4: Reset adapter
 [541020.099402] ixgbe :01:00.0 eth4: NIC Link is Up 10 Gbps, Flow 
 Control: RX/TX

 ( .. it continue but without the trace later .. )

 [567457.771728] ixgbe :01:00.0 eth4: NIC Link is Down
 [567458.140112] ixgbe :01:00.0 eth4: NIC Link is Up 10 Gbps, Flow 
 Control: RX/TX
 [567561.611941] ixgbe :01:00.0 eth4: NIC Link is Down
 [567568.188422] ixgbe :01:00.0 eth4: NIC Link is Up 10 Gbps, Flow 
 Control: RX/TX
 [570130.483823] ixgbe :01:00.0 eth4: initiating reset due to tx timeout
 [570130.483924] ixgbe :01:00.0 eth4: Reset adapter

The reset is a side effect of the Tx hang - the driver is trying to recover 
from the hang by resetting the interface.

If you could open up a ticket at e1000.sf.net with details about your setup and 
how you configure the interfaces that would help us get a better idea of the 
issue. You can also upload the stats, kernel config and any other logs that may 
be relevant.

Thanks,
Emil


--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

RE: [PATCH] ixgbe: Re-enable relaxed ordering as part of init/restart sequence for non-DCA config

[Tantilov, Emil S]

>-Original Message-
>From: Sowmini Varadhan [mailto:sowmini.varad...@oracle.com] 
>Relaxed ordering is disabled by default at driver initialization
>and re-enabled when DCA is used. The reason it is disabled  was
>due to an issue on some chipsets (see comments in ixgbe_update_tx_dca()).
>But when DCA is not used, RO needs to be re-enabled, else we have
>a serialization bottleneck on platforms like SPARC.
>
>This patch eliminates the bottleneck for ixgbe when DCA is not configured.
>
>Signed-off-by: Sowmini Varadhan 
>Cc: Emil Tantilov 
>---
> drivers/net/ethernet/intel/ixgbe/ixgbe_82598.c  |1 +
> drivers/net/ethernet/intel/ixgbe/ixgbe_common.c |   20 
> drivers/net/ethernet/intel/ixgbe/ixgbe_common.h |1 +
> drivers/net/ethernet/intel/ixgbe/ixgbe_main.c   |   11 +++
> drivers/net/ethernet/intel/ixgbe/ixgbe_type.h   |1 +
> drivers/net/ethernet/intel/ixgbe/ixgbe_x540.c   |1 +
> 6 files changed, 35 insertions(+), 0 deletions(-)
>
>diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_82598.c 
>b/drivers/net/ethernet/intel/ixgbe/ixgbe_82598.c
>index c5c97b4..85c7a28 100644
>--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_82598.c
>+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_82598.c
>@@ -1161,6 +1161,7 @@ static struct ixgbe_mac_operations mac_ops_82598 = {
>   .clear_hw_cntrs = _clear_hw_cntrs_generic,
>   .get_media_type = _get_media_type_82598,
>   .enable_rx_dma  = _enable_rx_dma_generic,
>+  .enable_relaxed_ordering = _enable_relaxed_ordering,

The IXGBE_DCA_TXCTRL register for 82598 is at a different offset. Also there is 
a limit of 16 registers. The function we have in our code for 82598 is as 
follows:

/**
 *  ixgbe_enable_relaxed_ordering_82598 - enable relaxed ordering
 *  @hw: pointer to hardware structure
 *
 **/
void ixgbe_enable_relaxed_ordering_82598(struct ixgbe_hw *hw)
{
u32 regval;
u32 i;

/* Enable relaxed ordering */
for (i = 0; ((i < hw->mac.max_tx_queues) &&
 (i < IXGBE_DCA_MAX_QUEUES_82598)); i++) {
regval = IXGBE_READ_REG(hw, IXGBE_DCA_TXCTRL(i));
regval |= IXGBE_DCA_TXCTRL_DESC_WRO_EN;
IXGBE_WRITE_REG(hw, IXGBE_DCA_TXCTRL(i), regval);
}

for (i = 0; ((i < hw->mac.max_rx_queues) &&
 (i < IXGBE_DCA_MAX_QUEUES_82598)); i++) {
regval = IXGBE_READ_REG(hw, IXGBE_DCA_RXCTRL(i));
regval |= IXGBE_DCA_RXCTRL_DATA_WRO_EN |
  IXGBE_DCA_RXCTRL_HEAD_WRO_EN;
IXGBE_WRITE_REG(hw, IXGBE_DCA_RXCTRL(i), regval);
}
}

Thanks,
Emil
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

RE: [PATCH] ixgbe: Re-enable relaxed ordering as part of init/restart sequence for non-DCA config

[Tantilov, Emil S]

-Original Message-
From: Sowmini Varadhan [mailto:sowmini.varad...@oracle.com] 
Relaxed ordering is disabled by default at driver initialization
and re-enabled when DCA is used. The reason it is disabled  was
due to an issue on some chipsets (see comments in ixgbe_update_tx_dca()).
But when DCA is not used, RO needs to be re-enabled, else we have
a serialization bottleneck on platforms like SPARC.

This patch eliminates the bottleneck for ixgbe when DCA is not configured.

Signed-off-by: Sowmini Varadhan sowmini.varad...@oracle.com
Cc: Emil Tantilov emil.s.tanti...@intel.com
---
 drivers/net/ethernet/intel/ixgbe/ixgbe_82598.c  |1 +
 drivers/net/ethernet/intel/ixgbe/ixgbe_common.c |   20 
 drivers/net/ethernet/intel/ixgbe/ixgbe_common.h |1 +
 drivers/net/ethernet/intel/ixgbe/ixgbe_main.c   |   11 +++
 drivers/net/ethernet/intel/ixgbe/ixgbe_type.h   |1 +
 drivers/net/ethernet/intel/ixgbe/ixgbe_x540.c   |1 +
 6 files changed, 35 insertions(+), 0 deletions(-)

diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_82598.c 
b/drivers/net/ethernet/intel/ixgbe/ixgbe_82598.c
index c5c97b4..85c7a28 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_82598.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_82598.c
@@ -1161,6 +1161,7 @@ static struct ixgbe_mac_operations mac_ops_82598 = {
   .clear_hw_cntrs = ixgbe_clear_hw_cntrs_generic,
   .get_media_type = ixgbe_get_media_type_82598,
   .enable_rx_dma  = ixgbe_enable_rx_dma_generic,
+  .enable_relaxed_ordering = ixgbe_enable_relaxed_ordering,

The IXGBE_DCA_TXCTRL register for 82598 is at a different offset. Also there is 
a limit of 16 registers. The function we have in our code for 82598 is as 
follows:

/**
 *  ixgbe_enable_relaxed_ordering_82598 - enable relaxed ordering
 *  @hw: pointer to hardware structure
 *
 **/
void ixgbe_enable_relaxed_ordering_82598(struct ixgbe_hw *hw)
{
u32 regval;
u32 i;

/* Enable relaxed ordering */
for (i = 0; ((i  hw-mac.max_tx_queues) 
 (i  IXGBE_DCA_MAX_QUEUES_82598)); i++) {
regval = IXGBE_READ_REG(hw, IXGBE_DCA_TXCTRL(i));
regval |= IXGBE_DCA_TXCTRL_DESC_WRO_EN;
IXGBE_WRITE_REG(hw, IXGBE_DCA_TXCTRL(i), regval);
}

for (i = 0; ((i  hw-mac.max_rx_queues) 
 (i  IXGBE_DCA_MAX_QUEUES_82598)); i++) {
regval = IXGBE_READ_REG(hw, IXGBE_DCA_RXCTRL(i));
regval |= IXGBE_DCA_RXCTRL_DATA_WRO_EN |
  IXGBE_DCA_RXCTRL_HEAD_WRO_EN;
IXGBE_WRITE_REG(hw, IXGBE_DCA_RXCTRL(i), regval);
}
}

Thanks,
Emil
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

RE: Problems with ixgbe driver

[Tantilov, Emil S]

>-Original Message-
>From: Holger Kiehl [mailto:holger.ki...@dwd.de]
>Sent: Monday, June 17, 2013 2:12 AM
>To: Tantilov, Emil S
>Cc: e1000-de...@lists.sf.net; linux-kernel; net...@vger.kernel.org
>Subject: RE: Problems with ixgbe driver
>
>Hello,
>
>first, thank you for the quick help!
>
>On Fri, 14 Jun 2013, Tantilov, Emil S wrote:
>
>>> -Original Message-
>>> From: netdev-ow...@vger.kernel.org [mailto:netdev-ow...@vger.kernel.org]
>On
>>> Behalf Of Holger Kiehl
>>> Sent: Friday, June 14, 2013 4:50 AM
>>> To: e1000-de...@lists.sf.net
>>> Cc: linux-kernel; net...@vger.kernel.org
>>> Subject: Problems with ixgbe driver
>>>
>>> Hello,
>>>
>>> I have dual port 10Gb Intel network card on a 2 socket (Xeon X5690) with
>>> a total of 12 cores. Hyperthreading is enabled so there are 24 cores.
>>> The problem I have is that when other systems send large amount of data
>>> the network with the intel ixgbe driver gets very slow. Ping times go up
>>> from 0.2ms to appr. 60ms. Some FTP connections stall for more then 2
>>> minutes. What is strange is that heatbeat is configured on the system
>>> with a serial connection to another node and kernel always reports
>>
>> If the network slows down so much there should be some indication in
>dmesg. Like Tx hangs perhaps.
>> Can you provide the output of dmesg and ethtool -S from the offending
>interface after the issue occurs?
>>
>No, there is absolute no indication in dmesg or /var/log/messages. But here
>the ethtool output when ping times go up:
>
>root@helena:~# ethtool -S eth6
>NIC statistics:
> rx_packets: 4410779
> tx_packets: 8902514
> rx_bytes: 2014041824
> tx_bytes: 13199913202
> rx_errors: 0
> tx_errors: 0
> rx_dropped: 0
> tx_dropped: 0
> multicast: 4245
> collisions: 0
> rx_over_errors: 0
> rx_crc_errors: 0
> rx_frame_errors: 0
> rx_fifo_errors: 0
> rx_missed_errors: 28143
> tx_aborted_errors: 0
> tx_carrier_errors: 0
> tx_fifo_errors: 0
> tx_heartbeat_errors: 0
> rx_pkts_nic: 2401276937
> tx_pkts_nic: 3868619482
> rx_bytes_nic: 868282794731
> tx_bytes_nic: 5743382228649
> lsc_int: 4
> tx_busy: 0
> non_eop_descs: 743957
> broadcast: 1745556
> rx_no_buffer_count: 0
> tx_timeout_count: 0
> tx_restart_queue: 425
> rx_long_length_errors: 0
> rx_short_length_errors: 0
> tx_flow_control_xon: 171
> rx_flow_control_xon: 0
> tx_flow_control_xoff: 277
> rx_flow_control_xoff: 0
> rx_csum_offload_errors: 0
> alloc_rx_page_failed: 0
> alloc_rx_buff_failed: 0
> lro_aggregated: 0
> lro_flushed: 0
> rx_no_dma_resources: 0
> hw_rsc_aggregated: 1153374
> hw_rsc_flushed: 129169
> fdir_match: 2424508153
> fdir_miss: 1706029
> fdir_overflow: 33
> os2bmc_rx_by_bmc: 0
> os2bmc_tx_by_bmc: 0
> os2bmc_tx_by_host: 0
> os2bmc_rx_by_host: 0
> tx_queue_0_packets: 470182
> tx_queue_0_bytes: 690123121
> tx_queue_1_packets: 797784
> tx_queue_1_bytes: 1203968369
> tx_queue_2_packets: 648692
> tx_queue_2_bytes: 950171718
> tx_queue_3_packets: 647434
> tx_queue_3_bytes: 948647518
> tx_queue_4_packets: 263216
> tx_queue_4_bytes: 394806409
> tx_queue_5_packets: 426786
> tx_queue_5_bytes: 629387628
> tx_queue_6_packets: 253708
> tx_queue_6_bytes: 371774276
> tx_queue_7_packets: 544634
> tx_queue_7_bytes: 812223169
> tx_queue_8_packets: 279056
> tx_queue_8_bytes: 407792510
> tx_queue_9_packets: 735792
> tx_queue_9_bytes: 1092693961
> tx_queue_10_packets: 393576
> tx_queue_10_bytes: 583283986
> tx_queue_11_packets: 712565
> tx_queue_11_bytes: 1037740789
> tx_queue_12_packets: 264445
> tx_queue_12_bytes: 386010613
> tx_queue_13_packets: 246828
> tx_queue_13_bytes: 370387352
> tx_queue_14_packets: 191789
> tx_queue_14_bytes: 281160607
> tx_queue_15_packets: 384581
> tx_queue_15_bytes: 579890782
> tx_queue_16_packets: 175119
> tx_queue_16_bytes: 261312970
> tx_queue_17_packets: 151219
> tx_queue_17_byte

RE: Problems with ixgbe driver

[Tantilov, Emil S]

-Original Message-
From: Holger Kiehl [mailto:holger.ki...@dwd.de]
Sent: Monday, June 17, 2013 2:12 AM
To: Tantilov, Emil S
Cc: e1000-de...@lists.sf.net; linux-kernel; net...@vger.kernel.org
Subject: RE: Problems with ixgbe driver

Hello,

first, thank you for the quick help!

On Fri, 14 Jun 2013, Tantilov, Emil S wrote:

 -Original Message-
 From: netdev-ow...@vger.kernel.org [mailto:netdev-ow...@vger.kernel.org]
On
 Behalf Of Holger Kiehl
 Sent: Friday, June 14, 2013 4:50 AM
 To: e1000-de...@lists.sf.net
 Cc: linux-kernel; net...@vger.kernel.org
 Subject: Problems with ixgbe driver

 Hello,

 I have dual port 10Gb Intel network card on a 2 socket (Xeon X5690) with
 a total of 12 cores. Hyperthreading is enabled so there are 24 cores.
 The problem I have is that when other systems send large amount of data
 the network with the intel ixgbe driver gets very slow. Ping times go up
 from 0.2ms to appr. 60ms. Some FTP connections stall for more then 2
 minutes. What is strange is that heatbeat is configured on the system
 with a serial connection to another node and kernel always reports

 If the network slows down so much there should be some indication in
dmesg. Like Tx hangs perhaps.
 Can you provide the output of dmesg and ethtool -S from the offending
interface after the issue occurs?

No, there is absolute no indication in dmesg or /var/log/messages. But here
the ethtool output when ping times go up:

root@helena:~# ethtool -S eth6
NIC statistics:
 rx_packets: 4410779
 tx_packets: 8902514
 rx_bytes: 2014041824
 tx_bytes: 13199913202
 rx_errors: 0
 tx_errors: 0
 rx_dropped: 0
 tx_dropped: 0
 multicast: 4245
 collisions: 0
 rx_over_errors: 0
 rx_crc_errors: 0
 rx_frame_errors: 0
 rx_fifo_errors: 0
 rx_missed_errors: 28143
 tx_aborted_errors: 0
 tx_carrier_errors: 0
 tx_fifo_errors: 0
 tx_heartbeat_errors: 0
 rx_pkts_nic: 2401276937
 tx_pkts_nic: 3868619482
 rx_bytes_nic: 868282794731
 tx_bytes_nic: 5743382228649
 lsc_int: 4
 tx_busy: 0
 non_eop_descs: 743957
 broadcast: 1745556
 rx_no_buffer_count: 0
 tx_timeout_count: 0
 tx_restart_queue: 425
 rx_long_length_errors: 0
 rx_short_length_errors: 0
 tx_flow_control_xon: 171
 rx_flow_control_xon: 0
 tx_flow_control_xoff: 277
 rx_flow_control_xoff: 0
 rx_csum_offload_errors: 0
 alloc_rx_page_failed: 0
 alloc_rx_buff_failed: 0
 lro_aggregated: 0
 lro_flushed: 0
 rx_no_dma_resources: 0
 hw_rsc_aggregated: 1153374
 hw_rsc_flushed: 129169
 fdir_match: 2424508153
 fdir_miss: 1706029
 fdir_overflow: 33
 os2bmc_rx_by_bmc: 0
 os2bmc_tx_by_bmc: 0
 os2bmc_tx_by_host: 0
 os2bmc_rx_by_host: 0
 tx_queue_0_packets: 470182
 tx_queue_0_bytes: 690123121
 tx_queue_1_packets: 797784
 tx_queue_1_bytes: 1203968369
 tx_queue_2_packets: 648692
 tx_queue_2_bytes: 950171718
 tx_queue_3_packets: 647434
 tx_queue_3_bytes: 948647518
 tx_queue_4_packets: 263216
 tx_queue_4_bytes: 394806409
 tx_queue_5_packets: 426786
 tx_queue_5_bytes: 629387628
 tx_queue_6_packets: 253708
 tx_queue_6_bytes: 371774276
 tx_queue_7_packets: 544634
 tx_queue_7_bytes: 812223169
 tx_queue_8_packets: 279056
 tx_queue_8_bytes: 407792510
 tx_queue_9_packets: 735792
 tx_queue_9_bytes: 1092693961
 tx_queue_10_packets: 393576
 tx_queue_10_bytes: 583283986
 tx_queue_11_packets: 712565
 tx_queue_11_bytes: 1037740789
 tx_queue_12_packets: 264445
 tx_queue_12_bytes: 386010613
 tx_queue_13_packets: 246828
 tx_queue_13_bytes: 370387352
 tx_queue_14_packets: 191789
 tx_queue_14_bytes: 281160607
 tx_queue_15_packets: 384581
 tx_queue_15_bytes: 579890782
 tx_queue_16_packets: 175119
 tx_queue_16_bytes: 261312970
 tx_queue_17_packets: 151219
 tx_queue_17_bytes: 220259675
 tx_queue_18_packets: 467746
 tx_queue_18_bytes: 707472612
 tx_queue_19_packets: 30642
 tx_queue_19_bytes: 44896997
 tx_queue_20_packets: 157957
 tx_queue_20_bytes: 238772784
 tx_queue_21_packets: 287819
 tx_queue_21_bytes: 434965075
 tx_queue_22_packets: 269298
 tx_queue_22_bytes: 407637986
 tx_queue_23_packets: 102344
 tx_queue_23_bytes: 145542751
 rx_queue_0_packets: 219438
 rx_queue_0_bytes: 273936020
 rx_queue_1_packets: 398269
 rx_queue_1_bytes: 52080243
 rx_queue_2_packets: 285870
 rx_queue_2_bytes

RE: Problems with ixgbe driver

[Tantilov, Emil S]

>-Original Message-
>From: netdev-ow...@vger.kernel.org [mailto:netdev-ow...@vger.kernel.org] On
>Behalf Of Holger Kiehl
>Sent: Friday, June 14, 2013 4:50 AM
>To: e1000-de...@lists.sf.net
>Cc: linux-kernel; net...@vger.kernel.org
>Subject: Problems with ixgbe driver
>
>Hello,
>
>I have dual port 10Gb Intel network card on a 2 socket (Xeon X5690) with
>a total of 12 cores. Hyperthreading is enabled so there are 24 cores.
>The problem I have is that when other systems send large amount of data
>the network with the intel ixgbe driver gets very slow. Ping times go up
>from 0.2ms to appr. 60ms. Some FTP connections stall for more then 2
>minutes. What is strange is that heatbeat is configured on the system
>with a serial connection to another node and kernel always reports

If the network slows down so much there should be some indication in dmesg. 
Like Tx hangs perhaps.
Can you provide the output of dmesg and ethtool -S from the offending interface 
after the issue occurs?

>
> ttyS0: 4 input overrun(s)
>
>when lot of data is send and the ping time goes up.
>
>On the network there are three vlan's configured. The network is bonded
>(active-backup) together with another HP NC523SFP 10Gb 2-port Server
>Adapter. When I switch the network to this card the problem goes away.
>Also the ttyS0 input overruns disappear. Note also both network cards
>are connected to the same switch.
>
>The system uses Scientific Linux 6.4 with kernel.org kernel. I noticed
>this behavior with kernel 3.9.5 and 3.9.6-rc1. Before I did not notice
>it because traffic always went over the HP NC523SFP qlcnic card.
>
>In search for a solution to the problem I found a newer ixgbe driver
>3.15.1 (3.9.6-rc1. has 3.11.33-k) and tried that. But it has the same
>problem. However when I load the module as follows:
>
> modprobe ixgbe RSS=8,8
>
>the problem goes away. The kernel.org ixgbe driver does not offer this
>option. Why? It seems that both drivers have problems on systems with

If you are using newer kernel and ethtool version you can use `ethtool -L ethX 
combined Y` to control the number of queues per interface.

>24 cpu's. But I cannot believe that I am the only one who noticed this,
>since ixgbe is widely used.

We run traffic with multiple queues all the time and I don't think what you are 
reporting is a generic issue. Most likely it's something related to your 
setup/system.

>
>It would really be nice if one could set the RSS=8,8 option for kernel.org
>ixgbe driver too. Or if someone could tell me where I can force the driver
>to Receive Side Scaling to 8 even if it means editing the source code.
>
>Below I have added some additional information. Please CC me since I
>am not subscribed to any of these lists. And please do not hesitate
>to ask if more information is needed.

I would suggest that you open up a bug at e1000.sf.net - describe your 
configuration and attach the relevant info (dmesg, ethtool -S, lspci etc). This 
would make it easier for us to follow.

Thanks,
Emil
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

RE: Problems with ixgbe driver

[Tantilov, Emil S]

-Original Message-
From: netdev-ow...@vger.kernel.org [mailto:netdev-ow...@vger.kernel.org] On
Behalf Of Holger Kiehl
Sent: Friday, June 14, 2013 4:50 AM
To: e1000-de...@lists.sf.net
Cc: linux-kernel; net...@vger.kernel.org
Subject: Problems with ixgbe driver

Hello,

I have dual port 10Gb Intel network card on a 2 socket (Xeon X5690) with
a total of 12 cores. Hyperthreading is enabled so there are 24 cores.
The problem I have is that when other systems send large amount of data
the network with the intel ixgbe driver gets very slow. Ping times go up
from 0.2ms to appr. 60ms. Some FTP connections stall for more then 2
minutes. What is strange is that heatbeat is configured on the system
with a serial connection to another node and kernel always reports

If the network slows down so much there should be some indication in dmesg. 
Like Tx hangs perhaps.
Can you provide the output of dmesg and ethtool -S from the offending interface 
after the issue occurs?


 ttyS0: 4 input overrun(s)

when lot of data is send and the ping time goes up.

On the network there are three vlan's configured. The network is bonded
(active-backup) together with another HP NC523SFP 10Gb 2-port Server
Adapter. When I switch the network to this card the problem goes away.
Also the ttyS0 input overruns disappear. Note also both network cards
are connected to the same switch.

The system uses Scientific Linux 6.4 with kernel.org kernel. I noticed
this behavior with kernel 3.9.5 and 3.9.6-rc1. Before I did not notice
it because traffic always went over the HP NC523SFP qlcnic card.

In search for a solution to the problem I found a newer ixgbe driver
3.15.1 (3.9.6-rc1. has 3.11.33-k) and tried that. But it has the same
problem. However when I load the module as follows:

 modprobe ixgbe RSS=8,8

the problem goes away. The kernel.org ixgbe driver does not offer this
option. Why? It seems that both drivers have problems on systems with

If you are using newer kernel and ethtool version you can use `ethtool -L ethX 
combined Y` to control the number of queues per interface.

24 cpu's. But I cannot believe that I am the only one who noticed this,
since ixgbe is widely used.

We run traffic with multiple queues all the time and I don't think what you are 
reporting is a generic issue. Most likely it's something related to your 
setup/system.


It would really be nice if one could set the RSS=8,8 option for kernel.org
ixgbe driver too. Or if someone could tell me where I can force the driver
to Receive Side Scaling to 8 even if it means editing the source code.

Below I have added some additional information. Please CC me since I
am not subscribed to any of these lists. And please do not hesitate
to ask if more information is needed.

I would suggest that you open up a bug at e1000.sf.net - describe your 
configuration and attach the relevant info (dmesg, ethtool -S, lspci etc). This 
would make it easier for us to follow.

Thanks,
Emil
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

RE: 3.6.10: Intel: ixgbe 0000:01:00.0 eth4: Detected Tx Unit Hang

[Tantilov, Emil S]

>-Original Message-
>From: linux-kernel-ow...@vger.kernel.org [mailto:linux-kernel-
>ow...@vger.kernel.org] On Behalf Of Justin Piszcz
>Sent: Saturday, December 15, 2012 7:49 AM
>To: linux-kernel@vger.kernel.org
>Subject: 3.6.10: Intel: ixgbe :01:00.0 eth4: Detected Tx Unit Hang
>
>Hello,
>
>Kernel 3.6.10, first time I have seen this that I can remember (on 10GbE)
>anyway, is this a known issue with 3.6.10?
>
>When the link went down is when I rebooted/etc the remote host attached on
>the other end.
>I've not changed anything physically with the hardware and have been on
>3.6.0-3.6.9 and noticed this when I moved to 3.6.10.
>
>[10270.229200] ixgbe :01:00.0 eth4: NIC Link is Down
>[10276.124937] ixgbe :01:00.0 eth4: NIC Link is Up 10 Gbps, Flow
>Control: RX/TX
>[24529.430997] ixgbe :01:00.0 eth4: Detected Tx Unit Hang
>[24529.430997]   Tx Queue <10>
>[24529.430997]   TDH, TDT <4e>, <51>
>[24529.430997]   next_to_use  <51>
>[24529.430997]   next_to_clean<4e>
>[24529.430997] tx_buffer_info[next_to_clean]
>[24529.430997]   time_stamp   <10172668f>
>[24529.430997]   jiffies  <101726ea4>
>[24529.431011] ixgbe :01:00.0 eth4: tx hang 1 detected on queue 10,
>resetting adapter
>[24529.431028] ixgbe :01:00.0 eth4: Reset adapter
>
>Thoughts?

I don't believe we have seen Tx hangs in validation. If you could narrow down 
the conditions that lead to the Tx hang that would help a lot. Also the output 
of ethtool -S eth4 after the Tx hang occurs can be useful to get an idea of the 
load on the interface.

Thanks,
Emil


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

RE: 3.6.10: Intel: ixgbe 0000:01:00.0 eth4: Detected Tx Unit Hang

[Tantilov, Emil S]

-Original Message-
From: linux-kernel-ow...@vger.kernel.org [mailto:linux-kernel-
ow...@vger.kernel.org] On Behalf Of Justin Piszcz
Sent: Saturday, December 15, 2012 7:49 AM
To: linux-kernel@vger.kernel.org
Subject: 3.6.10: Intel: ixgbe :01:00.0 eth4: Detected Tx Unit Hang

Hello,

Kernel 3.6.10, first time I have seen this that I can remember (on 10GbE)
anyway, is this a known issue with 3.6.10?

When the link went down is when I rebooted/etc the remote host attached on
the other end.
I've not changed anything physically with the hardware and have been on
3.6.0-3.6.9 and noticed this when I moved to 3.6.10.

[10270.229200] ixgbe :01:00.0 eth4: NIC Link is Down
[10276.124937] ixgbe :01:00.0 eth4: NIC Link is Up 10 Gbps, Flow
Control: RX/TX
[24529.430997] ixgbe :01:00.0 eth4: Detected Tx Unit Hang
[24529.430997]   Tx Queue 10
[24529.430997]   TDH, TDT 4e, 51
[24529.430997]   next_to_use  51
[24529.430997]   next_to_clean4e
[24529.430997] tx_buffer_info[next_to_clean]
[24529.430997]   time_stamp   10172668f
[24529.430997]   jiffies  101726ea4
[24529.431011] ixgbe :01:00.0 eth4: tx hang 1 detected on queue 10,
resetting adapter
[24529.431028] ixgbe :01:00.0 eth4: Reset adapter

Thoughts?

I don't believe we have seen Tx hangs in validation. If you could narrow down 
the conditions that lead to the Tx hang that would help a lot. Also the output 
of ethtool -S eth4 after the Tx hang occurs can be useful to get an idea of the 
load on the interface.

Thanks,
Emil


--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/