Re: [PATCH v10 08/22] IB/hns: Add icm support

2016-06-21 Thread Wei Hu (Xavier) 



On 2016/6/21 19:55, Leon Romanovsky wrote:

On Tue, Jun 21, 2016 at 12:37:39PM +0800, Wei Hu (Xavier) wrote:


On 2016/6/20 21:04, Leon Romanovsky wrote:

On Mon, Jun 20, 2016 at 05:48:15PM +0800, Wei Hu (Xavier) wrote:

On 2016/6/20 17:27, Leon Romanovsky wrote:

On Mon, Jun 20, 2016 at 03:49:24PM +0800, Wei Hu (Xavier) wrote:

On 2016/6/20 14:06, Leon Romanovsky wrote:

On Mon, Jun 20, 2016 at 12:37:40PM +0800, Wei Hu (Xavier) wrote:

On 2016/6/17 17:58, Leon Romanovsky wrote:

On Thu, Jun 16, 2016 at 10:35:16PM +0800, Lijun Ou wrote:

This patch mainly added icm support for RoCE. It initializes icm
which managers the relative memory blocks for RoCE. The data
structures of RoCE will be located in it. For example, CQ table,
QP table and MTPT table so on.

Signed-off-by: Wei Hu 
Signed-off-by: Nenglong Zhao 
Signed-off-by: Lijun Ou 
---

<...>


+

Another question which you didn't answer [1].

"I wonder if you have the same needs for ICM as it is in mlx4 device.
Do you have firmware?"

[1] http://marc.info/?l=linux-rdma=146545553104913=2

Hi, Leon
 Now we haven't firmware.
 But hardware still need memory for QPC\CQC\MTPT\mtt etc.

ICM stands for InfiniHost (Interconnect) Context Memory is a specific
memory place to share between host <-> FW and host <-> HW if HW is
aware of specific structures.

I assume that in your case, it is enough to allocate memory region and
supply it to HW. Am I right?

For Our hardware,
1. ICM has a memory management method, It's very good for QPC\CQC\MTPT\mtt
etc. we need it.

You need special HW to leverage its. AFAIK it is Mellanox specific.

For our hardware, we use ICM to memory management, the memory shared with
host and HW.
QPC\CQC\MTPT\mtt has specific memory requirement.
QPC\CQC\MTPT need continuous memory. we use ICM to management the block of
memory. It's very good！

I wasn't convinced why do you need to copy whole ICM logic which is
specific to Mellanox. Your requirements can be implemented by standard CMA
and/or DMA.

Hi, Leon

In hip06 soc,
Hardware need multiple memory blocks for QPC\CQC\MTPT, every block has 
continuous memory xxKbyte (like 128Kbyte),

We need to configure the first address of 128Kbyte to hardware.

For example:
//
example 1:
In create qp,
1. If the xx Kbyte memory that include QPC related with qpn, has not 
been allocated, do step 2.

   else do step 3.
2. dma_alloc xx Kbyte memory for QPC,  and configure the first address 
of xx Kbyte to hardware.

3. find the QPC memory in xx Kbyte, get the dma_addr.
4. send mailbox command to hardware to create QP.

In step 2, we call xx_table_get function as below to perform logic.
int hns_roce_table_get(struct hns_roce_dev *hr_dev,
   struct hns_roce_icm_table *table, unsigned long obj)
{

//dma_alloc_coherent 128Kbyte memory
hns_roce_alloc_icm(hr_dev,
  HNS_ROCE_TABLE_CHUNK_SIZE >> PAGE_SHIFT, );

/*configure the first address of xx Kbyte to hardware*/
hns_roce_map_icm(hr_dev, table, obj);

}

In step 3, we call xx_table_find function to perform logic.
void *hns_roce_table_find(struct hns_roce_icm_table *table, unsigned 
long obj,

  dma_addr_t *dma_handle);


example 2:
In modify qp:
1. find the QPC memory,  get the virtual addr.
2. modify the fields of QPC.
3. send mailbox command to hardware to modify QP.

In step 1,  we call xx_table_find function to perform logic.
//--


so, now we haven't a firmware, but ICM algorithm still suitable for 
hip06 soc perfectly.


Regards
Wei Hu

2. The meomry for QPC\CQC\MTPT\mtt only used for RoCE hardware and driver,
we don't want use MR.

I didn't mean Infiniband MR, but memory region returned from standard
allocation functions (kmalloc, ...).


3. Now we haven't firmware, maybe we need it next version.

You are always invited to add support once it will be needed, no need to
add it in advance.

Thanks

Re: [PATCH v10 08/22] IB/hns: Add icm support

2016-06-21 Thread Wei Hu (Xavier) 



On 2016/6/21 19:55, Leon Romanovsky wrote:

On Tue, Jun 21, 2016 at 12:37:39PM +0800, Wei Hu (Xavier) wrote:


On 2016/6/20 21:04, Leon Romanovsky wrote:

On Mon, Jun 20, 2016 at 05:48:15PM +0800, Wei Hu (Xavier) wrote:

On 2016/6/20 17:27, Leon Romanovsky wrote:

On Mon, Jun 20, 2016 at 03:49:24PM +0800, Wei Hu (Xavier) wrote:

On 2016/6/20 14:06, Leon Romanovsky wrote:

On Mon, Jun 20, 2016 at 12:37:40PM +0800, Wei Hu (Xavier) wrote:

On 2016/6/17 17:58, Leon Romanovsky wrote:

On Thu, Jun 16, 2016 at 10:35:16PM +0800, Lijun Ou wrote:

This patch mainly added icm support for RoCE. It initializes icm
which managers the relative memory blocks for RoCE. The data
structures of RoCE will be located in it. For example, CQ table,
QP table and MTPT table so on.

Signed-off-by: Wei Hu 
Signed-off-by: Nenglong Zhao 
Signed-off-by: Lijun Ou 
---

<...>


+

Another question which you didn't answer [1].

"I wonder if you have the same needs for ICM as it is in mlx4 device.
Do you have firmware?"

[1] http://marc.info/?l=linux-rdma=146545553104913=2

Hi, Leon
 Now we haven't firmware.
 But hardware still need memory for QPC\CQC\MTPT\mtt etc.

ICM stands for InfiniHost (Interconnect) Context Memory is a specific
memory place to share between host <-> FW and host <-> HW if HW is
aware of specific structures.

I assume that in your case, it is enough to allocate memory region and
supply it to HW. Am I right?

For Our hardware,
1. ICM has a memory management method, It's very good for QPC\CQC\MTPT\mtt
etc. we need it.

You need special HW to leverage its. AFAIK it is Mellanox specific.

For our hardware, we use ICM to memory management, the memory shared with
host and HW.
QPC\CQC\MTPT\mtt has specific memory requirement.
QPC\CQC\MTPT need continuous memory. we use ICM to management the block of
memory. It's very good！

I wasn't convinced why do you need to copy whole ICM logic which is
specific to Mellanox. Your requirements can be implemented by standard CMA
and/or DMA.

Hi, Leon

In hip06 soc,
Hardware need multiple memory blocks for QPC\CQC\MTPT, every block has 
continuous memory xxKbyte (like 128Kbyte),

We need to configure the first address of 128Kbyte to hardware.

For example:
//
example 1:
In create qp,
1. If the xx Kbyte memory that include QPC related with qpn, has not 
been allocated, do step 2.

   else do step 3.
2. dma_alloc xx Kbyte memory for QPC,  and configure the first address 
of xx Kbyte to hardware.

3. find the QPC memory in xx Kbyte, get the dma_addr.
4. send mailbox command to hardware to create QP.

In step 2, we call xx_table_get function as below to perform logic.
int hns_roce_table_get(struct hns_roce_dev *hr_dev,
   struct hns_roce_icm_table *table, unsigned long obj)
{

//dma_alloc_coherent 128Kbyte memory
hns_roce_alloc_icm(hr_dev,
  HNS_ROCE_TABLE_CHUNK_SIZE >> PAGE_SHIFT, );

/*configure the first address of xx Kbyte to hardware*/
hns_roce_map_icm(hr_dev, table, obj);

}

In step 3, we call xx_table_find function to perform logic.
void *hns_roce_table_find(struct hns_roce_icm_table *table, unsigned 
long obj,

  dma_addr_t *dma_handle);


example 2:
In modify qp:
1. find the QPC memory,  get the virtual addr.
2. modify the fields of QPC.
3. send mailbox command to hardware to modify QP.

In step 1,  we call xx_table_find function to perform logic.
//--


so, now we haven't a firmware, but ICM algorithm still suitable for 
hip06 soc perfectly.


Regards
Wei Hu

2. The meomry for QPC\CQC\MTPT\mtt only used for RoCE hardware and driver,
we don't want use MR.

I didn't mean Infiniband MR, but memory region returned from standard
allocation functions (kmalloc, ...).


3. Now we haven't firmware, maybe we need it next version.

You are always invited to add support once it will be needed, no need to
add it in advance.

Thanks

Re: [PATCH v10 08/22] IB/hns: Add icm support

2016-06-21 Thread Leon Romanovsky 
On Tue, Jun 21, 2016 at 12:37:39PM +0800, Wei Hu (Xavier) wrote:
> 
> 
> On 2016/6/20 21:04, Leon Romanovsky wrote:
> >On Mon, Jun 20, 2016 at 05:48:15PM +0800, Wei Hu (Xavier) wrote:
> >>
> >>On 2016/6/20 17:27, Leon Romanovsky wrote:
> >>>On Mon, Jun 20, 2016 at 03:49:24PM +0800, Wei Hu (Xavier) wrote:
> On 2016/6/20 14:06, Leon Romanovsky wrote:
> >On Mon, Jun 20, 2016 at 12:37:40PM +0800, Wei Hu (Xavier) wrote:
> >>On 2016/6/17 17:58, Leon Romanovsky wrote:
> >>>On Thu, Jun 16, 2016 at 10:35:16PM +0800, Lijun Ou wrote:
> This patch mainly added icm support for RoCE. It initializes icm
> which managers the relative memory blocks for RoCE. The data
> structures of RoCE will be located in it. For example, CQ table,
> QP table and MTPT table so on.
> 
> Signed-off-by: Wei Hu 
> Signed-off-by: Nenglong Zhao 
> Signed-off-by: Lijun Ou 
> ---
> >>><...>
> >>>
> +
> >Another question which you didn't answer [1].
> >
> >"I wonder if you have the same needs for ICM as it is in mlx4 device.
> >Do you have firmware?"
> >
> >[1] http://marc.info/?l=linux-rdma=146545553104913=2
> Hi, Leon
>  Now we haven't firmware.
>  But hardware still need memory for QPC\CQC\MTPT\mtt etc.
> >>>ICM stands for InfiniHost (Interconnect) Context Memory is a specific
> >>>memory place to share between host <-> FW and host <-> HW if HW is
> >>>aware of specific structures.
> >>>
> >>>I assume that in your case, it is enough to allocate memory region and
> >>>supply it to HW. Am I right?
> >>For Our hardware,
> >>1. ICM has a memory management method, It's very good for QPC\CQC\MTPT\mtt
> >>etc. we need it.
> >You need special HW to leverage its. AFAIK it is Mellanox specific.
> For our hardware, we use ICM to memory management, the memory shared with
> host and HW.
> QPC\CQC\MTPT\mtt has specific memory requirement.
> QPC\CQC\MTPT need continuous memory. we use ICM to management the block of
> memory. It's very good！

I wasn't convinced why do you need to copy whole ICM logic which is
specific to Mellanox. Your requirements can be implemented by standard CMA
and/or DMA.

> >>2. The meomry for QPC\CQC\MTPT\mtt only used for RoCE hardware and driver,
> >>we don't want use MR.
> >I didn't mean Infiniband MR, but memory region returned from standard
> >allocation functions (kmalloc, ...).
> >
> >>3. Now we haven't firmware, maybe we need it next version.
> >You are always invited to add support once it will be needed, no need to
> >add it in advance.
> >
> >Thanks
> 
> 


signature.asc
Description: Digital signature

Re: [PATCH v10 08/22] IB/hns: Add icm support

2016-06-21 Thread Leon Romanovsky 
On Tue, Jun 21, 2016 at 12:37:39PM +0800, Wei Hu (Xavier) wrote:
> 
> 
> On 2016/6/20 21:04, Leon Romanovsky wrote:
> >On Mon, Jun 20, 2016 at 05:48:15PM +0800, Wei Hu (Xavier) wrote:
> >>
> >>On 2016/6/20 17:27, Leon Romanovsky wrote:
> >>>On Mon, Jun 20, 2016 at 03:49:24PM +0800, Wei Hu (Xavier) wrote:
> On 2016/6/20 14:06, Leon Romanovsky wrote:
> >On Mon, Jun 20, 2016 at 12:37:40PM +0800, Wei Hu (Xavier) wrote:
> >>On 2016/6/17 17:58, Leon Romanovsky wrote:
> >>>On Thu, Jun 16, 2016 at 10:35:16PM +0800, Lijun Ou wrote:
> This patch mainly added icm support for RoCE. It initializes icm
> which managers the relative memory blocks for RoCE. The data
> structures of RoCE will be located in it. For example, CQ table,
> QP table and MTPT table so on.
> 
> Signed-off-by: Wei Hu 
> Signed-off-by: Nenglong Zhao 
> Signed-off-by: Lijun Ou 
> ---
> >>><...>
> >>>
> +
> >Another question which you didn't answer [1].
> >
> >"I wonder if you have the same needs for ICM as it is in mlx4 device.
> >Do you have firmware?"
> >
> >[1] http://marc.info/?l=linux-rdma=146545553104913=2
> Hi, Leon
>  Now we haven't firmware.
>  But hardware still need memory for QPC\CQC\MTPT\mtt etc.
> >>>ICM stands for InfiniHost (Interconnect) Context Memory is a specific
> >>>memory place to share between host <-> FW and host <-> HW if HW is
> >>>aware of specific structures.
> >>>
> >>>I assume that in your case, it is enough to allocate memory region and
> >>>supply it to HW. Am I right?
> >>For Our hardware,
> >>1. ICM has a memory management method, It's very good for QPC\CQC\MTPT\mtt
> >>etc. we need it.
> >You need special HW to leverage its. AFAIK it is Mellanox specific.
> For our hardware, we use ICM to memory management, the memory shared with
> host and HW.
> QPC\CQC\MTPT\mtt has specific memory requirement.
> QPC\CQC\MTPT need continuous memory. we use ICM to management the block of
> memory. It's very good！

I wasn't convinced why do you need to copy whole ICM logic which is
specific to Mellanox. Your requirements can be implemented by standard CMA
and/or DMA.

> >>2. The meomry for QPC\CQC\MTPT\mtt only used for RoCE hardware and driver,
> >>we don't want use MR.
> >I didn't mean Infiniband MR, but memory region returned from standard
> >allocation functions (kmalloc, ...).
> >
> >>3. Now we haven't firmware, maybe we need it next version.
> >You are always invited to add support once it will be needed, no need to
> >add it in advance.
> >
> >Thanks
> 
> 


signature.asc
Description: Digital signature

Re: [PATCH v10 08/22] IB/hns: Add icm support

2016-06-20 Thread Wei Hu (Xavier) 



On 2016/6/20 21:04, Leon Romanovsky wrote:

On Mon, Jun 20, 2016 at 05:48:15PM +0800, Wei Hu (Xavier) wrote:


On 2016/6/20 17:27, Leon Romanovsky wrote:

On Mon, Jun 20, 2016 at 03:49:24PM +0800, Wei Hu (Xavier) wrote:

On 2016/6/20 14:06, Leon Romanovsky wrote:

On Mon, Jun 20, 2016 at 12:37:40PM +0800, Wei Hu (Xavier) wrote:

On 2016/6/17 17:58, Leon Romanovsky wrote:

On Thu, Jun 16, 2016 at 10:35:16PM +0800, Lijun Ou wrote:

This patch mainly added icm support for RoCE. It initializes icm
which managers the relative memory blocks for RoCE. The data
structures of RoCE will be located in it. For example, CQ table,
QP table and MTPT table so on.

Signed-off-by: Wei Hu 
Signed-off-by: Nenglong Zhao 
Signed-off-by: Lijun Ou 
---

<...>


+

Another question which you didn't answer [1].

"I wonder if you have the same needs for ICM as it is in mlx4 device.
Do you have firmware?"

[1] http://marc.info/?l=linux-rdma=146545553104913=2

Hi, Leon
 Now we haven't firmware.
 But hardware still need memory for QPC\CQC\MTPT\mtt etc.

ICM stands for InfiniHost (Interconnect) Context Memory is a specific
memory place to share between host <-> FW and host <-> HW if HW is
aware of specific structures.

I assume that in your case, it is enough to allocate memory region and
supply it to HW. Am I right?

For Our hardware,
1. ICM has a memory management method, It's very good for QPC\CQC\MTPT\mtt
etc. we need it.

You need special HW to leverage its. AFAIK it is Mellanox specific.
For our hardware, we use ICM to memory management, the memory shared 
with host and HW.

QPC\CQC\MTPT\mtt has specific memory requirement.
QPC\CQC\MTPT need continuous memory. we use ICM to management the block 
of memory. It's very good！

2. The meomry for QPC\CQC\MTPT\mtt only used for RoCE hardware and driver,
we don't want use MR.

I didn't mean Infiniband MR, but memory region returned from standard
allocation functions (kmalloc, ...).


3. Now we haven't firmware, maybe we need it next version.

You are always invited to add support once it will be needed, no need to
add it in advance.

Thanks

Re: [PATCH v10 08/22] IB/hns: Add icm support

2016-06-20 Thread Wei Hu (Xavier) 



On 2016/6/20 21:04, Leon Romanovsky wrote:

On Mon, Jun 20, 2016 at 05:48:15PM +0800, Wei Hu (Xavier) wrote:


On 2016/6/20 17:27, Leon Romanovsky wrote:

On Mon, Jun 20, 2016 at 03:49:24PM +0800, Wei Hu (Xavier) wrote:

On 2016/6/20 14:06, Leon Romanovsky wrote:

On Mon, Jun 20, 2016 at 12:37:40PM +0800, Wei Hu (Xavier) wrote:

On 2016/6/17 17:58, Leon Romanovsky wrote:

On Thu, Jun 16, 2016 at 10:35:16PM +0800, Lijun Ou wrote:

This patch mainly added icm support for RoCE. It initializes icm
which managers the relative memory blocks for RoCE. The data
structures of RoCE will be located in it. For example, CQ table,
QP table and MTPT table so on.

Signed-off-by: Wei Hu 
Signed-off-by: Nenglong Zhao 
Signed-off-by: Lijun Ou 
---

<...>


+

Another question which you didn't answer [1].

"I wonder if you have the same needs for ICM as it is in mlx4 device.
Do you have firmware?"

[1] http://marc.info/?l=linux-rdma=146545553104913=2

Hi, Leon
 Now we haven't firmware.
 But hardware still need memory for QPC\CQC\MTPT\mtt etc.

ICM stands for InfiniHost (Interconnect) Context Memory is a specific
memory place to share between host <-> FW and host <-> HW if HW is
aware of specific structures.

I assume that in your case, it is enough to allocate memory region and
supply it to HW. Am I right?

For Our hardware,
1. ICM has a memory management method, It's very good for QPC\CQC\MTPT\mtt
etc. we need it.

You need special HW to leverage its. AFAIK it is Mellanox specific.
For our hardware, we use ICM to memory management, the memory shared 
with host and HW.

QPC\CQC\MTPT\mtt has specific memory requirement.
QPC\CQC\MTPT need continuous memory. we use ICM to management the block 
of memory. It's very good！

2. The meomry for QPC\CQC\MTPT\mtt only used for RoCE hardware and driver,
we don't want use MR.

I didn't mean Infiniband MR, but memory region returned from standard
allocation functions (kmalloc, ...).


3. Now we haven't firmware, maybe we need it next version.

You are always invited to add support once it will be needed, no need to
add it in advance.

Thanks

Re: [PATCH v10 08/22] IB/hns: Add icm support

2016-06-20 Thread Leon Romanovsky 
On Mon, Jun 20, 2016 at 05:48:15PM +0800, Wei Hu (Xavier) wrote:
> 
> 
> On 2016/6/20 17:27, Leon Romanovsky wrote:
> >On Mon, Jun 20, 2016 at 03:49:24PM +0800, Wei Hu (Xavier) wrote:
> >>
> >>On 2016/6/20 14:06, Leon Romanovsky wrote:
> >>>On Mon, Jun 20, 2016 at 12:37:40PM +0800, Wei Hu (Xavier) wrote:
> On 2016/6/17 17:58, Leon Romanovsky wrote:
> >On Thu, Jun 16, 2016 at 10:35:16PM +0800, Lijun Ou wrote:
> >>This patch mainly added icm support for RoCE. It initializes icm
> >>which managers the relative memory blocks for RoCE. The data
> >>structures of RoCE will be located in it. For example, CQ table,
> >>QP table and MTPT table so on.
> >>
> >>Signed-off-by: Wei Hu 
> >>Signed-off-by: Nenglong Zhao 
> >>Signed-off-by: Lijun Ou 
> >>---
> ><...>
> >
> >>+
> >>>Another question which you didn't answer [1].
> >>>
> >>>"I wonder if you have the same needs for ICM as it is in mlx4 device.
> >>>Do you have firmware?"
> >>>
> >>>[1] http://marc.info/?l=linux-rdma=146545553104913=2
> >>Hi, Leon
> >> Now we haven't firmware.
> >> But hardware still need memory for QPC\CQC\MTPT\mtt etc.
> >ICM stands for InfiniHost (Interconnect) Context Memory is a specific
> >memory place to share between host <-> FW and host <-> HW if HW is
> >aware of specific structures.
> >
> >I assume that in your case, it is enough to allocate memory region and
> >supply it to HW. Am I right?
> For Our hardware,
> 1. ICM has a memory management method, It's very good for QPC\CQC\MTPT\mtt
> etc. we need it.

You need special HW to leverage its. AFAIK it is Mellanox specific.

> 2. The meomry for QPC\CQC\MTPT\mtt only used for RoCE hardware and driver,
> we don't want use MR.

I didn't mean Infiniband MR, but memory region returned from standard
allocation functions (kmalloc, ...).

> 3. Now we haven't firmware, maybe we need it next version.

You are always invited to add support once it will be needed, no need to
add it in advance.

Thanks


signature.asc
Description: Digital signature

Re: [PATCH v10 08/22] IB/hns: Add icm support

2016-06-20 Thread Leon Romanovsky 
On Mon, Jun 20, 2016 at 05:48:15PM +0800, Wei Hu (Xavier) wrote:
> 
> 
> On 2016/6/20 17:27, Leon Romanovsky wrote:
> >On Mon, Jun 20, 2016 at 03:49:24PM +0800, Wei Hu (Xavier) wrote:
> >>
> >>On 2016/6/20 14:06, Leon Romanovsky wrote:
> >>>On Mon, Jun 20, 2016 at 12:37:40PM +0800, Wei Hu (Xavier) wrote:
> On 2016/6/17 17:58, Leon Romanovsky wrote:
> >On Thu, Jun 16, 2016 at 10:35:16PM +0800, Lijun Ou wrote:
> >>This patch mainly added icm support for RoCE. It initializes icm
> >>which managers the relative memory blocks for RoCE. The data
> >>structures of RoCE will be located in it. For example, CQ table,
> >>QP table and MTPT table so on.
> >>
> >>Signed-off-by: Wei Hu 
> >>Signed-off-by: Nenglong Zhao 
> >>Signed-off-by: Lijun Ou 
> >>---
> ><...>
> >
> >>+
> >>>Another question which you didn't answer [1].
> >>>
> >>>"I wonder if you have the same needs for ICM as it is in mlx4 device.
> >>>Do you have firmware?"
> >>>
> >>>[1] http://marc.info/?l=linux-rdma=146545553104913=2
> >>Hi, Leon
> >> Now we haven't firmware.
> >> But hardware still need memory for QPC\CQC\MTPT\mtt etc.
> >ICM stands for InfiniHost (Interconnect) Context Memory is a specific
> >memory place to share between host <-> FW and host <-> HW if HW is
> >aware of specific structures.
> >
> >I assume that in your case, it is enough to allocate memory region and
> >supply it to HW. Am I right?
> For Our hardware,
> 1. ICM has a memory management method, It's very good for QPC\CQC\MTPT\mtt
> etc. we need it.

You need special HW to leverage its. AFAIK it is Mellanox specific.

> 2. The meomry for QPC\CQC\MTPT\mtt only used for RoCE hardware and driver,
> we don't want use MR.

I didn't mean Infiniband MR, but memory region returned from standard
allocation functions (kmalloc, ...).

> 3. Now we haven't firmware, maybe we need it next version.

You are always invited to add support once it will be needed, no need to
add it in advance.

Thanks


signature.asc
Description: Digital signature

Re: [PATCH v10 08/22] IB/hns: Add icm support

2016-06-20 Thread Wei Hu (Xavier) 



On 2016/6/20 17:27, Leon Romanovsky wrote:

On Mon, Jun 20, 2016 at 03:49:24PM +0800, Wei Hu (Xavier) wrote:


On 2016/6/20 14:06, Leon Romanovsky wrote:

On Mon, Jun 20, 2016 at 12:37:40PM +0800, Wei Hu (Xavier) wrote:

On 2016/6/17 17:58, Leon Romanovsky wrote:

On Thu, Jun 16, 2016 at 10:35:16PM +0800, Lijun Ou wrote:

This patch mainly added icm support for RoCE. It initializes icm
which managers the relative memory blocks for RoCE. The data
structures of RoCE will be located in it. For example, CQ table,
QP table and MTPT table so on.

Signed-off-by: Wei Hu 
Signed-off-by: Nenglong Zhao 
Signed-off-by: Lijun Ou 
---

<...>


+

Another question which you didn't answer [1].

"I wonder if you have the same needs for ICM as it is in mlx4 device.
Do you have firmware?"

[1] http://marc.info/?l=linux-rdma=146545553104913=2

Hi, Leon
 Now we haven't firmware.
 But hardware still need memory for QPC\CQC\MTPT\mtt etc.

ICM stands for InfiniHost (Interconnect) Context Memory is a specific
memory place to share between host <-> FW and host <-> HW if HW is
aware of specific structures.

I assume that in your case, it is enough to allocate memory region and
supply it to HW. Am I right?

For Our hardware,
1. ICM has a memory management method, It's very good for 
QPC\CQC\MTPT\mtt etc. we need it.
2. The meomry for QPC\CQC\MTPT\mtt only used for RoCE hardware and 
driver, we don't want use MR.

3. Now we haven't firmware, maybe we need it next version.

Thanks
Wei Hu.

Thanks
Wei Hu

Regards
Wei Hu

Re: [PATCH v10 08/22] IB/hns: Add icm support

2016-06-20 Thread Wei Hu (Xavier) 



On 2016/6/20 17:27, Leon Romanovsky wrote:

On Mon, Jun 20, 2016 at 03:49:24PM +0800, Wei Hu (Xavier) wrote:


On 2016/6/20 14:06, Leon Romanovsky wrote:

On Mon, Jun 20, 2016 at 12:37:40PM +0800, Wei Hu (Xavier) wrote:

On 2016/6/17 17:58, Leon Romanovsky wrote:

On Thu, Jun 16, 2016 at 10:35:16PM +0800, Lijun Ou wrote:

This patch mainly added icm support for RoCE. It initializes icm
which managers the relative memory blocks for RoCE. The data
structures of RoCE will be located in it. For example, CQ table,
QP table and MTPT table so on.

Signed-off-by: Wei Hu 
Signed-off-by: Nenglong Zhao 
Signed-off-by: Lijun Ou 
---

<...>


+

Another question which you didn't answer [1].

"I wonder if you have the same needs for ICM as it is in mlx4 device.
Do you have firmware?"

[1] http://marc.info/?l=linux-rdma=146545553104913=2

Hi, Leon
 Now we haven't firmware.
 But hardware still need memory for QPC\CQC\MTPT\mtt etc.

ICM stands for InfiniHost (Interconnect) Context Memory is a specific
memory place to share between host <-> FW and host <-> HW if HW is
aware of specific structures.

I assume that in your case, it is enough to allocate memory region and
supply it to HW. Am I right?

For Our hardware,
1. ICM has a memory management method, It's very good for 
QPC\CQC\MTPT\mtt etc. we need it.
2. The meomry for QPC\CQC\MTPT\mtt only used for RoCE hardware and 
driver, we don't want use MR.

3. Now we haven't firmware, maybe we need it next version.

Thanks
Wei Hu.

Thanks
Wei Hu

Regards
Wei Hu

Re: [PATCH v10 08/22] IB/hns: Add icm support

2016-06-20 Thread Leon Romanovsky 
On Mon, Jun 20, 2016 at 03:49:24PM +0800, Wei Hu (Xavier) wrote:
> 
> 
> On 2016/6/20 14:06, Leon Romanovsky wrote:
> >On Mon, Jun 20, 2016 at 12:37:40PM +0800, Wei Hu (Xavier) wrote:
> >>
> >>On 2016/6/17 17:58, Leon Romanovsky wrote:
> >>>On Thu, Jun 16, 2016 at 10:35:16PM +0800, Lijun Ou wrote:
> This patch mainly added icm support for RoCE. It initializes icm
> which managers the relative memory blocks for RoCE. The data
> structures of RoCE will be located in it. For example, CQ table,
> QP table and MTPT table so on.
> 
> Signed-off-by: Wei Hu 
> Signed-off-by: Nenglong Zhao 
> Signed-off-by: Lijun Ou 
> ---
> >>><...>
> >>>
> +
> >Another question which you didn't answer [1].
> >
> >"I wonder if you have the same needs for ICM as it is in mlx4 device.
> >Do you have firmware?"
> >
> >[1] http://marc.info/?l=linux-rdma=146545553104913=2
> Hi, Leon
> Now we haven't firmware.
> But hardware still need memory for QPC\CQC\MTPT\mtt etc.

ICM stands for InfiniHost (Interconnect) Context Memory is a specific
memory place to share between host <-> FW and host <-> HW if HW is
aware of specific structures.

I assume that in your case, it is enough to allocate memory region and
supply it to HW. Am I right?

> 
> Thanks
> Wei Hu
> >>Regards
> >>Wei Hu
> >>
> 
> 


signature.asc
Description: Digital signature

Re: [PATCH v10 08/22] IB/hns: Add icm support

2016-06-20 Thread Leon Romanovsky 
On Mon, Jun 20, 2016 at 03:49:24PM +0800, Wei Hu (Xavier) wrote:
> 
> 
> On 2016/6/20 14:06, Leon Romanovsky wrote:
> >On Mon, Jun 20, 2016 at 12:37:40PM +0800, Wei Hu (Xavier) wrote:
> >>
> >>On 2016/6/17 17:58, Leon Romanovsky wrote:
> >>>On Thu, Jun 16, 2016 at 10:35:16PM +0800, Lijun Ou wrote:
> This patch mainly added icm support for RoCE. It initializes icm
> which managers the relative memory blocks for RoCE. The data
> structures of RoCE will be located in it. For example, CQ table,
> QP table and MTPT table so on.
> 
> Signed-off-by: Wei Hu 
> Signed-off-by: Nenglong Zhao 
> Signed-off-by: Lijun Ou 
> ---
> >>><...>
> >>>
> +
> >Another question which you didn't answer [1].
> >
> >"I wonder if you have the same needs for ICM as it is in mlx4 device.
> >Do you have firmware?"
> >
> >[1] http://marc.info/?l=linux-rdma=146545553104913=2
> Hi, Leon
> Now we haven't firmware.
> But hardware still need memory for QPC\CQC\MTPT\mtt etc.

ICM stands for InfiniHost (Interconnect) Context Memory is a specific
memory place to share between host <-> FW and host <-> HW if HW is
aware of specific structures.

I assume that in your case, it is enough to allocate memory region and
supply it to HW. Am I right?

> 
> Thanks
> Wei Hu
> >>Regards
> >>Wei Hu
> >>
> 
> 


signature.asc
Description: Digital signature

Re: [PATCH v10 08/22] IB/hns: Add icm support

2016-06-20 Thread Wei Hu (Xavier) 



On 2016/6/20 14:06, Leon Romanovsky wrote:

On Mon, Jun 20, 2016 at 12:37:40PM +0800, Wei Hu (Xavier) wrote:


On 2016/6/17 17:58, Leon Romanovsky wrote:

On Thu, Jun 16, 2016 at 10:35:16PM +0800, Lijun Ou wrote:

This patch mainly added icm support for RoCE. It initializes icm
which managers the relative memory blocks for RoCE. The data
structures of RoCE will be located in it. For example, CQ table,
QP table and MTPT table so on.

Signed-off-by: Wei Hu 
Signed-off-by: Nenglong Zhao 
Signed-off-by: Lijun Ou 
---

<...>


+
+static int hns_roce_alloc_icm_pages(struct scatterlist *mem, int order,
+   gfp_t gfp_mask)
+{
+   struct page *page;
+
+   page = alloc_pages(gfp_mask, order);
+   if (!page)
+   return -ENOMEM;
+
+   sg_set_page(mem, page, PAGE_SIZE << order, 0);
+
+   return 0;
+}
+
+static int hns_roce_alloc_icm_coherent(struct device *dev,
+  struct scatterlist *mem, int order,
+  gfp_t gfp_mask)
+{
+   void *buf = dma_alloc_coherent(dev, PAGE_SIZE << order,
+  _dma_address(mem), gfp_mask);
+   if (!buf)
+   return -ENOMEM;
+
+   sg_set_buf(mem, buf, PAGE_SIZE << order);
+   WARN_ON(mem->offset);
+   sg_dma_len(mem) = PAGE_SIZE << order;
+   return 0;
+}
+

<...>


+
+static void hns_roce_free_icm_pages(struct hns_roce_dev *hr_dev,
+   struct hns_roce_icm_chunk *chunk)
+{
+   int i;
+
+   if (chunk->nsg > 0)
+   dma_unmap_sg(_dev->pdev->dev, chunk->mem, chunk->npages,
+DMA_BIDIRECTIONAL);
+
+   for (i = 0; i < chunk->npages; ++i)
+   __free_pages(sg_page(>mem[i]),
+get_order(chunk->mem[i].length));

You used alloc_pages for this allocation, so why are you using
__free_pages instead of free_pages?

Hi, Leon
 The function prototype of these functions as below：
 static inline struct page * alloc_pages(gfp_t gfp_mask, unsigned int
order);
 void free_pages(unsigned long addr, unsigned int order);
 void __free_pages(struct page *page, unsigned int order);

 The type of the first parameter of free_pages is same with the type of
return value of alloc_pages.
 Maybe it is better to call __free_pages to release memory that allocated
by calling alloc_pages.

OK, I see.

Another question which you didn't answer [1].

"I wonder if you have the same needs for ICM as it is in mlx4 device.
Do you have firmware?"

[1] http://marc.info/?l=linux-rdma=146545553104913=2

Hi, Leon
Now we haven't firmware.
But hardware still need memory for QPC\CQC\MTPT\mtt etc.

Thanks
Wei Hu

Regards
Wei Hu

Re: [PATCH v10 08/22] IB/hns: Add icm support

2016-06-20 Thread Wei Hu (Xavier) 



On 2016/6/20 14:06, Leon Romanovsky wrote:

On Mon, Jun 20, 2016 at 12:37:40PM +0800, Wei Hu (Xavier) wrote:


On 2016/6/17 17:58, Leon Romanovsky wrote:

On Thu, Jun 16, 2016 at 10:35:16PM +0800, Lijun Ou wrote:

This patch mainly added icm support for RoCE. It initializes icm
which managers the relative memory blocks for RoCE. The data
structures of RoCE will be located in it. For example, CQ table,
QP table and MTPT table so on.

Signed-off-by: Wei Hu 
Signed-off-by: Nenglong Zhao 
Signed-off-by: Lijun Ou 
---

<...>


+
+static int hns_roce_alloc_icm_pages(struct scatterlist *mem, int order,
+   gfp_t gfp_mask)
+{
+   struct page *page;
+
+   page = alloc_pages(gfp_mask, order);
+   if (!page)
+   return -ENOMEM;
+
+   sg_set_page(mem, page, PAGE_SIZE << order, 0);
+
+   return 0;
+}
+
+static int hns_roce_alloc_icm_coherent(struct device *dev,
+  struct scatterlist *mem, int order,
+  gfp_t gfp_mask)
+{
+   void *buf = dma_alloc_coherent(dev, PAGE_SIZE << order,
+  _dma_address(mem), gfp_mask);
+   if (!buf)
+   return -ENOMEM;
+
+   sg_set_buf(mem, buf, PAGE_SIZE << order);
+   WARN_ON(mem->offset);
+   sg_dma_len(mem) = PAGE_SIZE << order;
+   return 0;
+}
+

<...>


+
+static void hns_roce_free_icm_pages(struct hns_roce_dev *hr_dev,
+   struct hns_roce_icm_chunk *chunk)
+{
+   int i;
+
+   if (chunk->nsg > 0)
+   dma_unmap_sg(_dev->pdev->dev, chunk->mem, chunk->npages,
+DMA_BIDIRECTIONAL);
+
+   for (i = 0; i < chunk->npages; ++i)
+   __free_pages(sg_page(>mem[i]),
+get_order(chunk->mem[i].length));

You used alloc_pages for this allocation, so why are you using
__free_pages instead of free_pages?

Hi, Leon
 The function prototype of these functions as below：
 static inline struct page * alloc_pages(gfp_t gfp_mask, unsigned int
order);
 void free_pages(unsigned long addr, unsigned int order);
 void __free_pages(struct page *page, unsigned int order);

 The type of the first parameter of free_pages is same with the type of
return value of alloc_pages.
 Maybe it is better to call __free_pages to release memory that allocated
by calling alloc_pages.

OK, I see.

Another question which you didn't answer [1].

"I wonder if you have the same needs for ICM as it is in mlx4 device.
Do you have firmware?"

[1] http://marc.info/?l=linux-rdma=146545553104913=2

Hi, Leon
Now we haven't firmware.
But hardware still need memory for QPC\CQC\MTPT\mtt etc.

Thanks
Wei Hu

Regards
Wei Hu

Re: [PATCH v10 08/22] IB/hns: Add icm support

2016-06-20 Thread Leon Romanovsky 
On Mon, Jun 20, 2016 at 12:37:40PM +0800, Wei Hu (Xavier) wrote:
> 
> 
> On 2016/6/17 17:58, Leon Romanovsky wrote:
> >On Thu, Jun 16, 2016 at 10:35:16PM +0800, Lijun Ou wrote:
> >>This patch mainly added icm support for RoCE. It initializes icm
> >>which managers the relative memory blocks for RoCE. The data
> >>structures of RoCE will be located in it. For example, CQ table,
> >>QP table and MTPT table so on.
> >>
> >>Signed-off-by: Wei Hu 
> >>Signed-off-by: Nenglong Zhao 
> >>Signed-off-by: Lijun Ou 
> >>---
> ><...>
> >
> >>+
> >>+static int hns_roce_alloc_icm_pages(struct scatterlist *mem, int order,
> >>+   gfp_t gfp_mask)
> >>+{
> >>+   struct page *page;
> >>+
> >>+   page = alloc_pages(gfp_mask, order);
> >>+   if (!page)
> >>+   return -ENOMEM;
> >>+
> >>+   sg_set_page(mem, page, PAGE_SIZE << order, 0);
> >>+
> >>+   return 0;
> >>+}
> >>+
> >>+static int hns_roce_alloc_icm_coherent(struct device *dev,
> >>+  struct scatterlist *mem, int order,
> >>+  gfp_t gfp_mask)
> >>+{
> >>+   void *buf = dma_alloc_coherent(dev, PAGE_SIZE << order,
> >>+  _dma_address(mem), gfp_mask);
> >>+   if (!buf)
> >>+   return -ENOMEM;
> >>+
> >>+   sg_set_buf(mem, buf, PAGE_SIZE << order);
> >>+   WARN_ON(mem->offset);
> >>+   sg_dma_len(mem) = PAGE_SIZE << order;
> >>+   return 0;
> >>+}
> >>+
> ><...>
> >
> >>+
> >>+static void hns_roce_free_icm_pages(struct hns_roce_dev *hr_dev,
> >>+   struct hns_roce_icm_chunk *chunk)
> >>+{
> >>+   int i;
> >>+
> >>+   if (chunk->nsg > 0)
> >>+   dma_unmap_sg(_dev->pdev->dev, chunk->mem, chunk->npages,
> >>+DMA_BIDIRECTIONAL);
> >>+
> >>+   for (i = 0; i < chunk->npages; ++i)
> >>+   __free_pages(sg_page(>mem[i]),
> >>+get_order(chunk->mem[i].length));
> >You used alloc_pages for this allocation, so why are you using
> >__free_pages instead of free_pages?
> Hi, Leon
> The function prototype of these functions as below：
> static inline struct page * alloc_pages(gfp_t gfp_mask, unsigned int
> order);
> void free_pages(unsigned long addr, unsigned int order);
> void __free_pages(struct page *page, unsigned int order);
> 
> The type of the first parameter of free_pages is same with the type of
> return value of alloc_pages.
> Maybe it is better to call __free_pages to release memory that allocated
> by calling alloc_pages.

OK, I see.

Another question which you didn't answer [1].

"I wonder if you have the same needs for ICM as it is in mlx4 device.
Do you have firmware?"

[1] http://marc.info/?l=linux-rdma=146545553104913=2

> 
> Regards
> Wei Hu
> 
> >
> 


signature.asc
Description: Digital signature

Re: [PATCH v10 08/22] IB/hns: Add icm support

2016-06-20 Thread Leon Romanovsky 
On Mon, Jun 20, 2016 at 12:37:40PM +0800, Wei Hu (Xavier) wrote:
> 
> 
> On 2016/6/17 17:58, Leon Romanovsky wrote:
> >On Thu, Jun 16, 2016 at 10:35:16PM +0800, Lijun Ou wrote:
> >>This patch mainly added icm support for RoCE. It initializes icm
> >>which managers the relative memory blocks for RoCE. The data
> >>structures of RoCE will be located in it. For example, CQ table,
> >>QP table and MTPT table so on.
> >>
> >>Signed-off-by: Wei Hu 
> >>Signed-off-by: Nenglong Zhao 
> >>Signed-off-by: Lijun Ou 
> >>---
> ><...>
> >
> >>+
> >>+static int hns_roce_alloc_icm_pages(struct scatterlist *mem, int order,
> >>+   gfp_t gfp_mask)
> >>+{
> >>+   struct page *page;
> >>+
> >>+   page = alloc_pages(gfp_mask, order);
> >>+   if (!page)
> >>+   return -ENOMEM;
> >>+
> >>+   sg_set_page(mem, page, PAGE_SIZE << order, 0);
> >>+
> >>+   return 0;
> >>+}
> >>+
> >>+static int hns_roce_alloc_icm_coherent(struct device *dev,
> >>+  struct scatterlist *mem, int order,
> >>+  gfp_t gfp_mask)
> >>+{
> >>+   void *buf = dma_alloc_coherent(dev, PAGE_SIZE << order,
> >>+  _dma_address(mem), gfp_mask);
> >>+   if (!buf)
> >>+   return -ENOMEM;
> >>+
> >>+   sg_set_buf(mem, buf, PAGE_SIZE << order);
> >>+   WARN_ON(mem->offset);
> >>+   sg_dma_len(mem) = PAGE_SIZE << order;
> >>+   return 0;
> >>+}
> >>+
> ><...>
> >
> >>+
> >>+static void hns_roce_free_icm_pages(struct hns_roce_dev *hr_dev,
> >>+   struct hns_roce_icm_chunk *chunk)
> >>+{
> >>+   int i;
> >>+
> >>+   if (chunk->nsg > 0)
> >>+   dma_unmap_sg(_dev->pdev->dev, chunk->mem, chunk->npages,
> >>+DMA_BIDIRECTIONAL);
> >>+
> >>+   for (i = 0; i < chunk->npages; ++i)
> >>+   __free_pages(sg_page(>mem[i]),
> >>+get_order(chunk->mem[i].length));
> >You used alloc_pages for this allocation, so why are you using
> >__free_pages instead of free_pages?
> Hi, Leon
> The function prototype of these functions as below：
> static inline struct page * alloc_pages(gfp_t gfp_mask, unsigned int
> order);
> void free_pages(unsigned long addr, unsigned int order);
> void __free_pages(struct page *page, unsigned int order);
> 
> The type of the first parameter of free_pages is same with the type of
> return value of alloc_pages.
> Maybe it is better to call __free_pages to release memory that allocated
> by calling alloc_pages.

OK, I see.

Another question which you didn't answer [1].

"I wonder if you have the same needs for ICM as it is in mlx4 device.
Do you have firmware?"

[1] http://marc.info/?l=linux-rdma=146545553104913=2

> 
> Regards
> Wei Hu
> 
> >
> 


signature.asc
Description: Digital signature

Re: [PATCH v10 08/22] IB/hns: Add icm support

2016-06-17 Thread Leon Romanovsky 
On Thu, Jun 16, 2016 at 10:35:16PM +0800, Lijun Ou wrote:
> This patch mainly added icm support for RoCE. It initializes icm
> which managers the relative memory blocks for RoCE. The data
> structures of RoCE will be located in it. For example, CQ table,
> QP table and MTPT table so on.
> 
> Signed-off-by: Wei Hu 
> Signed-off-by: Nenglong Zhao 
> Signed-off-by: Lijun Ou 
> ---

<...>

> +
> +static int hns_roce_alloc_icm_pages(struct scatterlist *mem, int order,
> + gfp_t gfp_mask)
> +{
> + struct page *page;
> +
> + page = alloc_pages(gfp_mask, order);
> + if (!page)
> + return -ENOMEM;
> +
> + sg_set_page(mem, page, PAGE_SIZE << order, 0);
> +
> + return 0;
> +}
> +
> +static int hns_roce_alloc_icm_coherent(struct device *dev,
> +struct scatterlist *mem, int order,
> +gfp_t gfp_mask)
> +{
> + void *buf = dma_alloc_coherent(dev, PAGE_SIZE << order,
> +_dma_address(mem), gfp_mask);
> + if (!buf)
> + return -ENOMEM;
> +
> + sg_set_buf(mem, buf, PAGE_SIZE << order);
> + WARN_ON(mem->offset);
> + sg_dma_len(mem) = PAGE_SIZE << order;
> + return 0;
> +}
> +

<...>

> +
> +static void hns_roce_free_icm_pages(struct hns_roce_dev *hr_dev,
> + struct hns_roce_icm_chunk *chunk)
> +{
> + int i;
> +
> + if (chunk->nsg > 0)
> + dma_unmap_sg(_dev->pdev->dev, chunk->mem, chunk->npages,
> +  DMA_BIDIRECTIONAL);
> +
> + for (i = 0; i < chunk->npages; ++i)
> + __free_pages(sg_page(>mem[i]),
> +  get_order(chunk->mem[i].length));

You used alloc_pages for this allocation, so why are you using
__free_pages instead of free_pages?




signature.asc
Description: Digital signature

Re: [PATCH v10 08/22] IB/hns: Add icm support

2016-06-17 Thread Leon Romanovsky 
On Thu, Jun 16, 2016 at 10:35:16PM +0800, Lijun Ou wrote:
> This patch mainly added icm support for RoCE. It initializes icm
> which managers the relative memory blocks for RoCE. The data
> structures of RoCE will be located in it. For example, CQ table,
> QP table and MTPT table so on.
> 
> Signed-off-by: Wei Hu 
> Signed-off-by: Nenglong Zhao 
> Signed-off-by: Lijun Ou 
> ---

<...>

> +
> +static int hns_roce_alloc_icm_pages(struct scatterlist *mem, int order,
> + gfp_t gfp_mask)
> +{
> + struct page *page;
> +
> + page = alloc_pages(gfp_mask, order);
> + if (!page)
> + return -ENOMEM;
> +
> + sg_set_page(mem, page, PAGE_SIZE << order, 0);
> +
> + return 0;
> +}
> +
> +static int hns_roce_alloc_icm_coherent(struct device *dev,
> +struct scatterlist *mem, int order,
> +gfp_t gfp_mask)
> +{
> + void *buf = dma_alloc_coherent(dev, PAGE_SIZE << order,
> +_dma_address(mem), gfp_mask);
> + if (!buf)
> + return -ENOMEM;
> +
> + sg_set_buf(mem, buf, PAGE_SIZE << order);
> + WARN_ON(mem->offset);
> + sg_dma_len(mem) = PAGE_SIZE << order;
> + return 0;
> +}
> +

<...>

> +
> +static void hns_roce_free_icm_pages(struct hns_roce_dev *hr_dev,
> + struct hns_roce_icm_chunk *chunk)
> +{
> + int i;
> +
> + if (chunk->nsg > 0)
> + dma_unmap_sg(_dev->pdev->dev, chunk->mem, chunk->npages,
> +  DMA_BIDIRECTIONAL);
> +
> + for (i = 0; i < chunk->npages; ++i)
> + __free_pages(sg_page(>mem[i]),
> +  get_order(chunk->mem[i].length));

You used alloc_pages for this allocation, so why are you using
__free_pages instead of free_pages?




signature.asc
Description: Digital signature

[PATCH v10 08/22] IB/hns: Add icm support

2016-06-16 Thread Lijun Ou 
This patch mainly added icm support for RoCE. It initializes icm
which managers the relative memory blocks for RoCE. The data
structures of RoCE will be located in it. For example, CQ table,
QP table and MTPT table so on.

Signed-off-by: Wei Hu 
Signed-off-by: Nenglong Zhao 
Signed-off-by: Lijun Ou 
---
PATCH v9/v8/v7/v6:
- No change over the PATCH v5

PATCH v5:
- The initial patch which was redesigned based on the second patch
  in PATCH v4
---
---
 drivers/infiniband/hw/hns/hns_roce_common.h |  19 ++
 drivers/infiniband/hw/hns/hns_roce_device.h |  30 ++
 drivers/infiniband/hw/hns/hns_roce_icm.c| 460 
 drivers/infiniband/hw/hns/hns_roce_icm.h| 119 +++
 drivers/infiniband/hw/hns/hns_roce_main.c   |  84 +
 5 files changed, 712 insertions(+)
 create mode 100644 drivers/infiniband/hw/hns/hns_roce_icm.c
 create mode 100644 drivers/infiniband/hw/hns/hns_roce_icm.h

diff --git a/drivers/infiniband/hw/hns/hns_roce_common.h 
b/drivers/infiniband/hw/hns/hns_roce_common.h
index 4805852..f15bf1b 100644
--- a/drivers/infiniband/hw/hns/hns_roce_common.h
+++ b/drivers/infiniband/hw/hns/hns_roce_common.h
@@ -53,6 +53,22 @@
 #define roce_set_bit(origin, shift, val) \
roce_set_field((origin), (1ul << (shift)), (shift), (val))
 
+#define ROCEE_BT_CMD_H_ROCEE_BT_CMD_IN_MDF_S 0
+#define ROCEE_BT_CMD_H_ROCEE_BT_CMD_IN_MDF_M   \
+   (((1UL << 19) - 1) << ROCEE_BT_CMD_H_ROCEE_BT_CMD_IN_MDF_S)
+
+#define ROCEE_BT_CMD_H_ROCEE_BT_CMD_S 19
+
+#define ROCEE_BT_CMD_H_ROCEE_BT_CMD_MDF_S 20
+#define ROCEE_BT_CMD_H_ROCEE_BT_CMD_MDF_M   \
+   (((1UL << 2) - 1) << ROCEE_BT_CMD_H_ROCEE_BT_CMD_MDF_S)
+
+#define ROCEE_BT_CMD_H_ROCEE_BT_CMD_BA_H_S 22
+#define ROCEE_BT_CMD_H_ROCEE_BT_CMD_BA_H_M   \
+   (((1UL << 5) - 1) << ROCEE_BT_CMD_H_ROCEE_BT_CMD_BA_H_S)
+
+#define ROCEE_BT_CMD_H_ROCEE_BT_CMD_HW_SYNS_S 31
+
 #define ROCEE_CAEP_AEQC_AEQE_SHIFT_CAEP_AEQC_STATE_S 0
 #define ROCEE_CAEP_AEQC_AEQE_SHIFT_CAEP_AEQC_STATE_M   \
(((1UL << 2) - 1) << ROCEE_CAEP_AEQC_AEQE_SHIFT_CAEP_AEQC_STATE_S)
@@ -93,6 +109,8 @@
 #define ROCEE_SYS_IMAGE_GUID_L_REG 0xC
 #define ROCEE_SYS_IMAGE_GUID_H_REG 0x10
 
+#define ROCEE_BT_CMD_H_REG 0x204
+
 #define ROCEE_CAEP_AEQE_CONS_IDX_REG   0x3AC
 #define ROCEE_CAEP_CEQC_CONS_IDX_0_REG 0x3BC
 
@@ -105,6 +123,7 @@
 
 #define ROCEE_CAEP_CE_INTERVAL_CFG_REG 0x190
 #define ROCEE_CAEP_CE_BURST_NUM_CFG_REG0x194
+#define ROCEE_BT_CMD_L_REG 0x200
 
 #define ROCEE_MB1_REG  0x210
 
diff --git a/drivers/infiniband/hw/hns/hns_roce_device.h 
b/drivers/infiniband/hw/hns/hns_roce_device.h
index 57184ab..ab9ba61 100644
--- a/drivers/infiniband/hw/hns/hns_roce_device.h
+++ b/drivers/infiniband/hw/hns/hns_roce_device.h
@@ -111,6 +111,26 @@ enum {
HNS_ROCE_CMD_SUCCESS= 1,
 };
 
+struct hns_roce_icm_table {
+   /* ICM type: 0 = qpc 1 = mtt 2 = cqc 3 = srq 4 = other */
+   u32 type;
+   /* ICM array elment num */
+   unsigned long   num_icm;
+   /* ICM entry record obj total num */
+   unsigned long   num_obj;
+   /*Single obj size */
+   unsigned long   obj_size;
+   int lowmem;
+   int coherent;
+   struct mutexmutex;
+   struct hns_roce_icm **icm;
+};
+
+struct hns_roce_mr_table {
+   struct hns_roce_icm_table   mtt_table;
+   struct hns_roce_icm_table   mtpt_table;
+};
+
 struct hns_roce_buf_list {
void*buf;
dma_addr_t  map;
@@ -126,11 +146,14 @@ struct hns_roce_cq {
 
 struct hns_roce_qp_table {
spinlock_t  lock;
+   struct hns_roce_icm_table   qp_table;
+   struct hns_roce_icm_table   irrl_table;
 };
 
 struct hns_roce_cq_table {
spinlock_t  lock;
struct radix_tree_root  tree;
+   struct hns_roce_icm_table   table;
 };
 
 struct hns_roce_cmd_context {
@@ -259,6 +282,7 @@ struct hns_roce_dev {
struct ib_deviceib_dev;
struct platform_device  *pdev;
const char  *irq_names;
+   spinlock_t  bt_cmd_lock;
struct hns_roce_ib_iboe iboe;
 
int irq[HNS_ROCE_MAX_IRQ_NUM];
@@ -273,6 +297,7 @@ struct hns_roce_dev {
u32 hw_rev;
 
struct hns_roce_cmdqcmd;
+   struct hns_roce_mr_table  mr_table;
struct hns_roce_cq_table  cq_table;
struct hns_roce_qp_table  qp_table;
struct hns_roce_eq_table  eq_table;
@@ -282,6 +307,11 @@ struct hns_roce_dev {
struct hns_roce_hw  *hw;
 };
 
+static inline void hns_roce_write64_k(__be32 val[2], void __iomem *dest)
+{
+   __raw_writeq(*(u64 *) val, dest);
+}
+
 static inline struct hns_roce_qp

[PATCH v10 08/22] IB/hns: Add icm support

2016-06-16 Thread Lijun Ou 
This patch mainly added icm support for RoCE. It initializes icm
which managers the relative memory blocks for RoCE. The data
structures of RoCE will be located in it. For example, CQ table,
QP table and MTPT table so on.

Signed-off-by: Wei Hu 
Signed-off-by: Nenglong Zhao 
Signed-off-by: Lijun Ou 
---
PATCH v9/v8/v7/v6:
- No change over the PATCH v5

PATCH v5:
- The initial patch which was redesigned based on the second patch
  in PATCH v4
---
---
 drivers/infiniband/hw/hns/hns_roce_common.h |  19 ++
 drivers/infiniband/hw/hns/hns_roce_device.h |  30 ++
 drivers/infiniband/hw/hns/hns_roce_icm.c| 460 
 drivers/infiniband/hw/hns/hns_roce_icm.h| 119 +++
 drivers/infiniband/hw/hns/hns_roce_main.c   |  84 +
 5 files changed, 712 insertions(+)
 create mode 100644 drivers/infiniband/hw/hns/hns_roce_icm.c
 create mode 100644 drivers/infiniband/hw/hns/hns_roce_icm.h

diff --git a/drivers/infiniband/hw/hns/hns_roce_common.h 
b/drivers/infiniband/hw/hns/hns_roce_common.h
index 4805852..f15bf1b 100644
--- a/drivers/infiniband/hw/hns/hns_roce_common.h
+++ b/drivers/infiniband/hw/hns/hns_roce_common.h
@@ -53,6 +53,22 @@
 #define roce_set_bit(origin, shift, val) \
roce_set_field((origin), (1ul << (shift)), (shift), (val))
 
+#define ROCEE_BT_CMD_H_ROCEE_BT_CMD_IN_MDF_S 0
+#define ROCEE_BT_CMD_H_ROCEE_BT_CMD_IN_MDF_M   \
+   (((1UL << 19) - 1) << ROCEE_BT_CMD_H_ROCEE_BT_CMD_IN_MDF_S)
+
+#define ROCEE_BT_CMD_H_ROCEE_BT_CMD_S 19
+
+#define ROCEE_BT_CMD_H_ROCEE_BT_CMD_MDF_S 20
+#define ROCEE_BT_CMD_H_ROCEE_BT_CMD_MDF_M   \
+   (((1UL << 2) - 1) << ROCEE_BT_CMD_H_ROCEE_BT_CMD_MDF_S)
+
+#define ROCEE_BT_CMD_H_ROCEE_BT_CMD_BA_H_S 22
+#define ROCEE_BT_CMD_H_ROCEE_BT_CMD_BA_H_M   \
+   (((1UL << 5) - 1) << ROCEE_BT_CMD_H_ROCEE_BT_CMD_BA_H_S)
+
+#define ROCEE_BT_CMD_H_ROCEE_BT_CMD_HW_SYNS_S 31
+
 #define ROCEE_CAEP_AEQC_AEQE_SHIFT_CAEP_AEQC_STATE_S 0
 #define ROCEE_CAEP_AEQC_AEQE_SHIFT_CAEP_AEQC_STATE_M   \
(((1UL << 2) - 1) << ROCEE_CAEP_AEQC_AEQE_SHIFT_CAEP_AEQC_STATE_S)
@@ -93,6 +109,8 @@
 #define ROCEE_SYS_IMAGE_GUID_L_REG 0xC
 #define ROCEE_SYS_IMAGE_GUID_H_REG 0x10
 
+#define ROCEE_BT_CMD_H_REG 0x204
+
 #define ROCEE_CAEP_AEQE_CONS_IDX_REG   0x3AC
 #define ROCEE_CAEP_CEQC_CONS_IDX_0_REG 0x3BC
 
@@ -105,6 +123,7 @@
 
 #define ROCEE_CAEP_CE_INTERVAL_CFG_REG 0x190
 #define ROCEE_CAEP_CE_BURST_NUM_CFG_REG0x194
+#define ROCEE_BT_CMD_L_REG 0x200
 
 #define ROCEE_MB1_REG  0x210
 
diff --git a/drivers/infiniband/hw/hns/hns_roce_device.h 
b/drivers/infiniband/hw/hns/hns_roce_device.h
index 57184ab..ab9ba61 100644
--- a/drivers/infiniband/hw/hns/hns_roce_device.h
+++ b/drivers/infiniband/hw/hns/hns_roce_device.h
@@ -111,6 +111,26 @@ enum {
HNS_ROCE_CMD_SUCCESS= 1,
 };
 
+struct hns_roce_icm_table {
+   /* ICM type: 0 = qpc 1 = mtt 2 = cqc 3 = srq 4 = other */
+   u32 type;
+   /* ICM array elment num */
+   unsigned long   num_icm;
+   /* ICM entry record obj total num */
+   unsigned long   num_obj;
+   /*Single obj size */
+   unsigned long   obj_size;
+   int lowmem;
+   int coherent;
+   struct mutexmutex;
+   struct hns_roce_icm **icm;
+};
+
+struct hns_roce_mr_table {
+   struct hns_roce_icm_table   mtt_table;
+   struct hns_roce_icm_table   mtpt_table;
+};
+
 struct hns_roce_buf_list {
void*buf;
dma_addr_t  map;
@@ -126,11 +146,14 @@ struct hns_roce_cq {
 
 struct hns_roce_qp_table {
spinlock_t  lock;
+   struct hns_roce_icm_table   qp_table;
+   struct hns_roce_icm_table   irrl_table;
 };
 
 struct hns_roce_cq_table {
spinlock_t  lock;
struct radix_tree_root  tree;
+   struct hns_roce_icm_table   table;
 };
 
 struct hns_roce_cmd_context {
@@ -259,6 +282,7 @@ struct hns_roce_dev {
struct ib_deviceib_dev;
struct platform_device  *pdev;
const char  *irq_names;
+   spinlock_t  bt_cmd_lock;
struct hns_roce_ib_iboe iboe;
 
int irq[HNS_ROCE_MAX_IRQ_NUM];
@@ -273,6 +297,7 @@ struct hns_roce_dev {
u32 hw_rev;
 
struct hns_roce_cmdqcmd;
+   struct hns_roce_mr_table  mr_table;
struct hns_roce_cq_table  cq_table;
struct hns_roce_qp_table  qp_table;
struct hns_roce_eq_table  eq_table;
@@ -282,6 +307,11 @@ struct hns_roce_dev {
struct hns_roce_hw  *hw;
 };
 
+static inline void hns_roce_write64_k(__be32 val[2], void __iomem *dest)
+{
+   __raw_writeq(*(u64 *) val, dest);
+}
+
 static inline struct hns_roce_qp
*__hns_roce_qp_lookup(struct hns_roce_dev *hr_dev, u32 qpn)
 {
diff --git