Re: [PATCHv9 0/3] rdmacg: IB/core: rdma controller support

[Parav Pandit]

Hi Tejun,

On Wed, Mar 2, 2016 at 11:10 PM, Tejun Heo  wrote:
> Hello, Parav.
>
> It doesn't look like my reviews are getting through.  For now,
>
I have addressed all the review comments that you provided in v5 patch.
I admit that few comments have not followed CodingStyle and I owe to
fix it, which is bad on my part.
I have few 2 questions on your comments, I will ask there.

For cgroup lock - out of 3 proposals, you acknowledged that you are ok
with cgroup specific spin_lock, though you don't see that as big
saving.
Even though it can be made course gained by having single lock for the
whole rdma cgroup subsystem it raises contention among processes of
different cgroups, which is preferable to avoid.
So I have continued to have cgroup specific fine grain lock, which I
believe is ok as this is little efficient for first cut.

Including above one we agreed on almost all the points at design and
implementation level in previous patches.
If you see any issue, I am open to resolve them.

I will address comments given in patch v9.

> Nacked-by: Tejun Heo 
>
> Thanks.
>
> --
> tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCHv9 0/3] rdmacg: IB/core: rdma controller support

[Tejun Heo]

Hello, Parav.

It doesn't look like my reviews are getting through.  For now,

Nacked-by: Tejun Heo 

Thanks.

-- 
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCHv9 0/3] rdmacg: IB/core: rdma controller support

[Parav Pandit]

Patch is generated and tested against below Doug's linux-rdma
git tree as it merges cleanly with both the git tree
(linux-rdma and linux-cgroup).

URL: git://git.kernel.org/pub/scm/linux/kernel/git/dledford/rdma.git
Branch: k.o/for-4.6

Patch 1/3, 3/3 also compiles against Tejun's cgroup tree as well
and tested with little different patch of 2/3 as both the
trees are little different.

Overview:
Currently user space applications can easily take away all the rdma
device specific resources such as AH, CQ, QP, MR etc. Due to which other
applications in other cgroup or kernel space ULPs may not even get chance
to allocate any rdma resources. This results into service unavailibility.

RDMA cgroup addresses this issue by allowing resource accounting,
limit enforcement on per cgroup, per rdma device basis.

Resources are not defined by the RDMA cgroup. Instead they are defined
by RDMA/IB stack. This allows rdma cgroup to remain constant while RDMA/IB
stack can evolve without the need of rdma cgroup update. A new
resource can be easily added by the RDMA/IB stack without touching
rdma cgroup.

RDMA uverbs layer will enforce limits on well defined RDMA verb
resources without any HCA vendor device driver involvement.

RDMA uverbs layer will not do limit enforcement of HCA hw vendor
specific resources. Instead rdma cgroup provides set of APIs
through which vendor specific drivers can do resource accounting
by making use of rdma cgroup.

Resource limit enforcement is hierarchical.

When process is migrated with active RDMA resources, rdma cgroup
continues to uncharge original cgroup for allocated resource. New resource
is charged to current process's cgroup, which means if the process is
migrated with active resources, for new resources it will be charged to
new cgroup and old resources will be correctly uncharged from old cgroup.

Changes from v8:
 * Fixed compilation error.
 * Fixed warning reported by checkpatch script.

Changes from v7:
 * (To address comments from Haggai)
   1. Removed max_limit from query_limit function as it is
  unnecessary.
   2. Kept existing printk as it is to instead of replacing all
  with pr_warn except newly added printk.

Changes from v6:
 * (To address comments from Haggai)
   1. Made functions as void wherever necessary.
   2. Code cleanup related to correting few spelling mistakes
  in comments, correcting comments to reflect the code.
   3. Removed max_count parameter from query_limit as its not
  necessary.
   4. Fixed printk to pr_warn.
   5. Removed dependency on pd, instead relying on ib_dev.
   6. Added more documentation to reflect that IB stack honors
  configured limit during query_device operation.
   7. Added pr_warn and avoided system crash in case of
  IB stack or rdma cgroup bug.
 * (To address comments from Leon)
   1. Removed #ifdef CONFIG_CGROUP_RDMA from .c files and added
  necessary dummy functions in header file.
   2. Removed unwanted forward declaration.
 * Fixed uncharing to rdma controller after resource is released
   from verb layer, instead of uncharing first. This ensures that
   uncharging doesn't complete while resource is still allocated.
 
Changes from v5:
 * (To address comments from Tejun)
   1. Removed two type of resource pool, made is single type (as Tejun
  described in past comment)
   2. Removed match tokens and have array definition like "qp", "mr",
  "cq" etc.
   3. Wrote small parser and avoided match_token API as that won't work
  due to different array definitions
   4. Removed one-off remove API to unconfigure cgroup, instead all
  resource should be set to max.
   5. Removed resource pool type (user/default), instead having
  max_num_cnt, when ref_cnt drops to zero and
  max_num_cnt = total_rescource_cnt, pool is freed.
   6. Resource definition ownership is now only with IB stack at single
  header file, no longer in each low level driver.
  This goes through IB maintainer and other reviewers eyes.
  This continue to give flexibility to not force kernel upgrade for
  few enums additions for new resource type.
   7. Wherever possible pool lock is pushed out, except for hierarchical
  charging/unchanging points, as it not possible to do so, due to
  iterative process involves blocking allocations of rpool. Coming up
  more levels up to release locks doesn't make any sense either.
  This is anyway slow path where rpool is not allocated. Except for
  typical first resource allocation, this is less travelled path.
   8. Avoided %d manipulation due to removal of match_token and replaced
  with seq_putc etc friend functions.
 * Other minor cleanups.
 * Fixed rdmacg_register_device to return error in case of IB stack
   tries to register for than 64 resources.
 * Fixed not allowing negative value on resource setting.
 * Fixed cleaning up resource pools during device removal.
 * Simplfied and rename table length field to use ARRAY_SIZE macro.
 * Updated