[ewg] Re: [ofa-general] RFC: SRC API
On Sun, Jul 29, 2007 at 05:04:31PM +0300, Michael S. Tsirkin wrote:
Hello!
Here is an API proposal for support of the SRC
(scalable reliable connected) protocol extension in libibverbs.
This adds APIs to:
- manage SRC domains
- share SRC domains between processes,
by means of creating a 1:1 association
between an SRC domain and a file.
Notes:
- The file is specified by means of a file descriptor,
this makes it possible for the user to manage file
creation/deletion in the most flexible manner
(e.g. tmpfile can be used).
- I envision implementing this sharing mechanism in kernel by means
of a per-device tree, with inode as a key and domain object
as a value.
Please comment.
Can you provide a pseudo code of an application using this API?
Especially QP sharing part.
Signed-off-by: Michael S. Tsirkin [EMAIL PROTECTED]
---
diff --git a/include/infiniband/verbs.h b/include/infiniband/verbs.h
index acc1b82..503f201 100644
--- a/include/infiniband/verbs.h
+++ b/include/infiniband/verbs.h
@@ -370,6 +370,11 @@ struct ibv_ah_attr {
uint8_t port_num;
};
+struct ibv_src_domain {
+ struct ibv_context *context;
+ uint32_thandle;
+};
+
enum ibv_srq_attr_mask {
IBV_SRQ_MAX_WR = 1 0,
IBV_SRQ_LIMIT = 1 1
@@ -389,7 +394,8 @@ struct ibv_srq_init_attr {
enum ibv_qp_type {
IBV_QPT_RC = 2,
IBV_QPT_UC,
- IBV_QPT_UD
+ IBV_QPT_UD,
+ IBV_QPT_SRC
};
struct ibv_qp_cap {
@@ -408,6 +414,7 @@ struct ibv_qp_init_attr {
struct ibv_qp_cap cap;
enum ibv_qp_typeqp_type;
int sq_sig_all;
+ struct ibv_src_domain *src_domain;
};
enum ibv_qp_attr_mask {
@@ -526,6 +533,7 @@ struct ibv_send_wr {
uint32_tremote_qkey;
} ud;
} wr;
+ uint32_tsrc_remote_srq_num;
};
struct ibv_recv_wr {
@@ -553,6 +561,10 @@ struct ibv_srq {
pthread_mutex_t mutex;
pthread_cond_t cond;
uint32_tevents_completed;
+
+ uint32_tsrc_srq_num;
+ struct ibv_src_domain *src_domain;
+ struct ibv_cq *src_cq;
};
struct ibv_qp {
@@ -570,6 +582,8 @@ struct ibv_qp {
pthread_mutex_t mutex;
pthread_cond_t cond;
uint32_tevents_completed;
+
+ struct ibv_src_domain *src_domain;
};
struct ibv_comp_channel {
@@ -912,6 +926,25 @@ struct ibv_srq *ibv_create_srq(struct ibv_pd *pd,
struct ibv_srq_init_attr *srq_init_attr);
/**
+ * ibv_create_src_srq - Creates a SRQ associated with the specified
protection
+ * domain and src domain.
+ * @pd: The protection domain associated with the SRQ.
+ * @src_domain: The SRC domain associated with the SRQ.
+ * @src_cq: CQ to report completions for SRC packets on.
+ *
+ * @srq_init_attr: A list of initial attributes required to create the SRQ.
+ *
+ * srq_attr-max_wr and srq_attr-max_sge are read the determine the
+ * requested size of the SRQ, and set to the actual values allocated
+ * on return. If ibv_create_srq() succeeds, then max_wr and max_sge
+ * will always be at least as large as the requested values.
+ */
+struct ibv_srq *ibv_create_src_srq(struct ibv_pd *pd,
+struct ibv_src_domain *src_domain,
+struct ibv_cq *src_cq,
+struct ibv_srq_init_attr *srq_init_attr);
+
+/**
* ibv_modify_srq - Modifies the attributes for the specified SRQ.
* @srq: The SRQ to modify.
* @srq_attr: On input, specifies the SRQ attributes to modify. On output,
@@ -1074,6 +1107,44 @@ int ibv_detach_mcast(struct ibv_qp *qp, union ibv_gid
*gid, uint16_t lid);
*/
int ibv_fork_init(void);
+/**
+ * ibv_alloc_src_domain - Allocate an SRC domain
+ * Returns a reference to an SRC domain.
+ * Use ibv_put_src_domain to free the reference.
+ * @context: Device context
+ */
+struct ibv_src_domain *ibv_get_new_src_domain(struct ibv_context *context);
+
+/**
+ * ibv_share_src_domain - associate the src domain with a file.
+ * Establishes a connection between an SRC domain object and a file
descriptor.
+ *
+ * @d: SRC domain to share
+ * @fd: descriptor for a file to associate with the domain
+ */
+int ibv_share_src_domain(struct ibv_src_domain *d, int fd);
+
+/**
+ * ibv_unshare_src_domain - disassociate the src domain from a file.
+ * Subsequent calls to ibv_get_shared_src_domain will fail.
+ * @d: SRC domain to unshare
+ */
+int ibv_unshare_src_domain(struct ibv_src_domain *d);
+
+/**
+ * ibv_get_src_domain - get a reference to shared SRC domain
+ * @context: Device context
+ * @fd: descriptor for a file associated with the domain
+ */
+struct ibv_src_domain *ibv_get_shared_src_domain(struct
[ewg] Re: [ofa-general] RFC: SRC API
On Sun, Jul 29, 2007 at 05:04:31PM +0300, Michael S. Tsirkin wrote: Hello! Here is an API proposal for support of the SRC (scalable reliable connected) protocol extension in libibverbs. This adds APIs to: - manage SRC domains - share SRC domains between processes, by means of creating a 1:1 association between an SRC domain and a file. Notes: - The file is specified by means of a file descriptor, this makes it possible for the user to manage file creation/deletion in the most flexible manner (e.g. tmpfile can be used). - I envision implementing this sharing mechanism in kernel by means of a per-device tree, with inode as a key and domain object as a value. Please comment. Can you provide a pseudo code of an application using this API? Especially QP sharing part. There's no QP sharing here. You mean SRC domain sharing? -- MST ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] Re: [ofa-general] RFC: SRC API
On Mon, Jul 30, 2007 at 11:52:21AM +0300, Michael S. Tsirkin wrote: On Sun, Jul 29, 2007 at 05:04:31PM +0300, Michael S. Tsirkin wrote: Hello! Here is an API proposal for support of the SRC (scalable reliable connected) protocol extension in libibverbs. This adds APIs to: - manage SRC domains - share SRC domains between processes, by means of creating a 1:1 association between an SRC domain and a file. Notes: - The file is specified by means of a file descriptor, this makes it possible for the user to manage file creation/deletion in the most flexible manner (e.g. tmpfile can be used). - I envision implementing this sharing mechanism in kernel by means of a per-device tree, with inode as a key and domain object as a value. Please comment. Can you provide a pseudo code of an application using this API? Especially QP sharing part. There's no QP sharing here. You mean SRC domain sharing? Yes. Sorry. -- Gleb. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] Re: [ofa-general] RFC: SRC API
Some code examples: /* create a domain and share it: */ struct ibv_src_domain * d = ibv_get_new_src_domain(ctx); int fd = open(path, O_CREAT | O_RDWR, mode); ibv_share_src_domain(d, fd); /* get a reference to a shared domain: */ int fd = open(path, O_CREAT | O_RDWR, mode); struct ibv_src_domain * d = ibv_get_shared_src_domain(ctx, fd); /* once done: */ ibv_put_src_domain(d); Note: when all users do put, domain is destroyed. -- MST ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] Re: [ofa-general] RFC: SRC API
On Mon, Jul 30, 2007 at 12:01:40PM +0300, Michael S. Tsirkin wrote: Some code examples: /* create a domain and share it: */ struct ibv_src_domain * d = ibv_get_new_src_domain(ctx); int fd = open(path, O_CREAT | O_RDWR, mode); ibv_share_src_domain(d, fd); /* get a reference to a shared domain: */ int fd = open(path, O_CREAT | O_RDWR, mode); struct ibv_src_domain * d = ibv_get_shared_src_domain(ctx, fd); /* once done: */ ibv_put_src_domain(d); Note: when all users do put, domain is destroyed. OK. I am more interested in how SRC is connected to a QP in different processes. How a process will be able to do sends to different processes through one QP, etc... -- Gleb. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] Re: [ofa-general] RFC: SRC API
More code examples: Create an SRC QP, part of SRC domain: attr.qp_type = IBV_QPT_SRC; attr.src_domain = d; qp = ibv_create_qp(pd, attr); Given remote SRQ number, send data to this SRQ over an SRC QP: wr.src_remote_srq_num = src_remote_srq_num; ib_post_send(qp, wr); Note: SRQ number needs to be exchanged as part of CM private data or some other protocol. -- MST ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] Re: [ofa-general] RFC: SRC API
On Mon, Jul 30, 2007 at 12:16:39PM +0300, Michael S. Tsirkin wrote: More code examples: Create an SRC QP, part of SRC domain: attr.qp_type = IBV_QPT_SRC; attr.src_domain = d; qp = ibv_create_qp(pd, attr); Given remote SRQ number, send data to this SRQ over an SRC QP: wr.src_remote_srq_num = src_remote_srq_num; ib_post_send(qp, wr); Note: SRQ number needs to be exchanged as part of CM private data or some other protocol. You are too brief. I can come up with one linears based on the API by myself. I am trying to understand how sharing of SRC between processes will work and your example doesn't show this. Can I connected the same SRC to different QPs? If yes, can I send packet to any SRQ connected to the SRC through any QP connected to the same SRC? If yes how is this different from having regular QPs? -- Gleb. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] Re: [ofa-general] RFC: SRC API
Quoting Gleb Natapov [EMAIL PROTECTED]: Subject: Re: [ofa-general] RFC: SRC API On Mon, Jul 30, 2007 at 12:16:39PM +0300, Michael S. Tsirkin wrote: More code examples: Create an SRC QP, part of SRC domain: attr.qp_type = IBV_QPT_SRC; attr.src_domain = d; qp = ibv_create_qp(pd, attr); Given remote SRQ number, send data to this SRQ over an SRC QP: wr.src_remote_srq_num = src_remote_srq_num; ib_post_send(qp, wr); Note: SRQ number needs to be exchanged as part of CM private data or some other protocol. You are too brief. I can come up with one linears based on the API by myself. I am trying to understand how sharing of SRC between processes will work and your example doesn't show this. It seems what you are missing is what SRC is, not how to use the API. I'll have a working example when I get closer to implementation. For now you'll have to look up Dror's preso if you want to understand what SRC is. Can I connected the same SRC to different QPs? If yes, can I send packet to any SRQ connected to the SRC through any QP connected to the same SRC? Yes to both. If yes how is this different from having regular QPs? With regular QP you can only send to a single SRQ. But again, look at Dror's preso. -- MST ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] Re: [ofa-general] RFC: SRC API
It seems what you are missing is what SRC is, not how to use the API. So tell us. This calls for a separate document. From feedback from Sonoma I really assumed people have it figured out. Let's open a separate thread, and there I will try writing up what SRC is from the protocol point of view. -- MST ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] Re: [ofa-general] RFC: SRC API
On Mon, Jul 30, 2007 at 03:10:57PM +0300, Michael S. Tsirkin wrote: It seems what you are missing is what SRC is, not how to use the API. So tell us. This calls for a separate document. From feedback from Sonoma I really assumed people have it figured out. Let's open a separate thread, and there I will try writing up what SRC is from the protocol point of view. No problem. Start it :) -- Gleb. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
