All of the questions that you are asking can be answered by the IB spec.
But i will try to do my best to answer anyway...
Kalbag, Ashwin wrote:
Question 1:
Here's a section of the man page for ibv_post_send:
"The attribute send_flags describes the properties of the WR. It is
either 0 or the bitwise OR of one or more of the following flags:
IBV_SEND_FENCE Set the fence indicator. Valid only for QPs with
Transport Service Type IBV_QPT_RC
IBV_SEND_SIGNALED Set the completion notification indicator.
Relevant only if QP was created with sq_sig_all=0
IBV_SEND_SOLICITED Set the solicited event indicator. Valid only
for Send and RDMA Write with immediate
IBV_SEND_INLINE Send data in given gather list as inline data
in a send WQE. Valid only for Send and RDMA Write. The
L_Key will not be checked."
a. What is the fence indicator? Under what circumstances would I use
this?
Here is the line from the IB spec describing the use of the fence indicator:
"When the Fence Indicator has been set in a Work Request, the
Send Queue shall not begin processing that Work Request until all prior
RDMA Read and Atomic Operations on that Send Queue have completed."
b. How is a solicited event different from a signaled event? Under what
circumstances would I use this?
Signal influence about the requestor (sender) and solicited influence
about the responser (receiver).
There isn't any connection between them and they can used alone (only
one of them will be set)
or together.
Signaled means that a completion will be created when did SR will be
finished
(if the QP was created with sq_sig_all=0 this mens that only ended SR
with the SIGNAL indicator will
create a completion. if sq_sig_all=1 EVERY SR will create a completion
when it ends).
The solicited event means that a special event called solicited event
will occur in the OTHER side
and he will be able to wait of completion event with the solicited bit set
(ibv_req_notify_cq(cq, 1) will wait until such completion is being created).
Using the solicited bit the sender can influence when the receiver side
will handle the received completions.
c. What is not apparent from this man page is whether the signal is
generated on the sending side on send completion or on the receiving
side on completion of the corresponding posted recv. It's not
explicitly stated, but I am assuming that the signaling refers to send
completion on the sending side. Is it true that regardless of whether
the send is signaled on the sending side, it will generate a signal on
the receiving side when the recv operation completes?
I don't quite understand your question, but i think that the answered
that i gave will answer this.
If not, can you please rephrase the question?
Question 2:
Say I'm trying to optimize between polling and completion event
notification. Could you please see whether I'm conceptualizing this
correctly?
At one extreme, you could poll continuously (without signaling sends),
until you want to send the next message. The polling would take care of
recycling memory region elements on send completions, handling received
messages for recv completions. At the other extreme, every event would
be signaled. You could set sg_sig_all=1 while creating the completion
queue, or alternatively always use the IBV_SEND_SIGNALED flag when
posting sends.
FYI: The term is every "Send Request".
And yes, completion of WRs (send or receiver request) means that
the operation was completed and you can reclaim the buffers that it
accessed and reuse/free them.
If sg_sig_all=0 you can make sure that completions won't be created for
all of the WR.
To take the middle road, one possibility is to generate completions for
every nth send. If the messages are being sent rapidly, you can afford
to signal fewer sends. If sends are fewer, you would need to either
signal more sends, say every send, or compensate by expending CPU in
polling. Seems like the signaling needs to be adaptive to the rate of
sending.
You are right, this is what this mechanism is all about.
If this is the case, you still have a choice in how you process send
completions, assuming that signals will always be generated on the
receiving side.
In the Receiver side completions will be created for EVERY WR.
Even if you didn't signal every send, you could still
process send completions by relying exclusively on signaling instead of
polling. If you did this, you'd need to signal at least once per "send
queue depth", correct? There is no urgency in processing send
completions so long as you have some available depth in the send queue
and elements to post sends with. But there is urgency in processing
recv completions. So, it may be more optimal to process send
completions with fewer signals, say 2 signals per send queue depth, for
a margin of safety, so at any time, you will have half the send queue
available. This asymmetry would imply it is better to use separate
completion queues for sending and receiving. Am I right in surmising
this?
In the receiver side you have completion for every WR because you don't
know when a message is available,
but in the sender side you knows when you send the message, so you don't
have to create a completion
for every created SR.
You can separate the Send CQ and the Receive CQ (for example: one of
them can be used with polling and the other
with completion events). This depend of the way you write your code.
Maybe handling one CQ will be easier for you ...
On the recv side, you want to process completions in a hurry and post
more recvs, to keep up with the incoming message traffic. Now here's
where you might benefit from polling even after your recv event
completion was signaled, so you avoid context switching out and back in,
and the concomitant delay, just in case there are actually more recvs
that completed. Is this correct?
This is Infiniband, you don't have any context switch in the data
operations.
Post SR and Poll completions are being executed in the context you are
in ....
You are right that incoming messages need to have more priority than
messages that you send.
I hope that i helped a little bit.
If there is any point that i missed, you are welcome to ask again
Dotan
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - -
This message is intended only for the personal and confidential use of the
designated recipient(s) named above. If you are not the intended recipient of
this message you are hereby notified that any review, dissemination,
distribution or copying of this message is strictly prohibited. This
communication is for information purposes only and should not be regarded as an
offer to sell or as a solicitation of an offer to buy any financial product, an
official confirmation of any transaction, or as an official statement of Lehman
Brothers. Email transmission cannot be guaranteed to be secure or error-free.
Therefore, we do not represent that this information is complete or accurate
and it should not be relied upon as such. All information is subject to change
without notice.
--------
IRS Circular 230 Disclosure:
Please be advised that any discussion of U.S. tax matters contained within this
communication (including any attachments) is not intended or written to be used
and cannot be used for the purpose of (i) avoiding U.S. tax related penalties
or (ii) promoting, marketing or recommending to another party any transaction
or matter addressed herein.
_______________________________________________
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
_______________________________________________
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
