alxrxs commented on code in PR #3569:
URL: https://github.com/apache/brpc/pull/3569#discussion_r4197452003
##########
src/brpc/rdma/rdma_endpoint.cpp:
##########
@@ -1128,7 +1161,65 @@ int RdmaEndpoint::PostRecv(uint32_t num, bool zerocopy) {
return 0;
}
-static ibv_qp* AllocateQp(ibv_cq* send_cq, ibv_cq* recv_cq, uint32_t sq_size,
uint32_t rq_size) {
+// Inline size to request: on the first try, size capped at the vendor's first
+// known limit; after a refusal, the vendor's next smaller known limit, else
+// one step smaller. Not static: exposed for UT.
+uint32_t InlineDataToRequest(uint32_t vendor_id, uint32_t size, bool refused) {
+ for (const auto& known : KNOWN_INLINE_LIMITS) {
+ if (known.vendor_id == vendor_id &&
+ (!refused || known.max_inline < size)) {
+ return std::min(known.max_inline, size);
+ }
+ }
+ if (refused) {
+ size = size > INLINE_DATA_STEP ? size - INLINE_DATA_STEP : 0;
+ }
+ return size;
+}
+
+// The inline size the last QP was created with (UINT32_MAX: none yet). All
+// QPs use the same device and attributes apart from their queue sizes, so
+// later QPs start here instead of repeating the step-down. Not static:
+// exposed for UT.
+butil::atomic<uint32_t> g_inline_data_request(UINT32_MAX);
+
+// Creates a QP with attr, asking for the inline size the last QP was created
+// with (at first BF_MAX_INLINE_DATA, capped by the vendor's known limit), or
+// less if the device refuses that size, and sets *max_inline_data to the
+// largest message to post inline on it. Not static: exposed for UT.
+ibv_qp* CreateQpWithInlineData(ibv_pd* pd, ibv_qp_init_attr* attr,
+ uint32_t vendor_id, uint32_t* max_inline_data) {
+ uint32_t inline_size =
+ g_inline_data_request.load(butil::memory_order_relaxed);
+ if (inline_size == UINT32_MAX) {
+ inline_size = InlineDataToRequest(vendor_id, BF_MAX_INLINE_DATA,
false);
+ }
+ attr->cap.max_inline_data = inline_size;
+ ibv_qp* qp = IbvCreateQp(pd, attr);
+ // A refused inline size fails with EINVAL; any other failure is left to
+ // the caller, which reports its errno.
+ while (qp == nullptr && errno == EINVAL && inline_size > 0) {
+ uint32_t next = InlineDataToRequest(vendor_id, inline_size, true);
+ LOG_IF(INFO, FLAGS_rdma_trace_verbose)
+ << "ibv_create_qp refused " << inline_size
+ << " bytes of inline data, trying " << next;
+ inline_size = next;
+ attr->cap.max_inline_data = inline_size;
+ qp = IbvCreateQp(pd, attr);
+ }
+ if (qp != nullptr) {
+ g_inline_data_request.store(inline_size, butil::memory_order_relaxed);
+ // ibv_create_qp writes the granted inline data size back into attr
+ *max_inline_data = attr->cap.max_inline_data;
+ if (vendor_id == MELLANOX_VENDOR_ID) {
+ *max_inline_data = std::min(*max_inline_data, BF_MAX_INLINE_DATA);
+ }
Review Comment:
This is on purpose. `ibv_create_qp()` either fails or grants at least what
was asked: ibv_create_qp(3) says "the values will be greater than or equal to
the values requested". So a size that was accepted once is accepted again on
the next QP with the same attributes, and the step-down doesn't repeat. Caching
the request keeps every QP asking for exactly what the first one asked for.
Each QP still reads its own grant from `attr.cap.max_inline_data`, and the
Mellanox cap is applied to that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
