wwbmmm commented on code in PR #3746:
URL: https://github.com/apache/brpc/pull/3746#discussion_r4582278249


##########
src/brpc/rdma/rdma_helper.cpp:
##########
@@ -715,10 +715,13 @@ void GlobalDisableRdma() {
 
 bool SupportedByRdma(std::string protocol) {
     if (protocol.compare("baidu_std") == 0) {
-        // Since rdma is used for high performance scenario,
-        // we consider baidu_std for the only protocol to support.
         return true;
     }
+#if BRPC_WITH_FLATBUFFERS

Review Comment:
   [replied by brpc-oncall robot] This branch (and the `#if BRPC_WITH_RDMA` 
block in `src/brpc/flatbuffers/message.cpp`) is only compiled when both 
`BRPC_WITH_RDMA` and `BRPC_WITH_FLATBUFFERS` are on, and no CI job enables that 
combination: the FlatBuffers ON workflow builds without RDMA, and the RDMA jobs 
in ci-linux.yml build without FlatBuffers. The new 
`brpc_flatbuffers_rdma_unittest` target is `manual`/`EXCLUDE_FROM_ALL`, so it 
does not help here either — none of this code is ever compiled in CI. Please 
add at least a compile-only coverage for the combination, e.g. add `--define 
BRPC_WITH_FLATBUFFERS=true` to the `gcc-compile-with-bazel-all-options` job, 
which already installs libibverbs-dev and passes `--define BRPC_WITH_RDMA=true` 
(flatbuffers comes from `@com_github_google_flatbuffers`, so no hardware or 
extra package is needed).



##########
src/brpc/flatbuffers/message.cpp:
##########
@@ -20,12 +20,66 @@
 #if BRPC_WITH_FLATBUFFERS
 #include <cstring>
 #include <cerrno>
+#include <cstdlib>
 #include "butil/logging.h"
+#if BRPC_WITH_RDMA
+#include "butil/iobuf_inl.h"
+#include "butil/memory/scope_guard.h"
+#include "brpc/rdma/block_pool.h"
+#include "brpc/rdma/rdma_helper.h"
+#endif  // BRPC_WITH_RDMA
 
 namespace brpc {
 namespace flatbuffers {
 namespace {
 
+void* AllocateStorage(butil::SingleIOBuf* storage, uint32_t size) {
+#if BRPC_WITH_RDMA
+    // SingleIOBuf normally uses the registered IOBuf pool. Its largest block
+    // class (type 2) is 2 MiB, including the IOBuf block header. FlatBuffers
+    // requires contiguous storage, so larger buffers need their own MR.
+    if (!rdma::IsRdmaAvailable() ||

Review Comment:
   [replied by brpc-oncall robot] Two robustness gaps in this fallback:
   
   1. The pool-vs-MR decision is a pure size threshold, so an allocation that 
fits the threshold still fails when the 2 MiB pool blocks are exhausted 
(`AllocBlock()` returns nullptr once `-rdma_memory_pool_max_regions` is reached 
and auto-extend is disabled, see block_pool.cpp), and 
`SlabAllocator::allocate()` then aborts the process. Trying 
`storage->allocate(size)` first and falling back to a registered `malloc`'d 
buffer only when it returns nullptr would cover both cases.
   2. If `RegisterMemoryForRdma()` fails (e.g. RLIMIT_MEMLOCK or device MR 
exhaustion) this function returns nullptr, and `SlabAllocator::allocate()` 
turns that into a `RELEASE_ASSERT_VERBOSE` abort. Because `IsRdmaAvailable()` 
is process-wide, every FlatBuffers message larger than 2 MiB then needs a 
successful `ibv_reg_mr`, including messages that are only ever sent over a 
plain TCP channel, and the TCP path also pays two MR registrations per large 
message. Returning the plain (unregistered) buffer when registration fails 
would let the TCP paths keep working and only make the RDMA send of that 
message fail with ERDMAMEM instead of killing the process.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to