wwbmmm commented on code in PR #3746:
URL: https://github.com/apache/brpc/pull/3746#discussion_r4582278249
##########
src/brpc/rdma/rdma_helper.cpp:
##########
@@ -715,10 +715,13 @@ void GlobalDisableRdma() {
bool SupportedByRdma(std::string protocol) {
if (protocol.compare("baidu_std") == 0) {
- // Since rdma is used for high performance scenario,
- // we consider baidu_std for the only protocol to support.
return true;
}
+#if BRPC_WITH_FLATBUFFERS
Review Comment:
[replied by brpc-oncall robot] This branch (and the `#if BRPC_WITH_RDMA`
block in `src/brpc/flatbuffers/message.cpp`) is only compiled when both
`BRPC_WITH_RDMA` and `BRPC_WITH_FLATBUFFERS` are on, and no CI job enables that
combination: the FlatBuffers ON workflow builds without RDMA, and the RDMA jobs
in ci-linux.yml build without FlatBuffers. The new
`brpc_flatbuffers_rdma_unittest` target is `manual`/`EXCLUDE_FROM_ALL`, so it
does not help here either — none of this code is ever compiled in CI. Please
add at least a compile-only coverage for the combination, e.g. add `--define
BRPC_WITH_FLATBUFFERS=true` to the `gcc-compile-with-bazel-all-options` job,
which already installs libibverbs-dev and passes `--define BRPC_WITH_RDMA=true`
(flatbuffers comes from `@com_github_google_flatbuffers`, so no hardware or
extra package is needed).
##########
src/brpc/flatbuffers/message.cpp:
##########
@@ -20,12 +20,66 @@
#if BRPC_WITH_FLATBUFFERS
#include <cstring>
#include <cerrno>
+#include <cstdlib>
#include "butil/logging.h"
+#if BRPC_WITH_RDMA
+#include "butil/iobuf_inl.h"
+#include "butil/memory/scope_guard.h"
+#include "brpc/rdma/block_pool.h"
+#include "brpc/rdma/rdma_helper.h"
+#endif // BRPC_WITH_RDMA
namespace brpc {
namespace flatbuffers {
namespace {
+void* AllocateStorage(butil::SingleIOBuf* storage, uint32_t size) {
+#if BRPC_WITH_RDMA
+ // SingleIOBuf normally uses the registered IOBuf pool. Its largest block
+ // class (type 2) is 2 MiB, including the IOBuf block header. FlatBuffers
+ // requires contiguous storage, so larger buffers need their own MR.
+ if (!rdma::IsRdmaAvailable() ||
Review Comment:
[replied by brpc-oncall robot] Two robustness gaps in this fallback:
1. The pool-vs-MR decision is a pure size threshold, so an allocation that
fits the threshold still fails when the 2 MiB pool blocks are exhausted
(`AllocBlock()` returns nullptr once `-rdma_memory_pool_max_regions` is reached
and auto-extend is disabled, see block_pool.cpp), and
`SlabAllocator::allocate()` then aborts the process. Trying
`storage->allocate(size)` first and falling back to a registered `malloc`'d
buffer only when it returns nullptr would cover both cases.
2. If `RegisterMemoryForRdma()` fails (e.g. RLIMIT_MEMLOCK or device MR
exhaustion) this function returns nullptr, and `SlabAllocator::allocate()`
turns that into a `RELEASE_ASSERT_VERBOSE` abort. Because `IsRdmaAvailable()`
is process-wide, every FlatBuffers message larger than 2 MiB then needs a
successful `ibv_reg_mr`, including messages that are only ever sent over a
plain TCP channel, and the TCP path also pays two MR registrations per large
message. Returning the plain (unregistered) buffer when registration fails
would let the TCP paths keep working and only make the RDMA send of that
message fail with ERDMAMEM instead of killing the process.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]