Re: [PR] support jemalloc profiler (brpc)
yanglimingcn merged PR #2737: URL: https://github.com/apache/brpc/pull/2737 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For additional commands, e-mail: dev-h...@brpc.apache.org
Re: [PR] support jemalloc profiler (brpc)
chenBright commented on PR #2737: URL: https://github.com/apache/brpc/pull/2737#issuecomment-2367124060 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For additional commands, e-mail: dev-h...@brpc.apache.org
Re: [PR] support jemalloc profiler (brpc)
yanglimingcn commented on PR #2737: URL: https://github.com/apache/brpc/pull/2737#issuecomment-2367097259 这个PR还有别的问题吗 @chenBright -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For additional commands, e-mail: dev-h...@brpc.apache.org
Re: [PR] support jemalloc profiler (brpc)
GreateCode commented on code in PR #2737:
URL: https://github.com/apache/brpc/pull/2737#discussion_r1744808688
##
src/brpc/builtin/memory_service.cpp:
##
@@ -58,6 +59,15 @@ static void get_tcmalloc_memory_info(butil::IOBuf& out) {
os.move_to(out);
}
+static void get_jemalloc_memory_info(Controller* cntl) {
+const brpc::URI& uri = cntl->http_request().uri();
+cntl->http_response().set_content_type("text/plain");
+
+const std::string* uri_opts = uri.GetQuery("opts");
+std::string opts = !uri_opts || uri_opts->empty() ? "Ja" : *uri_opts;
Review Comment:
可以的，不加就用默认的Ja，呈现的是所有arena合并后的信息。
这样写兼顾内置页面，还可以脱离内置页面自由选择参数。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org
For additional commands, e-mail: dev-h...@brpc.apache.org
Re: [PR] support jemalloc profiler (brpc)
GreateCode commented on code in PR #2737:
URL: https://github.com/apache/brpc/pull/2737#discussion_r1744808688
##
src/brpc/builtin/memory_service.cpp:
##
@@ -58,6 +59,15 @@ static void get_tcmalloc_memory_info(butil::IOBuf& out) {
os.move_to(out);
}
+static void get_jemalloc_memory_info(Controller* cntl) {
+const brpc::URI& uri = cntl->http_request().uri();
+cntl->http_response().set_content_type("text/plain");
+
+const std::string* uri_opts = uri.GetQuery("opts");
+std::string opts = !uri_opts || uri_opts->empty() ? "Ja" : *uri_opts;
Review Comment:
可以的，不加就用默认的Ja。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org
For additional commands, e-mail: dev-h...@brpc.apache.org
Re: [PR] support jemalloc profiler (brpc)
chenBright commented on code in PR #2737:
URL: https://github.com/apache/brpc/pull/2737#discussion_r1744760692
##
src/brpc/builtin/memory_service.cpp:
##
@@ -58,6 +59,15 @@ static void get_tcmalloc_memory_info(butil::IOBuf& out) {
os.move_to(out);
}
+static void get_jemalloc_memory_info(Controller* cntl) {
+const brpc::URI& uri = cntl->http_request().uri();
+cntl->http_response().set_content_type("text/plain");
+
+const std::string* uri_opts = uri.GetQuery("opts");
+std::string opts = !uri_opts || uri_opts->empty() ? "Ja" : *uri_opts;
Review Comment:
用户是用过内置服务页面访问MemoryService的，还得手动加上`opts=Ja`参数。使用上不方便吧。MemoryService可以不需要`opts=Ja`参数吗？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org
For additional commands, e-mail: dev-h...@brpc.apache.org
Re: [PR] support jemalloc profiler (brpc)
GreateCode commented on PR #2737: URL: https://github.com/apache/brpc/pull/2737#issuecomment-2323803247 > 在[MemoryService](https://github.com/apache/brpc/blob/master/src/brpc/builtin/memory_service.cpp#L61-L78)也支持一下jemalloc stats？ done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For additional commands, e-mail: dev-h...@brpc.apache.org
Re: [PR] support jemalloc profiler (brpc)
chenBright commented on PR #2737: URL: https://github.com/apache/brpc/pull/2737#issuecomment-2323662220 在[MemoryService](https://github.com/apache/brpc/blob/master/src/brpc/builtin/memory_service.cpp#L61-L78)也支持一下jemalloc stats？ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For additional commands, e-mail: dev-h...@brpc.apache.org
Re: [PR] support jemalloc profiler (brpc)
GreateCode commented on PR #2737: URL: https://github.com/apache/brpc/pull/2737#issuecomment-2322865921 @wwbmmm @yanglimingcn @chenBright 接口新增了jemalloc stats信息，帮助查看memory细节如 各arena，metadata等，麻烦看下还有问题吗？谢谢~ https://github.com/user-attachments/assets/f1d76f97-5820-48d9-ac2f-45f04886e7d4";> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For additional commands, e-mail: dev-h...@brpc.apache.org
Re: [PR] support jemalloc profiler (brpc)
GreateCode commented on code in PR #2737:
URL: https://github.com/apache/brpc/pull/2737#discussion_r1739699067
##
src/brpc/details/jemalloc_profiler.cpp:
##
@@ -0,0 +1,289 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+#include "brpc/controller.h"
+#include "brpc/http_header.h"
+#include "brpc/reloadable_flags.h"
+#include "brpc/uri.h"
+#include "butil/files/file_path.h"
+#include "butil/iobuf.h"
+#include "butil/popen.h"
+#include "gflags/gflags.h"
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+extern "C" {
+// weak symbol: resolved at runtime by the linker if we are using jemalloc,
nullptr otherwise
+int mallctl(const char*, void*, size_t*, void*, size_t)
__attribute__((__weak__));
+}
+
+namespace brpc {
+
+// https://jemalloc.net/jemalloc.3.html
+DEFINE_bool(je_prof_active, false, "control jemalloc prof.active, jemalloc
profiling enabled but inactive,"
+"it toggle profiling at any time during process running");
+DEFINE_int32(je_prof_dump, 0, "control jemalloc prof.dump, change this only
dump profile");
+DEFINE_int32(je_prof_reset, 19, "control jemalloc prof.reset, reset all memory
profile statistics, "
+ "and optionally update the sample rate, default 2^19 B");
+
+// define in src/brpc/builtin/pprof_service.cpp
+extern int MakeProfName(ProfilingType type, char* buf, size_t buf_len);
+
+static bool HasInit(const std::string& fn) {
+static std::set fns;
+if (fns.count(fn) > 0) {
+return true;
+}
+fns.insert(fn);
+return false;
+}
+
+bool HasJemalloc() {
+return mallctl != nullptr;
+}
+
+bool HasEnableJemallocProfile() {
+bool prof = false;
+size_t len = sizeof(prof);
+int ret = mallctl("opt.prof", &prof, &len, nullptr, 0);
+if (ret != 0) {
+LOG(WARNING) << "mallctl get opt.prof err, ret:" << ret;
+return false;
+}
+return prof;
+}
+
+static int JeProfileActive(bool active) {
+if (!HasJemalloc()) {
+LOG(WARNING) << "no jemalloc";
+return -1;
+}
+
+if (!HasEnableJemallocProfile()) {
+LOG(WARNING) << "jemalloc have not set opt.prof before start";
+return -1;
+}
+
+int ret = mallctl("prof.active", nullptr, nullptr, &active,
sizeof(active));
+if (ret != 0) {
+LOG(WARNING) << "mallctl set prof.active:" << active << " err, ret:"
<< ret;
+return ret;
+}
+LOG(INFO) << "mallctl set prof.active:" << active << " succ";
+return 0;
+}
+
+static std::string JeProfileDump() {
+if (!HasJemalloc()) {
+LOG(WARNING) << "no jemalloc";
+return "";
+}
+
+if (!HasEnableJemallocProfile()) {
+LOG(WARNING) << "jemalloc have not set opt.prof before start";
+return "";
+}
+
+char prof_name[256];
+if (MakeProfName(PROFILING_HEAP, prof_name, sizeof(prof_name)) != 0) {
+LOG(WARNING) << "Fail to create .prof file, " << berror();
+return "";
+}
+butil::File::Error error;
+const butil::FilePath dir = butil::FilePath(prof_name).DirName();
+if (!butil::CreateDirectoryAndGetError(dir, &error)) {
+LOG(WARNING) << "Fail to create directory= " << dir.value();
+return "";
+}
+
+const char* p_prof_name = prof_name;
+int ret = mallctl("prof.dump", NULL, NULL, (void*)&p_prof_name,
sizeof(p_prof_name));
+if (ret != 0) {
+LOG(WARNING) << "mallctl set prof.dump:" << p_prof_name << " err,
ret:" << ret;
+return "";
+}
+LOG(INFO) << "heap profile dump:" << prof_name << " succ";
+return prof_name;
+}
+
+static int JeProfileReset(size_t lg_sample) {
+if (!HasJemalloc()) {
+LOG(WARNING) << "no jemalloc";
+return -1;
+}
+
+if (!HasEnableJemallocProfile()) {
+LOG(WARNING) << "jemalloc have not set opt.prof before start";
+return -1;
+}
+
+int ret = mallctl("prof.reset", nullptr, nullptr, &lg_sample,
sizeof(lg_sample));
+if (ret != 0) {
+LOG(WARNING) << "mallctl set prof.reset:" << lg_sample << " err, ret:"
<< ret;
+return ret;
+}
+LOG(INFO) << "mallctl set prof.reset:" << lg_sample << " succ";
+
+FLAGS_je_prof_active = false;
+
Re: [PR] support jemalloc profiler (brpc)
chenBright commented on code in PR #2737:
URL: https://github.com/apache/brpc/pull/2737#discussion_r1736655448
##
src/brpc/details/jemalloc_profiler.cpp:
##
@@ -0,0 +1,289 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+#include "brpc/controller.h"
+#include "brpc/http_header.h"
+#include "brpc/reloadable_flags.h"
+#include "brpc/uri.h"
+#include "butil/files/file_path.h"
+#include "butil/iobuf.h"
+#include "butil/popen.h"
+#include "gflags/gflags.h"
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+extern "C" {
+// weak symbol: resolved at runtime by the linker if we are using jemalloc,
nullptr otherwise
+int mallctl(const char*, void*, size_t*, void*, size_t)
__attribute__((__weak__));
Review Comment:
要
##
src/brpc/details/jemalloc_profiler.cpp:
##
@@ -0,0 +1,289 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+#include "brpc/controller.h"
+#include "brpc/http_header.h"
+#include "brpc/reloadable_flags.h"
+#include "brpc/uri.h"
+#include "butil/files/file_path.h"
+#include "butil/iobuf.h"
+#include "butil/popen.h"
+#include "gflags/gflags.h"
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+extern "C" {
+// weak symbol: resolved at runtime by the linker if we are using jemalloc,
nullptr otherwise
+int mallctl(const char*, void*, size_t*, void*, size_t)
__attribute__((__weak__));
+}
+
+namespace brpc {
+
+// https://jemalloc.net/jemalloc.3.html
+DEFINE_bool(je_prof_active, false, "control jemalloc prof.active, jemalloc
profiling enabled but inactive,"
+"it toggle profiling at any time during process running");
+DEFINE_int32(je_prof_dump, 0, "control jemalloc prof.dump, change this only
dump profile");
+DEFINE_int32(je_prof_reset, 19, "control jemalloc prof.reset, reset all memory
profile statistics, "
+ "and optionally update the sample rate, default 2^19 B");
+
+// define in src/brpc/builtin/pprof_service.cpp
+extern int MakeProfName(ProfilingType type, char* buf, size_t buf_len);
+
+static bool HasInit(const std::string& fn) {
+static std::set fns;
+if (fns.count(fn) > 0) {
+return true;
+}
+fns.insert(fn);
+return false;
+}
+
+bool HasJemalloc() {
+return mallctl != nullptr;
+}
+
+bool HasEnableJemallocProfile() {
+bool prof = false;
+size_t len = sizeof(prof);
+int ret = mallctl("opt.prof", &prof, &len, nullptr, 0);
+if (ret != 0) {
+LOG(WARNING) << "mallctl get opt.prof err, ret:" << ret;
+return false;
+}
+return prof;
+}
+
+static int JeProfileActive(bool active) {
+if (!HasJemalloc()) {
+LOG(WARNING) << "no jemalloc";
+return -1;
+}
+
+if (!HasEnableJemallocProfile()) {
+LOG(WARNING) << "jemalloc have not set opt.prof before start";
+return -1;
+}
+
+int ret = mallctl("prof.active", nullptr, nullptr, &active,
sizeof(active));
+if (ret != 0) {
+LOG(WARNING) << "mallctl set prof.active:" << active << " err, ret:"
<< ret;
+return ret;
+}
+LOG(INFO) << "mallctl set prof.active:" << active << " succ";
+return 0;
+}
+
+static std::string JeProfileDump() {
+if (!HasJemalloc()) {
+LOG(WARNING) << "no jemalloc";
+return "";
+}
+
+if (!HasEnableJemallocProfile()) {
+LOG(WARNING) << "jemalloc have not set opt.prof before start";
+return "";
+}
+
+c
Re: [PR] support jemalloc profiler (brpc)
chenBright commented on code in PR #2737:
URL: https://github.com/apache/brpc/pull/2737#discussion_r1736642295
##
src/brpc/details/jemalloc_profiler.cpp:
##
@@ -0,0 +1,289 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+#include "brpc/controller.h"
+#include "brpc/http_header.h"
+#include "brpc/reloadable_flags.h"
+#include "brpc/uri.h"
+#include "butil/files/file_path.h"
+#include "butil/iobuf.h"
+#include "butil/popen.h"
+#include "gflags/gflags.h"
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+extern "C" {
+// weak symbol: resolved at runtime by the linker if we are using jemalloc,
nullptr otherwise
+int mallctl(const char*, void*, size_t*, void*, size_t)
__attribute__((__weak__));
Review Comment:
要
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org
For additional commands, e-mail: dev-h...@brpc.apache.org
Re: [PR] support jemalloc profiler (brpc)
chenBright commented on code in PR #2737:
URL: https://github.com/apache/brpc/pull/2737#discussion_r1736642295
##
src/brpc/details/jemalloc_profiler.cpp:
##
@@ -0,0 +1,289 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+#include "brpc/controller.h"
+#include "brpc/http_header.h"
+#include "brpc/reloadable_flags.h"
+#include "brpc/uri.h"
+#include "butil/files/file_path.h"
+#include "butil/iobuf.h"
+#include "butil/popen.h"
+#include "gflags/gflags.h"
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+extern "C" {
+// weak symbol: resolved at runtime by the linker if we are using jemalloc,
nullptr otherwise
+int mallctl(const char*, void*, size_t*, void*, size_t)
__attribute__((__weak__));
Review Comment:
要
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org
For additional commands, e-mail: dev-h...@brpc.apache.org
Re: [PR] support jemalloc profiler (brpc)
GreateCode commented on code in PR #2737:
URL: https://github.com/apache/brpc/pull/2737#discussion_r1736507246
##
src/brpc/details/jemalloc_profiler.cpp:
##
@@ -0,0 +1,289 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+#include "brpc/controller.h"
+#include "brpc/http_header.h"
+#include "brpc/reloadable_flags.h"
+#include "brpc/uri.h"
+#include "butil/files/file_path.h"
+#include "butil/iobuf.h"
+#include "butil/popen.h"
+#include "gflags/gflags.h"
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+extern "C" {
+// weak symbol: resolved at runtime by the linker if we are using jemalloc,
nullptr otherwise
+int mallctl(const char*, void*, size_t*, void*, size_t)
__attribute__((__weak__));
+}
+
+namespace brpc {
+
+// https://jemalloc.net/jemalloc.3.html
+DEFINE_bool(je_prof_active, false, "control jemalloc prof.active, jemalloc
profiling enabled but inactive,"
+"it toggle profiling at any time during process running");
+DEFINE_int32(je_prof_dump, 0, "control jemalloc prof.dump, change this only
dump profile");
+DEFINE_int32(je_prof_reset, 19, "control jemalloc prof.reset, reset all memory
profile statistics, "
+ "and optionally update the sample rate, default 2^19 B");
+
+// define in src/brpc/builtin/pprof_service.cpp
+extern int MakeProfName(ProfilingType type, char* buf, size_t buf_len);
+
+static bool HasInit(const std::string& fn) {
+static std::set fns;
+if (fns.count(fn) > 0) {
+return true;
+}
+fns.insert(fn);
+return false;
+}
+
+bool HasJemalloc() {
+return mallctl != nullptr;
+}
+
+bool HasEnableJemallocProfile() {
+bool prof = false;
+size_t len = sizeof(prof);
+int ret = mallctl("opt.prof", &prof, &len, nullptr, 0);
+if (ret != 0) {
+LOG(WARNING) << "mallctl get opt.prof err, ret:" << ret;
+return false;
+}
+return prof;
+}
+
+static int JeProfileActive(bool active) {
+if (!HasJemalloc()) {
+LOG(WARNING) << "no jemalloc";
+return -1;
+}
+
+if (!HasEnableJemallocProfile()) {
+LOG(WARNING) << "jemalloc have not set opt.prof before start";
+return -1;
+}
+
+int ret = mallctl("prof.active", nullptr, nullptr, &active,
sizeof(active));
+if (ret != 0) {
+LOG(WARNING) << "mallctl set prof.active:" << active << " err, ret:"
<< ret;
+return ret;
+}
+LOG(INFO) << "mallctl set prof.active:" << active << " succ";
+return 0;
+}
+
+static std::string JeProfileDump() {
+if (!HasJemalloc()) {
+LOG(WARNING) << "no jemalloc";
+return "";
+}
+
+if (!HasEnableJemallocProfile()) {
+LOG(WARNING) << "jemalloc have not set opt.prof before start";
+return "";
+}
+
+char prof_name[256];
+if (MakeProfName(PROFILING_HEAP, prof_name, sizeof(prof_name)) != 0) {
+LOG(WARNING) << "Fail to create .prof file, " << berror();
+return "";
+}
+butil::File::Error error;
+const butil::FilePath dir = butil::FilePath(prof_name).DirName();
+if (!butil::CreateDirectoryAndGetError(dir, &error)) {
+LOG(WARNING) << "Fail to create directory= " << dir.value();
+return "";
+}
+
+const char* p_prof_name = prof_name;
+int ret = mallctl("prof.dump", NULL, NULL, (void*)&p_prof_name,
sizeof(p_prof_name));
+if (ret != 0) {
+LOG(WARNING) << "mallctl set prof.dump:" << p_prof_name << " err,
ret:" << ret;
+return "";
+}
+LOG(INFO) << "heap profile dump:" << prof_name << " succ";
+return prof_name;
+}
+
+static int JeProfileReset(size_t lg_sample) {
+if (!HasJemalloc()) {
+LOG(WARNING) << "no jemalloc";
+return -1;
+}
+
+if (!HasEnableJemallocProfile()) {
+LOG(WARNING) << "jemalloc have not set opt.prof before start";
+return -1;
+}
+
+int ret = mallctl("prof.reset", nullptr, nullptr, &lg_sample,
sizeof(lg_sample));
+if (ret != 0) {
+LOG(WARNING) << "mallctl set prof.reset:" << lg_sample << " err, ret:"
<< ret;
+return ret;
+}
+LOG(INFO) << "mallctl set prof.reset:" << lg_sample << " succ";
+
+FLAGS_je_prof_active = false;
+
Re: [PR] support jemalloc profiler (brpc)
GreateCode commented on code in PR #2737:
URL: https://github.com/apache/brpc/pull/2737#discussion_r1736527722
##
src/brpc/details/jemalloc_profiler.cpp:
##
@@ -0,0 +1,289 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+#include "brpc/controller.h"
+#include "brpc/http_header.h"
+#include "brpc/reloadable_flags.h"
+#include "brpc/uri.h"
+#include "butil/files/file_path.h"
+#include "butil/iobuf.h"
+#include "butil/popen.h"
+#include "gflags/gflags.h"
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+extern "C" {
+// weak symbol: resolved at runtime by the linker if we are using jemalloc,
nullptr otherwise
+int mallctl(const char*, void*, size_t*, void*, size_t)
__attribute__((__weak__));
Review Comment:
改了，用BAIDU_WEAK 后，CMakeLists.txt是不是还得加 "-Wl,-U,_mallctl" ？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org
For additional commands, e-mail: dev-h...@brpc.apache.org
Re: [PR] support jemalloc profiler (brpc)
GreateCode commented on code in PR #2737:
URL: https://github.com/apache/brpc/pull/2737#discussion_r1736507246
##
src/brpc/details/jemalloc_profiler.cpp:
##
@@ -0,0 +1,289 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+#include "brpc/controller.h"
+#include "brpc/http_header.h"
+#include "brpc/reloadable_flags.h"
+#include "brpc/uri.h"
+#include "butil/files/file_path.h"
+#include "butil/iobuf.h"
+#include "butil/popen.h"
+#include "gflags/gflags.h"
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+extern "C" {
+// weak symbol: resolved at runtime by the linker if we are using jemalloc,
nullptr otherwise
+int mallctl(const char*, void*, size_t*, void*, size_t)
__attribute__((__weak__));
+}
+
+namespace brpc {
+
+// https://jemalloc.net/jemalloc.3.html
+DEFINE_bool(je_prof_active, false, "control jemalloc prof.active, jemalloc
profiling enabled but inactive,"
+"it toggle profiling at any time during process running");
+DEFINE_int32(je_prof_dump, 0, "control jemalloc prof.dump, change this only
dump profile");
+DEFINE_int32(je_prof_reset, 19, "control jemalloc prof.reset, reset all memory
profile statistics, "
+ "and optionally update the sample rate, default 2^19 B");
+
+// define in src/brpc/builtin/pprof_service.cpp
+extern int MakeProfName(ProfilingType type, char* buf, size_t buf_len);
+
+static bool HasInit(const std::string& fn) {
+static std::set fns;
+if (fns.count(fn) > 0) {
+return true;
+}
+fns.insert(fn);
+return false;
+}
+
+bool HasJemalloc() {
+return mallctl != nullptr;
+}
+
+bool HasEnableJemallocProfile() {
+bool prof = false;
+size_t len = sizeof(prof);
+int ret = mallctl("opt.prof", &prof, &len, nullptr, 0);
+if (ret != 0) {
+LOG(WARNING) << "mallctl get opt.prof err, ret:" << ret;
+return false;
+}
+return prof;
+}
+
+static int JeProfileActive(bool active) {
+if (!HasJemalloc()) {
+LOG(WARNING) << "no jemalloc";
+return -1;
+}
+
+if (!HasEnableJemallocProfile()) {
+LOG(WARNING) << "jemalloc have not set opt.prof before start";
+return -1;
+}
+
+int ret = mallctl("prof.active", nullptr, nullptr, &active,
sizeof(active));
+if (ret != 0) {
+LOG(WARNING) << "mallctl set prof.active:" << active << " err, ret:"
<< ret;
+return ret;
+}
+LOG(INFO) << "mallctl set prof.active:" << active << " succ";
+return 0;
+}
+
+static std::string JeProfileDump() {
+if (!HasJemalloc()) {
+LOG(WARNING) << "no jemalloc";
+return "";
+}
+
+if (!HasEnableJemallocProfile()) {
+LOG(WARNING) << "jemalloc have not set opt.prof before start";
+return "";
+}
+
+char prof_name[256];
+if (MakeProfName(PROFILING_HEAP, prof_name, sizeof(prof_name)) != 0) {
+LOG(WARNING) << "Fail to create .prof file, " << berror();
+return "";
+}
+butil::File::Error error;
+const butil::FilePath dir = butil::FilePath(prof_name).DirName();
+if (!butil::CreateDirectoryAndGetError(dir, &error)) {
+LOG(WARNING) << "Fail to create directory= " << dir.value();
+return "";
+}
+
+const char* p_prof_name = prof_name;
+int ret = mallctl("prof.dump", NULL, NULL, (void*)&p_prof_name,
sizeof(p_prof_name));
+if (ret != 0) {
+LOG(WARNING) << "mallctl set prof.dump:" << p_prof_name << " err,
ret:" << ret;
+return "";
+}
+LOG(INFO) << "heap profile dump:" << prof_name << " succ";
+return prof_name;
+}
+
+static int JeProfileReset(size_t lg_sample) {
+if (!HasJemalloc()) {
+LOG(WARNING) << "no jemalloc";
+return -1;
+}
+
+if (!HasEnableJemallocProfile()) {
+LOG(WARNING) << "jemalloc have not set opt.prof before start";
+return -1;
+}
+
+int ret = mallctl("prof.reset", nullptr, nullptr, &lg_sample,
sizeof(lg_sample));
+if (ret != 0) {
+LOG(WARNING) << "mallctl set prof.reset:" << lg_sample << " err, ret:"
<< ret;
+return ret;
+}
+LOG(INFO) << "mallctl set prof.reset:" << lg_sample << " succ";
+
+FLAGS_je_prof_active = false;
+
Re: [PR] support jemalloc profiler (brpc)
chenBright commented on code in PR #2737:
URL: https://github.com/apache/brpc/pull/2737#discussion_r1736471882
##
src/brpc/details/jemalloc_profiler.cpp:
##
@@ -0,0 +1,289 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+#include "brpc/controller.h"
+#include "brpc/http_header.h"
+#include "brpc/reloadable_flags.h"
+#include "brpc/uri.h"
+#include "butil/files/file_path.h"
+#include "butil/iobuf.h"
+#include "butil/popen.h"
+#include "gflags/gflags.h"
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+extern "C" {
+// weak symbol: resolved at runtime by the linker if we are using jemalloc,
nullptr otherwise
+int mallctl(const char*, void*, size_t*, void*, size_t)
__attribute__((__weak__));
+}
+
+namespace brpc {
+
+// https://jemalloc.net/jemalloc.3.html
+DEFINE_bool(je_prof_active, false, "control jemalloc prof.active, jemalloc
profiling enabled but inactive,"
+"it toggle profiling at any time during process running");
+DEFINE_int32(je_prof_dump, 0, "control jemalloc prof.dump, change this only
dump profile");
+DEFINE_int32(je_prof_reset, 19, "control jemalloc prof.reset, reset all memory
profile statistics, "
+ "and optionally update the sample rate, default 2^19 B");
+
+// define in src/brpc/builtin/pprof_service.cpp
+extern int MakeProfName(ProfilingType type, char* buf, size_t buf_len);
+
+static bool HasInit(const std::string& fn) {
+static std::set fns;
+if (fns.count(fn) > 0) {
+return true;
+}
+fns.insert(fn);
+return false;
+}
+
+bool HasJemalloc() {
+return mallctl != nullptr;
+}
+
+bool HasEnableJemallocProfile() {
+bool prof = false;
+size_t len = sizeof(prof);
+int ret = mallctl("opt.prof", &prof, &len, nullptr, 0);
+if (ret != 0) {
+LOG(WARNING) << "mallctl get opt.prof err, ret:" << ret;
+return false;
+}
+return prof;
+}
+
+static int JeProfileActive(bool active) {
+if (!HasJemalloc()) {
+LOG(WARNING) << "no jemalloc";
+return -1;
+}
+
+if (!HasEnableJemallocProfile()) {
+LOG(WARNING) << "jemalloc have not set opt.prof before start";
+return -1;
+}
+
+int ret = mallctl("prof.active", nullptr, nullptr, &active,
sizeof(active));
+if (ret != 0) {
+LOG(WARNING) << "mallctl set prof.active:" << active << " err, ret:"
<< ret;
+return ret;
+}
+LOG(INFO) << "mallctl set prof.active:" << active << " succ";
+return 0;
+}
+
+static std::string JeProfileDump() {
+if (!HasJemalloc()) {
+LOG(WARNING) << "no jemalloc";
+return "";
+}
+
+if (!HasEnableJemallocProfile()) {
+LOG(WARNING) << "jemalloc have not set opt.prof before start";
+return "";
+}
+
+char prof_name[256];
+if (MakeProfName(PROFILING_HEAP, prof_name, sizeof(prof_name)) != 0) {
+LOG(WARNING) << "Fail to create .prof file, " << berror();
+return "";
+}
+butil::File::Error error;
+const butil::FilePath dir = butil::FilePath(prof_name).DirName();
+if (!butil::CreateDirectoryAndGetError(dir, &error)) {
+LOG(WARNING) << "Fail to create directory= " << dir.value();
+return "";
+}
+
+const char* p_prof_name = prof_name;
+int ret = mallctl("prof.dump", NULL, NULL, (void*)&p_prof_name,
sizeof(p_prof_name));
+if (ret != 0) {
+LOG(WARNING) << "mallctl set prof.dump:" << p_prof_name << " err,
ret:" << ret;
+return "";
+}
+LOG(INFO) << "heap profile dump:" << prof_name << " succ";
+return prof_name;
+}
+
+static int JeProfileReset(size_t lg_sample) {
+if (!HasJemalloc()) {
+LOG(WARNING) << "no jemalloc";
+return -1;
+}
+
+if (!HasEnableJemallocProfile()) {
+LOG(WARNING) << "jemalloc have not set opt.prof before start";
+return -1;
+}
+
+int ret = mallctl("prof.reset", nullptr, nullptr, &lg_sample,
sizeof(lg_sample));
+if (ret != 0) {
+LOG(WARNING) << "mallctl set prof.reset:" << lg_sample << " err, ret:"
<< ret;
+return ret;
+}
+LOG(INFO) << "mallctl set prof.reset:" << lg_sample << " succ";
+
+FLAGS_je_prof_active = false;
+
Re: [PR] support jemalloc profiler (brpc)
chenBright commented on code in PR #2737:
URL: https://github.com/apache/brpc/pull/2737#discussion_r1736471882
##
src/brpc/details/jemalloc_profiler.cpp:
##
@@ -0,0 +1,289 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+#include "brpc/controller.h"
+#include "brpc/http_header.h"
+#include "brpc/reloadable_flags.h"
+#include "brpc/uri.h"
+#include "butil/files/file_path.h"
+#include "butil/iobuf.h"
+#include "butil/popen.h"
+#include "gflags/gflags.h"
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+extern "C" {
+// weak symbol: resolved at runtime by the linker if we are using jemalloc,
nullptr otherwise
+int mallctl(const char*, void*, size_t*, void*, size_t)
__attribute__((__weak__));
+}
+
+namespace brpc {
+
+// https://jemalloc.net/jemalloc.3.html
+DEFINE_bool(je_prof_active, false, "control jemalloc prof.active, jemalloc
profiling enabled but inactive,"
+"it toggle profiling at any time during process running");
+DEFINE_int32(je_prof_dump, 0, "control jemalloc prof.dump, change this only
dump profile");
+DEFINE_int32(je_prof_reset, 19, "control jemalloc prof.reset, reset all memory
profile statistics, "
+ "and optionally update the sample rate, default 2^19 B");
+
+// define in src/brpc/builtin/pprof_service.cpp
+extern int MakeProfName(ProfilingType type, char* buf, size_t buf_len);
+
+static bool HasInit(const std::string& fn) {
+static std::set fns;
+if (fns.count(fn) > 0) {
+return true;
+}
+fns.insert(fn);
+return false;
+}
+
+bool HasJemalloc() {
+return mallctl != nullptr;
+}
+
+bool HasEnableJemallocProfile() {
+bool prof = false;
+size_t len = sizeof(prof);
+int ret = mallctl("opt.prof", &prof, &len, nullptr, 0);
+if (ret != 0) {
+LOG(WARNING) << "mallctl get opt.prof err, ret:" << ret;
+return false;
+}
+return prof;
+}
+
+static int JeProfileActive(bool active) {
+if (!HasJemalloc()) {
+LOG(WARNING) << "no jemalloc";
+return -1;
+}
+
+if (!HasEnableJemallocProfile()) {
+LOG(WARNING) << "jemalloc have not set opt.prof before start";
+return -1;
+}
+
+int ret = mallctl("prof.active", nullptr, nullptr, &active,
sizeof(active));
+if (ret != 0) {
+LOG(WARNING) << "mallctl set prof.active:" << active << " err, ret:"
<< ret;
+return ret;
+}
+LOG(INFO) << "mallctl set prof.active:" << active << " succ";
+return 0;
+}
+
+static std::string JeProfileDump() {
+if (!HasJemalloc()) {
+LOG(WARNING) << "no jemalloc";
+return "";
+}
+
+if (!HasEnableJemallocProfile()) {
+LOG(WARNING) << "jemalloc have not set opt.prof before start";
+return "";
+}
+
+char prof_name[256];
+if (MakeProfName(PROFILING_HEAP, prof_name, sizeof(prof_name)) != 0) {
+LOG(WARNING) << "Fail to create .prof file, " << berror();
+return "";
+}
+butil::File::Error error;
+const butil::FilePath dir = butil::FilePath(prof_name).DirName();
+if (!butil::CreateDirectoryAndGetError(dir, &error)) {
+LOG(WARNING) << "Fail to create directory= " << dir.value();
+return "";
+}
+
+const char* p_prof_name = prof_name;
+int ret = mallctl("prof.dump", NULL, NULL, (void*)&p_prof_name,
sizeof(p_prof_name));
+if (ret != 0) {
+LOG(WARNING) << "mallctl set prof.dump:" << p_prof_name << " err,
ret:" << ret;
+return "";
+}
+LOG(INFO) << "heap profile dump:" << prof_name << " succ";
+return prof_name;
+}
+
+static int JeProfileReset(size_t lg_sample) {
+if (!HasJemalloc()) {
+LOG(WARNING) << "no jemalloc";
+return -1;
+}
+
+if (!HasEnableJemallocProfile()) {
+LOG(WARNING) << "jemalloc have not set opt.prof before start";
+return -1;
+}
+
+int ret = mallctl("prof.reset", nullptr, nullptr, &lg_sample,
sizeof(lg_sample));
+if (ret != 0) {
+LOG(WARNING) << "mallctl set prof.reset:" << lg_sample << " err, ret:"
<< ret;
+return ret;
+}
+LOG(INFO) << "mallctl set prof.reset:" << lg_sample << " succ";
+
+FLAGS_je_prof_active = false;
+
Re: [PR] support jemalloc profiler (brpc)
GreateCode commented on code in PR #2737:
URL: https://github.com/apache/brpc/pull/2737#discussion_r1736362684
##
src/brpc/details/jemalloc_profiler.cpp:
##
@@ -0,0 +1,289 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+#include "brpc/controller.h"
+#include "brpc/http_header.h"
+#include "brpc/reloadable_flags.h"
+#include "brpc/uri.h"
+#include "butil/files/file_path.h"
+#include "butil/iobuf.h"
+#include "butil/popen.h"
+#include "gflags/gflags.h"
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+extern "C" {
+// weak symbol: resolved at runtime by the linker if we are using jemalloc,
nullptr otherwise
+int mallctl(const char*, void*, size_t*, void*, size_t)
__attribute__((__weak__));
Review Comment:
BAIDU_WEAK只对gcc有效，clang也需要设置
```
// Mark function as weak. This is GCC only feature.
#if defined(COMPILER_GCC)
# define BAIDU_WEAK __attribute__((weak))
#else
# define BAIDU_WEAK
#endif
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org
For additional commands, e-mail: dev-h...@brpc.apache.org
Re: [PR] support jemalloc profiler (brpc)
GreateCode commented on code in PR #2737:
URL: https://github.com/apache/brpc/pull/2737#discussion_r1736348931
##
src/brpc/details/jemalloc_profiler.cpp:
##
@@ -0,0 +1,289 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+#include "brpc/controller.h"
+#include "brpc/http_header.h"
+#include "brpc/reloadable_flags.h"
+#include "brpc/uri.h"
+#include "butil/files/file_path.h"
+#include "butil/iobuf.h"
+#include "butil/popen.h"
+#include "gflags/gflags.h"
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+extern "C" {
+// weak symbol: resolved at runtime by the linker if we are using jemalloc,
nullptr otherwise
+int mallctl(const char*, void*, size_t*, void*, size_t)
__attribute__((__weak__));
+}
+
+namespace brpc {
+
+// https://jemalloc.net/jemalloc.3.html
+DEFINE_bool(je_prof_active, false, "control jemalloc prof.active, jemalloc
profiling enabled but inactive,"
+"it toggle profiling at any time during process running");
+DEFINE_int32(je_prof_dump, 0, "control jemalloc prof.dump, change this only
dump profile");
+DEFINE_int32(je_prof_reset, 19, "control jemalloc prof.reset, reset all memory
profile statistics, "
+ "and optionally update the sample rate, default 2^19 B");
+
+// define in src/brpc/builtin/pprof_service.cpp
+extern int MakeProfName(ProfilingType type, char* buf, size_t buf_len);
+
+static bool HasInit(const std::string& fn) {
+static std::set fns;
+if (fns.count(fn) > 0) {
+return true;
+}
+fns.insert(fn);
+return false;
+}
+
+bool HasJemalloc() {
+return mallctl != nullptr;
+}
+
+bool HasEnableJemallocProfile() {
+bool prof = false;
+size_t len = sizeof(prof);
+int ret = mallctl("opt.prof", &prof, &len, nullptr, 0);
+if (ret != 0) {
+LOG(WARNING) << "mallctl get opt.prof err, ret:" << ret;
+return false;
+}
+return prof;
+}
+
+static int JeProfileActive(bool active) {
+if (!HasJemalloc()) {
+LOG(WARNING) << "no jemalloc";
+return -1;
+}
+
+if (!HasEnableJemallocProfile()) {
+LOG(WARNING) << "jemalloc have not set opt.prof before start";
+return -1;
+}
+
+int ret = mallctl("prof.active", nullptr, nullptr, &active,
sizeof(active));
+if (ret != 0) {
+LOG(WARNING) << "mallctl set prof.active:" << active << " err, ret:"
<< ret;
+return ret;
+}
+LOG(INFO) << "mallctl set prof.active:" << active << " succ";
+return 0;
+}
+
+static std::string JeProfileDump() {
+if (!HasJemalloc()) {
+LOG(WARNING) << "no jemalloc";
+return "";
+}
+
+if (!HasEnableJemallocProfile()) {
+LOG(WARNING) << "jemalloc have not set opt.prof before start";
+return "";
+}
+
+char prof_name[256];
+if (MakeProfName(PROFILING_HEAP, prof_name, sizeof(prof_name)) != 0) {
+LOG(WARNING) << "Fail to create .prof file, " << berror();
+return "";
+}
+butil::File::Error error;
+const butil::FilePath dir = butil::FilePath(prof_name).DirName();
+if (!butil::CreateDirectoryAndGetError(dir, &error)) {
+LOG(WARNING) << "Fail to create directory= " << dir.value();
+return "";
+}
+
+const char* p_prof_name = prof_name;
+int ret = mallctl("prof.dump", NULL, NULL, (void*)&p_prof_name,
sizeof(p_prof_name));
+if (ret != 0) {
+LOG(WARNING) << "mallctl set prof.dump:" << p_prof_name << " err,
ret:" << ret;
+return "";
+}
+LOG(INFO) << "heap profile dump:" << prof_name << " succ";
+return prof_name;
+}
+
+static int JeProfileReset(size_t lg_sample) {
+if (!HasJemalloc()) {
+LOG(WARNING) << "no jemalloc";
+return -1;
+}
+
+if (!HasEnableJemallocProfile()) {
+LOG(WARNING) << "jemalloc have not set opt.prof before start";
+return -1;
+}
+
+int ret = mallctl("prof.reset", nullptr, nullptr, &lg_sample,
sizeof(lg_sample));
+if (ret != 0) {
+LOG(WARNING) << "mallctl set prof.reset:" << lg_sample << " err, ret:"
<< ret;
+return ret;
+}
+LOG(INFO) << "mallctl set prof.reset:" << lg_sample << " succ";
+
+FLAGS_je_prof_active = false;
+
Re: [PR] support jemalloc profiler (brpc)
GreateCode commented on code in PR #2737:
URL: https://github.com/apache/brpc/pull/2737#discussion_r1736339795
##
src/brpc/details/jemalloc_profiler.cpp:
##
@@ -0,0 +1,289 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+#include "brpc/controller.h"
+#include "brpc/http_header.h"
+#include "brpc/reloadable_flags.h"
+#include "brpc/uri.h"
+#include "butil/files/file_path.h"
+#include "butil/iobuf.h"
+#include "butil/popen.h"
+#include "gflags/gflags.h"
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+extern "C" {
+// weak symbol: resolved at runtime by the linker if we are using jemalloc,
nullptr otherwise
+int mallctl(const char*, void*, size_t*, void*, size_t)
__attribute__((__weak__));
+}
+
+namespace brpc {
+
+// https://jemalloc.net/jemalloc.3.html
+DEFINE_bool(je_prof_active, false, "control jemalloc prof.active, jemalloc
profiling enabled but inactive,"
+"it toggle profiling at any time during process running");
+DEFINE_int32(je_prof_dump, 0, "control jemalloc prof.dump, change this only
dump profile");
+DEFINE_int32(je_prof_reset, 19, "control jemalloc prof.reset, reset all memory
profile statistics, "
+ "and optionally update the sample rate, default 2^19 B");
+
+// define in src/brpc/builtin/pprof_service.cpp
+extern int MakeProfName(ProfilingType type, char* buf, size_t buf_len);
+
+static bool HasInit(const std::string& fn) {
+static std::set fns;
+if (fns.count(fn) > 0) {
+return true;
+}
+fns.insert(fn);
+return false;
+}
+
+bool HasJemalloc() {
+return mallctl != nullptr;
+}
+
+bool HasEnableJemallocProfile() {
+bool prof = false;
+size_t len = sizeof(prof);
+int ret = mallctl("opt.prof", &prof, &len, nullptr, 0);
+if (ret != 0) {
+LOG(WARNING) << "mallctl get opt.prof err, ret:" << ret;
+return false;
+}
+return prof;
+}
+
+static int JeProfileActive(bool active) {
+if (!HasJemalloc()) {
+LOG(WARNING) << "no jemalloc";
+return -1;
+}
+
+if (!HasEnableJemallocProfile()) {
+LOG(WARNING) << "jemalloc have not set opt.prof before start";
+return -1;
+}
+
+int ret = mallctl("prof.active", nullptr, nullptr, &active,
sizeof(active));
+if (ret != 0) {
+LOG(WARNING) << "mallctl set prof.active:" << active << " err, ret:"
<< ret;
+return ret;
+}
+LOG(INFO) << "mallctl set prof.active:" << active << " succ";
+return 0;
+}
+
+static std::string JeProfileDump() {
+if (!HasJemalloc()) {
+LOG(WARNING) << "no jemalloc";
+return "";
+}
+
+if (!HasEnableJemallocProfile()) {
+LOG(WARNING) << "jemalloc have not set opt.prof before start";
+return "";
+}
+
+char prof_name[256];
+if (MakeProfName(PROFILING_HEAP, prof_name, sizeof(prof_name)) != 0) {
+LOG(WARNING) << "Fail to create .prof file, " << berror();
+return "";
+}
+butil::File::Error error;
+const butil::FilePath dir = butil::FilePath(prof_name).DirName();
+if (!butil::CreateDirectoryAndGetError(dir, &error)) {
+LOG(WARNING) << "Fail to create directory= " << dir.value();
+return "";
+}
+
+const char* p_prof_name = prof_name;
+int ret = mallctl("prof.dump", NULL, NULL, (void*)&p_prof_name,
sizeof(p_prof_name));
+if (ret != 0) {
+LOG(WARNING) << "mallctl set prof.dump:" << p_prof_name << " err,
ret:" << ret;
+return "";
+}
+LOG(INFO) << "heap profile dump:" << prof_name << " succ";
+return prof_name;
+}
+
+static int JeProfileReset(size_t lg_sample) {
+if (!HasJemalloc()) {
+LOG(WARNING) << "no jemalloc";
+return -1;
+}
+
+if (!HasEnableJemallocProfile()) {
+LOG(WARNING) << "jemalloc have not set opt.prof before start";
+return -1;
+}
+
+int ret = mallctl("prof.reset", nullptr, nullptr, &lg_sample,
sizeof(lg_sample));
+if (ret != 0) {
+LOG(WARNING) << "mallctl set prof.reset:" << lg_sample << " err, ret:"
<< ret;
+return ret;
+}
+LOG(INFO) << "mallctl set prof.reset:" << lg_sample << " succ";
+
+FLAGS_je_prof_active = false;
+
Re: [PR] support jemalloc profiler (brpc)
chenBright commented on code in PR #2737:
URL: https://github.com/apache/brpc/pull/2737#discussion_r1736075671
##
src/brpc/details/jemalloc_profiler.cpp:
##
@@ -0,0 +1,289 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+#include "brpc/controller.h"
+#include "brpc/http_header.h"
+#include "brpc/reloadable_flags.h"
+#include "brpc/uri.h"
+#include "butil/files/file_path.h"
+#include "butil/iobuf.h"
+#include "butil/popen.h"
+#include "gflags/gflags.h"
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+extern "C" {
+// weak symbol: resolved at runtime by the linker if we are using jemalloc,
nullptr otherwise
+int mallctl(const char*, void*, size_t*, void*, size_t)
__attribute__((__weak__));
Review Comment:
`__attribute__((__weak__))` -> `BAIDU_WEAK`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org
For additional commands, e-mail: dev-h...@brpc.apache.org
Re: [PR] support jemalloc profiler (brpc)
chenBright commented on code in PR #2737:
URL: https://github.com/apache/brpc/pull/2737#discussion_r1736068717
##
src/brpc/details/jemalloc_profiler.cpp:
##
@@ -0,0 +1,289 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+#include "brpc/controller.h"
+#include "brpc/http_header.h"
+#include "brpc/reloadable_flags.h"
+#include "brpc/uri.h"
+#include "butil/files/file_path.h"
+#include "butil/iobuf.h"
+#include "butil/popen.h"
+#include "gflags/gflags.h"
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+extern "C" {
+// weak symbol: resolved at runtime by the linker if we are using jemalloc,
nullptr otherwise
+int mallctl(const char*, void*, size_t*, void*, size_t)
__attribute__((__weak__));
+}
+
+namespace brpc {
+
+// https://jemalloc.net/jemalloc.3.html
+DEFINE_bool(je_prof_active, false, "control jemalloc prof.active, jemalloc
profiling enabled but inactive,"
+"it toggle profiling at any time during process running");
+DEFINE_int32(je_prof_dump, 0, "control jemalloc prof.dump, change this only
dump profile");
+DEFINE_int32(je_prof_reset, 19, "control jemalloc prof.reset, reset all memory
profile statistics, "
+ "and optionally update the sample rate, default 2^19 B");
+
+// define in src/brpc/builtin/pprof_service.cpp
+extern int MakeProfName(ProfilingType type, char* buf, size_t buf_len);
+
+static bool HasInit(const std::string& fn) {
+static std::set fns;
+if (fns.count(fn) > 0) {
+return true;
+}
+fns.insert(fn);
+return false;
+}
+
+bool HasJemalloc() {
+return mallctl != nullptr;
+}
+
+bool HasEnableJemallocProfile() {
+bool prof = false;
+size_t len = sizeof(prof);
+int ret = mallctl("opt.prof", &prof, &len, nullptr, 0);
+if (ret != 0) {
+LOG(WARNING) << "mallctl get opt.prof err, ret:" << ret;
+return false;
+}
+return prof;
+}
+
+static int JeProfileActive(bool active) {
+if (!HasJemalloc()) {
+LOG(WARNING) << "no jemalloc";
+return -1;
+}
+
+if (!HasEnableJemallocProfile()) {
+LOG(WARNING) << "jemalloc have not set opt.prof before start";
+return -1;
+}
+
+int ret = mallctl("prof.active", nullptr, nullptr, &active,
sizeof(active));
+if (ret != 0) {
+LOG(WARNING) << "mallctl set prof.active:" << active << " err, ret:"
<< ret;
+return ret;
+}
+LOG(INFO) << "mallctl set prof.active:" << active << " succ";
+return 0;
+}
+
+static std::string JeProfileDump() {
+if (!HasJemalloc()) {
+LOG(WARNING) << "no jemalloc";
+return "";
+}
+
+if (!HasEnableJemallocProfile()) {
+LOG(WARNING) << "jemalloc have not set opt.prof before start";
+return "";
+}
+
+char prof_name[256];
+if (MakeProfName(PROFILING_HEAP, prof_name, sizeof(prof_name)) != 0) {
+LOG(WARNING) << "Fail to create .prof file, " << berror();
+return "";
+}
+butil::File::Error error;
+const butil::FilePath dir = butil::FilePath(prof_name).DirName();
+if (!butil::CreateDirectoryAndGetError(dir, &error)) {
+LOG(WARNING) << "Fail to create directory= " << dir.value();
+return "";
+}
+
+const char* p_prof_name = prof_name;
+int ret = mallctl("prof.dump", NULL, NULL, (void*)&p_prof_name,
sizeof(p_prof_name));
+if (ret != 0) {
+LOG(WARNING) << "mallctl set prof.dump:" << p_prof_name << " err,
ret:" << ret;
+return "";
+}
+LOG(INFO) << "heap profile dump:" << prof_name << " succ";
+return prof_name;
+}
+
+static int JeProfileReset(size_t lg_sample) {
+if (!HasJemalloc()) {
+LOG(WARNING) << "no jemalloc";
+return -1;
+}
+
+if (!HasEnableJemallocProfile()) {
+LOG(WARNING) << "jemalloc have not set opt.prof before start";
+return -1;
+}
+
+int ret = mallctl("prof.reset", nullptr, nullptr, &lg_sample,
sizeof(lg_sample));
+if (ret != 0) {
+LOG(WARNING) << "mallctl set prof.reset:" << lg_sample << " err, ret:"
<< ret;
+return ret;
+}
+LOG(INFO) << "mallctl set prof.reset:" << lg_sample << " succ";
+
+FLAGS_je_prof_active = false;
+
Re: [PR] support jemalloc profiler (brpc)
chenBright commented on PR #2737: URL: https://github.com/apache/brpc/pull/2737#issuecomment-2317430500 > > 可以集成到内置服务heap页面吗？ > > 尝试了，放弃了。。 你帮忙改下？ 那后面再看看怎么支持吧 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For additional commands, e-mail: dev-h...@brpc.apache.org
Re: [PR] support jemalloc profiler (brpc)
GreateCode commented on PR #2737: URL: https://github.com/apache/brpc/pull/2737#issuecomment-2311767165 > 可以集成到内置服务heap页面吗？ 尝试了，放弃了。。 你帮忙改下？ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For additional commands, e-mail: dev-h...@brpc.apache.org
Re: [PR] support jemalloc profiler (brpc)
GreateCode commented on code in PR #2737: URL: https://github.com/apache/brpc/pull/2737#discussion_r1732289243 ## docs/cn/heap_profiler.md: ## @@ -107,3 +107,34 @@ brpc还提供一个类似的growth profiler分析内存的分配去向（不考 1. 安装[standalone pprof](https://github.com/google/pprof)，并把下载的pprof二进制文件路径写入环境变量GOOGLE_PPROF_BINARY_PATH中 2. 安装llvm-symbolizer（将函数符号转化为函数名），直接用brew安装即可：`brew install llvm` + +# Jemalloc Heap Profiler + +## 开启方法 + +1. 编译[jemalloc](https://github.com/jemalloc/jemalloc)时需--enable-prof以支持profiler, 安装完成后bin目录下会有jeprof文件。 +2. 启动进程前最好配置env `JEPROF_FILE=/xxx/jeprof`，否则进程默认用$PATH里的jeprof解析。 +3. 启动进程并开启profiler功能：`MALLOC_CONF="prof:true" LD_PRELOAD=/xxx/lib/libjemalloc.so ./bin/test_server`，MALLOC_CONF是env项，此时只做一些初始化动作，并不会采样。若启动就采样可设置`MALLOC_CONF="prof:true,prof_active:true,lg_prof_sample:19"`，或通过下面的gflags控制，gflags不会反应MALLOC_CONF值。 +4. 相关gflags说明： +- FLAGS_je_prof_active：true:开启采样，false:关闭采样。 +- FLAGS_je_prof_dump：修改值会生成heap文件，用于手动操作jeprof分析。 +- FLAGS_je_prof_reset：清理已采样数据和重置profiler选项，并且动态设置采样率，[默认](https://jemalloc.net/jemalloc.3.html#opt.lg_prof_sample)2^19B（512K）。 +5. 若要做memory leak，需`MALLOC_CONF="prof:true,prof_leak:true,prof_final:true"`，进程退出时生成heap文件，注：可`kill pid`，不可`kill -9 pid`。 + +注：每次dump的都是从采样至今的所有数据，若触发了reset，接来下dump的是从reset至今的所有数据，方便做diff。 +更多jemalloc profiler选项请参考[官网](https://jemalloc.net/jemalloc.3.html)，如prof_leak_error:true则检测到内存泄漏，进程立即退出。 + +## 样例 + +- jeprof命令`jeprof 127.0.0.1:12345/pprof/heap`。 + + + +- curl生成text格式`curl 127.0.0.1:12345/pprof/heap?display=text`。 + + + +- curl生成svg图片格式`curl 127.0.0.1:12345/pprof/heap?display=svg`。 Review Comment: 用JEPROF_FILE指定或$PATH的jeprof转换成svg格式图片。 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For additional commands, e-mail: dev-h...@brpc.apache.org
Re: [PR] support jemalloc profiler (brpc)
GreateCode commented on code in PR #2737:
URL: https://github.com/apache/brpc/pull/2737#discussion_r1732285273
##
src/brpc/details/jemalloc_profiler.cpp:
##
@@ -0,0 +1,289 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+#include "brpc/controller.h"
+#include "brpc/http_header.h"
+#include "brpc/reloadable_flags.h"
+#include "brpc/uri.h"
+#include "butil/files/file_path.h"
+#include "butil/iobuf.h"
+#include "butil/popen.h"
+#include "gflags/gflags.h"
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+extern "C" {
+// weak symbol: resolved at runtime by the linker if we are using jemalloc,
nullptr otherwise
+int mallctl(const char*, void*, size_t*, void*, size_t)
__attribute__((__weak__));
+}
+
+namespace brpc {
+
+// https://jemalloc.net/jemalloc.3.html
+DEFINE_bool(je_prof_active, false, "control jemalloc prof.active, jemalloc
profiling enabled but inactive,"
+"it toggle profiling at any time during process running");
+DEFINE_int32(je_prof_dump, 0, "control jemalloc prof.dump, change this only
dump profile");
+DEFINE_int32(je_prof_reset, 19, "control jemalloc prof.reset, reset all memory
profile statistics, "
+ "and optionally update the sample rate, default 2^19 B");
+
+// define in src/brpc/builtin/pprof_service.cpp
+extern int MakeProfName(ProfilingType type, char* buf, size_t buf_len);
+
+static bool HasInit(const std::string& fn) {
+static std::set fns;
+if (fns.count(fn) > 0) {
+return true;
+}
+fns.insert(fn);
+return false;
+}
+
+bool HasJemalloc() {
+return mallctl != nullptr;
+}
+
+bool HasEnableJemallocProfile() {
+bool prof = false;
+size_t len = sizeof(prof);
+int ret = mallctl("opt.prof", &prof, &len, nullptr, 0);
+if (ret != 0) {
+LOG(WARNING) << "mallctl get opt.prof err, ret:" << ret;
+return false;
+}
+return prof;
+}
+
+static int JeProfileActive(bool active) {
+if (!HasJemalloc()) {
+LOG(WARNING) << "no jemalloc";
+return -1;
+}
+
+if (!HasEnableJemallocProfile()) {
+LOG(WARNING) << "jemalloc have not set opt.prof before start";
+return -1;
+}
+
+int ret = mallctl("prof.active", nullptr, nullptr, &active,
sizeof(active));
+if (ret != 0) {
+LOG(WARNING) << "mallctl set prof.active:" << active << " err, ret:"
<< ret;
+return ret;
+}
+LOG(INFO) << "mallctl set prof.active:" << active << " succ";
+return 0;
+}
+
+static std::string JeProfileDump() {
+if (!HasJemalloc()) {
+LOG(WARNING) << "no jemalloc";
+return "";
+}
+
+if (!HasEnableJemallocProfile()) {
+LOG(WARNING) << "jemalloc have not set opt.prof before start";
+return "";
+}
+
+char prof_name[256];
+if (MakeProfName(PROFILING_HEAP, prof_name, sizeof(prof_name)) != 0) {
+LOG(WARNING) << "Fail to create .prof file, " << berror();
+return "";
+}
+butil::File::Error error;
+const butil::FilePath dir = butil::FilePath(prof_name).DirName();
+if (!butil::CreateDirectoryAndGetError(dir, &error)) {
+LOG(WARNING) << "Fail to create directory= " << dir.value();
+return "";
+}
+
+const char* p_prof_name = prof_name;
+int ret = mallctl("prof.dump", NULL, NULL, (void*)&p_prof_name,
sizeof(p_prof_name));
+if (ret != 0) {
+LOG(WARNING) << "mallctl set prof.dump:" << p_prof_name << " err,
ret:" << ret;
+return "";
+}
+LOG(INFO) << "heap profile dump:" << prof_name << " succ";
+return prof_name;
+}
+
+static int JeProfileReset(size_t lg_sample) {
+if (!HasJemalloc()) {
+LOG(WARNING) << "no jemalloc";
+return -1;
+}
+
+if (!HasEnableJemallocProfile()) {
+LOG(WARNING) << "jemalloc have not set opt.prof before start";
+return -1;
+}
+
+int ret = mallctl("prof.reset", nullptr, nullptr, &lg_sample,
sizeof(lg_sample));
+if (ret != 0) {
+LOG(WARNING) << "mallctl set prof.reset:" << lg_sample << " err, ret:"
<< ret;
+return ret;
+}
+LOG(INFO) << "mallctl set prof.reset:" << lg_sample << " succ";
+
+FLAGS_je_prof_active = false;
+
Re: [PR] support jemalloc profiler (brpc)
chenBright commented on code in PR #2737: URL: https://github.com/apache/brpc/pull/2737#discussion_r1732220875 ## docs/cn/heap_profiler.md: ## @@ -107,3 +107,34 @@ brpc还提供一个类似的growth profiler分析内存的分配去向（不考 1. 安装[standalone pprof](https://github.com/google/pprof)，并把下载的pprof二进制文件路径写入环境变量GOOGLE_PPROF_BINARY_PATH中 2. 安装llvm-symbolizer（将函数符号转化为函数名），直接用brew安装即可：`brew install llvm` + +# Jemalloc Heap Profiler + +## 开启方法 + +1. 编译[jemalloc](https://github.com/jemalloc/jemalloc)时需--enable-prof以支持profiler, 安装完成后bin目录下会有jeprof文件。 +2. 启动进程前最好配置env `JEPROF_FILE=/xxx/jeprof`，否则进程默认用$PATH里的jeprof解析。 +3. 启动进程并开启profiler功能：`MALLOC_CONF="prof:true" LD_PRELOAD=/xxx/lib/libjemalloc.so ./bin/test_server`，MALLOC_CONF是env项，此时只做一些初始化动作，并不会采样。若启动就采样可设置`MALLOC_CONF="prof:true,prof_active:true,lg_prof_sample:19"`，或通过下面的gflags控制，gflags不会反应MALLOC_CONF值。 +4. 相关gflags说明： +- FLAGS_je_prof_active：true:开启采样，false:关闭采样。 +- FLAGS_je_prof_dump：修改值会生成heap文件，用于手动操作jeprof分析。 +- FLAGS_je_prof_reset：清理已采样数据和重置profiler选项，并且动态设置采样率，[默认](https://jemalloc.net/jemalloc.3.html#opt.lg_prof_sample)2^19B（512K）。 +5. 若要做memory leak，需`MALLOC_CONF="prof:true,prof_leak:true,prof_final:true"`，进程退出时生成heap文件，注：可`kill pid`，不可`kill -9 pid`。 + +注：每次dump的都是从采样至今的所有数据，若触发了reset，接来下dump的是从reset至今的所有数据，方便做diff。 +更多jemalloc profiler选项请参考[官网](https://jemalloc.net/jemalloc.3.html)，如prof_leak_error:true则检测到内存泄漏，进程立即退出。 + +## 样例 + +- jeprof命令`jeprof 127.0.0.1:12345/pprof/heap`。 + + + +- curl生成text格式`curl 127.0.0.1:12345/pprof/heap?display=text`。 + + + +- curl生成svg图片格式`curl 127.0.0.1:12345/pprof/heap?display=svg`。 Review Comment: ip:port/pprof/heap?display=svg ？ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For additional commands, e-mail: dev-h...@brpc.apache.org
Re: [PR] support jemalloc profiler (brpc)
chenBright commented on code in PR #2737:
URL: https://github.com/apache/brpc/pull/2737#discussion_r1718535889
##
src/brpc/details/jemalloc_profiler.cpp:
##
@@ -0,0 +1,289 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+#include "brpc/controller.h"
+#include "brpc/http_header.h"
+#include "brpc/reloadable_flags.h"
+#include "brpc/uri.h"
+#include "butil/files/file_path.h"
+#include "butil/iobuf.h"
+#include "butil/popen.h"
+#include "gflags/gflags.h"
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+extern "C" {
+// weak symbol: resolved at runtime by the linker if we are using jemalloc,
nullptr otherwise
+int mallctl(const char*, void*, size_t*, void*, size_t)
__attribute__((__weak__));
+}
+
+namespace brpc {
+
+// https://jemalloc.net/jemalloc.3.html
+DEFINE_bool(je_prof_active, false, "control jemalloc prof.active, jemalloc
profiling enabled but inactive,"
+"it toggle profiling at any time during process running");
+DEFINE_int32(je_prof_dump, 0, "control jemalloc prof.dump, change this only
dump profile");
+DEFINE_int32(je_prof_reset, 19, "control jemalloc prof.reset, reset all memory
profile statistics, "
+ "and optionally update the sample rate, default 2^19 B");
+
+// define in src/brpc/builtin/pprof_service.cpp
+extern int MakeProfName(ProfilingType type, char* buf, size_t buf_len);
+
+static bool HasInit(const std::string& fn) {
+static std::set fns;
+if (fns.count(fn) > 0) {
+return true;
+}
+fns.insert(fn);
+return false;
+}
+
+bool HasJemalloc() {
+return mallctl != nullptr;
+}
+
+bool HasEnableJemallocProfile() {
+bool prof = false;
+size_t len = sizeof(prof);
+int ret = mallctl("opt.prof", &prof, &len, nullptr, 0);
+if (ret != 0) {
+LOG(WARNING) << "mallctl get opt.prof err, ret:" << ret;
+return false;
+}
+return prof;
+}
+
+static int JeProfileActive(bool active) {
+if (!HasJemalloc()) {
+LOG(WARNING) << "no jemalloc";
+return -1;
+}
+
+if (!HasEnableJemallocProfile()) {
+LOG(WARNING) << "jemalloc have not set opt.prof before start";
+return -1;
+}
+
+int ret = mallctl("prof.active", nullptr, nullptr, &active,
sizeof(active));
+if (ret != 0) {
+LOG(WARNING) << "mallctl set prof.active:" << active << " err, ret:"
<< ret;
+return ret;
+}
+LOG(INFO) << "mallctl set prof.active:" << active << " succ";
+return 0;
+}
+
+static std::string JeProfileDump() {
+if (!HasJemalloc()) {
+LOG(WARNING) << "no jemalloc";
+return "";
+}
+
+if (!HasEnableJemallocProfile()) {
+LOG(WARNING) << "jemalloc have not set opt.prof before start";
+return "";
+}
+
+char prof_name[256];
+if (MakeProfName(PROFILING_HEAP, prof_name, sizeof(prof_name)) != 0) {
+LOG(WARNING) << "Fail to create .prof file, " << berror();
+return "";
+}
+butil::File::Error error;
+const butil::FilePath dir = butil::FilePath(prof_name).DirName();
+if (!butil::CreateDirectoryAndGetError(dir, &error)) {
+LOG(WARNING) << "Fail to create directory= " << dir.value();
+return "";
+}
+
+const char* p_prof_name = prof_name;
+int ret = mallctl("prof.dump", NULL, NULL, (void*)&p_prof_name,
sizeof(p_prof_name));
+if (ret != 0) {
+LOG(WARNING) << "mallctl set prof.dump:" << p_prof_name << " err,
ret:" << ret;
+return "";
+}
+LOG(INFO) << "heap profile dump:" << prof_name << " succ";
+return prof_name;
+}
+
+static int JeProfileReset(size_t lg_sample) {
+if (!HasJemalloc()) {
+LOG(WARNING) << "no jemalloc";
+return -1;
+}
+
+if (!HasEnableJemallocProfile()) {
+LOG(WARNING) << "jemalloc have not set opt.prof before start";
+return -1;
+}
+
+int ret = mallctl("prof.reset", nullptr, nullptr, &lg_sample,
sizeof(lg_sample));
+if (ret != 0) {
+LOG(WARNING) << "mallctl set prof.reset:" << lg_sample << " err, ret:"
<< ret;
+return ret;
+}
+LOG(INFO) << "mallctl set prof.reset:" << lg_sample << " succ";
+
+FLAGS_je_prof_active = false;
+
Re: [PR] support jemalloc profiler (brpc)
chenBright commented on code in PR #2737:
URL: https://github.com/apache/brpc/pull/2737#discussion_r1718535889
##
src/brpc/details/jemalloc_profiler.cpp:
##
@@ -0,0 +1,289 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+#include "brpc/controller.h"
+#include "brpc/http_header.h"
+#include "brpc/reloadable_flags.h"
+#include "brpc/uri.h"
+#include "butil/files/file_path.h"
+#include "butil/iobuf.h"
+#include "butil/popen.h"
+#include "gflags/gflags.h"
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+extern "C" {
+// weak symbol: resolved at runtime by the linker if we are using jemalloc,
nullptr otherwise
+int mallctl(const char*, void*, size_t*, void*, size_t)
__attribute__((__weak__));
+}
+
+namespace brpc {
+
+// https://jemalloc.net/jemalloc.3.html
+DEFINE_bool(je_prof_active, false, "control jemalloc prof.active, jemalloc
profiling enabled but inactive,"
+"it toggle profiling at any time during process running");
+DEFINE_int32(je_prof_dump, 0, "control jemalloc prof.dump, change this only
dump profile");
+DEFINE_int32(je_prof_reset, 19, "control jemalloc prof.reset, reset all memory
profile statistics, "
+ "and optionally update the sample rate, default 2^19 B");
+
+// define in src/brpc/builtin/pprof_service.cpp
+extern int MakeProfName(ProfilingType type, char* buf, size_t buf_len);
+
+static bool HasInit(const std::string& fn) {
+static std::set fns;
+if (fns.count(fn) > 0) {
+return true;
+}
+fns.insert(fn);
+return false;
+}
+
+bool HasJemalloc() {
+return mallctl != nullptr;
+}
+
+bool HasEnableJemallocProfile() {
+bool prof = false;
+size_t len = sizeof(prof);
+int ret = mallctl("opt.prof", &prof, &len, nullptr, 0);
+if (ret != 0) {
+LOG(WARNING) << "mallctl get opt.prof err, ret:" << ret;
+return false;
+}
+return prof;
+}
+
+static int JeProfileActive(bool active) {
+if (!HasJemalloc()) {
+LOG(WARNING) << "no jemalloc";
+return -1;
+}
+
+if (!HasEnableJemallocProfile()) {
+LOG(WARNING) << "jemalloc have not set opt.prof before start";
+return -1;
+}
+
+int ret = mallctl("prof.active", nullptr, nullptr, &active,
sizeof(active));
+if (ret != 0) {
+LOG(WARNING) << "mallctl set prof.active:" << active << " err, ret:"
<< ret;
+return ret;
+}
+LOG(INFO) << "mallctl set prof.active:" << active << " succ";
+return 0;
+}
+
+static std::string JeProfileDump() {
+if (!HasJemalloc()) {
+LOG(WARNING) << "no jemalloc";
+return "";
+}
+
+if (!HasEnableJemallocProfile()) {
+LOG(WARNING) << "jemalloc have not set opt.prof before start";
+return "";
+}
+
+char prof_name[256];
+if (MakeProfName(PROFILING_HEAP, prof_name, sizeof(prof_name)) != 0) {
+LOG(WARNING) << "Fail to create .prof file, " << berror();
+return "";
+}
+butil::File::Error error;
+const butil::FilePath dir = butil::FilePath(prof_name).DirName();
+if (!butil::CreateDirectoryAndGetError(dir, &error)) {
+LOG(WARNING) << "Fail to create directory= " << dir.value();
+return "";
+}
+
+const char* p_prof_name = prof_name;
+int ret = mallctl("prof.dump", NULL, NULL, (void*)&p_prof_name,
sizeof(p_prof_name));
+if (ret != 0) {
+LOG(WARNING) << "mallctl set prof.dump:" << p_prof_name << " err,
ret:" << ret;
+return "";
+}
+LOG(INFO) << "heap profile dump:" << prof_name << " succ";
+return prof_name;
+}
+
+static int JeProfileReset(size_t lg_sample) {
+if (!HasJemalloc()) {
+LOG(WARNING) << "no jemalloc";
+return -1;
+}
+
+if (!HasEnableJemallocProfile()) {
+LOG(WARNING) << "jemalloc have not set opt.prof before start";
+return -1;
+}
+
+int ret = mallctl("prof.reset", nullptr, nullptr, &lg_sample,
sizeof(lg_sample));
+if (ret != 0) {
+LOG(WARNING) << "mallctl set prof.reset:" << lg_sample << " err, ret:"
<< ret;
+return ret;
+}
+LOG(INFO) << "mallctl set prof.reset:" << lg_sample << " succ";
+
+FLAGS_je_prof_active = false;
+
Re: [PR] support jemalloc profiler (brpc)
chenBright commented on PR #2737: URL: https://github.com/apache/brpc/pull/2737#issuecomment-2311672748 可以集成到内置服务heap页面吗？ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For additional commands, e-mail: dev-h...@brpc.apache.org
Re: [PR] support jemalloc profiler (brpc)
wwbmmm commented on PR #2737: URL: https://github.com/apache/brpc/pull/2737#issuecomment-2309180938 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For additional commands, e-mail: dev-h...@brpc.apache.org
Re: [PR] support jemalloc profiler (brpc)
GreateCode commented on PR #2737:
URL: https://github.com/apache/brpc/pull/2737#issuecomment-2303563862
补充下，用静态方式链接jemalloc，进程加载其他so库比如dlopen("/xxx/libjvm.so",
xx)申请的内存，可能不会被profiler采样到。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org
For additional commands, e-mail: dev-h...@brpc.apache.org
Re: [PR] support jemalloc profiler (brpc)
yanglimingcn commented on PR #2737: URL: https://github.com/apache/brpc/pull/2737#issuecomment-2303551884 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For additional commands, e-mail: dev-h...@brpc.apache.org
Re: [PR] support jemalloc profiler (brpc)
GreateCode commented on PR #2737: URL: https://github.com/apache/brpc/pull/2737#issuecomment-2303538640 > bazel编译，也是动态链接libjemalloc.so库吗 不强制，和编译方式无关，只需profiler时能找到mallctl即可。 所以静态链接jemalloc时，不用LD_PRELOAD=/xxx/lib/libjemalloc.so，直接MALLOC_CONF="prof:true" bin/xxx。 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For additional commands, e-mail: dev-h...@brpc.apache.org
Re: [PR] support jemalloc profiler (brpc)
yanglimingcn commented on PR #2737: URL: https://github.com/apache/brpc/pull/2737#issuecomment-2303519050 bazel编译，也是动态链接libjemalloc.so库吗 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For additional commands, e-mail: dev-h...@brpc.apache.org
Re: [PR] support jemalloc profiler (brpc)
GreateCode commented on PR #2737: URL: https://github.com/apache/brpc/pull/2737#issuecomment-2300412120 > 感谢贡献！ 能否顺便更新一下使用文档？ done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For additional commands, e-mail: dev-h...@brpc.apache.org
Re: [PR] support jemalloc profiler (brpc)
GreateCode commented on PR #2737: URL: https://github.com/apache/brpc/pull/2737#issuecomment-2297824620 好的，我近期更新下 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For additional commands, e-mail: dev-h...@brpc.apache.org
Re: [PR] support jemalloc profiler (brpc)
wwbmmm commented on PR #2737: URL: https://github.com/apache/brpc/pull/2737#issuecomment-2296183777 感谢贡献！ 能否顺便更新一下使用文档？ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For additional commands, e-mail: dev-h...@brpc.apache.org
