This is an automated email from the ASF dual-hosted git repository.
panxiaolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 1afe090486 [improvement](memory) modify jemalloc conf in be.conf
(#21943)
1afe090486 is described below
commit 1afe0904868d47ec0d22d5d194bfd2fab2fb6dc3
Author: Xinyi Zou <[email protected]>
AuthorDate: Thu Jul 20 10:34:31 2023 +0800
[improvement](memory) modify jemalloc conf in be.conf (#21943)
modify jemalloc conf in be.conf
disable je_purge_all_arena_dirty_pages
---
be/src/util/mem_info.h | 18 ++++-----
bin/start_be.sh | 13 +++++-
conf/be.conf | 5 +++
docs/en/community/developer-guide/debug-tool.md | 47 ++++++++++++----------
docs/zh-CN/community/developer-guide/debug-tool.md | 47 ++++++++++++----------
5 files changed, 77 insertions(+), 53 deletions(-)
diff --git a/be/src/util/mem_info.h b/be/src/util/mem_info.h
index c83068befd..179295fd69 100644
--- a/be/src/util/mem_info.h
+++ b/be/src/util/mem_info.h
@@ -109,15 +109,15 @@ public:
static inline void je_purge_all_arena_dirty_pages() {
#ifdef USE_JEMALLOC
- try {
- // Purge all unused dirty pages for arena <i>, or for all arenas
if <i> equals MALLCTL_ARENAS_ALL.
- jemallctl(fmt::format("arena.{}.purge",
MALLCTL_ARENAS_ALL).c_str(), nullptr, nullptr,
- nullptr, 0);
- } catch (...) {
- // https://github.com/jemalloc/jemalloc/issues/2470
- // Occasional core dump during stress test.
- LOG(WARNING) << "Purge all unused dirty pages for all arenas
failed";
- }
+ // https://github.com/jemalloc/jemalloc/issues/2470
+ // Occasional core dump during stress test, purge should be turned on
after the heap corruption is resolved.
+ // try {
+ // // Purge all unused dirty pages for arena <i>, or for all
arenas if <i> equals MALLCTL_ARENAS_ALL.
+ // jemallctl(fmt::format("arena.{}.purge",
MALLCTL_ARENAS_ALL).c_str(), nullptr, nullptr,
+ // nullptr, 0);
+ // } catch (...) {
+ // LOG(WARNING) << "Purge all unused dirty pages for all arenas
failed";
+ // }
#endif
}
diff --git a/bin/start_be.sh b/bin/start_be.sh
index 60d2677399..0aeea47961 100755
--- a/bin/start_be.sh
+++ b/bin/start_be.sh
@@ -303,8 +303,17 @@ export LIBHDFS_OPTS="${final_java_opt}"
#echo "LD_LIBRARY_PATH: ${LD_LIBRARY_PATH}"
#echo "LIBHDFS_OPTS: ${LIBHDFS_OPTS}"
-# see
https://github.com/apache/doris/blob/master/docs/zh-CN/community/developer-guide/debug-tool.md#jemalloc-heap-profile
-export
JEMALLOC_CONF="percpu_arena:percpu,background_thread:true,metadata_thp:auto,muzzy_decay_ms:30000,dirty_decay_ms:30000,oversize_threshold:0,lg_tcache_max:16,prof_prefix:jeprof.out"
+if [[ -z ${JEMALLOC_CONF} ]]; then
+
JEMALLOC_CONF="percpu_arena:percpu,background_thread:true,metadata_thp:auto,muzzy_decay_ms:15000,dirty_decay_ms:15000,oversize_threshold:0,lg_tcache_max:20,prof:false,lg_prof_interval:32,lg_prof_sample:19,prof_gdump:false,prof_accum:false,prof_leak:false,prof_final:false,confirm_conf:true"
+fi
+
+if [[ -z ${JEMALLOC_PROF_PRFIX} ]]; then
+ export JEMALLOC_CONF="${JEMALLOC_CONF}"
+else
+ JEMALLOC_PROF_PRFIX="${DORIS_HOME}/log/${JEMALLOC_PROF_PRFIX}"
+ export JEMALLOC_CONF="${JEMALLOC_CONF},prof_prefix:${JEMALLOC_PROF_PRFIX}"
+fi
+
export AWS_EC2_METADATA_DISABLED=true
export AWS_MAX_ATTEMPTS=2
diff --git a/conf/be.conf b/conf/be.conf
index 220a1122ad..ddc7a2b517 100644
--- a/conf/be.conf
+++ b/conf/be.conf
@@ -27,6 +27,11 @@ JAVA_OPTS_FOR_JDK_9="-Xmx1024m
-DlogPath=$DORIS_HOME/log/jni.log -Xlog:gc:$DORIS
# since 1.2, the JAVA_HOME need to be set to run BE process.
# JAVA_HOME=/path/to/jdk/
+#
https://github.com/apache/doris/blob/master/docs/zh-CN/community/developer-guide/debug-tool.md#jemalloc-heap-profile
+# https://jemalloc.net/jemalloc.3.html
+JEMALLOC_CONF="percpu_arena:percpu,background_thread:true,metadata_thp:auto,muzzy_decay_ms:15000,dirty_decay_ms:15000,oversize_threshold:0,lg_tcache_max:20,prof:false,lg_prof_interval:32,lg_prof_sample:19,prof_gdump:false,prof_accum:false,prof_leak:false,prof_final:false,confirm_conf:true"
+JEMALLOC_PROF_PRFIX=""
+
# INFO, WARNING, ERROR, FATAL
sys_log_level = INFO
diff --git a/docs/en/community/developer-guide/debug-tool.md
b/docs/en/community/developer-guide/debug-tool.md
index d4eeda8ac1..ae9a161f66 100644
--- a/docs/en/community/developer-guide/debug-tool.md
+++ b/docs/en/community/developer-guide/debug-tool.md
@@ -220,21 +220,39 @@ The output of this command is the same as the output and
view mode of heap profi
##### JEMALLOC
-###### 1. runtime heap dump by http
-Add `,prof:true,lg_prof_sample:10` to `JEMALLOC_CONF` in `start_be.sh` and
restart BE, then use the jemalloc heap dump http interface to generate a heap
dump file on the corresponding BE machine.
-
-The directory where the heap dump file is located can be configured through
the ``jeprofile_dir`` variable in ``be.conf``, and the default is
``${DORIS_HOME}/log``
+###### 1. realtime heap dump
+Change `prof:false` of `JEMALLOC_CONF` in `be.conf` to `prof:true` and restart
BE, then use the jemalloc heap dump http interface to generate a heap dump file
on the corresponding BE machine.
```shell
curl http://be_host:be_webport/jeheap/dump
```
-`prof`: After opening, jemalloc will generate a heap dump file according to
the current memory usage. There is a small amount of performance loss in heap
profile sampling, which can be turned off during performance testing.
-`lg_prof_sample`: heap profile sampling interval, the default value is 19,
that is, the default sampling interval is 512K (2^19 B), which will result in
only 10% of the memory recorded by the heap profile, `lg_prof_sample:10` can
reduce the sampling interval to 1K (2^10 B), more frequent sampling will make
the heap profile close to real memory, but this will bring greater performance
loss.
+The directory where the heap dump file is located can be configured through
the ``jeprofile_dir`` variable in ``be.conf``, and the default is
``${DORIS_HOME}/log``
-For detailed parameter description, refer to
https://linux.die.net/man/3/jemalloc.
+The default sampling interval is 512K, usually only 10% of memory is recorded
by heap dump, and the impact on performance is usually less than 10%. You can
modify `lg_prof_sample` of `JEMALLOC_CONF` in `be.conf`, and the default is
`19` (2^19 B = 512K), reducing `lg_prof_sample` can sample more frequently to
make the heap profile close to the real memory, but this will bring greater
performance loss.
+
+If you are doing profiling, keep `prof:false` to avoid the performance penalty
of heap dump.
+
+###### 2. regular heap dump
+Also change `prof:false` of `JEMALLOC_CONF` in `be.conf` to `prof:true`, and
modify `JEMALLOC_PROF_PRFIX` in `be.conf` to any value and restart BE.
+
+The directory where the heap dump file is located is `${DORIS_HOME}/log` by
default, and the file name prefix is `JEMALLOC_PROF_PRFIX`.
+
+1. Dump when accumulatively applying for a certain value of memory:
+
+ The default memory accumulatively applies for 4GB to generate a dump. You
can modify the `lg_prof_interval` of `JEMALLOC_CONF` in `be.conf` to adjust the
dump interval. The default value is `32` (2^32 B = 4GB).
+2. Dump every time the memory reaches a new high:
-##### 2. jemalloc heap dump profiling
+ Change `prof_gdump` of `JEMALLOC_CONF` in `be.conf` to `true` and restart
BE.
+3. Dump when the program exits, and detect memory leaks:
+
+ Change `prof_leak` and `prof_final` of `JEMALLOC_CONF` in `be.conf` to
`true` and restart BE.
+4. Dump memory cumulative value (growth), not real-time value:
+
+ Change `prof_accum` of `JEMALLOC_CONF` in `be.conf` to `true` and restart
BE.
+ Use `jeprof --alloc_space` to display heap dump accumulation.
+
+##### 3. jemalloc heap dump profiling
1. A single heap dump file generates plain text analysis results
```shell
@@ -264,19 +282,6 @@ For detailed parameter description, refer to
https://linux.die.net/man/3/jemallo
jeprof --pdf lib/doris_be --base=heap_dump_file_1 heap_dump_file_2 >
result.pdf
```
-###### 3. heap dump by JEMALLOC_CONF
-Periodic heap dump can also be done by changing the `JEMALLOC_CONF` variable
in `start_be.sh` and restarting BE
-
-1. Dump every 1MB:
-
- Two new variable settings `prof:true,lg_prof_interval:20` have been added
to the `JEMALLOC_CONF` variable, where `prof:true` is to enable profiling, and
`lg_prof_interval:20` means that a dump is generated every 1MB (2^20)
-2. Dump each time a new high is reached:
-
- Two new variable settings `prof:true,prof_gdump:true` have been added to
the `JEMALLOC_CONF` variable, where `prof:true` is to enable profiling, and
`prof_gdump:true` means to generate a dump when the memory usage reaches a new
high
-3. Memory leak dump when the program exits:
-
- Added three new variable settings `prof_leak: true, lg_prof_sample: 0,
prof_final: true` in the `JEMALLOC_CONF` variable.
-
#### LSAN
[LSAN](https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer)
is an address checking tool, GCC has been integrated. When we compile the code,
we can enable this function by turning on the corresponding compilation
options. When the program has a determinable memory leak, it prints the leak
stack. Doris be has integrated this tool, only need to compile with the
following command to generate be binary with memory leak detection version.
diff --git a/docs/zh-CN/community/developer-guide/debug-tool.md
b/docs/zh-CN/community/developer-guide/debug-tool.md
index 435005bc1b..40078edc43 100644
--- a/docs/zh-CN/community/developer-guide/debug-tool.md
+++ b/docs/zh-CN/community/developer-guide/debug-tool.md
@@ -223,21 +223,39 @@ Total: 1296.4 MB
##### JEMALLOC
-###### 1. runtime heap dump by http
-在`start_be.sh` 中`JEMALLOC_CONF` 增加 `,prof:true,lg_prof_sample:10`
并重启BE，然后使用jemalloc heap dump http接口，在对应的BE机器上生成heap dump文件。
-
-heap dump文件所在目录可以在 ``be.conf``
中通过``jeprofile_dir``变量进行配置，默认为``${DORIS_HOME}/log``
+###### 1. realtime heap dump
+将 `be.conf` 中 `JEMALLOC_CONF` 的 `prof:false` 修改为 `prof:true`
并重启BE，然后使用jemalloc heap dump http接口，在对应的BE机器上生成heap dump文件。
```shell
curl http://be_host:be_webport/jeheap/dump
```
-`prof`: 打开后jemalloc将根据当前内存使用情况生成heap dump文件，heap profile采样存在少量性能损耗，性能测试时可关闭。
-`lg_prof_sample`: heap profile采样间隔，默认值19，即默认采样间隔为512K(2^19 B)，这会导致heap
profile记录的内存通常只有10%，`lg_prof_sample:10`可以减少采样间隔到1K (2^10 B), 更频繁的采样会使heap
profile接近真实内存，但这会带来更大的性能损耗。
+heap dump文件所在目录可以在 ``be.conf``
中通过``jeprofile_dir``变量进行配置，默认为``${DORIS_HOME}/log``
+
+默认采样间隔为 512K，这通常只会有 10% 的内存被heap dump记录，对性能的影响通常小于 10%，可以修改 `be.conf` 中
`JEMALLOC_CONF` 的 `lg_prof_sample`，默认为 `19` (2^19 B = 512K)，减小 `lg_prof_sample`
可以更频繁的采样使 heap profile 接近真实内存，但这会带来更大的性能损耗。
+
+如果你在做性能测试，保持 `prof:false` 来避免 heap dump 的性能损耗。
+
+###### 2. regular heap dump
+同样要将 `be.conf` 中 `JEMALLOC_CONF` 的 `prof:false` 修改为 `prof:true`，同时将 `be.conf`
中 `JEMALLOC_PROF_PRFIX` 修改为任意值并重启BE。
+
+heap dump文件所在目录默认为 `${DORIS_HOME}/log`, 文件名前缀是 `JEMALLOC_PROF_PRFIX`。
+
+1. 内存累计申请一定值时dump:
-详细参数说明参考 https://linux.die.net/man/3/jemalloc。
+ 默认内存累计申请 4GB 生成一次dump，可以修改 `be.conf` 中 `JEMALLOC_CONF` 的 `lg_prof_interval`
调整dump间隔，默认值 `32` (2^32 B = 4GB)。
+2. 内存每次达到新高时dump:
-##### 2. jemalloc heap dump profiling
+ 将 `be.conf` 中 `JEMALLOC_CONF` 的 `prof_gdump` 修改为 `true` 并重启BE。
+3. 程序退出时dump, 并检测内存泄漏:
+
+ 将 `be.conf` 中 `JEMALLOC_CONF` 的 `prof_leak` 和 `prof_final` 修改为 `true` 并重启BE。
+4. dump内存累计值(growth)，而不是实时值:
+
+ 将 `be.conf` 中 `JEMALLOC_CONF` 的 `prof_accum` 修改为 `true` 并重启BE。
+ 使用 `jeprof --alloc_space` 展示 heap dump 累计值。
+
+##### 3. heap dump profiling
1. 单个heap dump文件生成纯文本分析结果
```shell
@@ -267,19 +285,6 @@ curl http://be_host:be_webport/jeheap/dump
jeprof --pdf lib/doris_be --base=heap_dump_file_1 heap_dump_file_2 >
result.pdf
```
-###### 3. heap dump by JEMALLOC_CONF
-也可通过更改`start_be.sh` 中`JEMALLOC_CONF` 变量后重新启动 BE 来进行定期 heap dump
-
-1. 每1MB dump一次:
-
- `JEMALLOC_CONF`变量中新增两个变量设置`prof:true,lg_prof_interval:20`
其中`prof:true`是打开profiling，`lg_prof_interval:20`中表示每1MB(2^20)生成一次dump
-2. 每次达到新高时dump:
-
- `JEMALLOC_CONF`变量中新增两个变量设置`prof:true,prof_gdump:true`
其中`prof:true`是打开profiling，`prof_gdump:true` 代表内存使用达到新高时生成dump
-3. 程序退出时内存泄漏dump:
-
- `JEMALLOC_CONF`变量中新增三个变量设置`prof_leak:true,lg_prof_sample:0,prof_final:true`
-
#### LSAN
[LSAN](https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer)是一个地址检查工具，GCC已经集成。在我们编译代码的时候开启相应的编译选项，就能够开启这个功能。当程序发生可以确定的内存泄露时，会将泄露堆栈打印。Doris
BE已经集成了这个工具，只需要在编译的时候使用如下的命令进行编译就能够生成带有内存泄露检测版本的BE二进制
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
