This is an automated email from the ASF dual-hosted git repository.
luzhijing pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 4b9959488a2 [Doc](developer-tools) Introduce Pipeline Tracing tools
(#594)
4b9959488a2 is described below
commit 4b9959488a29377403fc82857919c587e6689a7a
Author: zclllyybb <[email protected]>
AuthorDate: Sat May 4 22:21:02 2024 +0800
[Doc](developer-tools) Introduce Pipeline Tracing tools (#594)
---
community/developer-guide/pipeline-tracing.md | 79 +++++++++++++++++++++
.../current/developer-guide/pipeline-tracing.md | 79 +++++++++++++++++++++
sidebarsCommunity.json | 1 +
static/images/tracing1.png | Bin 0 -> 161844 bytes
static/images/tracing2.png | Bin 0 -> 551394 bytes
5 files changed, 159 insertions(+)
diff --git a/community/developer-guide/pipeline-tracing.md
b/community/developer-guide/pipeline-tracing.md
new file mode 100644
index 00000000000..a9c62c31315
--- /dev/null
+++ b/community/developer-guide/pipeline-tracing.md
@@ -0,0 +1,79 @@
+---
+{
+ "title": "Pipeline Tracing",
+ "language": "en"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Pipeline Tracing
+
+In the Pipeline execution engine, we split the execution plan tree of each
Instance into multiple small Pipeline Tasks and execute them under our custom
Pipeline scheduler. Therefore, in an environment with a large number of
Pipeline Tasks executing, how these Tasks are scheduled across threads and CPU
cores is an important factor for execution performance. We have developed a
specialised tool to observe the scheduling process on a particular query or
time period, which we call "Pipeline [...]
+
+## Usage steps
+
+### 1. Record Data
+
+First we need to log the Pipeline scheduling process. Whether and how to log
the scheduling process can be set via the HTTP interface. These settings are
associated with a specific BE:
+
+1. Turn off Pipeline Tracing record
+
+```shell
+curl -X POST http://{be_host}:{http_port}/api/pipeline/tracing?type=disable
+```
+
+2. Generate a record for each Query
+
+```shell
+curl -X POST http://{be_host}:{http_port}/api/pipeline/tracing?type=perquery
+```
+
+3. Generate Tracing records for a fixed period of time
+
+```shell
+curl -X POST http://{be_host}:{http_port}/api/pipeline/tracing?type=periodic
+```
+
+Set the time period (in seconds):
+```shell
+curl -X POST http://{be_host}:{http_port}/api/pipeline/tracing?dump_interval=60
+```
+
+### 2. Format Conversion
+
+The logged data will be generated in the `log/tracing` directory of the
corresponding BE. The next step is to convert the data to a file that matches
the format required by the visualisation tool. A conversion tool is provided to
convert the tracing logs generated by the BE, which can be executed directly:
+
+```shell
+cd doris/tools/pipeline-tracing/
+python3 origin-to-show.py -s <SOURCE_FILE> -d <DEST>.json
+```
+
+to generate a json file that can be displayed. For more detailed instructions,
see the `README.md` file in this directory.
+
+### 3. Visualisation
+
+Pipeline Tracing is visualised using [Perfetto](https://ui.perfetto.dev/).
After generating a file in the legal format, select "Open trace file" on its
page to open the file and view the results:
+
+
+
+The tool is very powerful. For example, it is easy to see how the same Task is
scheduled across cores.
+
+
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs-community/current/developer-guide/pipeline-tracing.md
b/i18n/zh-CN/docusaurus-plugin-content-docs-community/current/developer-guide/pipeline-tracing.md
new file mode 100644
index 00000000000..93393a970e5
--- /dev/null
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs-community/current/developer-guide/pipeline-tracing.md
@@ -0,0 +1,79 @@
+---
+{
+ "title": "Pipeline Tracing",
+ "language": "zh-CN"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Pipeline Tracing
+
+在 Pipeline 执行引擎中，我们会将每个 Instance 的执行计划树拆分成多个小的 Pipeline Task，并在我们自定义的 Pipeline
调度器调度下执行。因此，在拥有大量 Pipeline Task 的执行环境下，这些 Task 如何在线程和 CPU
核间进行调度，是执行性能的一个重要影响因素。我们开发了一个专门的工具用来观察特定查询或时间段上的调度过程，我们将这个工具称为 "Pipeline
Tracing"。
+
+## 使用步骤
+
+### 1. 记录数据
+
+首先我们需要对 Pipeline 调度过程进行记录。是否以及如何记录调度过程，可以通过 HTTP 接口设置。这些设置关联到特定 BE：
+
+1. 关闭 Pipeline Tracing 记录
+
+```shell
+curl -X POST http://{be_host}:{http_port}/api/pipeline/tracing?type=disable
+```
+
+2. 为每个 Query 产生一条记录
+
+```shell
+curl -X POST http://{be_host}:{http_port}/api/pipeline/tracing?type=perquery
+```
+
+3. 生成固定时间段内的 Tracing 记录
+
+```shell
+curl -X POST http://{be_host}:{http_port}/api/pipeline/tracing?type=periodic
+```
+
+设置时间周期（单位为秒）：
+```shell
+curl -X POST http://{be_host}:{http_port}/api/pipeline/tracing?dump_interval=60
+```
+
+### 2. 格式转换
+
+记录的数据将会生成到对应 BE 的 `log/tracing`
目录下。接下来需要对数据进行格式转换，生成符合可视化工具所需格式的文件。这里我们提供了一个转换工具直接对 BE 生成的 tracing
记录进行转换，可以直接执行：
+
+```shell
+cd doris/tools/pipeline-tracing/
+python3 origin-to-show.py -s <SOURCE_FILE> -d <DEST>.json
+```
+
+生成可以展示的 json 文件。更详细的使用说明，可以参考该目录下的 `README.md` 文件。
+
+### 3. 可视化结果
+
+Pipeline Tracing 的可视化使用
[Perfetto](https://ui.perfetto.dev/)。生成对应格式的文件后，在其页面上选择 "Open trace file"
打开该文件，即可查看结果：
+
+
+
+该工具的功能非常强大，例如可以方便查看同一个 Task 在各个核间的调度情况。
+
+
diff --git a/sidebarsCommunity.json b/sidebarsCommunity.json
index 437980cae94..4ffbfad7b1c 100644
--- a/sidebarsCommunity.json
+++ b/sidebarsCommunity.json
@@ -50,6 +50,7 @@
"developer-guide/debug-tool",
"developer-guide/docker-dev",
"developer-guide/benchmark-tool",
+ "developer-guide/pipeline-tracing",
"developer-guide/fe-eclipse-dev",
"developer-guide/fe-idea-dev",
"developer-guide/fe-vscode-dev",
diff --git a/static/images/tracing1.png b/static/images/tracing1.png
new file mode 100644
index 00000000000..b215de06b2c
Binary files /dev/null and b/static/images/tracing1.png differ
diff --git a/static/images/tracing2.png b/static/images/tracing2.png
new file mode 100644
index 00000000000..f65bceff583
Binary files /dev/null and b/static/images/tracing2.png differ
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
