This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new e41b03e5301 [Fix](multi-catalog) delete hdfs hedged configs at BE
side. (#25094)
e41b03e5301 is described below
commit e41b03e53017d7b364f471ffe171dc75a5f0233d
Author: Xiangyu Wang <[email protected]>
AuthorDate: Wed Oct 11 23:25:30 2023 +0800
[Fix](multi-catalog) delete hdfs hedged configs at BE side. (#25094)
Issue Number: close #25093
We can set hdfs hedged configs when creating catalog, just like this:
```
CREATE CATALOG `test_ctl` PROPERTIES (
...
"dfs.client.hedged.read.threadpool.size" = "128",
"dfs.client.hedged.read.threshold.millis" = "500",
...
);
```
It is redundant to set these configs at BE side, and it will brings an
occasional bug at #25093 .
---
be/src/common/config.cpp | 4 ----
be/src/common/config.h | 10 ----------
be/src/io/hdfs_builder.cpp | 11 -----------
docs/en/docs/lakehouse/faq.md | 10 ----------
docs/zh-CN/docs/lakehouse/faq.md | 11 +----------
5 files changed, 1 insertion(+), 45 deletions(-)
diff --git a/be/src/common/config.cpp b/be/src/common/config.cpp
index 78b0a2c79db..ffc4dae7683 100644
--- a/be/src/common/config.cpp
+++ b/be/src/common/config.cpp
@@ -1071,10 +1071,6 @@ DEFINE_mInt64(lookup_connection_cache_bytes_limit,
"4294967296");
// level of compression when using LZ4_HC, whose defalut value is
LZ4HC_CLEVEL_DEFAULT
DEFINE_mInt64(LZ4_HC_compression_level, "9");
-DEFINE_Bool(enable_hdfs_hedged_read, "false");
-DEFINE_Int32(hdfs_hedged_read_thread_num, "128");
-DEFINE_Int32(hdfs_hedged_read_threshold_time, "500");
-
DEFINE_mBool(enable_merge_on_write_correctness_check, "true");
// The secure path with user files, used in the `local` table function.
diff --git a/be/src/common/config.h b/be/src/common/config.h
index 72ff579a812..3010cf49763 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -1127,16 +1127,6 @@ DECLARE_mInt64(lookup_connection_cache_bytes_limit);
// level of compression when using LZ4_HC, whose defalut value is
LZ4HC_CLEVEL_DEFAULT
DECLARE_mInt64(LZ4_HC_compression_level);
-// whether to enable hdfs hedged read.
-// If set to true, it will be enabled even if user not enable it when creating
catalog
-DECLARE_Bool(enable_hdfs_hedged_read);
-// hdfs hedged read thread pool size, for
"dfs.client.hedged.read.threadpool.size"
-// Maybe overwritten by the value specified when creating catalog
-DECLARE_Int32(hdfs_hedged_read_thread_num);
-// the threshold of doing hedged read, for
"dfs.client.hedged.read.threshold.millis"
-// Maybe overwritten by the value specified when creating catalog
-DECLARE_Int32(hdfs_hedged_read_threshold_time);
-
DECLARE_mBool(enable_merge_on_write_correctness_check);
// The secure path with user files, used in the `local` table function.
diff --git a/be/src/io/hdfs_builder.cpp b/be/src/io/hdfs_builder.cpp
index 754eb61680b..41cac611de7 100644
--- a/be/src/io/hdfs_builder.cpp
+++ b/be/src/io/hdfs_builder.cpp
@@ -146,17 +146,6 @@ Status create_hdfs_builder(const THdfsParams& hdfsParams,
const std::string& fs_
}
}
-#ifdef USE_HADOOP_HDFS
- if (config::enable_hdfs_hedged_read) {
- hdfsBuilderConfSetStr(builder->get(),
"dfs.client.hedged.read.threadpool.size",
-
std::to_string(config::hdfs_hedged_read_thread_num).c_str());
- hdfsBuilderConfSetStr(builder->get(),
"dfs.client.hedged.read.threshold.millis",
-
std::to_string(config::hdfs_hedged_read_threshold_time).c_str());
- LOG(INFO) << "set hdfs hedged read config: " <<
config::hdfs_hedged_read_thread_num << ", "
- << config::hdfs_hedged_read_threshold_time;
- }
-#endif
-
hdfsBuilderConfSetStr(builder->get(),
"ipc.client.fallback-to-simple-auth-allowed", "true");
if (builder->is_need_kinit()) {
diff --git a/docs/en/docs/lakehouse/faq.md b/docs/en/docs/lakehouse/faq.md
index 8f6036de068..55e21a106d5 100644
--- a/docs/en/docs/lakehouse/faq.md
+++ b/docs/en/docs/lakehouse/faq.md
@@ -259,16 +259,6 @@ under the License.
`dfs.client.hedged.read.threshold.millis` is the read threshold in
milliseconds. When a read request exceeds this threshold and is not returned,
Hedged Read will be triggered.
- - Configure parameters in be.conf
-
- ```
- enable_hdfs_hedged_read = true
- hdfs_hedged_read_thread_num = 128
- hdfs_hedged_read_threshold_time = 500
- ```
-
- This method will enable Hedged Read globally on BE nodes (not enabled
by default). And ignore the Hedged Read property set when creating the Catalog.
-
After enabling it, you can see related parameters in Query Profile:
`TotalHedgedRead`: The number of Hedged Reads initiated.
diff --git a/docs/zh-CN/docs/lakehouse/faq.md b/docs/zh-CN/docs/lakehouse/faq.md
index 653e2177b6f..6e3df6daded 100644
--- a/docs/zh-CN/docs/lakehouse/faq.md
+++ b/docs/zh-CN/docs/lakehouse/faq.md
@@ -253,16 +253,7 @@ under the License.
`dfs.client.hedged.read.threadpool.size` 表示用于 Hedged Read 的线程数，这些线程由一个
HDFS Client 共享。通常情况下，针对一个 HDFS 集群，BE 节点会共享一个 HDFS Client。
`dfs.client.hedged.read.threshold.millis`
是读取阈值，单位毫秒。当一个读请求超过这个阈值未返回时，会触发 Hedged Read。
-
- - 在 be.conf 中配置参数
-
- ```
- enable_hdfs_hedged_read = true
- hdfs_hedged_read_thread_num = 128
- hdfs_hedged_read_threshold_time = 500
- ```
-
- 这种方式会在BE节点全局开启 Hedged Read（默认不开启）。并忽略在创建 Catalog 时设置的 Hedged Read 属性。
+
开启后，可以在 Query Profile 中看到相关参数：
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
