This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new e090b83e33 [improvemnt](script) support custom lib dir to save custom
libs (#23887)
e090b83e33 is described below
commit e090b83e33273ce0a70595291e79417569bd414e
Author: Mingyu Chen <[email protected]>
AuthorDate: Tue Sep 5 11:54:19 2023 +0800
[improvemnt](script) support custom lib dir to save custom libs (#23887)
Sometimes, user need to add some custom libs to the cluster, such lzo.jar,
orai18n.jar, etc.
In previous, these lib files are places in fe/lib or be/lib.
But when upgrading cluster, the lib dir will be replaced by new lib dir, so
that all custom libs are lost.
In this PR, I add new dir custom_lib for FE and BE, and user can place
custom lib files in it.
---
bin/start_be.sh | 7 +++++++
bin/start_fe.sh | 7 +++++++
docs/en/docs/lakehouse/faq.md | 2 ++
docs/en/docs/lakehouse/multi-catalog/jdbc.md | 4 +++-
docs/en/docs/lakehouse/multi-catalog/paimon.md | 5 ++++-
docs/zh-CN/docs/lakehouse/faq.md | 2 ++
docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md | 2 ++
docs/zh-CN/docs/lakehouse/multi-catalog/paimon.md | 7 +++++--
8 files changed, 32 insertions(+), 4 deletions(-)
diff --git a/bin/start_be.sh b/bin/start_be.sh
index 4dc6abd971..9841e57cda 100755
--- a/bin/start_be.sh
+++ b/bin/start_be.sh
@@ -112,6 +112,13 @@ if [[ -d "${DORIS_HOME}/lib/hadoop_hdfs/" ]]; then
done
fi
+# add custome_libs to CLASSPATH
+if [[ -d "${DORIS_HOME}/custom_lib" ]]; then
+ for f in "${DORIS_HOME}/custom_lib"/*.jar; do
+ DORIS_CLASSPATH="${DORIS_CLASSPATH}:${f}"
+ done
+fi
+
if [[ -n "${HADOOP_CONF_DIR}" ]]; then
export DORIS_CLASSPATH="${DORIS_CLASSPATH}:${HADOOP_CONF_DIR}"
fi
diff --git a/bin/start_fe.sh b/bin/start_fe.sh
index 3cdebe8f4b..82d0e78501 100755
--- a/bin/start_fe.sh
+++ b/bin/start_fe.sh
@@ -176,6 +176,13 @@ for f in "${DORIS_HOME}/lib"/*.jar; do
CLASSPATH="${f}:${CLASSPATH}"
done
+# add custome_libs to CLASSPATH
+if [[ -d "${DORIS_HOME}/custom_lib" ]]; then
+ for f in "${DORIS_HOME}/custom_lib"/*.jar; do
+ CLASSPATH="${f}:${CLASSPATH}"
+ done
+fi
+
# make sure the doris-fe.jar is at first order, so that some classed
# with same qualified name can be loaded priority from doris-fe.jar
CLASSPATH="${DORIS_FE_JAR}:${CLASSPATH}"
diff --git a/docs/en/docs/lakehouse/faq.md b/docs/en/docs/lakehouse/faq.md
index 997a59f371..d1a6e090f7 100644
--- a/docs/en/docs/lakehouse/faq.md
+++ b/docs/en/docs/lakehouse/faq.md
@@ -199,6 +199,8 @@ under the License.
Search in the hadoop environment hadoop-lzo-*.jar, and put it under
"${DORIS_HOME}/fe/lib/",then restart fe.
+ Starting from version 2.0.2, this file can be placed in BE's `custom_lib/`
directory (if it does not exist, just create it manually) to prevent the file
from being lost due to the replacement of the lib directory when upgrading the
cluster.
+
## HDFS
1. What to do with the`java.lang.VerifyError: xxx` error when accessing HDFS
3.x?
diff --git a/docs/en/docs/lakehouse/multi-catalog/jdbc.md
b/docs/en/docs/lakehouse/multi-catalog/jdbc.md
index ab1016cb06..dcab8e056b 100644
--- a/docs/en/docs/lakehouse/multi-catalog/jdbc.md
+++ b/docs/en/docs/lakehouse/multi-catalog/jdbc.md
@@ -791,7 +791,9 @@ DROP CATALOG <catalog_name>;
12. `Non supported character set (add orai18n.jar in your classpath):
ZHS16GBK` exception occurs when reading Oracle
- Download
[orai18n.jar](https://www.oracle.com/database/technologies/appdev/jdbc-downloads.html)
and put it in the lib directory of Doris FE and the lib/java_extensions
directory of BE (Doris versions before 2.0 need to be placed in the lib
directory of BE).
+ Download
[orai18n.jar](https://www.oracle.com/database/technologies/appdev/jdbc-downloads.html)
and put it in the lib directory of Doris FE and the `lib/java_extensions/`
directory of BE (Doris versions before 2.0 need to be placed in the lib
directory of BE).
+
+ Starting from version 2.0.2, this file can be placed in BE's `custom_lib/`
directory (if it does not exist, just create it manually) to prevent the file
from being lost due to the replacement of the lib directory when upgrading the
cluster.
13. `NoClassDefFoundError: net/jpountz/lz4/LZ4Factory` exception occurs when
reading Clickhouse data via jdbc catalog.
diff --git a/docs/en/docs/lakehouse/multi-catalog/paimon.md
b/docs/en/docs/lakehouse/multi-catalog/paimon.md
index 68bf4cb011..b095bdee3e 100644
--- a/docs/en/docs/lakehouse/multi-catalog/paimon.md
+++ b/docs/en/docs/lakehouse/multi-catalog/paimon.md
@@ -60,10 +60,13 @@ CREATE CATALOG `paimon_hdfs` PROPERTIES (
#### S3
->Note that.
+> Note that.
>
> user need download
> [paimon-s3-0.4.0-incubating.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-s3/0.4.0-incubating/paimon-s3-0.4.0-incubating.jar)
+>
> Place it in directory
> ${DORIS_HOME}/be/lib/java_extensions/preload-extensions and restart be
+>
+> Starting from version 2.0.2, this file can be placed in BE's `custom_lib/`
directory (if it does not exist, just create it manually) to prevent the file
from being lost due to the replacement of the lib directory when upgrading the
cluster.
```sql
CREATE CATALOG `paimon_s3` PROPERTIES (
diff --git a/docs/zh-CN/docs/lakehouse/faq.md b/docs/zh-CN/docs/lakehouse/faq.md
index 79d68a255f..01dd36eb44 100644
--- a/docs/zh-CN/docs/lakehouse/faq.md
+++ b/docs/zh-CN/docs/lakehouse/faq.md
@@ -192,6 +192,8 @@ under the License.
去hadoop环境搜索`hadoop-lzo-*.jar`放在`"${DORIS_HOME}/fe/lib/"`目录下并重启fe。
+ 从 2.0.2 版本起，可以将这个文件放置在BE的 `custom_lib/` 目录下（如不存在，手动创建即可），以防止升级集群时因为 lib
目录被替换而导致文件丢失。
+
## HDFS
1. 访问 HDFS 3.x 时报错：`java.lang.VerifyError: xxx`
diff --git a/docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md
b/docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md
index 73e0ac530d..35672e9835 100644
--- a/docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md
+++ b/docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md
@@ -788,6 +788,8 @@ DROP CATALOG <catalog_name>;
下载
[orai18n.jar](https://www.oracle.com/database/technologies/appdev/jdbc-downloads.html)
并放到 Doris FE 的 lib 目录以及 BE 的 lib/java_extensions 目录 (Doris 2.0 之前的版本需放到 BE 的
lib 目录下) 下即可。
+ 从 2.0.2 版本起，可以将这个文件放置在BE的 `custom_lib/` 目录下（如不存在，手动创建即可），以防止升级集群时因为 lib
目录被替换而导致文件丢失。
+
13. 通过jdbc catalog 读取Clickhouse数据出现`NoClassDefFoundError:
net/jpountz/lz4/LZ4Factory` 错误信息
可以先下载[lz4-1.3.0.jar](https://repo1.maven.org/maven2/net/jpountz/lz4/lz4/1.3.0/lz4-1.3.0.jar)包，然后放到DorisFE
lib 目录以及BE 的 `lib/lib/java_extensions`目录中（Doris 2.0 之前的版本需放到 BE 的 lib 目录下）。
diff --git a/docs/zh-CN/docs/lakehouse/multi-catalog/paimon.md
b/docs/zh-CN/docs/lakehouse/multi-catalog/paimon.md
index 73cab27a03..bd22d59af9 100644
--- a/docs/zh-CN/docs/lakehouse/multi-catalog/paimon.md
+++ b/docs/zh-CN/docs/lakehouse/multi-catalog/paimon.md
@@ -60,10 +60,13 @@ CREATE CATALOG `paimon_hdfs` PROPERTIES (
#### S3
->注意：
+> 注意：
>
> 用户需要手动下载[paimon-s3-0.4.0-incubating.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-s3/0.4.0-incubating/paimon-s3-0.4.0-incubating.jar)
-> 放在${DORIS_HOME}/be/lib/java_extensions/preload-extensions目录下并重启be
+
+> 放在${DORIS_HOME}/be/lib/java_extensions/preload-extensions目录下并重启be。
+>
+> 从 2.0.2 版本起，可以将这个文件放置在BE的 `custom_lib/` 目录下（如不存在，手动创建即可），以防止升级集群时因为 lib
目录被替换而导致文件丢失。
```sql
CREATE CATALOG `paimon_s3` PROPERTIES (
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
