This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit c27692fb3bd148b1a98a49d41ed1f6e7d99207c1
Author: zy-kkk <zhongy...@gmail.com>
AuthorDate: Wed Feb 21 12:07:31 2024 +0800
[Enhancement](jdbc catalog) Add security check on driver when creating Jdbc
Catalog (#31153)
---
docs/en/docs/lakehouse/multi-catalog/jdbc.md | 24 ++++++++++++++++++++--
docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md | 24 ++++++++++++++++++++--
.../main/java/org/apache/doris/common/Config.java | 8 ++++++++
.../org/apache/doris/catalog/JdbcResource.java | 19 +++++++++++++++--
.../doris/datasource/jdbc/JdbcExternalCatalog.java | 24 ++++++++++++++--------
.../org/apache/doris/catalog/JdbcResourceTest.java | 15 ++++++++++++++
6 files changed, 99 insertions(+), 15 deletions(-)
diff --git a/docs/en/docs/lakehouse/multi-catalog/jdbc.md
b/docs/en/docs/lakehouse/multi-catalog/jdbc.md
index a2fbde91768..b0019373f72 100644
--- a/docs/en/docs/lakehouse/multi-catalog/jdbc.md
+++ b/docs/en/docs/lakehouse/multi-catalog/jdbc.md
@@ -61,12 +61,32 @@ PROPERTIES ("key"="value", ...)
`driver_url` can be specified in three ways:
-1. File name. For example, `mysql-connector-java-5.1.47.jar`. Please place
the Jar file package in `jdbc_drivers/` under the FE/BE deployment directory
in advance so the system can locate the file. You can change the location of
the file by modifying `jdbc_drivers_dir` in fe.conf and be.conf.
+1. File name. For example, `mysql-connector-java-8.0.25.jar`. Please place
the Jar file package in `jdbc_drivers/` under the FE/BE deployment directory
in advance so the system can locate the file. You can change the location of
the file by modifying `jdbc_drivers_dir` in fe.conf and be.conf.
-2. Local absolute path. For example,
`file:///path/to/mysql-connector-java-5.1.47.jar`. Please place the Jar file
package in the specified paths of FE/BE node.
+2. Local absolute path. For example,
`file:///path/to/mysql-connector-java-8.0.25.jar`. Please place the Jar file
package in the specified paths of FE/BE node.
3. HTTP address. For example,
`https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/jdbc_driver/mysql-connector-java-8.0.25.jar`.
The system will download the Driver file from the HTTP address. This only
supports HTTP services with no authentication requirements.
+**Driver package security**
+
+In order to prevent the use of a Driver Jar package with an unallowed path
when creating the Catalog, Doris will perform path management and checksum
checking on the Jar package.
+
+1. For the above method 1, the `jdbc_drivers_dir` configured by the Doris
default user and all Jar packages in its directory are safe and will not be
path checked.
+
+2. For the above methods 2 and 3, Doris will check the source of the Jar
package. The checking rules are as follows:
+
+ * Control the allowed driver package paths through the FE configuration
item `jdbc_driver_secure_path`. This configuration item can configure multiple
paths, separated by semicolons. When this item is configured, Doris will check
whether the prefix of the driver_url path in the Catalog properties is in
`jdbc_driver_secure_path`. If not, it will refuse to create the Catalog.
+ * This parameter defaults to `*`, which means Jar packages of all paths are
allowed.
+ * If the configuration `jdbc_driver_secure_path` is empty, driver packages
for all paths are not allowed, which means that the driver package can only be
specified using method 1 above.
+
+ > If you configure `jdbc_driver_secure_path =
"file:///path/to/jdbc_drivers;http://path/to/jdbc_drivers"`, only
`file:///path/to/jdbc_drivers` or `http:// is allowed The driver package path
starting with path/to/jdbc_drivers`.
+
+3. When creating a Catalog, you can specify the checksum of the driver package
through the `checksum` parameter. Doris will verify the driver package after
loading the driver package. If the verification fails, it will refuse to create
the Catalog.
+
+:::warning
+The above verification will only be performed when the Catalog is created. For
already created Catalogs, verification will not be performed again.
+:::
+
### Lowercase name synchronization
When `lower_case_meta_names` is set to `true`, Doris maintains the mapping of
lowercase names to actual names in the remote system, enabling queries to use
lowercase to query non-lowercase databases, tables and columns of external data
sources.
diff --git a/docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md
b/docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md
index c426098772b..00ef1afcf65 100644
--- a/docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md
+++ b/docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md
@@ -61,12 +61,32 @@ PROPERTIES ("key"="value", ...)
`driver_url` 可以通过以下三种方式指定：
-1. 文件名。如 `mysql-connector-java-5.1.47.jar`。需将 Jar 包预先存放在 FE 和 BE 部署目录的
`jdbc_drivers/` 目录下。系统会自动在这个目录下寻找。该目录的位置，也可以由 fe.conf 和 be.conf 中的
`jdbc_drivers_dir` 配置修改。
+1. 文件名。如 `mysql-connector-java-8.0.25.jar`。需将 Jar 包预先存放在 FE 和 BE 部署目录的
`jdbc_drivers/` 目录下。系统会自动在这个目录下寻找。该目录的位置，也可以由 fe.conf 和 be.conf 中的
`jdbc_drivers_dir` 配置修改。
-2. 本地绝对路径。如 `file:///path/to/mysql-connector-java-5.1.47.jar`。需将 Jar 包预先存放在所有
FE/BE 节点指定的路径下。
+2. 本地绝对路径。如 `file:///path/to/mysql-connector-java-8.0.25.jar`。需将 Jar 包预先存放在所有
FE/BE 节点指定的路径下。
3. Http
地址。如：`https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/jdbc_driver/mysql-connector-java-8.0.25.jar`。系统会从这个
http 地址下载 Driver 文件。仅支持无认证的 http 服务。
+**驱动包安全性**
+
+为了防止在创建 Catalog 时使用了未允许路径的 Driver Jar 包，Doris 会对 Jar 包进行路径管理和校验和检查。
+
+1. 针对上述方式 1，Doris 默认用户配置的 `jdbc_drivers_dir` 和其目录下的所有 Jar 包都是安全的，不会对其进行路径检查。
+
+2. 针对上述方式 2、3 ，Doris 会对 Jar 包的来源进行检查，检查规则如下：
+
+ * 通过 FE 配置项 `jdbc_driver_secure_path`
来控制允许的驱动包路径，该配置项可配置多个路径，以分号分隔。当配置了该项时，Doris 会检查 Catalog properties 中 driver_url
的路径是的部分前缀是否在 `jdbc_driver_secure_path` 中，如果不在其中，则会拒绝创建 Catalog。
+ * 此参数默认为 `*` ，表示允许所有路径的 Jar 包。
+ * 如果配置 `jdbc_driver_secure_path` 为空，则不允许所有路径的驱动包，也就意味着只能使用上述方式 1 来指定驱动包。
+
+ > 如配置 `jdbc_driver_secure_path =
"file:///path/to/jdbc_drivers;http://path/to/jdbc_drivers"`, 则只允许以
`file:///path/to/jdbc_drivers` 或 `http://path/to/jdbc_drivers` 开头的驱动包路径。
+
+3. 在创建 Catalog 时，可以通过 `checksum` 参数来指定驱动包的校验和，Doris
会在加载驱动包后，对驱动包进行校验，如果校验失败，则会拒绝创建 Catalog。
+
+:::warning
+上述的校验只会在创建 Catalog 时进行，对于已经创建的 Catalog，不会再次进行校验。
+:::
+
### 小写名称同步
当 `lower_case_meta_names` 设置为 `true` 时，Doris
通过维护小写名称到远程系统中实际名称的映射，使查询时能够使用小写去查询外部数据源非小写的数据库和表以及列。
diff --git a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
index 2bb6173acd8..86ed91d3c08 100644
--- a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
+++ b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
@@ -142,6 +142,14 @@ public class Config extends ConfigBase {
+ "if the specified driver file path is not an absolute
path, Doris will find jars from this path"})
public static String jdbc_drivers_dir = System.getenv("DORIS_HOME") +
"/jdbc_drivers";
+ @ConfField(description = {"JDBC 驱动的安全路径。在创建 JDBC Catalog
时，允许使用的文件或者网络路径，可配置多个，使用分号分隔"
+ + "默认为 * 全部允许，如果设置为空责全部不允许",
+ "The safe path of the JDBC driver. When creating a JDBC Catalog,"
+ + "you can configure multiple files or network paths that
are allowed to be used,"
+ + "separated by semicolons"
+ + "The default is * to allow all, if set to empty, all are
not allowed"})
+ public static String jdbc_driver_secure_path = "*";
+
@ConfField(mutable = true, masterOnly = true, description = {"broker load
时，单个节点上 load 执行计划的默认并行度",
"The default parallelism of the load execution plan on a single
node when the broker load is submitted"})
public static int default_load_parallelism = 8;
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java
index 03bdd3d2f30..6b19137dff0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java
@@ -42,6 +42,7 @@ import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.time.LocalDateTime;
import java.time.ZoneId;
+import java.util.Arrays;
import java.util.Map;
@@ -277,14 +278,28 @@ public class JdbcResource extends Resource {
}
}
- public static String getFullDriverUrl(String driverUrl) {
+ public static String getFullDriverUrl(String driverUrl) throws
IllegalArgumentException {
try {
URI uri = new URI(driverUrl);
String schema = uri.getScheme();
if (schema == null && !driverUrl.startsWith("/")) {
return "file://" + Config.jdbc_drivers_dir + "/" + driverUrl;
+ } else {
+ if ("*".equals(Config.jdbc_driver_secure_path)) {
+ return driverUrl;
+ } else if (Config.jdbc_driver_secure_path.trim().isEmpty()) {
+ throw new IllegalArgumentException(
+ "jdbc_driver_secure_path is set to empty,
disallowing all driver URLs.");
+ } else {
+ boolean isAllowed =
Arrays.stream(Config.jdbc_driver_secure_path.split(";"))
+ .anyMatch(allowedPath ->
driverUrl.startsWith(allowedPath.trim()));
+ if (!isAllowed) {
+ throw new IllegalArgumentException("Driver URL does
not match any allowed paths: " + driverUrl);
+ } else {
+ return driverUrl;
+ }
+ }
}
- return driverUrl;
} catch (URISyntaxException e) {
LOG.warn("invalid jdbc driver url: " + driverUrl);
return driverUrl;
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/JdbcExternalCatalog.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/JdbcExternalCatalog.java
index ee2ca9e0e00..93ebb1b2134 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/JdbcExternalCatalog.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/JdbcExternalCatalog.java
@@ -119,11 +119,6 @@ public class JdbcExternalCatalog extends ExternalCatalog {
jdbcUrl = JdbcResource.handleJdbcUrl(jdbcUrl);
properties.put(JdbcResource.JDBC_URL, jdbcUrl);
}
-
- if (properties.containsKey(JdbcResource.DRIVER_URL) &&
!properties.containsKey(JdbcResource.CHECK_SUM)) {
- properties.put(JdbcResource.CHECK_SUM,
-
JdbcResource.computeObjectChecksum(properties.get(JdbcResource.DRIVER_URL)));
- }
return properties;
}
@@ -246,10 +241,21 @@ public class JdbcExternalCatalog extends ExternalCatalog {
if (isReplay) {
return;
}
- Map<String, String> properties = Maps.newHashMap();
- if (properties.containsKey(JdbcResource.DRIVER_URL) &&
!properties.containsKey(JdbcResource.CHECK_SUM)) {
- properties.put(JdbcResource.CHECK_SUM,
-
JdbcResource.computeObjectChecksum(properties.get(JdbcResource.DRIVER_URL)));
+ Map<String, String> properties = catalogProperty.getProperties();
+ if (properties.containsKey(JdbcResource.DRIVER_URL)) {
+ String computedChecksum =
JdbcResource.computeObjectChecksum(properties.get(JdbcResource.DRIVER_URL));
+ if (properties.containsKey(JdbcResource.CHECK_SUM)) {
+ String providedChecksum =
properties.get(JdbcResource.CHECK_SUM);
+ if (!providedChecksum.equals(computedChecksum)) {
+ throw new DdlException(
+ "The provided checksum (" + providedChecksum
+ + ") does not match the computed checksum
(" + computedChecksum
+ + ") for the driver_url."
+ );
+ }
+ } else {
+ catalogProperty.addProperty(JdbcResource.CHECK_SUM,
computedChecksum);
+ }
}
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/catalog/JdbcResourceTest.java
b/fe/fe-core/src/test/java/org/apache/doris/catalog/JdbcResourceTest.java
index 87dcecc79dd..8e004d4b236 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/catalog/JdbcResourceTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/catalog/JdbcResourceTest.java
@@ -20,6 +20,7 @@ package org.apache.doris.catalog;
import org.apache.doris.analysis.AccessTestUtil;
import org.apache.doris.analysis.Analyzer;
import org.apache.doris.analysis.CreateResourceStmt;
+import org.apache.doris.common.Config;
import org.apache.doris.common.DdlException;
import org.apache.doris.common.FeConstants;
import org.apache.doris.common.UserException;
@@ -201,4 +202,18 @@ public class JdbcResourceTest {
// Ensure the result URL still contains ';'
Assert.assertTrue(resultUrl.contains(";"));
}
+
+ @Test
+ public void testJdbcDriverPtah() {
+ String driverPath = "postgresql-42.5.0.jar";
+ Config.jdbc_driver_secure_path = "";
+ String fullPath = JdbcResource.getFullDriverUrl(driverPath);
+ Assert.assertEquals(fullPath, "file://" + Config.jdbc_drivers_dir +
"/" + driverPath);
+ Config.jdbc_driver_secure_path = "file:///jdbc/;http://jdbc";;
+ String driverPath2 = "file:///postgresql-42.5.0.jar";
+ Exception exception =
Assert.assertThrows(IllegalArgumentException.class, () -> {
+ JdbcResource.getFullDriverUrl(driverPath2);
+ });
+ Assert.assertEquals("Driver URL does not match any allowed paths:
file:///postgresql-42.5.0.jar", exception.getMessage());
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
