This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/dev/pr-13215-09a95daf02831f03ff0e4b6de0122315c4734624 in repository https://gitbox.apache.org/repos/asf/seatunnel.git
commit 7f6fe441c0e8d920799f663ff3011a32ba3320f1 Author: yidaimi <[email protected]> AuthorDate: Tue Nov 17 15:53:33 2026 +0000 [Feature][Connector-V2] Support per-job tls_ca_file in MongoDB sink (#13215) Co-authored-by: yidaimi <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> --- docs/en/connectors/sink/MongoDB.md | 31 +++ docs/zh/connectors/sink/MongoDB.md | 23 +++ .../seatunnel/mongodb/catalog/MongodbCatalog.java | 13 +- .../mongodb/config/MongodbSinkOptions.java | 12 ++ .../internal/MongodbClientSettingsFactory.java | 179 +++++++++++++++++ .../internal/MongodbCollectionProvider.java | 13 +- .../internal/MongodbSingleCollectionProvider.java | 15 +- .../seatunnel/mongodb/sink/MongodbSink.java | 5 +- .../mongodb/sink/MongodbSinkDryRunValidator.java | 24 ++- .../seatunnel/mongodb/sink/MongodbSinkFactory.java | 3 + .../seatunnel/mongodb/sink/MongodbWriter.java | 1 + .../mongodb/sink/MongodbWriterOptions.java | 42 +++- .../commit/MongodbSinkAggregatedCommitter.java | 1 + .../internal/MongodbClientSettingsFactoryTest.java | 185 +++++++++++++++++ .../internal/MongodbTlsCaFileHandshakeTest.java | 194 ++++++++++++++++++ .../mongodb/internal/TlsTestMaterial.java | 107 ++++++++++ .../mongodb/sink/MongodbSinkTlsCaFileTest.java | 221 +++++++++++++++++++++ .../src/test/resources/tls/generate-test-certs.sh | 66 ++++++ .../src/test/resources/tls/server.jks | Bin 0 -> 3049 bytes .../src/test/resources/tls/test-ca.jks | Bin 0 -> 890 bytes .../src/test/resources/tls/untrusted-ca.jks | Bin 0 -> 910 bytes .../src/test/resources/tls/wrong-host-server.jks | Bin 0 -> 3074 bytes 22 files changed, 1121 insertions(+), 14 deletions(-) diff --git a/docs/en/connectors/sink/MongoDB.md b/docs/en/connectors/sink/MongoDB.md index 340a6cfb63..335676621f 100644 --- a/docs/en/connectors/sink/MongoDB.md +++ b/docs/en/connectors/sink/MongoDB.md @@ -115,6 +115,7 @@ The following table lists the field data type mapping from SeaTunnel data type t | primary-key | List | No | - | The primary keys used for upsert/update. The list format is `["id","name",...]`. | | transaction | Boolean | No | false | Whether to use transactions in MongoSink (requires MongoDB 4.2+). | | data_save_mode | Enum | No | APPEND_DATA | The data saving mode for the MongoDB collection. Supported values: `DROP_DATA` (truncate the collection before writing), `APPEND_DATA` (append to existing data), `ERROR_WHEN_DATA_EXISTS` (fail if the collection already has data). | +| tls_ca_file | String | No | - | Path to a PEM file with the CA certificates trusted for TLS connections, for this job only. Setting it enables TLS. See [TLS With a Private CA](#tls-with-a-private-ca). | | multi_table_sink_replica | Int | No | 1 | The number of sink writer replicas per table in a multi-table job. Must be at least 1. | | common-options | | No | - | Sink plugin common parameters, please refer to [Sink Common Options](../common-options/sink-common-options.md) for details. | @@ -288,6 +289,36 @@ mongodb://192.168.0.1:27017,192.168.0.2:27017,192.168.0.3:27017/mydb > Note: The username and password in the URI must be URL-encoded before being > concatenated into > the connection string. +### TLS With a Private CA + +The MongoDB Java driver ignores `tlsCAFile` in the connection string, so servers whose certificates +are issued by a CA that is not in the JDK trust store (for example Amazon DocumentDB, which uses the +Amazon RDS CA, or a self-managed MongoDB with a corporate CA) cannot be reached by setting it in +`uri`. Use `tls_ca_file` instead: + +```hocon +sink { + MongoDB { + uri = "mongodb://user:passw...@docdb-cluster.cluster-xxxx.us-west-2.docdb.amazonaws.com:27017/?replicaSet=rs0&retryWrites=false" + database = "test_db" + collection = "test_collection" + tls_ca_file = "/opt/seatunnel/certs/global-bundle.pem" + } +} +``` + +- The file is read on every node that runs the sink, so it must exist at the same path on each of + them. It may contain several certificates (a bundle); all of them are trusted. +- Only the certificates of this file are trusted, and only for this sink. JVM-wide trust settings + such as `javax.net.ssl.trustStore` are neither used nor changed, so other jobs and connectors on + the same cluster are not affected. +- Setting `tls_ca_file` enables TLS. It cannot be combined with `tls=false` or `ssl=false` in `uri`. +- Hostname verification stays enabled. It can only be disabled explicitly with + `tlsAllowInvalidHostnames=true` in `uri`, as without `tls_ca_file`. +- The same settings are used for writing, transaction commits, the save mode (`data_save_mode`) + and the connectivity dry run. +- Amazon DocumentDB does not support retryable writes; keep `retryWrites=false` in `uri`. + ### Buffer Flush ```hocon diff --git a/docs/zh/connectors/sink/MongoDB.md b/docs/zh/connectors/sink/MongoDB.md index d55c04536d..a4b7ede804 100644 --- a/docs/zh/connectors/sink/MongoDB.md +++ b/docs/zh/connectors/sink/MongoDB.md @@ -108,6 +108,7 @@ MongoDB Sink 连接器将 SeaTunnel 行记录写入 MongoDB 集合。每行记 | primary-key | List | 否 | - | 用于 upsert 或更新的主键,格式为 `["id","name",...]`。 | | transaction | Boolean | 否 | false | 是否在 MongoSink 中启用事务(需要 MongoDB 4.2+)。 | | data_save_mode | Enum | 否 | APPEND_DATA | MongoDB 集合的数据写入模式:`DROP_DATA` 表示写入前清空集合;`APPEND_DATA` 表示追加写入;`ERROR_WHEN_DATA_EXISTS` 表示集合已有数据时直接报错。 | +| tls_ca_file | String | 否 | - | PEM 格式的 CA 证书文件路径,TLS 连接只信任该文件中的证书,且只对本作业生效。设置后会启用 TLS。详见 [使用私有 CA 的 TLS 连接](#使用私有-ca-的-tls-连接)。 | | multi_table_sink_replica | Int | 否 | 1 | 多表写入作业中每张表使用的写入器副本数,最小为 1。 | | common-options | - | 否 | - | 通用 Sink 插件参数,详见 [Sink Common Options](../common-options/sink-common-options.md)。 | @@ -276,6 +277,28 @@ mongodb://192.168.0.1:27017,192.168.0.2:27017,192.168.0.3:27017/mydb > 注意:URI 中的用户名与密码在拼接前必须进行 URL 编码。 +### 使用私有 CA 的 TLS 连接 + +MongoDB Java 驱动会忽略连接串中的 `tlsCAFile`。因此,如果服务端证书由 JDK 信任库之外的 CA 签发(例如使用 Amazon RDS CA 的 Amazon DocumentDB,或者使用企业内部 CA 的自建 MongoDB),在 `uri` 中设置 `tlsCAFile` 无法连接。请改用 `tls_ca_file`: + +```hocon +sink { + MongoDB { + uri = "mongodb://user:passw...@docdb-cluster.cluster-xxxx.us-west-2.docdb.amazonaws.com:27017/?replicaSet=rs0&retryWrites=false" + database = "test_db" + collection = "test_collection" + tls_ca_file = "/opt/seatunnel/certs/global-bundle.pem" + } +} +``` + +- 该文件在每个运行 Sink 的节点上读取,因此每个节点的相同路径下都必须有这个文件。文件中可以包含多个证书(证书包),其中的证书都会被信任。 +- 只信任该文件中的证书,并且只对本 Sink 生效。不会使用或修改 `javax.net.ssl.trustStore` 等 JVM 全局信任设置,同一集群上的其他作业和连接器不受影响。 +- 设置 `tls_ca_file` 会启用 TLS,不能与 `uri` 中的 `tls=false` 或 `ssl=false` 同时使用。 +- 主机名校验保持开启,只能像未设置 `tls_ca_file` 时一样,通过在 `uri` 中显式设置 `tlsAllowInvalidHostnames=true` 来关闭。 +- 数据写入、事务提交、保存模式(`data_save_mode`)和连通性 Dry Run 都使用同一套设置。 +- Amazon DocumentDB 不支持 retryable writes,请在 `uri` 中保留 `retryWrites=false`。 + ### Buffer Flush 示例 ```hocon diff --git a/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/catalog/MongodbCatalog.java b/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/catalog/MongodbCatalog.java index ceda30fdd1..4677a136f2 100644 --- a/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/catalog/MongodbCatalog.java +++ b/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/catalog/MongodbCatalog.java @@ -26,11 +26,11 @@ import org.apache.seatunnel.api.table.catalog.exception.DatabaseNotExistExceptio import org.apache.seatunnel.api.table.catalog.exception.TableAlreadyExistException; import org.apache.seatunnel.api.table.catalog.exception.TableNotExistException; import org.apache.seatunnel.common.exception.CommonError; +import org.apache.seatunnel.connectors.seatunnel.mongodb.internal.MongodbClientSettingsFactory; import org.bson.Document; import com.mongodb.client.MongoClient; -import com.mongodb.client.MongoClients; import com.mongodb.client.MongoCollection; import com.mongodb.client.MongoDatabase; @@ -44,17 +44,26 @@ public class MongodbCatalog implements Catalog { private transient MongoClient mongoClient; private final String defaultDatabase; + /** Optional PEM CA bundle for TLS connections; {@code null} when not configured. */ + private final String tlsCaFile; + public MongodbCatalog(String catalogName, String baseUrl, String defaultDatabase) { + this(catalogName, baseUrl, defaultDatabase, null); + } + + public MongodbCatalog( + String catalogName, String baseUrl, String defaultDatabase, String tlsCaFile) { this.catalogName = catalogName; this.baseUrl = baseUrl; this.defaultDatabase = defaultDatabase; + this.tlsCaFile = tlsCaFile; } @Override public void open() throws CatalogException { try { if (mongoClient == null) { - mongoClient = MongoClients.create(baseUrl); + mongoClient = MongodbClientSettingsFactory.createClient(baseUrl, tlsCaFile); } } catch (Exception e) { throw new CatalogException("Failed to open MongoDB Catalog: " + e.getMessage(), e); diff --git a/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/config/MongodbSinkOptions.java b/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/config/MongodbSinkOptions.java index 294efe1703..151f70c832 100644 --- a/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/config/MongodbSinkOptions.java +++ b/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/config/MongodbSinkOptions.java @@ -82,4 +82,16 @@ public class MongodbSinkOptions extends MongodbBaseOptions { Arrays.asList(DROP_DATA, APPEND_DATA, ERROR_WHEN_DATA_EXISTS)) .defaultValue(APPEND_DATA) .withDescription("The save mode of collection data"); + + public static final Option<String> TLS_CA_FILE = + Options.key("tls_ca_file") + .stringType() + .noDefaultValue() + .withDescription( + "Path to a PEM file with the CA certificates that the sink trusts for " + + "TLS connections, for example the Amazon RDS CA bundle for " + + "Amazon DocumentDB. Only these certificates are trusted, for " + + "this job only; JVM-wide trust settings are not used or " + + "changed. Setting it enables TLS. Hostname verification stays " + + "enabled unless the URI sets tlsAllowInvalidHostnames=true."); } diff --git a/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/internal/MongodbClientSettingsFactory.java b/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/internal/MongodbClientSettingsFactory.java new file mode 100644 index 0000000000..0411b24a77 --- /dev/null +++ b/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/internal/MongodbClientSettingsFactory.java @@ -0,0 +1,179 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.seatunnel.connectors.seatunnel.mongodb.internal; + +import org.apache.seatunnel.connectors.seatunnel.mongodb.exception.MongodbConnectorException; + +import com.mongodb.ConnectionString; +import com.mongodb.MongoClientSettings; +import com.mongodb.client.MongoClient; +import com.mongodb.client.MongoClients; + +import javax.net.ssl.SSLContext; +import javax.net.ssl.TrustManager; +import javax.net.ssl.TrustManagerFactory; +import javax.net.ssl.X509TrustManager; + +import java.io.IOException; +import java.io.InputStream; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.security.GeneralSecurityException; +import java.security.KeyStore; +import java.security.cert.Certificate; +import java.security.cert.CertificateFactory; +import java.util.Collection; + +import static org.apache.seatunnel.common.exception.CommonErrorCodeDeprecated.ILLEGAL_ARGUMENT; + +/** + * Builds the client settings shared by every MongoDB sink component that opens a connection (the + * writer, the transaction committer, the save-mode catalog and the dry-run validator), so that the + * optional {@code tls_ca_file} is applied consistently. + * + * <p>Without a CA file, clients are created exactly from the connection string, as before. With a + * CA file, TLS is enabled and a per-client {@link SSLContext} trusts only the certificates of that + * file; JVM-wide trust settings are neither used nor changed. All other settings, including {@code + * tlsAllowInvalidHostnames}, still come from the connection string, so hostname verification stays + * enabled by default. + * + * <p>Error messages never include the connection string, which can carry credentials. + */ +public final class MongodbClientSettingsFactory { + + private static final String TLS_CA_FILE_OPTION = "tls_ca_file"; + + private MongodbClientSettingsFactory() {} + + /** + * Creates a client for the given connection string, applying the CA file when one is set. + * + * @param connectionString MongoDB connection string + * @param tlsCaFile optional path to a PEM CA bundle; {@code null} keeps the previous behavior + */ + public static MongoClient createClient(String connectionString, String tlsCaFile) { + if (tlsCaFile == null) { + return MongoClients.create(connectionString); + } + return MongoClients.create(createSettings(connectionString, tlsCaFile)); + } + + /** + * Creates client settings from the connection string, applying the CA file when one is set. + * + * @param connectionString MongoDB connection string + * @param tlsCaFile optional path to a PEM CA bundle + * @return settings that can be further customized by the caller + */ + public static MongoClientSettings createSettings(String connectionString, String tlsCaFile) { + ConnectionString parsed = new ConnectionString(connectionString); + MongoClientSettings.Builder builder = + MongoClientSettings.builder().applyConnectionString(parsed); + if (tlsCaFile != null) { + if (Boolean.FALSE.equals(parsed.getSslEnabled())) { + throw new MongodbConnectorException( + ILLEGAL_ARGUMENT, + "MongoDB sink option '" + + TLS_CA_FILE_OPTION + + "' requires TLS, but the connection string disables it " + + "(tls=false or ssl=false)."); + } + SSLContext sslContext = createSslContext(tlsCaFile); + // Only enables TLS and replaces the trust material; the hostname verification + // setting applied from the connection string above is kept. + builder.applyToSslSettings(ssl -> ssl.enabled(true).context(sslContext)); + } + return builder.build(); + } + + /** Builds an isolated trust context from every X.509 certificate in the CA bundle. */ + static SSLContext createSslContext(String tlsCaFile) { + try { + SSLContext sslContext = SSLContext.getInstance("TLS"); + sslContext.init(null, new TrustManager[] {createTrustManager(tlsCaFile)}, null); + return sslContext; + } catch (GeneralSecurityException e) { + throw new MongodbConnectorException( + ILLEGAL_ARGUMENT, + "Failed to create the TLS context for MongoDB sink option '" + + TLS_CA_FILE_OPTION + + "': " + + tlsCaFile, + e); + } + } + + static X509TrustManager createTrustManager(String tlsCaFile) { + Path path = caFilePath(tlsCaFile); + try (InputStream inputStream = Files.newInputStream(path)) { + Collection<? extends Certificate> certificates = + CertificateFactory.getInstance("X.509").generateCertificates(inputStream); + if (certificates.isEmpty()) { + throw new MongodbConnectorException( + ILLEGAL_ARGUMENT, + "MongoDB sink option '" + + TLS_CA_FILE_OPTION + + "' contains no certificate: " + + tlsCaFile); + } + KeyStore trustStore = KeyStore.getInstance(KeyStore.getDefaultType()); + trustStore.load(null, null); + int index = 0; + for (Certificate certificate : certificates) { + trustStore.setCertificateEntry("mongodb-ca-" + index++, certificate); + } + TrustManagerFactory trustManagerFactory = + TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm()); + trustManagerFactory.init(trustStore); + for (TrustManager trustManager : trustManagerFactory.getTrustManagers()) { + if (trustManager instanceof X509TrustManager) { + return (X509TrustManager) trustManager; + } + } + throw new MongodbConnectorException( + ILLEGAL_ARGUMENT, "No X.509 trust manager available for " + tlsCaFile); + } catch (IOException | GeneralSecurityException e) { + throw new MongodbConnectorException( + ILLEGAL_ARGUMENT, + "Failed to load the PEM CA bundle of MongoDB sink option '" + + TLS_CA_FILE_OPTION + + "': " + + tlsCaFile, + e); + } + } + + private static Path caFilePath(String tlsCaFile) { + if (tlsCaFile.trim().isEmpty()) { + throw new MongodbConnectorException( + ILLEGAL_ARGUMENT, + "MongoDB sink option '" + TLS_CA_FILE_OPTION + "' must not be blank."); + } + Path path = Paths.get(tlsCaFile.trim()); + if (!Files.isRegularFile(path) || !Files.isReadable(path)) { + throw new MongodbConnectorException( + ILLEGAL_ARGUMENT, + "MongoDB sink option '" + + TLS_CA_FILE_OPTION + + "' is not a readable file: " + + tlsCaFile); + } + return path; + } +} diff --git a/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/internal/MongodbCollectionProvider.java b/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/internal/MongodbCollectionProvider.java index c68da4ea27..94465b3727 100644 --- a/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/internal/MongodbCollectionProvider.java +++ b/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/internal/MongodbCollectionProvider.java @@ -34,6 +34,8 @@ public class MongodbCollectionProvider { private String collection; + private String tlsCaFile; + public Builder connectionString(String connectionString) { this.connectionString = connectionString; return this; @@ -49,11 +51,20 @@ public class MongodbCollectionProvider { return this; } + /** + * Optional PEM CA bundle applied to the client, see {@link MongodbClientSettingsFactory}. + */ + public Builder tlsCaFile(String tlsCaFile) { + this.tlsCaFile = tlsCaFile; + return this; + } + public MongodbClientProvider build() { Preconditions.checkNotNull(connectionString, "Connection string must not be null"); Preconditions.checkNotNull(database, "Database must not be null"); Preconditions.checkNotNull(collection, "Collection must not be null"); - return new MongodbSingleCollectionProvider(connectionString, database, collection); + return new MongodbSingleCollectionProvider( + connectionString, database, collection, tlsCaFile); } } } diff --git a/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/internal/MongodbSingleCollectionProvider.java b/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/internal/MongodbSingleCollectionProvider.java index 0891a5b78f..6e9a81bc46 100644 --- a/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/internal/MongodbSingleCollectionProvider.java +++ b/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/internal/MongodbSingleCollectionProvider.java @@ -22,7 +22,6 @@ import org.apache.seatunnel.shade.com.google.common.base.Preconditions; import org.bson.BsonDocument; import com.mongodb.client.MongoClient; -import com.mongodb.client.MongoClients; import com.mongodb.client.MongoCollection; import com.mongodb.client.MongoDatabase; import lombok.extern.slf4j.Slf4j; @@ -36,6 +35,9 @@ public class MongodbSingleCollectionProvider implements MongodbClientProvider { private final String defaultCollection; + /** Optional PEM CA bundle; {@code null} creates the client from the connection string only. */ + private final String tlsCaFile; + private MongoClient client; private MongoDatabase database; @@ -44,19 +46,28 @@ public class MongodbSingleCollectionProvider implements MongodbClientProvider { public MongodbSingleCollectionProvider( String connectionString, String defaultDatabase, String defaultCollection) { + this(connectionString, defaultDatabase, defaultCollection, null); + } + + public MongodbSingleCollectionProvider( + String connectionString, + String defaultDatabase, + String defaultCollection, + String tlsCaFile) { Preconditions.checkNotNull(connectionString); Preconditions.checkNotNull(defaultDatabase); Preconditions.checkNotNull(defaultCollection); this.connectionString = connectionString; this.defaultDatabase = defaultDatabase; this.defaultCollection = defaultCollection; + this.tlsCaFile = tlsCaFile; } @Override public MongoClient getClient() { synchronized (this) { if (client == null) { - client = MongoClients.create(connectionString); + client = MongodbClientSettingsFactory.createClient(connectionString, tlsCaFile); } } return client; diff --git a/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/sink/MongodbSink.java b/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/sink/MongodbSink.java index 5d6a7e444b..a641937a59 100644 --- a/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/sink/MongodbSink.java +++ b/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/sink/MongodbSink.java @@ -112,7 +112,10 @@ public class MongodbSink Optional<Catalog> catalogOptional = Optional.of( new MongodbCatalog( - MongodbSinkOptions.CONNECTOR_IDENTITY, url, database)); + MongodbSinkOptions.CONNECTOR_IDENTITY, + url, + database, + options.getTlsCaFile())); try { DataSaveMode dataSaveMode = options.getDataSaveMode(); Catalog catalog = catalogOptional.get(); diff --git a/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/sink/MongodbSinkDryRunValidator.java b/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/sink/MongodbSinkDryRunValidator.java index a77e56f2c5..c6bfdd7b33 100644 --- a/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/sink/MongodbSinkDryRunValidator.java +++ b/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/sink/MongodbSinkDryRunValidator.java @@ -19,10 +19,11 @@ package org.apache.seatunnel.connectors.seatunnel.mongodb.sink; import org.apache.seatunnel.api.configuration.ReadonlyConfig; import org.apache.seatunnel.connectors.seatunnel.mongodb.config.MongodbSinkOptions; +import org.apache.seatunnel.connectors.seatunnel.mongodb.exception.MongodbConnectorException; +import org.apache.seatunnel.connectors.seatunnel.mongodb.internal.MongodbClientSettingsFactory; import org.bson.Document; -import com.mongodb.ConnectionString; import com.mongodb.MongoClientSettings; import com.mongodb.MongoInterruptedException; import com.mongodb.MongoSecurityException; @@ -48,7 +49,11 @@ final class MongodbSinkDryRunValidator { throw new IllegalStateException("MongoDB sink dry-run was interrupted."); } try (MongoClient client = - MongoClients.create(settings(options.get(MongodbSinkOptions.URI)))) { + MongoClients.create( + settings( + options.get(MongodbSinkOptions.URI), + options.getOptional(MongodbSinkOptions.TLS_CA_FILE) + .orElse(null)))) { // Writes select the primary regardless of the URI's read preference. client.getDatabase(options.get(MongodbSinkOptions.DATABASE)) .runCommand(new Document("ping", 1), ReadPreference.primary()); @@ -59,6 +64,10 @@ final class MongodbSinkDryRunValidator { throw new IllegalStateException("MongoDB sink dry-run authentication failed."); } catch (MongoTimeoutException e) { throw new IllegalStateException("MongoDB sink dry-run connection timed out."); + } catch (MongodbConnectorException e) { + // Raised for an invalid tls_ca_file; the message names the file, never the URI. + throw new IllegalStateException( + "MongoDB sink dry-run TLS configuration is invalid: " + e.getMessage()); } catch (RuntimeException e) { // Driver messages, causes and suppressed close failures can contain URI credentials. throw new IllegalStateException( @@ -66,12 +75,13 @@ final class MongodbSinkDryRunValidator { } } - /** Preserves URI settings, but bounds individual waits; this is not a total deadline. */ - private static MongoClientSettings settings(String uri) { + /** + * Uses the same client settings as the sink writer, including the optional CA file, but bounds + * individual waits; this is not a total deadline. + */ + private static MongoClientSettings settings(String uri, String tlsCaFile) { MongoClientSettings configured = - MongoClientSettings.builder() - .applyConnectionString(new ConnectionString(uri)) - .build(); + MongodbClientSettingsFactory.createSettings(uri, tlsCaFile); long selectionTimeout = bounded( configured diff --git a/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/sink/MongodbSinkFactory.java b/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/sink/MongodbSinkFactory.java index 71752bb292..467d8c08e8 100644 --- a/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/sink/MongodbSinkFactory.java +++ b/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/sink/MongodbSinkFactory.java @@ -60,6 +60,7 @@ public class MongodbSinkFactory implements TableSinkFactory, SupportSinkDryRunVa MongodbSinkOptions.UPSERT_ENABLE, MongodbSinkOptions.PRIMARY_KEY, MongodbSinkOptions.DATA_SAVE_MODE, + MongodbSinkOptions.TLS_CA_FILE, SinkConnectorCommonOptions.MULTI_TABLE_SINK_REPLICA) .build(); } @@ -100,6 +101,8 @@ public class MongodbSinkFactory implements TableSinkFactory, SupportSinkDryRunVa builder.withTransaction(readonlyConfig.get(MongodbSinkOptions.TRANSACTION)); } builder.withDataSaveMode(readonlyConfig.get(MongodbSinkOptions.DATA_SAVE_MODE)); + builder.withTlsCaFile( + readonlyConfig.getOptional(MongodbSinkOptions.TLS_CA_FILE).orElse(null)); CatalogTable catalogTable = context.getCatalogTable(); // sourceCatalogTable to sinkCatalogTable TableIdentifier tableIdentifier = diff --git a/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/sink/MongodbWriter.java b/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/sink/MongodbWriter.java index d0d356f10c..89e8669054 100644 --- a/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/sink/MongodbWriter.java +++ b/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/sink/MongodbWriter.java @@ -90,6 +90,7 @@ public class MongodbWriter .connectionString(options.getConnectString()) .database(options.getDatabase()) .collection(options.getCollection()) + .tlsCaFile(options.getTlsCaFile()) .build(); this.bulkActions = options.getFlushSize(); this.batchIntervalMs = options.getBatchIntervalMs(); diff --git a/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/sink/MongodbWriterOptions.java b/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/sink/MongodbWriterOptions.java index d1e4ee4aeb..313cc0d473 100644 --- a/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/sink/MongodbWriterOptions.java +++ b/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/sink/MongodbWriterOptions.java @@ -50,6 +50,9 @@ public class MongodbWriterOptions implements Serializable { protected final DataSaveMode dataSaveMode; + /** Optional PEM CA bundle for TLS connections; {@code null} when not configured. */ + protected final String tlsCaFile; + public MongodbWriterOptions( String connectString, String database, @@ -62,6 +65,34 @@ public class MongodbWriterOptions implements Serializable { long retryInterval, boolean transaction, DataSaveMode dataSaveMode) { + this( + connectString, + database, + collection, + flushSize, + batchIntervalMs, + upsertEnable, + primaryKey, + retryMax, + retryInterval, + transaction, + dataSaveMode, + null); + } + + public MongodbWriterOptions( + String connectString, + String database, + String collection, + int flushSize, + long batchIntervalMs, + boolean upsertEnable, + String[] primaryKey, + int retryMax, + long retryInterval, + boolean transaction, + DataSaveMode dataSaveMode, + String tlsCaFile) { this.connectString = connectString; this.database = database; this.collection = collection; @@ -73,6 +104,7 @@ public class MongodbWriterOptions implements Serializable { this.retryInterval = retryInterval; this.transaction = transaction; this.dataSaveMode = dataSaveMode; + this.tlsCaFile = tlsCaFile; } public static Builder builder() { @@ -103,6 +135,8 @@ public class MongodbWriterOptions implements Serializable { protected DataSaveMode dataSaveMode; + protected String tlsCaFile; + public Builder withConnectString(String connectString) { this.connectString = connectString; return this; @@ -158,6 +192,11 @@ public class MongodbWriterOptions implements Serializable { return this; } + public Builder withTlsCaFile(String tlsCaFile) { + this.tlsCaFile = tlsCaFile; + return this; + } + public MongodbWriterOptions build() { return new MongodbWriterOptions( connectString, @@ -170,7 +209,8 @@ public class MongodbWriterOptions implements Serializable { retryMax, retryInterval, transaction, - dataSaveMode); + dataSaveMode, + tlsCaFile); } } } diff --git a/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/sink/commit/MongodbSinkAggregatedCommitter.java b/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/sink/commit/MongodbSinkAggregatedCommitter.java index fb0195b61b..8ca36f4674 100644 --- a/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/sink/commit/MongodbSinkAggregatedCommitter.java +++ b/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/sink/commit/MongodbSinkAggregatedCommitter.java @@ -68,6 +68,7 @@ public class MongodbSinkAggregatedCommitter .connectionString(options.getConnectString()) .database(options.getDatabase()) .collection(options.getCollection()) + .tlsCaFile(options.getTlsCaFile()) .build(); } diff --git a/seatunnel-connectors-v2/connector-mongodb/src/test/java/org/apache/seatunnel/connectors/seatunnel/mongodb/internal/MongodbClientSettingsFactoryTest.java b/seatunnel-connectors-v2/connector-mongodb/src/test/java/org/apache/seatunnel/connectors/seatunnel/mongodb/internal/MongodbClientSettingsFactoryTest.java new file mode 100644 index 0000000000..508711792a --- /dev/null +++ b/seatunnel-connectors-v2/connector-mongodb/src/test/java/org/apache/seatunnel/connectors/seatunnel/mongodb/internal/MongodbClientSettingsFactoryTest.java @@ -0,0 +1,185 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.seatunnel.connectors.seatunnel.mongodb.internal; + +import org.apache.seatunnel.connectors.seatunnel.mongodb.exception.MongodbConnectorException; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; +import org.mockito.MockedStatic; + +import com.mongodb.MongoClientSettings; +import com.mongodb.client.MongoClients; +import com.mongodb.connection.SslSettings; + +import javax.net.ssl.X509TrustManager; + +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Arrays; +import java.util.Set; +import java.util.stream.Collectors; + +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.Mockito.mockStatic; +import static org.mockito.Mockito.never; + +class MongodbClientSettingsFactoryTest { + + private static final String SECRET_URI = "mongodb://user:secret@localhost:27017"; + + @TempDir Path tempDir; + + @Test + void withoutCaFileKeepsConnectionStringTlsSettings() { + SslSettings plain = ssl(MongodbClientSettingsFactory.createSettings(SECRET_URI, null)); + Assertions.assertFalse(plain.isEnabled()); + Assertions.assertNull(plain.getContext()); + + SslSettings tls = + ssl(MongodbClientSettingsFactory.createSettings(SECRET_URI + "/?tls=true", null)); + Assertions.assertTrue(tls.isEnabled()); + Assertions.assertNull(tls.getContext(), "JVM default trust must stay in use"); + } + + @Test + void withoutCaFileCreatesClientFromConnectionStringAsBefore() { + try (MockedStatic<MongoClients> clients = mockStatic(MongoClients.class)) { + MongodbClientSettingsFactory.createClient(SECRET_URI, null); + clients.verify(() -> MongoClients.create(SECRET_URI)); + clients.verify(() -> MongoClients.create(any(MongoClientSettings.class)), never()); + } + } + + @Test + void caFileEnablesTlsWithIsolatedTrustContext() throws Exception { + Path ca = TlsTestMaterial.writePem(tempDir, "ca.pem", TlsTestMaterial.testCa()); + + SslSettings ssl = + ssl(MongodbClientSettingsFactory.createSettings(SECRET_URI, ca.toString())); + + Assertions.assertTrue(ssl.isEnabled(), "setting tls_ca_file enables TLS"); + Assertions.assertNotNull(ssl.getContext()); + Assertions.assertFalse(ssl.isInvalidHostNameAllowed(), "hostname verification stays on"); + } + + @Test + void caFileKeepsHostnameSettingFromConnectionString() throws Exception { + Path ca = TlsTestMaterial.writePem(tempDir, "ca.pem", TlsTestMaterial.testCa()); + + SslSettings ssl = + ssl( + MongodbClientSettingsFactory.createSettings( + SECRET_URI + "/?tls=true&tlsAllowInvalidHostnames=true", + ca.toString())); + + Assertions.assertNotNull(ssl.getContext()); + Assertions.assertTrue(ssl.isInvalidHostNameAllowed()); + } + + @Test + void caFileCreatesClientFromSettings() throws Exception { + Path ca = TlsTestMaterial.writePem(tempDir, "ca.pem", TlsTestMaterial.testCa()); + try (MockedStatic<MongoClients> clients = mockStatic(MongoClients.class)) { + MongodbClientSettingsFactory.createClient(SECRET_URI, ca.toString()); + clients.verify(() -> MongoClients.create(any(MongoClientSettings.class))); + clients.verify(() -> MongoClients.create(anyString()), never()); + } + } + + @Test + void trustsEveryCertificateOfTheBundle() throws Exception { + Path bundle = + TlsTestMaterial.writePem( + tempDir, + "bundle.pem", + TlsTestMaterial.testCa(), + TlsTestMaterial.untrustedCa()); + + X509TrustManager trustManager = + MongodbClientSettingsFactory.createTrustManager(bundle.toString()); + + Set<String> issuers = + Arrays.stream(trustManager.getAcceptedIssuers()) + .map(certificate -> certificate.getSubjectX500Principal().getName()) + .collect(Collectors.toSet()); + Assertions.assertEquals(2, issuers.size()); + Assertions.assertTrue(issuers.contains("CN=SeaTunnel MongoDB Test CA")); + Assertions.assertTrue(issuers.contains("CN=SeaTunnel MongoDB Untrusted Test CA")); + } + + @Test + void rejectsCaFileWhenConnectionStringDisablesTls() throws Exception { + Path ca = TlsTestMaterial.writePem(tempDir, "ca.pem", TlsTestMaterial.testCa()); + + assertConfigError( + () -> + MongodbClientSettingsFactory.createSettings( + SECRET_URI + "/?tls=false", ca.toString()), + "requires TLS"); + } + + @Test + void rejectsMissingBlankAndInvalidCaFiles() throws Exception { + Path missing = tempDir.resolve("missing.pem"); + Path noCertificate = tempDir.resolve("empty.pem"); + Files.write(noCertificate, new byte[0]); + Path garbage = tempDir.resolve("garbage.pem"); + Files.write( + garbage, + "-----BEGIN CERTIFICATE-----\nnot a certificate\n-----END CERTIFICATE-----\n" + .getBytes(StandardCharsets.US_ASCII)); + + assertConfigError( + () -> MongodbClientSettingsFactory.createSettings(SECRET_URI, missing.toString()), + "is not a readable file: " + missing); + assertConfigError( + () -> MongodbClientSettingsFactory.createSettings(SECRET_URI, tempDir.toString()), + "is not a readable file"); + assertConfigError( + () -> MongodbClientSettingsFactory.createSettings(SECRET_URI, " "), + "must not be blank"); + assertConfigError( + () -> + MongodbClientSettingsFactory.createSettings( + SECRET_URI, noCertificate.toString()), + noCertificate.toString()); + assertConfigError( + () -> MongodbClientSettingsFactory.createSettings(SECRET_URI, garbage.toString()), + garbage.toString()); + } + + private static SslSettings ssl(MongoClientSettings settings) { + return settings.getSslSettings(); + } + + private static void assertConfigError(Runnable action, String expectedMessagePart) { + MongodbConnectorException e = + Assertions.assertThrows(MongodbConnectorException.class, action::run); + Assertions.assertTrue( + e.getMessage().contains("tls_ca_file"), "names the option: " + e.getMessage()); + Assertions.assertTrue( + e.getMessage().contains(expectedMessagePart), + "expected '" + expectedMessagePart + "' in: " + e.getMessage()); + Assertions.assertFalse( + e.getMessage().contains("secret"), "must not leak the URI: " + e.getMessage()); + } +} diff --git a/seatunnel-connectors-v2/connector-mongodb/src/test/java/org/apache/seatunnel/connectors/seatunnel/mongodb/internal/MongodbTlsCaFileHandshakeTest.java b/seatunnel-connectors-v2/connector-mongodb/src/test/java/org/apache/seatunnel/connectors/seatunnel/mongodb/internal/MongodbTlsCaFileHandshakeTest.java new file mode 100644 index 0000000000..667adb81bc --- /dev/null +++ b/seatunnel-connectors-v2/connector-mongodb/src/test/java/org/apache/seatunnel/connectors/seatunnel/mongodb/internal/MongodbTlsCaFileHandshakeTest.java @@ -0,0 +1,194 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.seatunnel.connectors.seatunnel.mongodb.internal; + +import org.bson.Document; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import com.mongodb.MongoTimeoutException; +import com.mongodb.client.MongoClient; + +import javax.net.ssl.SSLContext; +import javax.net.ssl.SSLServerSocket; +import javax.net.ssl.SSLSocket; + +import java.io.IOException; +import java.net.InetAddress; +import java.net.Socket; +import java.net.SocketException; +import java.nio.file.Path; +import java.util.concurrent.atomic.AtomicInteger; + +/** + * Proves the {@code tls_ca_file} trust behavior against real TLS endpoints. Two in-process TLS + * servers present certificates signed by the test CA: one for {@code DNS:localhost}, and one for + * {@code DNS:unrelated-host.invalid} only, which is used for hostname mismatches without relying on + * DNS or reverse lookups. The servers do not speak the MongoDB protocol, so every client call ends + * with a server selection timeout; the outcome of the TLS handshake is taken from the server side + * and from the client error. + */ +class MongodbTlsCaFileHandshakeTest { + + private static final String CLIENT_OPTIONS = + "/?directConnection=true&serverSelectionTimeoutMS=2000&connectTimeoutMS=2000"; + + @TempDir static Path tempDir; + + private static TlsServer server; + private static TlsServer wrongHostServer; + private static String trustedCa; + private static String untrustedCa; + + @BeforeAll + static void startServer() throws Exception { + server = new TlsServer(TlsTestMaterial.localhostServerContext()); + wrongHostServer = new TlsServer(TlsTestMaterial.wrongHostServerContext()); + trustedCa = + TlsTestMaterial.writePem(tempDir, "ca.pem", TlsTestMaterial.testCa()).toString(); + untrustedCa = + TlsTestMaterial.writePem(tempDir, "other.pem", TlsTestMaterial.untrustedCa()) + .toString(); + } + + @AfterAll + static void stopServer() throws IOException { + server.close(); + wrongHostServer.close(); + } + + @BeforeEach + void resetCounters() { + server.reset(); + wrongHostServer.reset(); + } + + @Test + void configuredCaIsTrusted() { + ping(server, "", trustedCa); + + Assertions.assertTrue( + server.completedHandshakes() > 0, "TLS handshake with the configured CA"); + } + + @Test + void untrustedCaIsRejected() { + String error = ping(server, "", untrustedCa); + + Assertions.assertEquals(0, server.completedHandshakes()); + Assertions.assertTrue(error.contains("PKIX path building failed"), error); + } + + @Test + void withoutCaFileTheJvmDefaultTrustIsUsedAsBefore() { + // The test CA is not in the JDK trust store, so the previous behavior rejects it. + String error = ping(server, "&tls=true", null); + + Assertions.assertEquals(0, server.completedHandshakes()); + Assertions.assertTrue(error.contains("PKIX path building failed"), error); + } + + @Test + void hostnameMismatchIsRejectedEvenWithTrustedCa() { + // Trusted CA, but the certificate is for unrelated-host.invalid, not localhost. + String error = ping(wrongHostServer, "", trustedCa); + + Assertions.assertEquals(0, wrongHostServer.completedHandshakes()); + Assertions.assertTrue(error.contains("No subject alternative DNS name matching"), error); + } + + @Test + void hostnameVerificationIsStillControlledByTheConnectionString() { + ping(wrongHostServer, "&tlsAllowInvalidHostnames=true", trustedCa); + + Assertions.assertTrue(wrongHostServer.completedHandshakes() > 0); + } + + /** Runs a ping and returns the client error (the server never answers MongoDB commands). */ + private static String ping(TlsServer target, String extraOptions, String caFile) { + String uri = "mongodb://localhost:" + target.port() + CLIENT_OPTIONS + extraOptions; + try (MongoClient client = MongodbClientSettingsFactory.createClient(uri, caFile)) { + MongoTimeoutException e = + Assertions.assertThrows( + MongoTimeoutException.class, + () -> client.getDatabase("admin").runCommand(new Document("ping", 1))); + return e.getMessage(); + } + } + + /** Accepts TLS connections and records whether each handshake completed. */ + private static final class TlsServer implements AutoCloseable { + private final SSLServerSocket socket; + private final AtomicInteger completedHandshakes = new AtomicInteger(); + private final Thread acceptor; + + TlsServer(SSLContext context) throws Exception { + socket = + (SSLServerSocket) + context.getServerSocketFactory() + .createServerSocket(0, 50, InetAddress.getLoopbackAddress()); + acceptor = new Thread(this::acceptLoop, "mongodb-tls-test-server"); + acceptor.setDaemon(true); + acceptor.start(); + } + + int port() { + return socket.getLocalPort(); + } + + int completedHandshakes() { + return completedHandshakes.get(); + } + + void reset() { + completedHandshakes.set(0); + } + + private void acceptLoop() { + while (!socket.isClosed()) { + try (Socket accepted = socket.accept()) { + accepted.setSoTimeout(5000); + try { + ((SSLSocket) accepted).startHandshake(); + // With TLS 1.3 the server side can finish before the client has + // validated the certificate. Only a client that accepted the + // certificate sends application data (its MongoDB hello). + if (accepted.getInputStream().read() >= 0) { + completedHandshakes.incrementAndGet(); + } + } catch (IOException handshakeFailure) { + // rejected by the client: untrusted CA or hostname mismatch + } + } catch (SocketException closed) { + return; + } catch (IOException ignored) { + // keep accepting + } + } + } + + @Override + public void close() throws IOException { + socket.close(); + } + } +} diff --git a/seatunnel-connectors-v2/connector-mongodb/src/test/java/org/apache/seatunnel/connectors/seatunnel/mongodb/internal/TlsTestMaterial.java b/seatunnel-connectors-v2/connector-mongodb/src/test/java/org/apache/seatunnel/connectors/seatunnel/mongodb/internal/TlsTestMaterial.java new file mode 100644 index 0000000000..d65686c0a3 --- /dev/null +++ b/seatunnel-connectors-v2/connector-mongodb/src/test/java/org/apache/seatunnel/connectors/seatunnel/mongodb/internal/TlsTestMaterial.java @@ -0,0 +1,107 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.seatunnel.connectors.seatunnel.mongodb.internal; + +import javax.net.ssl.KeyManagerFactory; +import javax.net.ssl.SSLContext; + +import java.io.IOException; +import java.io.InputStream; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.security.KeyStore; +import java.security.cert.X509Certificate; +import java.util.Base64; + +/** + * Test-only TLS material from {@code src/test/resources/tls} (see {@code generate-test-certs.sh}): + * a test CA, an unrelated CA, and two server key stores signed by the test CA, one for {@code + * DNS:localhost} and one for {@code DNS:unrelated-host.invalid} only. + */ +public final class TlsTestMaterial { + + private static final char[] PASSWORD = "changeit".toCharArray(); + + private TlsTestMaterial() {} + + /** The CA that signed the test server certificate. */ + public static X509Certificate testCa() { + return caCertificate("tls/test-ca.jks"); + } + + /** A CA that did not sign the test server certificate. */ + public static X509Certificate untrustedCa() { + return caCertificate("tls/untrusted-ca.jks"); + } + + /** Writes the certificates as a PEM bundle, the format expected by {@code tls_ca_file}. */ + public static Path writePem(Path directory, String fileName, X509Certificate... certificates) + throws Exception { + StringBuilder pem = new StringBuilder(); + Base64.Encoder encoder = + Base64.getMimeEncoder(64, "\n".getBytes(StandardCharsets.US_ASCII)); + for (X509Certificate certificate : certificates) { + pem.append("-----BEGIN CERTIFICATE-----\n") + .append(encoder.encodeToString(certificate.getEncoded())) + .append("\n-----END CERTIFICATE-----\n"); + } + Path file = directory.resolve(fileName); + Files.write(file, pem.toString().getBytes(StandardCharsets.US_ASCII)); + return file; + } + + /** Server-side TLS context presenting the certificate for {@code DNS:localhost}. */ + public static SSLContext localhostServerContext() throws Exception { + return serverContext("tls/server.jks"); + } + + /** Server-side TLS context whose certificate does not cover {@code localhost}. */ + public static SSLContext wrongHostServerContext() throws Exception { + return serverContext("tls/wrong-host-server.jks"); + } + + private static SSLContext serverContext(String resource) throws Exception { + KeyManagerFactory keyManagerFactory = + KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm()); + keyManagerFactory.init(load(resource), PASSWORD); + SSLContext context = SSLContext.getInstance("TLS"); + context.init(keyManagerFactory.getKeyManagers(), null, null); + return context; + } + + private static X509Certificate caCertificate(String resource) { + try { + return (X509Certificate) load(resource).getCertificate("ca"); + } catch (Exception e) { + throw new IllegalStateException("Cannot load test certificate " + resource, e); + } + } + + private static KeyStore load(String resource) throws Exception { + try (InputStream in = + TlsTestMaterial.class.getClassLoader().getResourceAsStream(resource)) { + if (in == null) { + throw new IOException("Missing test resource " + resource); + } + KeyStore keyStore = KeyStore.getInstance("JKS"); + keyStore.load(in, PASSWORD); + return keyStore; + } + } +} diff --git a/seatunnel-connectors-v2/connector-mongodb/src/test/java/org/apache/seatunnel/connectors/seatunnel/mongodb/sink/MongodbSinkTlsCaFileTest.java b/seatunnel-connectors-v2/connector-mongodb/src/test/java/org/apache/seatunnel/connectors/seatunnel/mongodb/sink/MongodbSinkTlsCaFileTest.java new file mode 100644 index 0000000000..b15252c823 --- /dev/null +++ b/seatunnel-connectors-v2/connector-mongodb/src/test/java/org/apache/seatunnel/connectors/seatunnel/mongodb/sink/MongodbSinkTlsCaFileTest.java @@ -0,0 +1,221 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.seatunnel.connectors.seatunnel.mongodb.sink; + +import org.apache.seatunnel.api.configuration.ReadonlyConfig; +import org.apache.seatunnel.api.sink.SinkWriter; +import org.apache.seatunnel.api.table.catalog.Catalog; +import org.apache.seatunnel.api.table.catalog.CatalogTableUtil; +import org.apache.seatunnel.api.table.factory.TableSinkFactoryContext; +import org.apache.seatunnel.api.table.type.BasicType; +import org.apache.seatunnel.api.table.type.SeaTunnelDataType; +import org.apache.seatunnel.api.table.type.SeaTunnelRowType; +import org.apache.seatunnel.connectors.seatunnel.mongodb.config.MongodbSinkOptions; +import org.apache.seatunnel.connectors.seatunnel.mongodb.internal.MongodbSingleCollectionProvider; +import org.apache.seatunnel.connectors.seatunnel.mongodb.internal.TlsTestMaterial; + +import org.bson.Document; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; +import org.mockito.ArgumentCaptor; +import org.mockito.MockedConstruction; +import org.mockito.MockedStatic; +import org.mockito.Mockito; + +import com.mongodb.MongoClientSettings; +import com.mongodb.ReadPreference; +import com.mongodb.client.MongoClient; +import com.mongodb.client.MongoClients; +import com.mongodb.client.MongoDatabase; +import com.mongodb.connection.SslSettings; + +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.mockStatic; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.when; + +/** + * Verifies that every MongoDB sink component that opens a connection (writer, transaction + * committer, save-mode catalog and dry-run validator) receives {@code tls_ca_file}, and that the + * previous behavior is kept when the option is not set. + */ +class MongodbSinkTlsCaFileTest { + + private static final String URI = "mongodb://user:secret@localhost:27017"; + + private static final SeaTunnelRowType ROW_TYPE = + new SeaTunnelRowType( + new String[] {"id"}, new SeaTunnelDataType<?>[] {BasicType.LONG_TYPE}); + + @TempDir Path tempDir; + + @Test + void optionIsRegisteredAsOptional() { + Assertions.assertTrue( + new MongodbSinkFactory() + .optionRule() + .getOptionalOptions() + .contains(MongodbSinkOptions.TLS_CA_FILE)); + } + + @Test + void writerAndTransactionCommitterUseCaFile() throws Exception { + String ca = caFile(); + Map<String, Object> config = config(ca); + config.put(MongodbSinkOptions.TRANSACTION.key(), true); + + Assertions.assertEquals(Arrays.asList(ca, ca), providerCaFiles(createSink(config))); + } + + @Test + void withoutCaFileProvidersKeepConnectionStringOnly() { + Map<String, Object> config = config(null); + config.put(MongodbSinkOptions.TRANSACTION.key(), true); + + Assertions.assertEquals(Arrays.asList(null, null), providerCaFiles(createSink(config))); + } + + @Test + void saveModeCatalogUsesCaFile() throws Exception { + MongodbSink sink = createSink(config(caFile())); + Catalog catalog = sink.getSaveModeHandler().get().getHandleCatalog(); + + try (MockedStatic<MongoClients> clients = mockStatic(MongoClients.class)) { + clients.when(() -> MongoClients.create(any(MongoClientSettings.class))) + .thenReturn(mock(MongoClient.class)); + catalog.open(); + catalog.close(); + + ArgumentCaptor<MongoClientSettings> settings = + ArgumentCaptor.forClass(MongoClientSettings.class); + clients.verify(() -> MongoClients.create(settings.capture())); + clients.verify(() -> MongoClients.create(anyString()), never()); + assertTrustsConfiguredCa(settings.getValue().getSslSettings()); + } + } + + @Test + void saveModeCatalogWithoutCaFileUsesConnectionStringAsBefore() { + MongodbSink sink = createSink(config(null)); + Catalog catalog = sink.getSaveModeHandler().get().getHandleCatalog(); + + try (MockedStatic<MongoClients> clients = mockStatic(MongoClients.class)) { + clients.when(() -> MongoClients.create(anyString())) + .thenReturn(mock(MongoClient.class)); + catalog.open(); + catalog.close(); + + clients.verify(() -> MongoClients.create(URI)); + clients.verify(() -> MongoClients.create(any(MongoClientSettings.class)), never()); + } + } + + @Test + void dryRunUsesCaFile() throws Exception { + MongoClient client = mock(MongoClient.class); + MongoDatabase database = mock(MongoDatabase.class); + when(client.getDatabase("test_database")).thenReturn(database); + when(database.runCommand(new Document("ping", 1), ReadPreference.primary())) + .thenReturn(new Document("ok", 1)); + + try (MockedStatic<MongoClients> clients = mockStatic(MongoClients.class)) { + ArgumentCaptor<MongoClientSettings> settings = + ArgumentCaptor.forClass(MongoClientSettings.class); + clients.when(() -> MongoClients.create(settings.capture())).thenReturn(client); + + new MongodbSinkFactory().validateConnectionForDryRun(context(config(caFile()))); + + assertTrustsConfiguredCa(settings.getValue().getSslSettings()); + } + } + + @Test + void dryRunReportsInvalidCaFileWithoutLeakingUri() { + String missing = tempDir.resolve("missing.pem").toString(); + + try (MockedStatic<MongoClients> clients = mockStatic(MongoClients.class)) { + IllegalStateException failure = + Assertions.assertThrows( + IllegalStateException.class, + () -> + new MongodbSinkFactory() + .validateConnectionForDryRun(context(config(missing)))); + + Assertions.assertTrue(failure.getMessage().contains("tls_ca_file")); + Assertions.assertTrue(failure.getMessage().contains(missing)); + Assertions.assertFalse(failure.toString().contains("secret")); + Assertions.assertNull(failure.getCause()); + clients.verify(() -> MongoClients.create(any(MongoClientSettings.class)), never()); + } + } + + private String caFile() throws Exception { + return TlsTestMaterial.writePem(tempDir, "ca.pem", TlsTestMaterial.testCa()).toString(); + } + + private static Map<String, Object> config(String caFile) { + Map<String, Object> config = new HashMap<>(); + config.put(MongodbSinkOptions.URI.key(), URI); + config.put(MongodbSinkOptions.DATABASE.key(), "test_database"); + config.put(MongodbSinkOptions.COLLECTION.key(), "test_collection"); + if (caFile != null) { + config.put(MongodbSinkOptions.TLS_CA_FILE.key(), caFile); + } + return config; + } + + private static TableSinkFactoryContext context(Map<String, Object> config) { + return new TableSinkFactoryContext( + CatalogTableUtil.getCatalogTable("test_table", ROW_TYPE), + ReadonlyConfig.fromMap(config), + MongodbSinkTlsCaFileTest.class.getClassLoader()); + } + + private static MongodbSink createSink(Map<String, Object> config) { + return (MongodbSink) new MongodbSinkFactory().createSink(context(config)).createSink(); + } + + /** Creates the writer and the aggregated committer and returns the CA file each received. */ + private static List<String> providerCaFiles(MongodbSink sink) { + List<String> caFiles = new ArrayList<>(); + try (MockedConstruction<MongodbSingleCollectionProvider> ignored = + Mockito.mockConstruction( + MongodbSingleCollectionProvider.class, + (provider, context) -> caFiles.add((String) context.arguments().get(3)))) { + sink.createWriter(mock(SinkWriter.Context.class)); + sink.createAggregatedCommitter().get(); + } + return caFiles; + } + + private static void assertTrustsConfiguredCa(SslSettings ssl) { + Assertions.assertTrue(ssl.isEnabled()); + Assertions.assertNotNull(ssl.getContext(), "per-client trust context from tls_ca_file"); + Assertions.assertFalse(ssl.isInvalidHostNameAllowed()); + } +} diff --git a/seatunnel-connectors-v2/connector-mongodb/src/test/resources/tls/generate-test-certs.sh b/seatunnel-connectors-v2/connector-mongodb/src/test/resources/tls/generate-test-certs.sh new file mode 100755 index 0000000000..a5e71d16d4 --- /dev/null +++ b/seatunnel-connectors-v2/connector-mongodb/src/test/resources/tls/generate-test-certs.sh @@ -0,0 +1,66 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# Regenerates the test-only TLS material used by the MongoDB sink tls_ca_file tests. +# +# test-ca.jks truststore with the test CA certificate +# untrusted-ca.jks truststore with an unrelated CA certificate +# server.jks key store with a server key and a certificate for DNS:localhost, +# signed by the test CA +# wrong-host-server.jks +# key store with a server key and a certificate for +# DNS:unrelated-host.invalid only, also signed by the test CA +# +# The stores use the JKS format because the repository file checks skip *.jks as binary. +# All stores use the password "changeit". CA private keys are not kept. Requires openssl and +# keytool. Run from this directory. +set -euo pipefail +DAYS=36500 +WORK=$(mktemp -d) +trap 'rm -rf "$WORK"' EXIT + +ca() { # $1 = file prefix, $2 = common name + openssl req -x509 -newkey rsa:2048 -nodes -sha256 -days "$DAYS" -subj "/CN=$2" \ + -addext "basicConstraints=critical,CA:TRUE" -addext "keyUsage=critical,keyCertSign,cRLSign" \ + -keyout "$WORK/$1.key" -out "$WORK/$1.crt" 2>/dev/null +} + +ca test-ca "SeaTunnel MongoDB Test CA" +ca untrusted-ca "SeaTunnel MongoDB Untrusted Test CA" + +server() { # $1 = output key store, $2 = DNS name + openssl req -newkey rsa:2048 -nodes -sha256 -subj "/CN=$2" \ + -keyout "$WORK/$1.key" -out "$WORK/$1.csr" 2>/dev/null + printf 'basicConstraints=CA:FALSE\nkeyUsage=critical,digitalSignature,keyEncipherment\nextendedKeyUsage=serverAuth\nsubjectAltName=DNS:%s\n' "$2" > "$WORK/$1.ext" + openssl x509 -req -sha256 -days "$DAYS" -in "$WORK/$1.csr" -CA "$WORK/test-ca.crt" \ + -CAkey "$WORK/test-ca.key" -CAcreateserial -extfile "$WORK/$1.ext" -out "$WORK/$1.crt" 2>/dev/null + openssl pkcs12 -export -name server -inkey "$WORK/$1.key" -in "$WORK/$1.crt" \ + -certfile "$WORK/test-ca.crt" -passout pass:changeit -out "$WORK/$1.p12" + rm -f "$1.jks" + keytool -importkeystore -noprompt -srckeystore "$WORK/$1.p12" -srcstoretype PKCS12 \ + -srcstorepass changeit -destkeystore "$1.jks" -deststoretype JKS -deststorepass changeit \ + -destkeypass changeit >/dev/null 2>&1 +} + +rm -f test-ca.jks untrusted-ca.jks +for name in test-ca untrusted-ca; do + keytool -importcert -noprompt -storetype JKS -keystore "$name.jks" -storepass changeit \ + -alias ca -file "$WORK/$name.crt" >/dev/null +done +server server localhost +server wrong-host-server unrelated-host.invalid diff --git a/seatunnel-connectors-v2/connector-mongodb/src/test/resources/tls/server.jks b/seatunnel-connectors-v2/connector-mongodb/src/test/resources/tls/server.jks new file mode 100644 index 0000000000..49a59fa0b6 Binary files /dev/null and b/seatunnel-connectors-v2/connector-mongodb/src/test/resources/tls/server.jks differ diff --git a/seatunnel-connectors-v2/connector-mongodb/src/test/resources/tls/test-ca.jks b/seatunnel-connectors-v2/connector-mongodb/src/test/resources/tls/test-ca.jks new file mode 100644 index 0000000000..ee3400f5d7 Binary files /dev/null and b/seatunnel-connectors-v2/connector-mongodb/src/test/resources/tls/test-ca.jks differ diff --git a/seatunnel-connectors-v2/connector-mongodb/src/test/resources/tls/untrusted-ca.jks b/seatunnel-connectors-v2/connector-mongodb/src/test/resources/tls/untrusted-ca.jks new file mode 100644 index 0000000000..9984662a2b Binary files /dev/null and b/seatunnel-connectors-v2/connector-mongodb/src/test/resources/tls/untrusted-ca.jks differ diff --git a/seatunnel-connectors-v2/connector-mongodb/src/test/resources/tls/wrong-host-server.jks b/seatunnel-connectors-v2/connector-mongodb/src/test/resources/tls/wrong-host-server.jks new file mode 100644 index 0000000000..d5202286f3 Binary files /dev/null and b/seatunnel-connectors-v2/connector-mongodb/src/test/resources/tls/wrong-host-server.jks differ
