This is an automated email from the ASF dual-hosted git repository.
davidzollo pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new 80d9f15bbc [Docs] Improve JDBC source guide (#11491)
80d9f15bbc is described below
commit 80d9f15bbcb607111504ff5964da3688ab53d783
Author: Daniel <[email protected]>
AuthorDate: Sun Jul 19 16:59:45 2026 +0800
[Docs] Improve JDBC source guide (#11491)
Co-authored-by: DanielLeens <[email protected]>
---
docs/en/connectors/source/Jdbc.md | 463 +++++++++++++++++++-----------------
docs/zh/connectors/source/Jdbc.md | 479 +++++++++++++++++++++++++-------------
2 files changed, 564 insertions(+), 378 deletions(-)
diff --git a/docs/en/connectors/source/Jdbc.md
b/docs/en/connectors/source/Jdbc.md
index 8931e85d73..b296186f6a 100644
--- a/docs/en/connectors/source/Jdbc.md
+++ b/docs/en/connectors/source/Jdbc.md
@@ -2,29 +2,123 @@ import ChangeLog from '../changelog/connector-jdbc.md';
# JDBC
-> JDBC source connector
-
## Description
-Read external data source data through JDBC.
+The JDBC Source connector reads tables or custom query results from databases
through a JDBC driver. It supports column projection, row filtering, parallel
snapshot reads, and reading multiple tables in one source configuration.
-:::tip
+JDBC Source is a bounded source: it reads the rows visible to the database
query and then finishes. Use a CDC connector instead when the job must continue
capturing later inserts, updates, and deletes.
-Warn: for license compliance, you have to provide database driver yourself,
copy to `$SEATUNNEL_HOME/lib/` directory in order to make them work.
+If this is your first JDBC Source job, start with [Choose a read
mode](#choose-a-read-mode) and the [Quick start](#quick-start-postgresql). The
complete option reference follows those sections.
-e.g. If you use MySQL, should download and copy `mysql-connector-java-xxx.jar`
to `$SEATUNNEL_HOME/lib/`. For Spark/Flink, you should also copy it to
`$SPARK_HOME/jars/` or `$FLINK_HOME/lib/`.
+## Using Dependency
-:::
+Install the `connector-jdbc` plugin first:
-## Using Dependency
+```plugin_config
+--seatunnel-connectors--
+connector-jdbc
+--end--
+```
+
+```bash
+cd "${SEATUNNEL_HOME}"
+sh bin/install-plugin.sh
+```
+
+JDBC driver licenses and redistribution terms vary by database vendor, and the
driver version must be compatible with both the database and the Java runtime.
SeaTunnel therefore does not bundle every JDBC driver. Download the appropriate
driver yourself and place its JAR in the engine-specific directory below before
starting the job.
+
+### Spark and Flink engines
+
+Place the JDBC driver in `${SEATUNNEL_HOME}/plugins/Jdbc/lib/` on every node
that runs SeaTunnel.
+
+### Zeta engine
+
+Place the JDBC driver in `${SEATUNNEL_HOME}/lib/` on every SeaTunnel node,
then restart the affected SeaTunnel processes so the driver is loaded.
+
+See the [driver reference](#driver-reference) for common driver class names
and download locations.
+
+## Choose a read mode
+
+Choose a single-table or multi-table layout before configuring parallelism. In
the single-table layout, `table_path` and `query` can be used separately or
together. The multi-table `table_list` layout is mutually exclusive with
top-level `table_path` and `query`.
+
+| Use case | Configuration | Behavior |
+|----------|---------------|----------|
+| Read one table with automatic schema discovery and dynamic splitting |
`table_path` | Recommended for a full-table snapshot. SeaTunnel reads table
metadata and uses `split.size` when the table has a usable split key. |
+| Control selected columns, joins, or database-side expressions | `query`,
optionally with `table_path` | SeaTunnel executes the SQL you provide. Add
`table_path` when explicit table identity and metadata are also needed. Query
key inference is unsafe for some joins; see [Query and primary-key
caution](#query-and-primary-key-caution). |
+| Read multiple tables or table-name patterns | `table_list` | Each entry can
define `table_path`, an optional `query`, and split settings. Top-level
`table_path` and `query` cannot be used together with `table_list`. |
+
+Use `where_condition` only for a common filter that should be added to every
selected table or query. Its value must start with `where`, for example `where
updated_at >= '2026-01-01'`.
+
+## Quick start: PostgreSQL
+
+This example reads three rows from PostgreSQL and prints them to the SeaTunnel
log.
+
+1. Put a compatible PostgreSQL JDBC driver in the directory described in
[Using Dependency](#using-dependency).
+
+2. As a PostgreSQL administrator, connect to an existing `sales` database,
create a dedicated tutorial table, and grant a read-only account access. If
`seatunnel_reader` already exists, omit the `CREATE ROLE` statement and reuse
the account:
+
+```sql
+CREATE ROLE seatunnel_reader WITH LOGIN PASSWORD 'change_me';
+
+DROP TABLE IF EXISTS public.seatunnel_jdbc_source_quick_start;
+
+CREATE TABLE public.seatunnel_jdbc_source_quick_start (
+ id BIGINT PRIMARY KEY,
+ customer_name VARCHAR(100) NOT NULL,
+ amount DECIMAL(10, 2) NOT NULL
+);
+
+INSERT INTO public.seatunnel_jdbc_source_quick_start VALUES
+ (1, 'Alice', 120.50),
+ (2, 'Bob', 80.00),
+ (3, 'Carol', 42.00);
+
+GRANT CONNECT ON DATABASE sales TO seatunnel_reader;
+GRANT USAGE ON SCHEMA public TO seatunnel_reader;
+GRANT SELECT ON TABLE public.seatunnel_jdbc_source_quick_start TO
seatunnel_reader;
+```
+
+The `DROP TABLE` makes the tutorial data repeatable. Do not use that statement
with a business table.
+
+3. Save the following job as
`${SEATUNNEL_HOME}/config/jdbc-source-quick-start.conf`. Replace the host,
credentials, and database name for your environment.
+
+```hocon
+env {
+ parallelism = 1
+ job.mode = "BATCH"
+}
+
+source {
+ Jdbc {
+ url = "jdbc:postgresql://postgresql.example.com:5432/sales"
+ driver = "org.postgresql.Driver"
+ username = "seatunnel_reader"
+ password = "change_me"
+ query = "SELECT id, customer_name, amount FROM
public.seatunnel_jdbc_source_quick_start ORDER BY id"
+ }
+}
+
+sink {
+ Console {}
+}
+```
+
+4. Run the job:
-### For Spark/Flink Engine
+```bash
+cd "${SEATUNNEL_HOME}"
+./bin/seatunnel.sh --config ./config/jdbc-source-quick-start.conf -m local
+```
-> 1. You need to ensure that the [jdbc driver jar
package](https://mvnrepository.com/artifact/mysql/mysql-connector-java) has
been placed in directory `${SEATUNNEL_HOME}/plugins/`.
+5. Confirm that the Console sink prints rows with the following values:
-### For SeaTunnel Zeta Engine
+| id | customer_name | amount |
+|----|---------------|-------:|
+| 1 | Alice | 120.50 |
+| 2 | Bob | 80.00 |
+| 3 | Carol | 42.00 |
-> 1. You need to ensure that the [jdbc driver jar
package](https://mvnrepository.com/artifact/mysql/mysql-connector-java) has
been placed in directory `${SEATUNNEL_HOME}/lib/`.
+If the job fails before reading rows, check
[Troubleshooting](#troubleshooting) first.
## Key features
@@ -33,7 +127,7 @@ e.g. If you use MySQL, should download and copy
`mysql-connector-java-xxx.jar` t
- [x] [exactly-once](../../introduction/concepts/connector-v2-features.md)
- [x] [column projection](../../introduction/concepts/connector-v2-features.md)
-supports query SQL and can achieve projection effect.
+Use `query` to select only the required columns.
- [x] [parallelism](../../introduction/concepts/connector-v2-features.md)
- [x] [support user-defined
split](../../introduction/concepts/connector-v2-features.md)
@@ -41,22 +135,24 @@ supports query SQL and can achieve projection effect.
## Options
+`url` and `driver` are always required. Authentication is optional because
some databases allow unauthenticated connections. Use either the top-level
single-table layout or `table_list`; top-level `table_path` and `query` may be
combined. `username` is the preferred account key, while legacy configurations
using `user` remain supported as a fallback.
+
| name | type | required | default
value | description
[...]
|--------------------------------------------|---------|-----------|-----------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
| url | String | Yes | -
| The URL of the JDBC connection. Refer to a case:
jdbc:postgresql://localhost/test
[...]
| driver | String | Yes | -
| The jdbc class name used to connect to the remote data source, if you
use MySQL the value is `com.mysql.cj.jdbc.Driver`.
[...]
-| username | String | No | -
| userName
[...]
-| password | String | No | -
| password
[...]
-| query | String | No | -
| Query statement
[...]
+| username | String | No | -
| Database account name. The legacy key `user` is still accepted as a
fallback.
[...]
+| password | String | No | -
| Password for the database account.
[...]
+| query | String | No | -
| SQL query to execute. It can be combined with `table_path` when
explicit table identity and metadata are also needed.
[...]
| compatible_mode | String | No | -
| The compatible mode of database, required when the database supports
multiple compatible modes.<br/> For example, when using OceanBase database, you
need to set it to 'mysql' or 'oracle'. <br/> when using starrocks, you need set
it to `starrocks`
[...]
| dialect | String | No | -
| The appointed dialect, if it does not exist, is still obtained
according to the url, and the priority is higher than the url. <br/> For
example,when using starrocks, you need set it to `starrocks`
[...]
| connection_check_timeout_sec | Int | No | 30
| The time in seconds to wait for the database operation used to validate
the connection to complete.
[...]
| connect_timeout_ms | Int | No | 86400000
| Connection timeout in milliseconds when establishing the JDBC
connection. `0` means no timeout.
[...]
| socket_timeout_ms | Int | No | 86400000
| Socket read timeout in milliseconds after the JDBC connection is
established. `0` means no timeout.
[...]
| partition_column | String | No | -
| The column name for split data.
[...]
-| partition_upper_bound | Long | No | -
| The partition_column max value for scan, if not set SeaTunnel will
query database get max value.
[...]
-| partition_lower_bound | Long | No | -
| The partition_column min value for scan, if not set SeaTunnel will
query database get min value.
[...]
-| partition_num | Int | No | job
parallelism | Not recommended for use, The correct approach is to control the
number of split through `split.size`<br/> **Note:** This parameter takes effect
only when using the `query` parameter. It does not take effect when using the
`table_path` parameter.
[...]
+| partition_upper_bound | String | No | -
| Inclusive upper value used for query-based partitioning. When omitted,
SeaTunnel queries the source for the maximum value.
[...]
+| partition_lower_bound | String | No | -
| Inclusive lower value used for query-based partitioning. When omitted,
SeaTunnel queries the source for the minimum value.
[...]
+| partition_num | Int | No | 10
| Number of splits when top-level `query` and `partition_column` select
the fixed splitter, whether or not `table_path` is also present. Dynamic
`table_path` and `table_list` layouts use `split.size` instead.
[...]
| decimal_type_narrowing | Boolean | No | true
| Decimal type narrowing, if true, the decimal type will be narrowed to
the int or long type if without loss of precision. Only support for Oracle at
now. Please refer to `decimal_type_narrowing` below
[...]
| int_type_narrowing | Boolean | No | true
| Int type narrowing, if true, the tinyint(1) type will be narrowed to
the boolean type if without loss of precision. Support for MySQL at now. Please
refer to `int_type_narrowing` below
[...]
| handle_blob_as_string | Boolean | No | false
| If true, BLOB type will be converted to STRING type. **Only supported
for Oracle database**. This is useful for handling large BLOB fields in Oracle
that exceed the default size limit. When transmitting Oracle's BLOB fields to
systems like Doris, setting this to true can make the data transfer more
efficient.
[...]
@@ -69,10 +165,10 @@ supports query SQL and can achieve projection effect.
| use_regex | Boolean | No | false
| Control regular expression matching for table_path. When set to `true`,
the table_path will be treated as a regular expression pattern. When set to
`false` or not specified, the table_path will be treated as an exact path (no
regex matching). |
| fetch_size | Int | No | 0
| For queries that return a large number of objects, you can configure
the row fetch size used in the query to improve performance by reducing the
number database hits required to satisfy the selection criteria. Zero means use
jdbc default value.
[...]
| properties | Map | No | -
| Additional connection configuration parameters,when properties and URL
have the same parameters, the priority is determined by the <br/>specific
implementation of the driver. For example, in MySQL, properties take precedence
over the URL.
[...]
-| table_path | String | No | -
| The path to the full path of table, you can use this configuration
instead of `query`. <br/>examples: <br/>`- mysql: "testdb.table1" `<br/>`-
oracle: "test_schema.table1" `<br/>`- sqlserver: "testdb.test_schema.table1"`
<br/>`- postgresql: "testdb.test_schema.table1"` <br/>`- iris:
"test_schema.table1"`
[...]
+| table_path | String | No | -
| Full table path. It can be used alone or together with `query` in the
single-table layout. <br/>Examples: <br/>`- mysql: "testdb.table1" `<br/>`-
oracle: "test_schema.table1" `<br/>`- sqlserver: "testdb.test_schema.table1"`
<br/>`- postgresql: "testdb.test_schema.table1"` <br/>`- iris:
"test_schema.table1"`
[...]
| table_list | Array | No | -
| The list of tables to be read, you can use this configuration instead
of `table_path`
[...]
| where_condition | String | No | -
| Common row filter conditions for all tables/queries, must start with
`where`. for example `where id > 100`
[...]
-| split.size | Int | No | 8096
| How many rows in one split, captured tables are split into multiple
splits when read of table. **Note**: This parameter takes effect only when
using the `table_path` parameter. It does not take effect when using the
`query` parameter.
[...]
+| split.size | Int | No | 8096
| Target rows per split when the dynamic splitter is used, including
`table_path` and `table_list` layouts. It does not control the top-level fixed
`query` plus `partition_column` mode.
[...]
| split.even-distribution.factor.lower-bound | Double | No | 0.05
| Not recommended for use.<br/> The lower bound of the chunk key
distribution factor. This factor is used to determine whether the table data is
evenly distributed. If the distribution factor is calculated to be greater than
or equal to this lower bound (i.e., (MAX(id) - MIN(id) + 1) / row count), the
table chunks would be optimized for even distribution. Otherwise, if the
distribution factor is less, th [...]
| split.even-distribution.factor.upper-bound | Double | No | 100
| Not recommended for use.<br/> The upper bound of the chunk key
distribution factor. This factor is used to determine whether the table data is
evenly distributed. If the distribution factor is calculated to be less than or
equal to this upper bound (i.e., (MAX(id) - MIN(id) + 1) / row count), the
table chunks would be optimized for even distribution. Otherwise, if the
distribution factor is greater, th [...]
| split.sample-sharding.threshold | Int | No | 1000
| This configuration specifies the threshold of estimated shard count to
trigger the sample sharding strategy. When the distribution factor is outside
the bounds specified by `chunk-key.even-distribution.factor.upper-bound` and
`chunk-key.even-distribution.factor.lower-bound`, and the estimated shard count
(calculated as approximate row count / chunk size) exceeds this threshold, the
sample sharding stra [...]
@@ -85,99 +181,24 @@ supports query SQL and can achieve projection effect.
### Table Matching
-The JDBC Source connector supports two ways to specify tables:
-
-#### Notes
-
-- Many JDBC drivers treat `DatabaseMetaData.getColumns(..., schemaPattern,
tableNamePattern, ...)` as SQL LIKE patterns.
- If your schema/table names contain `_` or `%`, column discovery may return
rows from other tables. SeaTunnel filters the
- returned metadata rows by exact schema/table identifier to avoid mixing
columns.
-- For case-sensitive databases, make sure the configured schema/table names
use the exact identifier case.
-
-1. **Exact Table Path**: Use `table_path` to specify a single table with its
full path.
- ```hocon
- table_path = "testdb.table1"
- ```
-
-2. **Regular Expression**: Use `table_path` with a regex pattern to match
multiple tables.
- ```hocon
- table_path = "testdb.table\\d+" # Matches table1, table2, table3, etc.
- use_regex = true
- ```
-
-#### Regular Expression Support for Table Names
-
-The JDBC connector supports using regular expressions to match multiple
tables. This feature allows you to process multiple tables with a single source
configuration.
-
-#### Configuration
+Use the full table path expected by the database dialect:
-To use regular expression matching for table paths:
+| Database family | Example |
+|-----------------|---------|
+| MySQL | `sales.orders` |
+| PostgreSQL and SQL Server | `sales.public.orders` |
+| Oracle | `SALES.ORDERS` |
-1. Set `use_regex = true` to enable regex matching
-2. If `use_regex` is not set or set to `false`, the connector will treat the
table_path as an exact path (no regex matching)
+`use_regex = false` performs exact matching and is the safest default. Set
`use_regex = true` only when the table part of `table_path` is intentionally a
regular expression:
-#### Regular Expression Syntax Notes
-
-- **Path Separator**: The dot (`.`) is treated as a separator between
database, schema, and table names.
-- **Escaped Dots**: If you need to use a dot (`.`) as a wildcard character in
your regular expression to match any character, you must escape it with a
backslash (`\.`).
-- **Path Format**: For paths like `database.table` or `database.schema.table`,
the last unescaped dot separates the table pattern from the database/schema
pattern.
-- **Pattern Examples**:
- - `test.table\\d+` - Matches tables like `table1`, `table2`, etc. in the
`test` database
- - `test.*` - Matches all tables in the `test` database (for whole database
synchronization)
- - `postgres.public.test_db_\.*` - Matches all tables that start with
`test_db_` in the `public` schema of the `postgres` database
-
-#### Example
-
-```hocon
-source {
- Jdbc {
- url = "jdbc:mysql://localhost:3306/test"
- driver = "com.mysql.cj.jdbc.Driver"
- user = "root"
- password = "password"
-
- table_list = [
- {
- # Regex matching - match any table in test database
- table_path = "test.*"
- use_regex = true
- },
- {
- # Regex matching - match tables with "user" followed by digits
- table_path = "test.user\\d+"
- use_regex = true
- },
- {
- # Exact matching - simple table name
- table_path = "test.config"
- # use_regex not specified, defaults to false
- },
- ]
- }
-}
+```text
+table_path = "sales.orders_\\d+"
+use_regex = true
```
-#### Multi-table Synchronization
-
-When using either regular expressions, the connector will read data from all
matching tables. Each table will be processed independently, and the data will
be combined in the output.
+In HOCON strings, a regular-expression backslash must be escaped, so `\d+` is
written as `\\d+` in the file. The final unescaped dot separates the
database/schema path from the table pattern.
-Example configuration for multi-table synchronization:
-```hocon
-Jdbc {
- url = "jdbc:mysql://localhost/test"
- driver = "com.mysql.cj.jdbc.Driver"
- user = "root"
- password = "123456"
-
- # Using regular expression with explicit configuration
- table_list = [
- {
- table_path = "testdb.table\\d+"
- use_regex = true
- }
- ]
-}
-```
+Many JDBC drivers treat schema and table arguments passed to
`DatabaseMetaData` as SQL `LIKE` patterns. SeaTunnel performs an exact
identifier check after metadata discovery, but you should still use the exact
case for case-sensitive database identifiers.
### decimal_type_narrowing
@@ -242,29 +263,27 @@ If one dialect not supported by SeaTunnel, it will use
the default dialect `Gene
## Parallel Reader
-The JDBC Source connector supports parallel reading of data from tables.
SeaTunnel will use certain rules to split the data in the table, which will be
handed over to readers for reading. The number of readers is determined by the
`parallelism` option.
+Parallelism determines how many readers can run at the same time; the split
configuration determines how many independent splits are available.
+
+### Recommended: `table_path` with dynamic splitting
+
+For a full-table snapshot, configure `table_path` and normally leave split-key
discovery to SeaTunnel. The connector uses `partition_column` when provided.
Otherwise it searches the primary key and then unique indexes for the first
supported column. String, numeric, and date columns are supported split-key
types.
-**Split Key Rules:**
+`split.size` is the target row count per split. It is not a hard row limit:
actual split sizes depend on key distribution and database statistics. If no
supported primary key, unique index, or explicit `partition_column` exists, the
table is read by one split even when job parallelism is greater than one.
-1. If `partition_column` is not null, It will be used to calculate split. The
column must in **Supported split data type**.
-2. If `partition_column` is null, seatunnel will read the schema from table
and get the Primary Key and Unique Index. If there are more than one column in
Primary Key and Unique Index, The first column which in the **supported split
data type** will be used to split data. For example, the table have Primary
Key(nn guid, name varchar), because `guid` id not in **supported split data
type**, so the column `name` will be used to split data.
+### Top-level `query` with fixed partitions
-**Supported split data type:**
-* String
-* Number(int, bigint, decimal, ...)
-* Date
+Only the top-level combination of `query` and `partition_column` selects the
legacy fixed splitter; `partition_num` then controls the number of splits. The
partition column must be present in the query result. Optional lower and upper
bounds avoid extra `MIN`/`MAX` queries, but incorrect bounds can omit source
rows, so use them only when the full data range is known.
-## tips
+Entries inside `table_list` continue to use dynamic splitting even when they
include `query` or partition settings. Do not expect `split.size` to affect the
top-level fixed partition mode.
-> If the table can not be split(for example, table have no Primary Key or
Unique Index, and `partition_column` is not set), it will run in single
concurrency.
->
-> Use `table_path` to replace `query` for single table reading. If you need to
read multiple tables, use `table_list`.
->
-> When inferring a primary key based on a `query`, the key is inherited from
the underlying table where the first column in the result set is located, and
its strictness for the overall join result set is not guaranteed (for example,
when the query contains joins or reads from multiple tables).
+### Query and primary-key caution
-## appendix
+When SeaTunnel infers a key for `query`, it inherits metadata from the
underlying table of the first result column. For joins or multi-table queries,
that key is not guaranteed to be unique across the complete result set. Prefer
a single-reader query or explicitly choose a partition column whose values
divide the result safely.
-there are some reference value for params above.
+## Driver reference
+
+The following values are starting points. Confirm the driver artifact,
license, database compatibility, and Java compatibility with the database
vendor before deployment.
| datasource | driver |
url | maven
|
|-------------------|-----------------------------------------------------|------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------|
@@ -275,14 +294,14 @@ there are some reference value for params above.
| sqlserver | com.microsoft.sqlserver.jdbc.SQLServerDriver |
jdbc:sqlserver://localhost:1433 |
https://mvnrepository.com/artifact/com.microsoft.sqlserver/mssql-jdbc
|
| oracle | oracle.jdbc.OracleDriver |
jdbc:oracle:thin:@localhost:1521/xepdb1 |
https://mvnrepository.com/artifact/com.oracle.database.jdbc/ojdbc8
|
| sqlite | org.sqlite.JDBC |
jdbc:sqlite:test.db |
https://mvnrepository.com/artifact/org.xerial/sqlite-jdbc
|
-| gbase8a | com.gbase.jdbc.Driver |
jdbc:gbase://e2e_gbase8aDb:5258/test |
https://cdn.gbase.cn/products/30/p5CiVwXBKQYIUGN8ecHvk/gbase-connector-java-9.5.0.7-build1-bin.jar
|
+| gbase8a | com.gbase.jdbc.Driver |
jdbc:gbase://localhost:5258/test |
https://cdn.gbase.cn/products/30/p5CiVwXBKQYIUGN8ecHvk/gbase-connector-java-9.5.0.7-build1-bin.jar
|
| starrocks | com.mysql.cj.jdbc.Driver |
jdbc:mysql://localhost:3306/test |
https://mvnrepository.com/artifact/mysql/mysql-connector-java
|
| db2 | com.ibm.db2.jcc.DB2Driver |
jdbc:db2://localhost:50000/testdb |
https://mvnrepository.com/artifact/com.ibm.db2.jcc/db2jcc/db2jcc4
|
-| tablestore | com.alicloud.openservices.tablestore.jdbc.OTSDriver |
"jdbc:ots:http s://myinstance.cn-hangzhou.ots.aliyuncs.com/myinstance" |
https://mvnrepository.com/artifact/com.aliyun.openservices/tablestore-jdbc
|
+| tablestore | com.alicloud.openservices.tablestore.jdbc.OTSDriver |
jdbc:ots:https://myinstance.cn-hangzhou.ots.aliyuncs.com/myinstance |
https://mvnrepository.com/artifact/com.aliyun.openservices/tablestore-jdbc
|
| saphana | com.sap.db.jdbc.Driver |
jdbc:sap://localhost:39015 |
https://mvnrepository.com/artifact/com.sap.cloud.db.jdbc/ngdbc
|
| doris | com.mysql.cj.jdbc.Driver |
jdbc:mysql://localhost:3306/test |
https://mvnrepository.com/artifact/mysql/mysql-connector-java
|
| teradata | com.teradata.jdbc.TeraDriver |
jdbc:teradata://localhost/DBS_PORT=1025,DATABASE=test |
https://mvnrepository.com/artifact/com.teradata.jdbc/terajdbc
|
-| Snowflake | net.snowflake.client.jdbc.SnowflakeDriver |
jdbc:snowflake://<account_name>.snowflakecomputing.com |
https://mvnrepository.com/artifact/net.snowflake/snowflake-jdbc
|
+| Snowflake | net.snowflake.client.jdbc.SnowflakeDriver |
jdbc:snowflake://<account_name>.snowflakecomputing.com |
https://mvnrepository.com/artifact/net.snowflake/snowflake-jdbc
|
| Redshift | com.amazon.redshift.jdbc42.Driver |
jdbc:redshift://localhost:5439/testdb?defaultRowFetchSize=1000 |
https://mvnrepository.com/artifact/com.amazon.redshift/redshift-jdbc42
|
| Vertica | com.vertica.jdbc.Driver |
jdbc:vertica://localhost:5433 |
https://repo1.maven.org/maven2/com/vertica/jdbc/vertica-jdbc/12.0.3-0/vertica-jdbc-12.0.3-0.jar
|
| Kingbase | com.kingbase8.Driver |
jdbc:kingbase8://localhost:54321/db_test |
https://repo1.maven.org/maven2/cn/com/kingbase/kingbase8/8.6.0/kingbase8-8.6.0.jar
|
@@ -296,56 +315,36 @@ there are some reference value for params above.
| Trino | io.trino.jdbc.TrinoDriver |
jdbc:trino://localhost:8080/trino |
https://repo1.maven.org/maven2/io/trino/trino-jdbc/460/trino-jdbc-460.jar
|
| YashanDB | com.yashandb.jdbc.Driver |
jdbc:yasdb://localhost:1688/SYS |
https://mvnrepository.com/artifact/com.yashandb/yashandb-jdbc
|
-## Example
-
-### simple
+## Common patterns
-#### Case 1
+### Custom query
-```
-Jdbc {
- url = "jdbc:mysql://localhost/test?serverTimezone=GMT%2b8"
- driver = "com.mysql.cj.jdbc.Driver"
- connection_check_timeout_sec = 100
- user = "root"
- password = "123456"
- query = "select * from type_bin"
+```hocon
+env {
+ parallelism = 1
+ job.mode = "BATCH"
}
-```
-
-#### Case 2 Use the select count(*) instead of analysis table for count table
rows in dynamic chunk split stage
-```
-Jdbc {
- url = "jdbc:mysql://localhost/test?serverTimezone=GMT%2b8"
+source {
+ Jdbc {
+ url = "jdbc:mysql://mysql.example.com:3306/sales"
driver = "com.mysql.cj.jdbc.Driver"
- connection_check_timeout_sec = 100
- user = "root"
- password = "123456"
- use_select_count = true
- query = "select * from type_bin"
+ username = "seatunnel_reader"
+ password = "change_me"
+ query = "SELECT id, customer_name, amount FROM orders WHERE status =
'PAID'"
+ }
}
-```
-
-#### Case 3 Use the select NUM_ROWS from all_tables for the table rows but
skip the analyze table.
-```
-Jdbc {
- url = "jdbc:mysql://localhost/test?serverTimezone=GMT%2b8"
- driver = "com.mysql.cj.jdbc.Driver"
- connection_check_timeout_sec = 100
- user = "root"
- password = "123456"
- skip_analyze = true
- query = "select * from type_bin"
+sink {
+ Console {}
}
```
-#### Case 4 Oracle Source with BLOB as string to Doris Sink
+### Oracle BLOB as STRING
-This example demonstrates how to handle Oracle's BLOB data as strings when
transferring to Doris. This is useful for large BLOB fields.
+Set `handle_blob_as_string = true` when Oracle BLOB values should be exposed
as SeaTunnel STRING values, for example before writing them to Doris.
-```
+```hocon
env {
parallelism = 1
job.mode = "BATCH"
@@ -354,36 +353,36 @@ env {
source {
Jdbc {
driver = oracle.jdbc.driver.OracleDriver
- url = "jdbc:oracle:thin:@oracle_host:1521/SERVICE_NAME"
- user = "username"
- password = "password"
+ url = "jdbc:oracle:thin:@oracle.example.com:1521/SERVICE_NAME"
+ username = "seatunnel_reader"
+ password = "change_me"
query = "SELECT ID, NAME, CONTENT_BLOB FROM MY_TABLE"
handle_blob_as_string = true # Enable BLOB to String conversion for Oracle
}
}
+
+sink {
+ Console {}
+}
```
-### parallel by partition_column
+### Custom query partitioned by a column
-```
+```hocon
env {
parallelism = 10
job.mode = "BATCH"
}
source {
Jdbc {
- url = "jdbc:mysql://localhost/test?serverTimezone=GMT%2b8"
+ url = "jdbc:mysql://mysql.example.com:3306/sales?serverTimezone=UTC"
driver = "com.mysql.cj.jdbc.Driver"
connection_check_timeout_sec = 100
- user = "root"
- password = "123456"
- query = "select * from type_bin"
+ username = "seatunnel_reader"
+ password = "change_me"
+ query = "SELECT id, customer_name, amount FROM orders"
partition_column = "id"
- partition_num = 10 # Replace split.size with partition_num
- # Read start boundary
- #partition_lower_bound = ...
- # Read end boundary
- #partition_upper_bound = ...
+ partition_num = 10
}
}
@@ -392,51 +391,53 @@ sink {
}
```
-### Parallel Boundary
+### Explicit partition boundaries
-> It is more efficient to specify the data within the upper and lower bounds
of the query. It is more efficient to read your data source according to the
upper and lower boundaries you configured.
+Specify bounds only when they cover the complete source range. SeaTunnel does
not read values outside the configured interval.
+
+```hocon
+env {
+ parallelism = 10
+ job.mode = "BATCH"
+}
-```
source {
Jdbc {
- url =
"jdbc:mysql://localhost:3306/test?serverTimezone=GMT%2b8&useUnicode=true&characterEncoding=UTF-8&rewriteBatchedStatements=true"
+ url = "jdbc:mysql://mysql.example.com:3306/sales?serverTimezone=UTC"
driver = "com.mysql.cj.jdbc.Driver"
connection_check_timeout_sec = 100
- user = "root"
- password = "123456"
- # Define query logic as required
- query = "select * from type_bin"
+ username = "seatunnel_reader"
+ password = "change_me"
+ query = "SELECT id, customer_name, amount FROM orders"
partition_column = "id"
- # Read start boundary
partition_lower_bound = 1
- # Read end boundary
partition_upper_bound = 500
partition_num = 10
- properties {
- useSSL=false
- }
}
}
+
+sink {
+ Console {}
+}
```
-### parallel by Primary Key or Unique Index
+### Dynamic splitting by primary key or unique index
-> Configuring `table_path` will turn on auto split, you can configure
`split.*` to adjust the split strategy
+This example uses `table_path` without the top-level `query` plus
`partition_column` combination, so it uses dynamic splitting. Start with the
default split settings, then tune `split.size` only after measuring the source
database load and job throughput.
-```
+```hocon
env {
parallelism = 10
job.mode = "BATCH"
}
source {
Jdbc {
- url = "jdbc:mysql://localhost/test?serverTimezone=GMT%2b8"
+ url = "jdbc:mysql://mysql.example.com:3306/sales?serverTimezone=UTC"
driver = "com.mysql.cj.jdbc.Driver"
connection_check_timeout_sec = 100
- user = "root"
- password = "123456"
- table_path = "testdb.table1"
- query = "select * from testdb.table1"
+ username = "seatunnel_reader"
+ password = "change_me"
+ table_path = "sales.orders"
split.size = 10000
}
}
@@ -446,43 +447,67 @@ sink {
}
```
-### multiple table read
+### Multiple tables
-***Configuring `table_list` will turn on auto split, you can configure
`split.*` to adjust the split strategy***
+Use `table_list` when different tables need different queries or matching
rules.
```hocon
-Jdbc {
- url = "jdbc:mysql://localhost/test?serverTimezone=GMT%2b8"
+env {
+ parallelism = 4
+ job.mode = "BATCH"
+}
+
+source {
+ Jdbc {
+ url = "jdbc:mysql://mysql.example.com:3306/sales?serverTimezone=UTC"
driver = "com.mysql.cj.jdbc.Driver"
connection_check_timeout_sec = 100
- user = "root"
- password = "123456"
+ username = "seatunnel_reader"
+ password = "change_me"
table_list = [
{
- # e.g. table_path = "testdb.table1"、table_path =
"test_schema.table1"、table_path = "testdb.test_schema.table1"
- table_path = "testdb.table1"
+ table_path = "sales.orders"
},
{
- table_path = "testdb.table2"
- # Use query filter rows & columns
- query = "select id, name from testdb.table2 where id > 100"
+ table_path = "sales.customers"
+ query = "SELECT id, name FROM customers WHERE id > 100"
},
{
- # Using regex to match multiple tables
- table_path = "testdb.user_table\\d+"
+ table_path = "sales.archive_\\d+"
use_regex = true
}
]
- #where_condition= "where id > 100"
- #split.size = 10000
- #split.even-distribution.factor.upper-bound = 100
- #split.even-distribution.factor.lower-bound = 0.05
- #split.sample-sharding.threshold = 1000
- #split.inverse-sampling.rate = 1000
+ }
+}
+
+sink {
+ Console {}
}
```
+## Troubleshooting
+
+### JDBC driver class cannot be found
+
+Confirm that the driver JAR is in the engine-specific directory on every
execution node, that the configured `driver` class exists in that JAR, and that
the affected process was restarted after the JAR was added.
+
+### Connection succeeds in a SQL client but fails in SeaTunnel
+
+The hostname must be reachable from the SeaTunnel process, not only from your
laptop. Check network routing, firewall rules, TLS settings, credentials,
database name, and `connection_check_timeout_sec`. Do not copy an example
hostname without replacing it.
+
+### Table or columns cannot be discovered
+
+Check the database-specific `table_path` format, identifier case, and the
account's metadata and `SELECT` privileges. For a custom query, first execute
the exact SQL with the same account in a database client.
+
+### Parallelism does not increase the number of readers
+
+For dynamic splitting, verify that the table has a supported primary key or
unique index, or configure `partition_column`. To select the legacy fixed
splitter for a top-level `query`, configure both `partition_column` and a
suitable `partition_num`. A table without a safe split key is intentionally
read by one split.
+
+### Rows are missing after setting partition bounds
+
+`partition_lower_bound` and `partition_upper_bound` define the range SeaTunnel
reads; they are not only performance hints. Remove them to let SeaTunnel
discover the bounds, or correct them so they include every required row.
+
## Changelog
<ChangeLog />
diff --git a/docs/zh/connectors/source/Jdbc.md
b/docs/zh/connectors/source/Jdbc.md
index b9d14bd429..2e3d19c862 100644
--- a/docs/zh/connectors/source/Jdbc.md
+++ b/docs/zh/connectors/source/Jdbc.md
@@ -2,29 +2,123 @@ import ChangeLog from '../changelog/connector-jdbc.md';
# JDBC
-> JDBC 源连接器
-
## 描述
-通过 JDBC 读取外部数据源数据。
+JDBC Source 通过数据库厂商提供的 JDBC 驱动读取表或自定义查询结果，支持列投影、行过滤、并行快照读取，以及在一个 Source
配置中读取多张表。
-:::tip
+JDBC Source 是有界数据源：它读取数据库查询当前可见的数据后结束。如果任务需要持续捕获后续的新增、更新和删除，请使用对应的 CDC
Connector。
-警告：为了符合许可证要求，您必须自己提供数据库驱动程序，复制到 `$SEATUNNEL_HOME/lib/` 目录以使其工作。
+第一次配置 JDBC Source
时，请先阅读[选择读取模式](#选择读取模式)和[快速入门](#快速入门postgresql)，再按需查阅后面的完整参数说明。
-例如，如果您使用 MySQL，应下载并复制 `mysql-connector-java-xxx.jar` 到
`$SEATUNNEL_HOME/lib/`。对于 Spark/Flink，您还应将其复制到 `$SPARK_HOME/jars/` 或
`$FLINK_HOME/lib/`。
+## 使用依赖
-:::
+先安装 `connector-jdbc` 插件：
-## 使用依赖
+```plugin_config
+--seatunnel-connectors--
+connector-jdbc
+--end--
+```
+
+```bash
+cd "${SEATUNNEL_HOME}"
+sh bin/install-plugin.sh
+```
+
+不同数据库厂商的 JDBC 驱动具有不同的许可证和再分发条款，而且驱动版本还必须同时兼容数据库和 Java 运行时，因此 SeaTunnel
不会统一内置所有 JDBC 驱动。请自行下载合适的驱动，并在启动任务前把 JAR 放入对应引擎的目录。
+
+### Spark 和 Flink 引擎
+
+把 JDBC 驱动放到每个 SeaTunnel 执行节点的 `${SEATUNNEL_HOME}/plugins/Jdbc/lib/`。
+
+### Zeta 引擎
+
+把 JDBC 驱动放到每个 SeaTunnel 节点的 `${SEATUNNEL_HOME}/lib/`，然后重启受影响的 SeaTunnel
进程，让驱动进入类路径。
+
+常用驱动类名和下载地址见[驱动参考](#驱动参考)。
+
+## 选择读取模式
+
+配置并行读取前，先选择单表或多表布局。单表布局中，`table_path` 和 `query` 可以单独使用，也可以组合使用；多表 `table_list`
与顶层 `table_path`、`query` 互斥。
+
+| 使用场景 | 配置方式 | 行为 |
+|----------|----------|------|
+| 读取单表，并自动发现 schema 和动态分片 | `table_path` | 推荐用于整表快照。SeaTunnel
读取表元数据；表存在可用分片键时，通过 `split.size` 控制分片大小。 |
+| 自定义读取列、JOIN 或数据库表达式 | `query`，可以同时配置 `table_path` | SeaTunnel 执行用户提供的
SQL。需要明确表身份和元数据时可同时配置 `table_path`。部分 JOIN 无法安全推断查询主键，详见 [Query
与主键注意事项](#query-与主键注意事项)。 |
+| 读取多张表或按表名模式匹配 | `table_list` | 每一项可以配置 `table_path`、可选 `query` 和分片参数。使用
`table_list` 时，不能再配置顶层 `table_path` 或 `query`。 |
+
+只有需要给所有选中表或查询追加同一过滤条件时才使用 `where_condition`。它必须以 `where` 开头，例如 `where
updated_at >= '2026-01-01'`。
+
+## 快速入门：PostgreSQL
+
+下面从 PostgreSQL 读取三行数据，并通过 Console Sink 输出到 SeaTunnel 日志。
+
+1. 按照[使用依赖](#使用依赖)的说明放置兼容版本的 PostgreSQL JDBC 驱动。
+
+2. 使用 PostgreSQL 管理员账号连接已有的 `sales` 数据库，创建专用教程表，并为只读账号授权。如果 `seatunnel_reader`
已经存在，请省略 `CREATE ROLE` 并复用该账号：
+
+```sql
+CREATE ROLE seatunnel_reader WITH LOGIN PASSWORD 'change_me';
+
+DROP TABLE IF EXISTS public.seatunnel_jdbc_source_quick_start;
+
+CREATE TABLE public.seatunnel_jdbc_source_quick_start (
+ id BIGINT PRIMARY KEY,
+ customer_name VARCHAR(100) NOT NULL,
+ amount DECIMAL(10, 2) NOT NULL
+);
-### 对于 Spark/Flink 引擎
+INSERT INTO public.seatunnel_jdbc_source_quick_start VALUES
+ (1, 'Alice', 120.50),
+ (2, 'Bob', 80.00),
+ (3, 'Carol', 42.00);
-> 1. 您需要确保 [jdbc 驱动程序 jar
包](https://mvnrepository.com/artifact/mysql/mysql-connector-java) 已放置在目录
`${SEATUNNEL_HOME}/plugins/` 中。
+GRANT CONNECT ON DATABASE sales TO seatunnel_reader;
+GRANT USAGE ON SCHEMA public TO seatunnel_reader;
+GRANT SELECT ON TABLE public.seatunnel_jdbc_source_quick_start TO
seatunnel_reader;
+```
+
+`DROP TABLE` 用于保证教程数据可以重复初始化，请勿对业务表执行这条语句。
+
+3. 把下面的任务保存为
`${SEATUNNEL_HOME}/config/jdbc-source-quick-start.conf`，并按实际环境替换主机、账号和数据库名。
-### 对于 SeaTunnel Zeta 引擎
+```hocon
+env {
+ parallelism = 1
+ job.mode = "BATCH"
+}
-> 1. 您需要确保 [jdbc 驱动程序 jar
包](https://mvnrepository.com/artifact/mysql/mysql-connector-java) 已放置在目录
`${SEATUNNEL_HOME}/lib/` 中。
+source {
+ Jdbc {
+ url = "jdbc:postgresql://postgresql.example.com:5432/sales"
+ driver = "org.postgresql.Driver"
+ username = "seatunnel_reader"
+ password = "change_me"
+ query = "SELECT id, customer_name, amount FROM
public.seatunnel_jdbc_source_quick_start ORDER BY id"
+ }
+}
+
+sink {
+ Console {}
+}
+```
+
+4. 运行任务：
+
+```bash
+cd "${SEATUNNEL_HOME}"
+./bin/seatunnel.sh --config ./config/jdbc-source-quick-start.conf -m local
+```
+
+5. 确认 Console Sink 输出包含以下字段值：
+
+| id | customer_name | amount |
+|----|---------------|-------:|
+| 1 | Alice | 120.50 |
+| 2 | Bob | 80.00 |
+| 3 | Carol | 42.00 |
+
+如果任务在读取数据前失败，请先检查[故障排查](#故障排查)。
## 关键特性
@@ -33,7 +127,7 @@ import ChangeLog from '../changelog/connector-jdbc.md';
- [x] [精确一次](../../introduction/concepts/connector-v2-features.md)
- [x] [列投影](../../introduction/concepts/connector-v2-features.md)
-支持查询 SQL 并可以实现投影效果。
+使用 `query` 可以只读取需要的列。
- [x] [并行性](../../introduction/concepts/connector-v2-features.md)
- [x] [支持用户自定义split](../../introduction/concepts/connector-v2-features.md)
@@ -41,22 +135,24 @@ import ChangeLog from '../changelog/connector-jdbc.md';
## 选项
+`url` 和 `driver` 始终必填。由于部分数据库允许无认证连接，账号和密码不是统一必填项。请选择顶层单表布局或 `table_list`；顶层
`table_path` 与 `query` 可以组合使用。账号建议使用规范键 `username`，历史配置中的 `user` 仍作为兼容键继续支持。
+
| 参数名 | 类型 | 必须 | 默认值 | 描述
[...]
|--------------------------------------------|---------|------|---------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
| url | String | 是 | - | JDBC
连接的 URL。参考示例：jdbc:postgresql://localhost/test
[...]
| driver | String | 是 | - |
用于连接到远程数据源的 jdbc 类名，如果您使用 MySQL，值为 `com.mysql.cj.jdbc.Driver`。
[...]
-| username | String | 否 | - |
用户名
[...]
-| password | String | 否 | - | 密码
[...]
-| query | String | 否 | - | 查询语句
[...]
+| username | String | 否 | - |
数据库账号。历史配置键 `user` 仍作为兼容键支持。
[...]
+| password | String | 否 | - |
数据库账号的密码。
[...]
+| query | String | 否 | - | 要执行的
SQL。需要同时明确表身份和元数据时，可以与 `table_path` 组合使用。
[...]
| compatible_mode | String | 否 | - |
数据库的兼容模式，当数据库支持多种兼容模式时需要。<br/> 例如，使用 OceanBase 数据库时，需要将其设置为 'mysql' 或
'oracle'。<br/> 使用 starrocks 时，需要将其设置为 `starrocks`
[...]
| dialect | String | 否 | - |
指定的方言，如果不存在，仍然根据 url 获取，优先级高于 url。<br/> 例如，使用 starrocks 时，需要将其设置为 `starrocks`
[...]
| connection_check_timeout_sec | Int | 否 | 30 |
等待用于验证连接的数据库操作完成的时间（秒）。
[...]
| connect_timeout_ms | Int | 否 | 86400000 | 建立
JDBC 连接时的连接超时时间，单位毫秒。`0` 表示不超时。
[...]
| socket_timeout_ms | Int | 否 | 86400000 |
JDBC 连接建立后的 socket 读取超时时间，单位毫秒。`0` 表示不超时。
[...]
| partition_column | String | 否 | - |
用于分割数据的列名。
[...]
-| partition_upper_bound | Long | 否 | - |
partition_column 的最大值用于扫描，如果未设置，SeaTunnel 将查询数据库获取最大值。
[...]
-| partition_lower_bound | Long | 否 | - |
partition_column 的最小值用于扫描，如果未设置，SeaTunnel 将查询数据库获取最小值。
[...]
-| partition_num | Int | 否 | job
parallelism | 不建议使用，正确的方法是通过 `split.size` 控制分割数量<br/> **注意：** 此参数仅在使用 `query`
参数时生效。使用 `table_path` 参数时不生效。
[...]
+| partition_upper_bound | String | 否 | - |
query 分区读取使用的闭区间上界。未配置时，SeaTunnel 会从源表查询最大值。
[...]
+| partition_lower_bound | String | 否 | - |
query 分区读取使用的闭区间下界。未配置时，SeaTunnel 会从源表查询最小值。
[...]
+| partition_num | Int | 否 | 10 | 顶层
`query` 与 `partition_column` 触发 fixed splitter 时的分片数，无论是否同时配置 `table_path`。动态
`table_path` 和 `table_list` 布局改用 `split.size`。
[...]
| decimal_type_narrowing | Boolean | 否 | true |
十进制类型缩小，如果为 true，十进制类型将缩小为 int 或 long 类型（如果没有精度损失）。目前仅支持 Oracle。请参考下面的
`decimal_type_narrowing`
[...]
| int_type_narrowing | Boolean | 否 | true | Int
类型缩小，如果为 true，tinyint(1) 类型将缩小为布尔类型（如果没有精度损失）。目前支持 MySQL。请参考下面的
`int_type_narrowing`
[...]
| handle_blob_as_string | Boolean | 否 | false | 如果为
true，BLOB 类型将转换为 STRING 类型。**仅支持 Oracle 数据库**。这对于处理超过默认大小限制的 Oracle 中的大 BLOB
字段很有用。将 Oracle 的 BLOB 字段传输到 Doris 等系统时，将其设置为 true 可以使数据传输更高效。
[...]
@@ -69,10 +165,10 @@ import ChangeLog from '../changelog/connector-jdbc.md';
| use_regex | Boolean | 否 | false | 控制
table_path 的正则表达式匹配。设置为 `true` 时，table_path 将被视为正则表达式模式。设置为 `false`
或未指定时，table_path 将被视为精确路径（无正则表达式匹配）。 |
| fetch_size | Int | 否 | 0 |
对于返回大量对象的查询，您可以配置查询中使用的行提取大小，以通过减少满足选择条件所需的数据库命中次数来提高性能。零表示使用 jdbc 默认值。
[...]
| properties | Map | 否 | - |
其他连接配置参数，当 properties 和 URL 具有相同参数时，优先级由<br/>驱动程序的具体实现确定。例如，在 MySQL
中，properties 优先于 URL。
[...]
-| table_path | String | 否 | - |
表的完整路径，您可以使用此配置代替 `query`。<br/>示例：<br/>`- mysql: "testdb.table1" `<br/>`-
oracle: "test_schema.table1" `<br/>`- sqlserver: "testdb.test_schema.table1"`
<br/>`- postgresql: "testdb.test_schema.table1"` <br/>`- iris:
"test_schema.table1"`
[...]
+| table_path | String | 否 | - |
表的完整路径。在单表布局中可以单独使用，也可以与 `query` 组合。<br/>示例：<br/>`- mysql: "testdb.table1"
`<br/>`- oracle: "test_schema.table1" `<br/>`- sqlserver:
"testdb.test_schema.table1"` <br/>`- postgresql: "testdb.test_schema.table1"`
<br/>`- iris: "test_schema.table1"`
[...]
| table_list | Array | 否 | - |
要读取的表列表，您可以使用此配置代替 `table_path`
[...]
| where_condition | String | 否 | - |
所有表/查询的通用行过滤条件，必须以 `where` 开头。例如 `where id > 100`
[...]
-| split.size | Int | 否 | 8096 |
一个分割中有多少行，捕获的表在读取时被分成多个分割。**注意**：此参数仅在使用 `table_path` 参数时生效。使用 `query` 参数时不生效。
[...]
+| split.size | Int | 否 | 8096 |
使用动态分片时每个 split 的目标行数，适用于 `table_path` 和 `table_list` 布局；不控制顶层 `query` 与
`partition_column` 的 fixed partition 模式。
[...]
| split.even-distribution.factor.lower-bound | Double | 否 | 0.05
| 不建议修改。<br/> 分片键分布因子的下界。该因子用于判断表数据是否均匀分布。如果计算出的分布因子大于或等于此下界（即 (MAX(id) -
MIN(id) + 1) / 行数），则将优化为均匀分布的分片方式。否则，如果分布因子更小，则表将被视为非均匀分布，当预估的分片数超过
`sample-sharding.threshold` 指定的值时，将使用基于采样的分片策略。默认值为 0.05。 |
| split.even-distribution.factor.upper-bound | Double | 否 | 100
| 不建议修改。<br/> 分片键分布因子的上界。该因子用于判断表数据是否均匀分布。如果计算出的分布因子小于或等于此上界（即 (MAX(id) -
MIN(id) + 1) / 行数），则将优化为均匀分布的分片方式。否则，如果分布因子更大，则表将被视为非均匀分布，当预估的分片数超过
`sample-sharding.threshold` 指定的值时，将使用基于采样的分片策略。默认值为 100.0。 |
| split.sample-sharding.threshold | Int | 否 | 1000
| 此配置指定触发采样分片策略的预估分片数阈值。当分布因子超出 `split.even-distribution.factor.upper-bound` 和
`split.even-distribution.factor.lower-bound` 指定的范围，且预估分片数（计算方式为近似行数 /
分片大小）超过此阈值时，将使用采样分片策略。这有助于更高效地处理大数据集。默认值为 1000。
|
@@ -85,99 +181,24 @@ import ChangeLog from '../changelog/connector-jdbc.md';
### 表匹配
-JDBC 源连接器支持两种方式指定表：
-
-#### 注意事项
+请按照数据库方言要求填写完整表路径：
-- 许多 JDBC 驱动会将 `DatabaseMetaData.getColumns(..., schemaPattern,
tableNamePattern, ...)` 视为 SQL LIKE 的模式匹配。
- 当 schema/table 名称中包含 `_` 或 `%` 时，列发现可能会返回其他表的列。SeaTunnel 会按精确的 schema/table
标识符对返回结果做二次过滤，
- 以避免混入其他表的列。
-- 对于大小写敏感的数据库，请确保配置的 schema/table 名称与数据库中实际标识符大小写一致。
+| 数据库类型 | 示例 |
+|------------|------|
+| MySQL | `sales.orders` |
+| PostgreSQL 和 SQL Server | `sales.public.orders` |
+| Oracle | `SALES.ORDERS` |
-1. **精确表路径**：使用 `table_path` 指定单个表及其完整路径。
- ```hocon
- table_path = "testdb.table1"
- ```
+`use_regex = false` 表示精确匹配，也是最安全的默认方式。只有明确需要把 `table_path` 的表名部分作为正则表达式时，才设置
`use_regex = true`：
-2. **正则表达式**：使用 `table_path` 与正则表达式模式匹配多个表。
- ```hocon
- table_path = "testdb.table\\d+" # 匹配 table1, table2, table3 等
- use_regex = true
- ```
-
-#### 表名的正则表达式支持
-
-JDBC 连接器支持使用正则表达式匹配多个表。此功能允许您使用单个源配置处理多个表。
-
-#### 配置
-
-要对表路径使用正则表达式匹配：
-
-1. 设置 `use_regex = true` 以启用正则表达式匹配
-2. 如果未设置 `use_regex` 或设置为 `false`，连接器将把 table_path 视为精确路径（无正则表达式匹配）
-
-#### 正则表达式语法注意事项
-
-- **路径分隔符**：点 (`.`) 被视为数据库、模式和表名之间的分隔符。
-- **转义点**：如果您需要在正则表达式中使用点 (`.`) 作为通配符来匹配任何字符，必须用反斜杠 (`\.`) 转义。
-- **路径格式**：对于 `database.table` 或 `database.schema.table`
之类的路径，最后一个未转义的点将表模式与数据库/模式模式分开。
-- **模式示例**：
- - `test.table\\d+` - 匹配 `test` 数据库中的 `table1`、`table2` 等表
- - `test.*` - 匹配 `test` 数据库中的所有表（用于整个数据库同步）
- - `postgres.public.test_db_\.*` - 匹配 `postgres` 数据库的 `public` 模式中以
`test_db_` 开头的所有表
-
-#### 示例
-
-```hocon
-source {
- Jdbc {
- url = "jdbc:mysql://localhost:3306/test"
- driver = "com.mysql.cj.jdbc.Driver"
- user = "root"
- password = "password"
-
- table_list = [
- {
- # 正则表达式匹配 - 匹配 test 数据库中的任何表
- table_path = "test.*"
- use_regex = true
- },
- {
- # 正则表达式匹配 - 匹配名称为 "user" 后跟数字的表
- table_path = "test.user\\d+"
- use_regex = true
- },
- {
- # 精确匹配 - 简单表名
- table_path = "test.config"
- # use_regex 未指定，默认为 false
- },
- ]
- }
-}
+```text
+table_path = "sales.orders_\\d+"
+use_regex = true
```
-#### 多表同步
+HOCON 字符串中的正则反斜杠需要转义，因此正则中的 `\d+` 在配置文件里要写成 `\\d+`。最后一个未转义的点用于分隔数据库/schema
路径和表名模式。
-使用正则表达式时，连接器将从所有匹配的表中读取数据。每个表将被独立处理，数据将在输出中合并。
-
-多表同步的示例配置：
-```hocon
-Jdbc {
- url = "jdbc:mysql://localhost/test"
- driver = "com.mysql.cj.jdbc.Driver"
- user = "root"
- password = "123456"
-
- # 使用显式配置的正则表达式
- table_list = [
- {
- table_path = "testdb.table\\d+"
- use_regex = true
- }
- ]
-}
-```
+许多 JDBC 驱动会把传给 `DatabaseMetaData` 的 schema 和表名参数当作 SQL `LIKE` 模式。SeaTunnel
会在发现元数据后再次精确核对标识符；对于大小写敏感的数据库，仍应确保配置与数据库中的真实标识符大小写完全一致。
### decimal_type_narrowing
@@ -241,109 +262,249 @@ int_type_narrowing = false
## 并行读取器
-JDBC 源连接器支持从表中并行读取数据。SeaTunnel 将使用某些规则分割表中的数据，这些数据将交给读取器进行读取。读取器的数量由
`parallelism` 选项确定。
+任务 `parallelism` 决定最多可以同时运行多少个 Reader；分片配置决定实际有多少个独立 split 可以分配给 Reader。
+
+### 推荐方式：`table_path` 动态分片
+
+整表快照建议配置 `table_path`，通常让 SeaTunnel 自动发现分片键。显式配置 `partition_column`
时优先使用该列；否则依次从主键和唯一索引中选择第一个受支持的列。支持 String、数值和 Date 类型的分片键。
+
+`split.size` 表示每个分片的目标行数，并不是严格的行数上限，实际大小会受到键值分布和数据库统计信息影响。如果表没有可用的主键、唯一索引或显式
`partition_column`，即使任务并行度大于 1，该表仍然只有一个读取分片。
-**分割键规则：**
+### 顶层 `query` 固定分区
-1. 如果 `partition_column` 不为 null，它将用于计算分割。该列必须在**支持的分割数据类型**中。
-2. 如果 `partition_column` 为 null，seatunnel
将从表中读取模式并获取主键和唯一索引。如果主键和唯一索引中有多个列，将使用**支持的分割数据类型**中的第一列来分割数据。例如，表有主键(nn guid,
name varchar)，因为 `guid` 不在**支持的分割数据类型**中，所以列 `name` 将用于分割数据。
+只有顶层同时配置 `query` 和 `partition_column` 时才会使用旧版 fixed splitter，再通过
`partition_num` 控制分片数。分区列必须包含在 query 结果中。配置上下界可以避免额外执行 `MIN`/`MAX`
查询，但错误的边界会漏读源数据，因此只有确认完整数据范围时才应设置。
-**支持的分割数据类型：**
-* String
-* Number(int, bigint, decimal, ...)
-* Date
+`table_list` 中的条目即使配置了 `query` 或分区参数，仍然使用动态分片。`split.size` 不影响顶层 fixed
partition 模式。
-## 提示
+### Query 与主键注意事项
-> 如果表无法分割（例如，表没有主键或唯一索引，且未设置 `partition_column`），它将以单并发运行。
->
-> 使用 `table_path` 替换 `query` 进行单表读取。如果需要读取多个表，请使用 `table_list`。
-> 当基于 `query` 推断主键时，主键继承自结果集中第一列所在的底层表；如果 `query` 包含多表 JOIN 或同时从多张表读取，该主键对整个
JOIN 结果集的唯一性不作严格保证。
+SeaTunnel 为 `query` 推断主键时，会继承结果集第一列所属底层表的元数据。对于 JOIN
或多表查询，该主键不保证在完整结果集中唯一。此类查询应使用单 Reader，或者显式选择能够安全划分查询结果的分区列。
-## 附录
+## 驱动参考
-以上参数有一些参考值。
+下表仅作为起点。部署前应向数据库厂商确认驱动制品、许可证、数据库版本兼容性和 Java 版本兼容性。
| 数据源 | 驱动 | URL
| Maven
|
|-------------|---------------------------------------------------|--------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------|
| mysql | com.mysql.cj.jdbc.Driver |
jdbc:mysql://localhost:3306/test |
https://mvnrepository.com/artifact/mysql/mysql-connector-java
|
| postgresql | org.postgresql.Driver |
jdbc:postgresql://localhost:5432/postgres |
https://mvnrepository.com/artifact/org.postgresql/postgresql
|
| dm | dm.jdbc.driver.DmDriver |
jdbc:dm://localhost:5236 |
https://mvnrepository.com/artifact/com.dameng/DmJdbcDriver18
|
+| phoenix | org.apache.phoenix.queryserver.client.Driver |
jdbc:phoenix:thin:url=http://localhost:8765;serialization=PROTOBUF |
https://mvnrepository.com/artifact/com.aliyun.phoenix/ali-phoenix-shaded-thin-client
|
| oracle | oracle.jdbc.OracleDriver |
jdbc:oracle:thin:@localhost:1521/xepdb1 |
https://mvnrepository.com/artifact/com.oracle.database.jdbc/ojdbc8
|
| sqlserver | com.microsoft.sqlserver.jdbc.SQLServerDriver |
jdbc:sqlserver://localhost:1433 |
https://mvnrepository.com/artifact/com.microsoft.sqlserver/mssql-jdbc
|
+| sqlite | org.sqlite.JDBC |
jdbc:sqlite:test.db |
https://mvnrepository.com/artifact/org.xerial/sqlite-jdbc
|
+| gbase8a | com.gbase.jdbc.Driver |
jdbc:gbase://localhost:5258/test |
https://cdn.gbase.cn/products/30/p5CiVwXBKQYIUGN8ecHvk/gbase-connector-java-9.5.0.7-build1-bin.jar
|
| starrocks | com.mysql.cj.jdbc.Driver |
jdbc:mysql://localhost:3306/test |
https://mvnrepository.com/artifact/mysql/mysql-connector-java
|
+| db2 | com.ibm.db2.jcc.DB2Driver |
jdbc:db2://localhost:50000/testdb |
https://mvnrepository.com/artifact/com.ibm.db2.jcc/db2jcc/db2jcc4
|
+| tablestore | com.alicloud.openservices.tablestore.jdbc.OTSDriver |
jdbc:ots:https://myinstance.cn-hangzhou.ots.aliyuncs.com/myinstance |
https://mvnrepository.com/artifact/com.aliyun.openservices/tablestore-jdbc
|
+| saphana | com.sap.db.jdbc.Driver |
jdbc:sap://localhost:39015 |
https://mvnrepository.com/artifact/com.sap.cloud.db.jdbc/ngdbc
|
+| doris | com.mysql.cj.jdbc.Driver |
jdbc:mysql://localhost:3306/test |
https://mvnrepository.com/artifact/mysql/mysql-connector-java
|
+| teradata | com.teradata.jdbc.TeraDriver |
jdbc:teradata://localhost/DBS_PORT=1025,DATABASE=test |
https://mvnrepository.com/artifact/com.teradata.jdbc/terajdbc
|
+| Snowflake | net.snowflake.client.jdbc.SnowflakeDriver |
jdbc:snowflake://<account_name>.snowflakecomputing.com |
https://mvnrepository.com/artifact/net.snowflake/snowflake-jdbc
|
+| Redshift | com.amazon.redshift.jdbc42.Driver |
jdbc:redshift://localhost:5439/testdb?defaultRowFetchSize=1000 |
https://mvnrepository.com/artifact/com.amazon.redshift/redshift-jdbc42
|
+| Vertica | com.vertica.jdbc.Driver |
jdbc:vertica://localhost:5433 |
https://repo1.maven.org/maven2/com/vertica/jdbc/vertica-jdbc/12.0.3-0/vertica-jdbc-12.0.3-0.jar
|
| kingbase | com.kingbase8.Driver |
jdbc:kingbase8://localhost:54321/db_test |
https://repo1.maven.org/maven2/cn/com/kingbase/kingbase8/8.6.0/kingbase8-8.6.0.jar
|
| oceanbase | com.oceanbase.jdbc.Driver |
jdbc:oceanbase://localhost:2881 |
https://repo1.maven.org/maven2/com/oceanbase/oceanbase-client/2.4.12/oceanbase-client-2.4.12.jar
|
| hive | org.apache.hive.jdbc.HiveDriver |
jdbc:hive2://localhost:10000 |
https://repo1.maven.org/maven2/org/apache/hive/hive-jdbc/3.1.3/hive-jdbc-3.1.3-standalone.jar
|
+| xugu | com.xugu.cloudjdbc.Driver |
jdbc:xugu://localhost:5138 |
https://repo1.maven.org/maven2/com/xugudb/xugu-jdbc/12.2.0/xugu-jdbc-12.2.0.jar
|
+| InterSystems IRIS | com.intersystems.jdbc.IRISDriver |
jdbc:IRIS://localhost:1972/%SYS |
https://raw.githubusercontent.com/intersystems-community/iris-driver-distribution/main/JDBC/JDK18/intersystems-jdbc-3.8.4.jar
|
+| opengauss | org.opengauss.Driver |
jdbc:opengauss://localhost:5432/postgres |
https://repo1.maven.org/maven2/org/opengauss/opengauss-jdbc/5.1.0-og/opengauss-jdbc-5.1.0-og.jar
|
+| Highgo | com.highgo.jdbc.Driver |
jdbc:highgo://localhost:5866/highgo |
https://repo1.maven.org/maven2/com/highgo/HgdbJdbc/6.2.3/HgdbJdbc-6.2.3.jar
|
+| Presto | com.facebook.presto.jdbc.PrestoDriver |
jdbc:presto://localhost:8080/presto |
https://repo1.maven.org/maven2/com/facebook/presto/presto-jdbc/0.279/presto-jdbc/0.279.jar
|
+| Trino | io.trino.jdbc.TrinoDriver |
jdbc:trino://localhost:8080/trino |
https://repo1.maven.org/maven2/io/trino/trino-jdbc/460/trino-jdbc-460.jar
|
| YashanDB | com.yashandb.jdbc.Driver |
jdbc:yasdb://localhost:1688/SYS |
https://mvnrepository.com/artifact/com.yashandb/yashandb-jdbc
|
-## 示例
+## 常用模式
-### 简单
+### 自定义查询
-#### 情况 1
+```hocon
+env {
+ parallelism = 1
+ job.mode = "BATCH"
+}
-```
-Jdbc {
- url = "jdbc:mysql://localhost/test?serverTimezone=GMT%2b8"
+source {
+ Jdbc {
+ url = "jdbc:mysql://mysql.example.com:3306/sales"
driver = "com.mysql.cj.jdbc.Driver"
- connection_check_timeout_sec = 100
- user = "root"
- password = "123456"
- query = "select * from type_bin"
+ username = "seatunnel_reader"
+ password = "change_me"
+ query = "SELECT id, customer_name, amount FROM orders WHERE status =
'PAID'"
+ }
+}
+
+sink {
+ Console {}
}
```
-#### 情况 2 在动态块分割阶段使用 select count(*) 代替分析表来计算表行数
+### Oracle BLOB 读取为 STRING
+需要把 Oracle BLOB 暴露为 SeaTunnel STRING 时，可以设置 `handle_blob_as_string =
true`，例如后续需要写入 Doris 的场景。
+
+```hocon
+env {
+ parallelism = 1
+ job.mode = "BATCH"
+}
+
+source {
+ Jdbc {
+ driver = oracle.jdbc.driver.OracleDriver
+ url = "jdbc:oracle:thin:@oracle.example.com:1521/SERVICE_NAME"
+ username = "seatunnel_reader"
+ password = "change_me"
+ query = "SELECT ID, NAME, CONTENT_BLOB FROM MY_TABLE"
+ handle_blob_as_string = true # 为 Oracle 启用 BLOB 到字符串转换
+ }
+}
+
+sink {
+ Console {}
+}
```
-Jdbc {
- url = "jdbc:mysql://localhost/test?serverTimezone=GMT%2b8"
+
+### 按列并行执行自定义 query
+
+```hocon
+env {
+ parallelism = 10
+ job.mode = "BATCH"
+}
+
+source {
+ Jdbc {
+ url = "jdbc:mysql://mysql.example.com:3306/sales?serverTimezone=UTC"
driver = "com.mysql.cj.jdbc.Driver"
- connection_check_timeout_sec = 100
- user = "root"
- password = "123456"
- use_select_count = true
- query = "select * from type_bin"
+ username = "seatunnel_reader"
+ password = "change_me"
+ query = "SELECT id, customer_name, amount FROM orders"
+ partition_column = "id"
+ partition_num = 10
+ }
+}
+
+sink {
+ Console {}
}
```
-#### 情况 3 使用 select NUM_ROWS from all_tables 获取表行数但跳过分析表
+### 显式配置分区边界
-```
-Jdbc {
- url = "jdbc:mysql://localhost/test?serverTimezone=GMT%2b8"
+只有上下界能够覆盖完整源数据范围时才应设置边界。SeaTunnel 不会读取配置区间之外的值。
+
+```hocon
+env {
+ parallelism = 10
+ job.mode = "BATCH"
+}
+
+source {
+ Jdbc {
+ url = "jdbc:mysql://mysql.example.com:3306/sales?serverTimezone=UTC"
driver = "com.mysql.cj.jdbc.Driver"
- connection_check_timeout_sec = 100
- user = "root"
- password = "123456"
- skip_analyze = true
- query = "select * from type_bin"
+ username = "seatunnel_reader"
+ password = "change_me"
+ query = "SELECT id, customer_name, amount FROM orders"
+ partition_column = "id"
+ partition_lower_bound = 1
+ partition_upper_bound = 500
+ partition_num = 10
+ }
+}
+
+sink {
+ Console {}
}
```
-#### 情况 4 Oracle 源与 BLOB 作为字符串到 Doris Sink
+### 通过主键或唯一索引动态分片
+
+本示例只配置 `table_path`，没有同时配置顶层 `query +
partition_column`，因此使用动态分片。建议先使用默认分片参数，再根据源库负载和任务吞吐实测结果调整 `split.size`。
+
+```hocon
+env {
+ parallelism = 10
+ job.mode = "BATCH"
+}
-此示例演示了在传输到 Doris 时如何将 Oracle 的 BLOB 数据作为字符串处理。这对于大型 BLOB 字段很有用。
+source {
+ Jdbc {
+ url = "jdbc:mysql://mysql.example.com:3306/sales?serverTimezone=UTC"
+ driver = "com.mysql.cj.jdbc.Driver"
+ username = "seatunnel_reader"
+ password = "change_me"
+ table_path = "sales.orders"
+ split.size = 10000
+ }
+}
+sink {
+ Console {}
+}
```
+
+### 读取多张表
+
+不同表需要不同 query 或匹配规则时使用 `table_list`：
+
+```hocon
env {
- parallelism = 1
+ parallelism = 4
job.mode = "BATCH"
}
source {
Jdbc {
- driver = oracle.jdbc.driver.OracleDriver
- url = "jdbc:oracle:thin:@oracle_host:1521/SERVICE_NAME"
- user = "username"
- password = "password"
- query = "SELECT ID, NAME, CONTENT_BLOB FROM MY_TABLE"
- handle_blob_as_string = true # 为 Oracle 启用 BLOB 到字符串转换
+ url = "jdbc:mysql://mysql.example.com:3306/sales?serverTimezone=UTC"
+ driver = "com.mysql.cj.jdbc.Driver"
+ username = "seatunnel_reader"
+ password = "change_me"
+
+ table_list = [
+ {
+ table_path = "sales.orders"
+ },
+ {
+ table_path = "sales.customers"
+ query = "SELECT id, name FROM customers WHERE id > 100"
+ },
+ {
+ table_path = "sales.archive_\\d+"
+ use_regex = true
+ }
+ ]
}
}
+
+sink {
+ Console {}
+}
```
+## 故障排查
+
+### 找不到 JDBC 驱动类
+
+确认每个执行节点的引擎对应目录中都有驱动 JAR，配置的 `driver` 类真实存在于该 JAR，并且添加 JAR 后已经重启受影响的进程。
+
+### SQL 客户端能连接，但 SeaTunnel 连接失败
+
+主机名必须能从 SeaTunnel 进程所在环境访问，而不只是能从个人电脑访问。检查网络路由、防火墙、TLS、账号、数据库名和
`connection_check_timeout_sec`，不要直接使用未替换的示例主机名。
+
+### 无法发现表或字段
+
+检查数据库对应的 `table_path` 格式、标识符大小写，以及账号是否具有元数据和 `SELECT` 权限。使用自定义 query
时，先在数据库客户端中用同一账号执行完全相同的 SQL。
+
+### 提高 parallelism 后 Reader 数量没有增加
+
+使用动态分片时，确认表存在受支持的主键或唯一索引，或者显式配置 `partition_column`。要让顶层 `query` 使用旧版 fixed
splitter，需要同时配置 `partition_column` 和合适的 `partition_num`。没有安全分片键的表会有意使用单个 split
读取。
+
+### 配置分区边界后出现漏数
+
+`partition_lower_bound` 和 `partition_upper_bound` 定义 SeaTunnel
实际读取的数据范围，并不只是性能提示。可以删除它们，让 SeaTunnel 自动发现边界；也可以修正边界，确保覆盖所有需要读取的行。
+
## 变更日志
<ChangeLog />
