Hisoka-X commented on code in PR #5581:
URL: https://github.com/apache/seatunnel/pull/5581#discussion_r1366538860
##########
docs/en/concept/connector-v2-features.md:
##########
@@ -49,6 +49,10 @@ In the **Parallelism Source Connector**, the source will be
split into multiple
User can config the split rule.
+### support multiple table reading
Review Comment:
```suggestion
### Support multiple table reading
```
##########
docs/en/concept/connector-v2-features.md:
##########
@@ -49,6 +49,10 @@ In the **Parallelism Source Connector**, the source will be
split into multiple
User can config the split rule.
+### support multiple table reading
+
+Supports reading multiple tables at one time
Review Comment:
```suggestion
Supports reading multiple tables in one SeaTunnel job
```
##########
docs/en/connector-v2/source/Jdbc.md:
##########
@@ -25,25 +25,33 @@ supports query SQL and can achieve projection effect.
- [x] [parallelism](../../concept/connector-v2-features.md)
- [x] [support user-defined split](../../concept/connector-v2-features.md)
+- [x] [support multiple table reading](../../concept/connector-v2-features.md)
## Options
-| name | type | required | default value |
-|------------------------------|--------|----------|-----------------|
-| url | String | Yes | - |
-| driver | String | Yes | - |
-| user | String | No | - |
-| password | String | No | - |
-| query | String | Yes | - |
-| compatible_mode | String | No | - |
-| connection_check_timeout_sec | Int | No | 30 |
-| partition_column | String | No | - |
-| partition_upper_bound | Long | No | - |
-| partition_lower_bound | Long | No | - |
-| partition_num | Int | No | job parallelism |
-| fetch_size | Int | No | 0 |
-| properties | Map | No | - |
-| common-options | | No | - |
+| name | type | required | default
value |
+|--------------------------------------------|--------|----------|-----------------|
+| url | String | Yes | -
|
+| driver | String | Yes | -
|
+| user | String | No | -
|
+| password | String | No | -
|
+| query | String | No | -
|
+| compatible_mode | String | No | -
|
+| connection_check_timeout_sec | Int | No | 30
|
+| partition_column | String | No | -
|
+| partition_upper_bound | Long | No | -
|
+| partition_lower_bound | Long | No | -
|
+| partition_num | Int | No | job
parallelism |
+| fetch_size | Int | No | 0
|
+| properties | Map | No | -
|
+| table_path | String | No | -
|
+| table_list | Array | No | -
|
+| split.size | Int | No | 8096
|
+| split.even-distribution.factor.lower-bound | Double | No | 0.05
|
+| split.even-distribution.factor.upper-bound | Double | No | 100
|
+| split.sample-sharding.threshold | Int | No | 1000
|
+| split.inverse-sampling.rate | Int | No | 1000
|
+| common-options | | No | -
|
Review Comment:
This is a new feature or just document missed? I think it's better create a
single PR for this.
##########
seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/config/JdbcOptions.java:
##########
@@ -67,7 +67,7 @@ public interface JdbcOptions {
Option<Integer> FETCH_SIZE =
Options.key("fetch_size")
.intType()
- .defaultValue(0)
+ .defaultValue(1024)
.withDescription(
"For queries that return a large number of
objects, "
Review Comment:
Please do not change this value. `0` mean use default value of jdbc driver.
Use `1024` as default value could affect some db read performance.
##########
docs/en/connector-v2/source/Jdbc.md:
##########
@@ -98,6 +106,54 @@ improve performance by reducing the number database hits
required to satisfy the
Additional connection configuration parameters,when properties and URL have
the same parameters, the priority is determined by the <br/>specific
implementation of the driver. For example, in MySQL, properties take precedence
over the URL.
+### table_path
+
+The path to the full path of table, you can use this configuration instead of
`query`.
+
+examples:
+- mysql: "testdb.table1"
+- oracle: "test_schema.table1"
+- sqlserver: "testdb.test_schema.table1"
+- postgresql: "testdb.test_schema.table1"
+
+### table_list
+
+The list of tables to be read, you can use this configuration instead of
`table_path`
+
+example
+
+```hocon
+table_list = [
+ {
+ table_path = "testdb.table1"
+ }
+ {
+ table_path = "testdb.table2"
+ query = "select * from testdb.table2 where id > 100"
+ }
+]
+```
Review Comment:
If we use multiple tables feature in jdbc source, how to deal with config
only for single table like: `partition_column`, `partition_upper_bound` etc? I
think you should share the behavior to the doc which when use multiple tables.
##########
seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/catalog/dm/DamengCatalog.java:
##########
@@ -148,4 +152,23 @@ protected String getUrlFromDatabaseName(String
databaseName) {
protected String getOptionTableName(TablePath tablePath) {
return tablePath.getSchemaAndTableName();
}
+
+ @Override
+ public boolean tableExists(TablePath tablePath) throws CatalogException {
+ try {
+ if (StringUtils.isNotBlank(tablePath.getDatabaseName())) {
+ return databaseExists(tablePath.getDatabaseName())
+ && listTables(tablePath.getDatabaseName())
+ .contains(tablePath.getSchemaAndTableName());
+ }
+ return listTables().contains(tablePath.getSchemaAndTableName());
+ } catch (DatabaseNotExistException e) {
+ return false;
+ }
+ }
Review Comment:
Shoud care about case?
##########
seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/catalog/oracle/OracleCatalog.java:
##########
@@ -223,4 +231,41 @@ protected String getUrlFromDatabaseName(String
databaseName) {
protected String getOptionTableName(TablePath tablePath) {
return tablePath.getSchemaAndTableName();
}
+
+ @Override
+ public boolean tableExists(TablePath tablePath) throws CatalogException {
+ try {
+ if (StringUtils.isNotBlank(tablePath.getDatabaseName())) {
+ return databaseExists(tablePath.getDatabaseName())
+ && listTables(tablePath.getDatabaseName())
+ .contains(tablePath.getSchemaAndTableName());
+ }
+ return listTables().contains(tablePath.getSchemaAndTableName());
+ } catch (DatabaseNotExistException e) {
+ return false;
+ }
+ }
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
