MartijnVisser commented on code in PR #220:
URL:
https://github.com/apache/flink-connector-jdbc/pull/220#discussion_r4377202554
##########
docs/content/docs/connectors/table/jdbc.md:
##########
@@ -38,8 +38,32 @@ The JDBC sink operate in upsert mode for exchange
UPDATE/DELETE messages with th
Dependencies
------------
+Since version 4.0 the JDBC connector is no longer published as a single
artifact. It is split into
+`flink-connector-jdbc-core`, which contains the shared runtime, and one
artifact per supported
+database, which contains the dialect and the catalog for that database. You
need the artifact of the
+database you are connecting to; it pulls in `flink-connector-jdbc-core`
transitively.
+
+| Database | Connector artifact |
+|:-----------|:----------------------------------|
+| MySQL | `flink-connector-jdbc-mysql` |
+| Oracle | `flink-connector-jdbc-oracle` |
+| PostgreSQL | `flink-connector-jdbc-postgres` |
+| SQL Server | `flink-connector-jdbc-sqlserver` |
+| CrateDB | `flink-connector-jdbc-cratedb` |
+| Db2 | `flink-connector-jdbc-db2` |
+| Trino | `flink-connector-jdbc-trino` |
+| OceanBase | `flink-connector-jdbc-oceanbase` |
+| Spanner | `flink-connector-jdbc-spanner` |
+| Derby | `flink-connector-jdbc-core` |
+
{{< sql_connector_download_table "jdbc" >}}
+None of these artifacts is a self-contained SQL uber jar. When using the SQL
Client or a session
+cluster, put both `flink-connector-jdbc-core` and the artifact for your
database into the `lib/`
Review Comment:
`flink-connector-jdbc-core` doesn't bundle `openlineage-sql-java` and
`openlineage-java`, and without them `JdbcSource.getLineageVertex()` throws
`NoClassDefFoundError` when the job is built. They need to go into `lib/` as
well.
##########
docs/content/docs/connectors/datastream/jdbc.md:
##########
@@ -463,63 +456,44 @@ Furthermore, XA needs to be enabled and/or configured in
some databases.
For PostgreSQL, you should set `max_prepared_transactions` to some value
greater than zero.
For MySQL v8+, you should grant `XA_RECOVER_ADMIN` to Flink DB user.
-**ATTENTION:** Currently, `JdbcSink.exactlyOnceSink` can ensure exactly once
semantics
+**ATTENTION:** Currently, the exactly-once sink can ensure exactly once
semantics
with `JdbcExecutionOptions.maxRetries == 0`; otherwise, duplicated results
maybe produced.
### `XADataSource` examples
PostgreSQL `XADataSource` example:
-{{< tabs "4ab65f13-608a-411a-8d24-e304f323ab3a" >}}
-{{< tab "Java" >}}
```java
PGXADataSource xaDataSource = new org.postgresql.xa.PGXADataSource();
xaDataSource.setUrl("jdbc:postgresql://localhost:5432/postgres");
xaDataSource.setUser(username);
xaDataSource.setPassword(password);
```
-{{< /tab >}}
-{{< tab "Python" >}}
-```python
-Still not supported in Python API.
-```
-{{< /tab >}}
-{{< /tabs >}}
MySQL `XADataSource` example:
-{{< tabs "4ab65f13-608a-411a-8d24-b213f323ca3c" >}}
-{{< tab "Java" >}}
```java
MysqlXADataSource xaDataSource = new com.mysql.cj.jdbc.MysqlXADataSource();
xaDataSource.setUrl("jdbc:mysql://localhost:3306/");
xaDataSource.setUser(username);
xaDataSource.setPassword(password);
```
-{{< /tab >}}
-{{< tab "Python" >}}
-```python
-Still not supported in Python API.
-```
-{{< /tab >}}
-{{< /tabs >}}
Oracle `XADataSource` example:
-{{< tabs "4ab65f13-608a-411a-8d24-b213f337ad5f" >}}
-{{< tab "Java" >}}
```java
OracleXADataSource xaDataSource = new oracle.jdbc.xa.OracleXADataSource();
xaDataSource.setURL("jdbc:oracle:oci8:@");
xaDataSource.setUser("scott");
xaDataSource.setPassword("tiger");
```
-{{< /tab >}}
-{{< tab "Python" >}}
-```python
-Still not supported in Python API.
-```
-{{< /tab >}}
-{{< /tabs >}}
Please also take Oracle connection pooling into account.
-Please refer to the `JdbcXaSinkFunction` documentation for more details.
+## Lineage
+
+`JdbcSource` and `JdbcSink` expose lineage information as described in
+[FLIP-314](https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener),
+as do the Table API source and lookup function. No configuration is needed on
the connector side; the
+information is delivered to any `JobStatusChangedListener` registered in the
cluster.
+
+The dataset namespace is derived from the JDBC URL and the dataset name from
the queries the job
Review Comment:
Only for a `SimpleJdbcConnectionProvider`. With `buildExactlyOnce` or a
custom provider the namespace is empty, I checked with a `PGXADataSource` sink.
Can you mention that?
##########
docs/content/docs/connectors/table/jdbc.md:
##########
@@ -480,15 +521,44 @@ Please refer to [Dependencies](#dependencies) section for
how to setup a JDBC co
The JDBC catalog supports the following options:
- `name`: required, name of the catalog.
-- `default-database`: required, default database to connect to.
- `username`: required, username of database account.
- `password`: required, password of the account.
-- `base-url`: required (should not contain the database name)
+- `base-url`: required
- for Postgres Catalog this should be `"jdbc:postgresql://<ip>:<port>"`
- for MySQL Catalog this should be `"jdbc:mysql://<ip>:<port>"`
- for OceanBase Catalog this should be `"jdbc:oceanbase://<ip>:<port>"`
- for Spanner Catalog this should be
`"jdbc:cloudspanner:/projects/<project>/instances/<instance>/databases/"`
-- `compatible-mode`: optional, the compatible mode of database.
+- `default-database`: optional, default database to connect to. Required for
the Spanner Catalog.
+- `compatible-mode`: optional, the compatible mode of database. Only OceanBase
supports this option,
+ see [Connector Options](#connector-options).
+
+Except for the Spanner Catalog, the database the catalog connects to has to be
identifiable from
+`base-url`, `default-database`, or both. Any one of the following is valid:
+
+- `default-database` is set and `base-url` carries no database name, for
example
+ `'base-url' = 'jdbc:postgresql://localhost:5432'` with `'default-database' =
'mydb'`
+- `base-url` carries the database name and `default-database` is omitted, for
example
Review Comment:
Without a query string this form pins the catalog to `mydb`, since
`urlFunction` returns the base-url for every database. I think it makes sense
to mention that.
##########
docs/content/docs/connectors/datastream/jdbc.md:
##########
@@ -93,12 +160,16 @@ remains unchanged in the whole lifecycle of
`JdbcSourceSplit` processing.
Unfortunately, this condition is not met in most databases and data scenarios.
See
[FLIP-239](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=217386271)
for more details.
+Using `DeliveryGuarantee.EXACTLY_ONCE` requires `setResultSetType` to be either
+`ResultSet.TYPE_SCROLL_INSENSITIVE` or `ResultSet.CONCUR_READ_ONLY`; other
values fail at `build()`
Review Comment:
`CONCUR_READ_ONLY` is a concurrency constant, not a result set type. The
check in `JdbcSourceBuilder.build()` compares against the wrong constant, so I
would only document `TYPE_SCROLL_INSENSITIVE` here.
##########
docs/content/docs/connectors/datastream/jdbc.md:
##########
@@ -463,63 +456,44 @@ Furthermore, XA needs to be enabled and/or configured in
some databases.
For PostgreSQL, you should set `max_prepared_transactions` to some value
greater than zero.
For MySQL v8+, you should grant `XA_RECOVER_ADMIN` to Flink DB user.
-**ATTENTION:** Currently, `JdbcSink.exactlyOnceSink` can ensure exactly once
semantics
+**ATTENTION:** Currently, the exactly-once sink can ensure exactly once
semantics
Review Comment:
This doesn't risk duplicates, `JdbcWriter` fails at startup for exactly-once
when `maxRetries` isn't 0. Since the default is 3, please say it has to be set
to 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
