[
https://issues.apache.org/jira/browse/FLINK-31820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17719641#comment-17719641
]
xingyuan cheng edited comment on FLINK-31820 at 5/5/23 6:03 AM:
----------------------------------------------------------------
[~martijnvisser] Hello, sorry for the late reply after the May Day holiday. I
did some simple research. At present, in the case of many single databases, in
order to reduce performance loss, some domestic companies adopt the scheme of
sub-database and sub-table, and foreign companies adopt schemes such as
distributed database. Regarding the difference between domestic and foreign
use, the reason for using sub-database and sub-table is because the storage
medium of the stock data has been determined, and the cost of migrating to a
distributed database is too high, and the enterprise business cannot tolerate
it. Therefore, it is necessary to transform the connector to support
sub-database and sub-table.
The reference given by [~Thesharing] is a powerful explanation of mysql
sub-database and sub-table. I will update the documentation in the near future,
and explain this logic in the form of a flow chart.
was (Author: complone):
[~martijnvisser] Hello, sorry for the late reply after the May Day holiday. I
did some simple research. At present, in the case of many single databases,
some domestic companies adopt the scheme of sub-database and sub-table, and
foreign companies adopt solutions such as distributed databases. For the
difference between domestic and foreign use, the reason for using sub-database
and sub-table is because the storage medium of the stock data has been
determined, and the cost of migrating to a distributed database is too high,
and the enterprise business cannot tolerate it, so it is necessary to transform
the connector to support sub-database sub table.
The reference given by [~Thesharing] is a powerful explanation of mysql
sub-database and sub-table. I will update the documentation in the near future,
and explain this logic in the form of a flow chart.
> Support data source sub-database and sub-table
> ----------------------------------------------
>
> Key: FLINK-31820
> URL: https://issues.apache.org/jira/browse/FLINK-31820
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / JDBC
> Reporter: xingyuan cheng
> Priority: Major
> Labels: pull-request-available
>
> At present, apache/flink-connector-jdbc does not support sub-database and
> table sub-database. Now three commonly used databases Mysql, Postgres and
> Oracle support sub-database and sub-table
>
> Taking oracle as an example, users only need to configure the following
> format to use
>
> {code:java}
> create table oracle_source (
> EMPLOYEE_ID BIGINT,
> START_DATE TIMESTAMP,
> END_DATE TIMESTAMP,
> JOB_ID VARCHAR,
> DEPARTMENT_ID VARCHAR
> ) with (
> type = 'oracle',
> url =
> 'jdbc:oracle:thin:@//localhost:3306/order_([0-9]{1,}),jdbc:oracle:thin:@//localhost:3306/order_([0-9]{1,})',
> userName = 'userName',
> password = 'password',
> dbName = 'hr',
> table-name = 'order_([0-9]{1,})',
> timeField = 'START_DATE',
> startTime = '2007-1-1 00:00:00'
> ); {code}
> In the above code, the dbName attribute corresponds to the schema-name
> attribute in oracle or postgres, and the mysql database needs to manually
> specify the dbName
>
> At the same time, I am also developing the CDAS whole database
> synchronization syntax for the company, and the data source supports
> sub-database and table as part of it. Add unit tests. For now, please keep
> this PR in draft status.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
