sandynz opened a new issue, #22500:
URL: https://github.com/apache/shardingsphere/issues/22500
Hi community,
We've planned data synchronization feature before, and we received feature
request of CDC (change data capture) from community recently, we'll add CDC
feature at next weeks.
CDC feature could be used for ingesting data from Apache ShardingSphere
proxy to OLTP, OLAP and MQ system etc.
After several times of discussion with @azexcy and @justbk2015 , review by
@tristaZero and @terrymanu , we've got a design for current stage and welcome
to feedback.
## Components
- CDC server
- CDC protocol
- CDC client
## CDC server
Supply incremental data. And inventory data is optional.
Part of tasks:
1. Add CDC configuration in`server.yaml`
```
cdc:
enabled: false
port: 33071
```
2. Start CDC server and listen on configured port
3. CDC feature is part of pipeline. pipeline-cdc core modules are in L1
layer. New sub-modules will be created.
4. Part of common classes in pipeline-core module will be refactored for CDC
## CDC protocol
Protocol buffers will be used for CDC server and client, for good
performance and easy serialization.
## CDC client
Common client classes will be implemented here, e.g.
- Network operation
- Protocol serialization
- API operation
- Basic interface of importing data to target system, it's extensible for
easier integration with existing ETL tools
## New modules
- data-pipeline-cdc
- data-pipeline-cdc-protocol
- data-pipeline-cdc-core
- data-pipeline-cdc-client
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail:
[email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
