sandynz opened a new issue #13875: URL: https://github.com/apache/shardingsphere/issues/13875
Hi community, Scaling is an individual tool before, we've integrated scaling with proxy and removed scaling bootstrap and distribution in #13276, but we didn't refactor scaling. We are planning to make scaling as part of Apache ShardingSphere [3 layers architecture](https://shardingsphere.apache.org/document/current/en/concepts/pluggable/#implementation ). After several times of discussion and review by @tristaZero and @terrymanu , we've got the final design. In general, it includes following parts: - Modules design - Configuration design Package name and class name refactoring won't be included here. We'll refactor it in several stages. ## Modules design ### Origianl related modules shardingsphere - shardingsphere-scaling (include all scaling features) ### Current related modules shardingsphere - shardingsphere-kernel (original module) - shardingsphere-schedule (new module) - shardingsphere-schedule-core : scheduling core, facade of ElasticJob - shardingsphere-migration-common : fundamental functions for migration scenarios: scaling, data encryption - shardingsphere-migration-distsql : common DistSQL for migration scenarios - shardingsphere-migration-sql-core : dialect SPI interfaces - shardingsphere-migration-sql-dialect : migration dialect implementation - shardingsphere-migration-mysql - shardingsphere-migration-postgresql - shardingsphere-migration-opengauss - shardingsphere-cdc : change data capture - shardingsphere-cdc-core : data producer and data channel, `Dumper` and `Channel` in scaling - shardingsphere-cdc-dialect - shardingsphere-cdc-mysql - shardingsphere-cdc-postgresql - shardingsphere-cdc-opengauss - shardingsphere-features (original module) - shardingsphere-encrypt (original module) - shardingsphere-encrypt-api : encrypt rule configuration upgrade - shardingsphere-encrypt-core : add encryption job here - shardingsphere-sharding (original module) - shardingsphere-sharding-api : sharding rule configuration upgrade - shardingsphere-sharding-core : move scaling job here, no shard could be considered as one shard rule - shardingsphere-scaling (original module, disappeared after refactoring) In summary, `shardingsphere-scaling` will be extracted into: - `shardingsphere-kernel`: most code into `shardingsphere-migration-xxx` and `shardingsphere-cdc`, scheduling into `shardingsphere-schedule-core`, which is in L1 and L3 layer. - `shardingsphere-sharding` and `shardingsphere-encrypt`: scenarios implementation (scaling and data encryption) based on underlying fundament, which is in L2 layer. ## Configuration design ### Original configuration in `server.yaml` ```yaml scaling: blockQueueSize: 10000 workerThread: 40 clusterAutoSwitchAlgorithm: type: IDLE props: incremental-task-idle-minute-threshold: 30 dataConsistencyCheckAlgorithm: type: DEFAULT ``` ### Current configuration in `config-sharding.yaml`, `migratorName` and `migrators` is added: ```yaml keyGenerators: snowflake: type: SNOWFLAKE props: worker-id: 123 migratorName: default_migrator migrators: default_migrator: blockQueueSize: 10000 workerThread: 40 resourceSwitcher: type: IDLE props: incremental-task-idle-minute-threshold: 30 dataConsistencyChecker: type: CRC32 props: chunk-size: 100000 ignored-column-names: tbl1.blob1,tbl1.clob1 ``` in `config-encrypt.yaml`, `migratorName` and `migrators` is added: ```yaml encryptors: aes_encryptor: type: AES props: aes-key-value: 123456abc md5_encryptor: type: MD5 migratorName: default_migrator migrators: default_migrator: blockQueueSize: 10000 workerThread: 40 resourceSwitcher: type: IDLE props: incremental-task-idle-minute-threshold: 30 dataConsistencyChecker: type: DATA_MATCH props: chunk-size: 10000 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
