Re: [DISCUSS] Add Apache ShardingSphere as the server storage for Apache SkyWalking
Hi Juan Pan I am updating the 8.0.0 core in the PR[1]. I removed the `-day`, `-month`, `-hour` tables. And *_inventory tables have been removed, with several metadata tables. Please recheck the storage, the config file is required to change. [1] https://github.com/apache/skywalking/pull/4599 Sheng Wu 吴晟 Twitter, wusheng1108 Sheng Wu 于2020年3月16日周一 上午8:57写道： > Thinks happen quickly, I think that PR is going to be merged soon. > You could wait 2-3 days. > > Sheng Wu 吴晟 > Twitter, wusheng1108 > > > Juan Pan 于2020年3月15日周日 下午3:39写道： > >> Hi, >> >> >> Thanks for your kind remainder. >> Actually i am on the way writing e2e tests for ShardingSphere. >> Considering this PR[1] has not been merged and the discussion is underway, >> do you think i should stop my current work (Based on the old one) and >> wait for your progress? >> >> >> Cheers, >> Trista >> >> >> Juan Pan (Trista) >> >> Senior DBA & PPMC of Apache ShardingSphere(Incubating) >> E-mail: panj...@apache.org >> >> >> >> >> On 03/15/2020 14:08，kezhenxu94 wrote： >> Hi Juan Pan, we've noticed that the E2E tests are now very unfriendly for >> new contributors, thus we have rewrote the tests now and, from my point of >> view, if you're familiar with Docker and docker compose, you should be >> quick to get started with the new E2E framework, please take a look at this >> pull request https://github.com/apache/skywalking/pull/4510 and check >> out the branch locally to play around it, and I hope to get feedback from >> E2E contributors like you whether it improves, compared with the old one, >> thanks. >> >> >> >> >> >> >> -- >> >> GitHub @kezhenxu94 >> Apache SkyWalking, Apache Dubbo >> >> >> >> At 2020-03-15 13:42:55, "Juan Pan" wrote: >> Hi community, >> >> >> Update task status and raise my questions. Could I get your bits of help? >> >> >> According to our schedule, the 1st to 3rd sub-tasks have been completed >> already[1]. Currently, I am working on sub-task 4th and 5th, i.e., >> 4. Provide configuration of Sharding Proxy[2] for Skywalking. >> 5. Provide e2e tests. >> >> >> Here are my questions, >> 1. `rc0-prepare.sh` is used only for one `image` of >> `docker-maven-plugin`, right? If so, what should i do if i plan to add >> `rc0-prepare.sh` for `shardingsphere/sharding-proxy` image (Not for >> `skyapm/e2e-container` image). >> 2. Who will call `rc0-prepare.sh` and when? >> >> >> I spent much time on learning more `e2e-test` and docker image from other >> e2e modules, which seems a blocker to me to work on sub-task 5th. Could >> anyone give me some ideas? Thanks in advance. :) >> >> >> [1] https://github.com/apache/skywalking/pull/4421 >> >> >> Juan Pan (Trista) >> >> Senior DBA & PPMC of Apache ShardingSphere(Incubating) >> E-mail: panj...@apache.org >> >> >> >> >> On 02/16/2020 18:36，Juan Pan wrote： >> Hi fellows, >> >> >> I am working on providing ShardingSphere[1] as one of the server storages >> for Skywalking. >> ShardingSphere will help shard the data collected by Skywalking into >> different databases(Popular RDBMS, like MySQL, PostgreSQL and so on) and >> tables to seek better query performance and do scale-out of database. >> >> >> Since it is not an easy job, and needs some helps from community, i put >> task schedule and discussion items here to welcome your comments and >> suggestions. >> If you are interested in this one, welcome to be my partner! :-) >> >> >> >> Here is the schedule, >> 1. Acquire all required tables and query SQLs from db logs, like general >> log of MySQL. [DONE] >> 2. Analyse those tables and query SQLs to explore appropriate sharding >> strategy.[DOING] >> 3. Provide class implementation of `PreciseShardingAlgorithm` and >> `RangeShardingAlgorithm` of ShardingSphere to do sharding.[TODO] >> 4. Provide configuration of Sharding Proxy[2] for Skywalking.[TODO] >> 5. Provide Acceptance test(maybe e2e tests of Skywalking?).[TODO] >> 6. Update readme[3].[TODO] >> 7. Update storage doc[4].[TODO] >> - >> >> >> It seems a long-term work, and i will do it step by step. Any progress >> will be updated here, also ISSUE [5] will give you thread of PRs and >> discussion detail. >> >> >> [1] https://shardingsphere.apache.org/ >> [2] >> https://shardingsphere.apache.org/document/current/en/quick-start/sharding-proxy-quick-start/ >> [3] https://github.com/apache/skywalking/tree/master/install >> [4 ] >> https://github.com/apache/skywalking/blob/master/docs/en/setup/backend/backend-storage.md >> [5] https://github.com/apache/skywalking/issues/4131 >> >> >> >> >> >> Juan Pan (Trista) >> >> Senior DBA & PPMC of Apache ShardingSphere(Incubating) >> E-mail: panj...@apache.org >> >> >> >>
Re: [DISCUSS] Add Apache ShardingSphere as the server storage for Apache SkyWalking
Thinks happen quickly, I think that PR is going to be merged soon. You could wait 2-3 days. Sheng Wu 吴晟 Twitter, wusheng1108 Juan Pan 于2020年3月15日周日 下午3:39写道： > Hi, > > > Thanks for your kind remainder. > Actually i am on the way writing e2e tests for ShardingSphere. Considering > this PR[1] has not been merged and the discussion is underway, > do you think i should stop my current work (Based on the old one) and wait > for your progress? > > > Cheers, > Trista > > > Juan Pan (Trista) > > Senior DBA & PPMC of Apache ShardingSphere(Incubating) > E-mail: panj...@apache.org > > > > > On 03/15/2020 14:08，kezhenxu94 wrote： > Hi Juan Pan, we've noticed that the E2E tests are now very unfriendly for > new contributors, thus we have rewrote the tests now and, from my point of > view, if you're familiar with Docker and docker compose, you should be > quick to get started with the new E2E framework, please take a look at this > pull request https://github.com/apache/skywalking/pull/4510 and check out > the branch locally to play around it, and I hope to get feedback from E2E > contributors like you whether it improves, compared with the old one, > thanks. > > > > > > > -- > > GitHub @kezhenxu94 > Apache SkyWalking, Apache Dubbo > > > > At 2020-03-15 13:42:55, "Juan Pan" wrote: > Hi community, > > > Update task status and raise my questions. Could I get your bits of help? > > > According to our schedule, the 1st to 3rd sub-tasks have been completed > already[1]. Currently, I am working on sub-task 4th and 5th, i.e., > 4. Provide configuration of Sharding Proxy[2] for Skywalking. > 5. Provide e2e tests. > > > Here are my questions, > 1. `rc0-prepare.sh` is used only for one `image` of `docker-maven-plugin`, > right? If so, what should i do if i plan to add `rc0-prepare.sh` for > `shardingsphere/sharding-proxy` image (Not for `skyapm/e2e-container` > image). > 2. Who will call `rc0-prepare.sh` and when? > > > I spent much time on learning more `e2e-test` and docker image from other > e2e modules, which seems a blocker to me to work on sub-task 5th. Could > anyone give me some ideas? Thanks in advance. :) > > > [1] https://github.com/apache/skywalking/pull/4421 > > > Juan Pan (Trista) > > Senior DBA & PPMC of Apache ShardingSphere(Incubating) > E-mail: panj...@apache.org > > > > > On 02/16/2020 18:36，Juan Pan wrote： > Hi fellows, > > > I am working on providing ShardingSphere[1] as one of the server storages > for Skywalking. > ShardingSphere will help shard the data collected by Skywalking into > different databases(Popular RDBMS, like MySQL, PostgreSQL and so on) and > tables to seek better query performance and do scale-out of database. > > > Since it is not an easy job, and needs some helps from community, i put > task schedule and discussion items here to welcome your comments and > suggestions. > If you are interested in this one, welcome to be my partner! :-) > > > > Here is the schedule, > 1. Acquire all required tables and query SQLs from db logs, like general > log of MySQL. [DONE] > 2. Analyse those tables and query SQLs to explore appropriate sharding > strategy.[DOING] > 3. Provide class implementation of `PreciseShardingAlgorithm` and > `RangeShardingAlgorithm` of ShardingSphere to do sharding.[TODO] > 4. Provide configuration of Sharding Proxy[2] for Skywalking.[TODO] > 5. Provide Acceptance test(maybe e2e tests of Skywalking?).[TODO] > 6. Update readme[3].[TODO] > 7. Update storage doc[4].[TODO] > - > > > It seems a long-term work, and i will do it step by step. Any progress > will be updated here, also ISSUE [5] will give you thread of PRs and > discussion detail. > > > [1] https://shardingsphere.apache.org/ > [2] > https://shardingsphere.apache.org/document/current/en/quick-start/sharding-proxy-quick-start/ > [3] https://github.com/apache/skywalking/tree/master/install > [4 ] > https://github.com/apache/skywalking/blob/master/docs/en/setup/backend/backend-storage.md > [5] https://github.com/apache/skywalking/issues/4131 > > > > > > Juan Pan (Trista) > > Senior DBA & PPMC of Apache ShardingSphere(Incubating) > E-mail: panj...@apache.org > > > >
Re:[DISCUSS] Add Apache ShardingSphere as the server storage for Apache SkyWalking
Hi, Thanks for your kind remainder. Actually i am on the way writing e2e tests for ShardingSphere. Considering this PR[1] has not been merged and the discussion is underway, do you think i should stop my current work (Based on the old one) and wait for your progress? Cheers, Trista Juan Pan (Trista) Senior DBA & PPMC of Apache ShardingSphere(Incubating) E-mail: panj...@apache.org On 03/15/2020 14:08，kezhenxu94 wrote： Hi Juan Pan, we've noticed that the E2E tests are now very unfriendly for new contributors, thus we have rewrote the tests now and, from my point of view, if you're familiar with Docker and docker compose, you should be quick to get started with the new E2E framework, please take a look at this pull request https://github.com/apache/skywalking/pull/4510 and check out the branch locally to play around it, and I hope to get feedback from E2E contributors like you whether it improves, compared with the old one, thanks. -- GitHub @kezhenxu94 Apache SkyWalking, Apache Dubbo At 2020-03-15 13:42:55, "Juan Pan" wrote: Hi community, Update task status and raise my questions. Could I get your bits of help? According to our schedule, the 1st to 3rd sub-tasks have been completed already[1]. Currently, I am working on sub-task 4th and 5th, i.e., 4. Provide configuration of Sharding Proxy[2] for Skywalking. 5. Provide e2e tests. Here are my questions, 1. `rc0-prepare.sh` is used only for one `image` of `docker-maven-plugin`, right? If so, what should i do if i plan to add `rc0-prepare.sh` for `shardingsphere/sharding-proxy` image (Not for `skyapm/e2e-container` image). 2. Who will call `rc0-prepare.sh` and when? I spent much time on learning more `e2e-test` and docker image from other e2e modules, which seems a blocker to me to work on sub-task 5th. Could anyone give me some ideas? Thanks in advance. :) [1] https://github.com/apache/skywalking/pull/4421 Juan Pan (Trista) Senior DBA & PPMC of Apache ShardingSphere(Incubating) E-mail: panj...@apache.org On 02/16/2020 18:36，Juan Pan wrote： Hi fellows, I am working on providing ShardingSphere[1] as one of the server storages for Skywalking. ShardingSphere will help shard the data collected by Skywalking into different databases(Popular RDBMS, like MySQL, PostgreSQL and so on) and tables to seek better query performance and do scale-out of database. Since it is not an easy job, and needs some helps from community, i put task schedule and discussion items here to welcome your comments and suggestions. If you are interested in this one, welcome to be my partner! :-) Here is the schedule, 1. Acquire all required tables and query SQLs from db logs, like general log of MySQL. [DONE] 2. Analyse those tables and query SQLs to explore appropriate sharding strategy.[DOING] 3. Provide class implementation of `PreciseShardingAlgorithm` and `RangeShardingAlgorithm` of ShardingSphere to do sharding.[TODO] 4. Provide configuration of Sharding Proxy[2] for Skywalking.[TODO] 5. Provide Acceptance test(maybe e2e tests of Skywalking?).[TODO] 6. Update readme[3].[TODO] 7. Update storage doc[4].[TODO] - It seems a long-term work, and i will do it step by step. Any progress will be updated here, also ISSUE [5] will give you thread of PRs and discussion detail. [1] https://shardingsphere.apache.org/ [2] https://shardingsphere.apache.org/document/current/en/quick-start/sharding-proxy-quick-start/ [3] https://github.com/apache/skywalking/tree/master/install [4 ]https://github.com/apache/skywalking/blob/master/docs/en/setup/backend/backend-storage.md [5] https://github.com/apache/skywalking/issues/4131 Juan Pan (Trista) Senior DBA & PPMC of Apache ShardingSphere(Incubating) E-mail: panj...@apache.org
Re:[DISCUSS] Add Apache ShardingSphere as the server storage for Apache SkyWalking
Hi community, Update task status and raise my questions. Could I get your bits of help? According to our schedule, the 1st to 3rd sub-tasks have been completed already[1]. Currently, I am working on sub-task 4th and 5th, i.e., 4. Provide configuration of Sharding Proxy[2] for Skywalking. 5. Provide e2e tests. Here are my questions, 1. `rc0-prepare.sh` is used only for one `image` of `docker-maven-plugin`, right? If so, what should i do if i plan to add `rc0-prepare.sh` for `shardingsphere/sharding-proxy` image (Not for `skyapm/e2e-container` image). 2. Who will call `rc0-prepare.sh` and when? I spent much time on learning more `e2e-test` and docker image from other e2e modules, which seems a blocker to me to work on sub-task 5th. Could anyone give me some ideas? Thanks in advance. :) [1] https://github.com/apache/skywalking/pull/4421 Juan Pan (Trista) Senior DBA & PPMC of Apache ShardingSphere(Incubating) E-mail: panj...@apache.org On 02/16/2020 18:36，Juan Pan wrote： Hi fellows, I am working on providing ShardingSphere[1] as one of the server storages for Skywalking. ShardingSphere will help shard the data collected by Skywalking into different databases(Popular RDBMS, like MySQL, PostgreSQL and so on) and tables to seek better query performance and do scale-out of database. Since it is not an easy job, and needs some helps from community, i put task schedule and discussion items here to welcome your comments and suggestions. If you are interested in this one, welcome to be my partner! :-) Here is the schedule, 1. Acquire all required tables and query SQLs from db logs, like general log of MySQL. [DONE] 2. Analyse those tables and query SQLs to explore appropriate sharding strategy.[DOING] 3. Provide class implementation of `PreciseShardingAlgorithm` and `RangeShardingAlgorithm` of ShardingSphere to do sharding.[TODO] 4. Provide configuration of Sharding Proxy[2] for Skywalking.[TODO] 5. Provide Acceptance test(maybe e2e tests of Skywalking?).[TODO] 6. Update readme[3].[TODO] 7. Update storage doc[4].[TODO] - It seems a long-term work, and i will do it step by step. Any progress will be updated here, also ISSUE [5] will give you thread of PRs and discussion detail. [1] https://shardingsphere.apache.org/ [2] https://shardingsphere.apache.org/document/current/en/quick-start/sharding-proxy-quick-start/ [3] https://github.com/apache/skywalking/tree/master/install [4 ]https://github.com/apache/skywalking/blob/master/docs/en/setup/backend/backend-storage.md [5] https://github.com/apache/skywalking/issues/4131 Juan Pan (Trista) Senior DBA & PPMC of Apache ShardingSphere(Incubating) E-mail: panj...@apache.org
Re: [DISCUSS] Add Apache ShardingSphere as the server storage for Apache SkyWalking
Hi Daming, That’s not a big issue, we can do it step by step and you can learn it more at link[1]. @Wu-Sheng and @Daming, 2nd sub-task concerning tables and SQLs analysis was presented at issue[2], please review it and give your comments. :-) [1] https://shardingsphere.apache.org/ [2] https://github.com/apache/skywalking/issues/4131#issuecomment-587302393 Sincerely, trista Juan Pan (Trista) Senior DBA & PPMC of Apache ShardingSphere(Incubating) E-mail: panj...@apache.org On 02/18/2020 11:32，Daming wrote： Hi Trista, I am not very familiar with ShardingSphere. I think I need to learn how to run it for e2e test, so I pick up the task. :-) ——— Daming(@dmsolr) Apache SkyWalking 在 2020年2月18日，上午10:49，Juan Pan 写道： Hi Daming, I am glad you can take part in, of course ,you could. Therefore i will take care of sub-task 1to 3. BTW, do you know something about Sharding Proxy[1]? To finish sub-task 4 and 5, you have to use Sharding Proxy to shard DBs and tables, so as to make e2e tests run well. Don’t worry, i will work with you together, and to be frank, it is a good chance to learn two of Apache projects and coordinate them together for a better solution! [1] https://shardingsphere.apache.org/document/current/en/quick-start/sharding-proxy-quick-start/ Juan Pan (Trista) Senior DBA & PPMC of Apache ShardingSphere(Incubating) E-mail: panj...@apache.org On 02/17/2020 10:33，Daming wrote： Could I pick up this two sub-tasks? 4. Provide configuration of Sharding Proxy[2] for Skywalking.[TODO] 5. Provide Acceptance test(maybe e2e tests of Skywalking?).[TODO] — Daming(@dmsolr) Apache SkyWalking 在 2020年2月17日，上午9:58，Juan Pan 写道： That exactly what i concerned, moreover thanks for your joining. Could you tell me which part you are interested in, so that we can share the task list and work together. :-) Juan Pan (Trista) Senior DBA & PPMC of Apache ShardingSphere(Incubating) E-mail: panj...@apache.org On 02/17/2020 00:34，吴晟 Sheng Wu wrote： Hi Which part or both? ShengWu吴晟 FromWuSheng'sphone. -- Original -- From: apmail-wusheng-owner https://shardingsphere.apache.org/ gt; [2] gt; https://shardingsphere.apache.org/document/current/en/quick-start/sharding-proxy-quick-start/ gt; [3] https://github.com/apache/skywalking/tree/master/install gt; [4 ] gt; https://github.com/apache/skywalking/blob/master/docs/en/setup/backend/backend-storage.md gt; [5] https://github.com/apache/skywalking/issues/4131 gt; gt; gt; gt; gt; gt;nbsp; Juan Pan (Trista) gt; gt; Senior DBA amp; PPMC of Apache ShardingSphere(Incubating) gt; E-mail: panj...@apache.org gt; gt; gt; gt;
Re: [DISCUSS] Add Apache ShardingSphere as the server storage for Apache SkyWalking
Hi Trista, I am not very familiar with ShardingSphere. I think I need to learn how to run it for e2e test, so I pick up the task. :-) ——— Daming(@dmsolr) Apache SkyWalking > 在 2020年2月18日，上午10:49，Juan Pan 写道： > > Hi Daming, > > > I am glad you can take part in, of course ,you could. Therefore i will take > care of sub-task 1to 3. > BTW, do you know something about Sharding Proxy[1]? To finish sub-task 4 and > 5, you have to use Sharding Proxy to shard DBs and tables, so as to make e2e > tests run well. > > > Don’t worry, i will work with you together, and to be frank, it is a good > chance to learn two of Apache projects and coordinate them together for a > better solution! > > > [1] > https://shardingsphere.apache.org/document/current/en/quick-start/sharding-proxy-quick-start/ > > > Juan Pan (Trista) > > Senior DBA & PPMC of Apache ShardingSphere(Incubating) > E-mail: panj...@apache.org > > > > > On 02/17/2020 10:33，Daming wrote： > > Could I pick up this two sub-tasks? > > 4. Provide configuration of Sharding Proxy[2] for Skywalking.[TODO] > 5. Provide Acceptance test(maybe e2e tests of Skywalking?).[TODO] > > > — > Daming(@dmsolr) > Apache SkyWalking > > > 在 2020年2月17日，上午9:58，Juan Pan 写道： > > That exactly what i concerned, moreover thanks for your joining. > Could you tell me which part you are interested in, so that we can share the > task list and work together. :-) > > > Juan Pan (Trista) > > Senior DBA & PPMC of Apache ShardingSphere(Incubating) > E-mail: panj...@apache.org > > > > > On 02/17/2020 00:34，吴晟 Sheng Wu wrote： > Hi > > > Which part or both? > > > > > ShengWu吴晟 > > FromWuSheng'sphone. > > > > -- Original -- > From: apmail-wusheng-owner Date: Mon,Feb 17,2020 0:26 AM > To: dev Subject: Re: [DISCUSS] Add Apache ShardingSphere as the server storage for > Apache SkyWalking > > > > hinbsp; > I am glad to take part in. > > > 发自我的iPhone > > -- Original -- > From: Sheng Wu Date: Sun,Feb 16,2020 10:20 PM > To: dev Subject: Re: [DISCUSS] Add Apache ShardingSphere as the server storage for > Apache SkyWalking > > > > Juan Pan > gt; Hi fellows, > gt; > gt; > gt; I am working on providing ShardingSphere[1] as one of the server > storages > gt; for Skywalking. > gt; ShardingSphere will help shard the data collected by Skywalking into > gt; different databases(Popular RDBMS, like MySQL, PostgreSQL and so on) > and > gt; tables to seek better query performance and do scale-out of database. > gt; > gt; > gt; Since it is not an easy job, and needs some helps from community, i > put > gt; task schedule and discussion items here to welcome your comments and > gt; suggestions. > gt; If you are interested in this one, welcome to be my partner! :-) > gt; > gt; > gt; > gt; Here is the schedule, > gt; 1. Acquire all required tables and query SQLs from db logs, like > general > gt; log of MySQL. [DONE] > gt; 2. Analyse those tables and query SQLs to explore appropriate > sharding > gt; strategy.[DOING] > gt; 3. Provide class implementation of `PreciseShardingAlgorithm` and > gt; `RangeShardingAlgorithm` of ShardingSphere to do sharding.[TODO] > gt; 4. Provide configuration of Sharding Proxy[2] for Skywalking.[TODO] > > 5. Provide Acceptance test(maybe e2e tests of Skywalking?).[TODO > > > Yes. We call it e2e. You could find many storage implementation tests > there. Zhenxu Ke and Haochao Zhuang are the two leading this part. > > BTW, your plan looks good to me. > > > Sheng Wu 吴晟 > Twitter, wusheng1108 > > > gt; ] > gt; 6. Update readme[3].[TODO] > gt; 7. Update storage doc[4].[TODO] > gt; - > gt; > gt; > gt; It seems a long-term work, and i will do it step by step. Any > progress > gt; will be updated here, also ISSUE [5] will give you thread of PRs and > gt; discussion detail. > gt; > gt; > gt; [1] https://shardingsphere.apache.org/ > gt; [2] > gt; > https://shardingsphere.apache.org/document/current/en/quick-start/sharding-proxy-quick-start/ > gt; [3] https://github.com/apache/skywalking/tree/master/install > gt; [4 ] > gt; > https://github.com/apache/skywalking/blob/master/docs/en/setup/backend/backend-storage.md > gt; [5] https://github.com/apache/skywalking/issues/4131 > gt; > gt; > gt; > gt; > gt; > gt;nbsp; Juan Pan (Trista) > gt; > gt; Senior DBA amp; PPMC of Apache ShardingSphere(Incubating) > gt; E-mail: panj...@apache.org > gt; > gt; > gt; > gt;
Re: [DISCUSS] Add Apache ShardingSphere as the server storage for Apache SkyWalking
Hi Daming I know your InfluxDB PR is closing to merge. But, there is still two left for you in the 7.0.0 1. https://github.com/apache/skywalking/issues/4330 for enhancing the tests. 2. InfluxDB performance tests. Are you sure you have enough time for 3rd task? :P I have a concern about your pressure. Sheng Wu 吴晟 Twitter, wusheng1108 Juan Pan 于2020年2月18日周二 上午11:00写道： > Hi Sheng and Daming, > > > Today, i guess i can propose one sharding strategy for us to discuss, > which means sub-task 2 > (Analyse those tables and query SQLs to explore appropriate sharding) will > be done. > Once the sharding strategy is settled down by us, > i am going to work on sub-task 3(Provide class implementation > ShardingAlgorithm), meanwhile, Daming can prepare configuration of Sharding > Proxy. > After that, e2e tests will come behind. > > > It is my plan draft, looking forward to your suggestions. > > > Juan Pan (Trista) > > Senior DBA & PPMC of Apache ShardingSphere(Incubating) > E-mail: panj...@apache.org > > > > > On 02/17/2020 14:01，Sheng Wu wrote： > Daming 于2020年2月17日周一 上午10:33写道： > > > Could I pick up this two sub-tasks? > > 4. Provide configuration of Sharding Proxy[2] for Skywalking.[TODO] > > > Hi Juan > > I think you have a draft about this already? > > Sheng Wu 吴晟 > Twitter, wusheng1108 > > > > 5. Provide Acceptance test(maybe e2e tests of Skywalking?).[TODO] > > > This would be helpful. You are more familiar with the e2e than her. > > > > > > — > Daming(@dmsolr) > Apache SkyWalking > > > 在 2020年2月17日，上午9:58，Juan Pan 写道： > > That exactly what i concerned, moreover thanks for your joining. > Could you tell me which part you are interested in, so that we can share > the task list and work together. :-) > > > Juan Pan (Trista) > > Senior DBA & PPMC of Apache ShardingSphere(Incubating) > E-mail: panj...@apache.org > > > > > On 02/17/2020 00:34，吴晟 Sheng Wu wrote： > Hi > > > Which part or both? > > > > > ShengWu吴晟 > > FromWuSheng'sphone. > > > > -- Original -- > From: apmail-wusheng-owner Date: Mon,Feb 17,2020 0:26 AM > To: dev Subject: Re: [DISCUSS] Add Apache ShardingSphere as the server storage > for Apache SkyWalking > > > > hinbsp; > I am glad to take part in. > > > 发自我的iPhone > > -- Original -- > From: Sheng Wu Date: Sun,Feb 16,2020 10:20 PM > To: dev Subject: Re: [DISCUSS] Add Apache ShardingSphere as the server storage > for Apache SkyWalking > > > > Juan Pan > gt; Hi fellows, > gt; > gt; > gt; I am working on providing ShardingSphere[1] as one of the > server storages > gt; for Skywalking. > gt; ShardingSphere will help shard the data collected by Skywalking > into > gt; different databases(Popular RDBMS, like MySQL, PostgreSQL and > so on) and > gt; tables to seek better query performance and do scale-out of > database. > gt; > gt; > gt; Since it is not an easy job, and needs some helps from > community, i put > gt; task schedule and discussion items here to welcome your > comments and > gt; suggestions. > gt; If you are interested in this one, welcome to be my partner! :-) > gt; > gt; > gt; > gt; Here is the schedule, > gt; 1. Acquire all required tables and query SQLs from db logs, > like general > gt; log of MySQL. [DONE] > gt; 2. Analyse those tables and query SQLs to explore appropriate > sharding > gt; strategy.[DOING] > gt; 3. Provide class implementation of `PreciseShardingAlgorithm` > and > gt; `RangeShardingAlgorithm` of ShardingSphere to do sharding.[TODO] > gt; 4. Provide configuration of Sharding Proxy[2] for > Skywalking.[TODO] > > 5. Provide Acceptance test(maybe e2e tests of Skywalking?).[TODO > > > Yes. We call it e2e. You could find many storage implementation tests > there. Zhenxu Ke and Haochao Zhuang are the two leading this part. > > BTW, your plan looks good to me. > > > Sheng Wu 吴晟 > Twitter, wusheng1108 > > > gt; ] > gt; 6. Update readme[3].[TODO] > gt; 7. Update storage doc[4].[TODO] > gt; - > gt; > gt; > gt; It seems a long-term work, and i will do it step by step. Any > progress > gt; will be updated here, also ISSUE [5] will give you thread of > PRs and > gt; discussion detail. > gt; > gt; > gt; [1] https://shardingsphere.apache.org/ > gt; [2] > gt; > > https://shardingsphere.apache.org/document/current/en/quick-start/sharding-proxy-quick-start/ > gt > <https://shardingsphere.apache.org/document/current/en/quick-start/sharding-proxy-quick
Re: [DISCUSS] Add Apache ShardingSphere as the server storage for Apache SkyWalking
Hi Sheng and Daming, Today, i guess i can propose one sharding strategy for us to discuss, which means sub-task 2 (Analyse those tables and query SQLs to explore appropriate sharding) will be done. Once the sharding strategy is settled down by us, i am going to work on sub-task 3(Provide class implementation ShardingAlgorithm), meanwhile, Daming can prepare configuration of Sharding Proxy. After that, e2e tests will come behind. It is my plan draft, looking forward to your suggestions. Juan Pan (Trista) Senior DBA & PPMC of Apache ShardingSphere(Incubating) E-mail: panj...@apache.org On 02/17/2020 14:01，Sheng Wu wrote： Daming 于2020年2月17日周一 上午10:33写道： Could I pick up this two sub-tasks? 4. Provide configuration of Sharding Proxy[2] for Skywalking.[TODO] Hi Juan I think you have a draft about this already? Sheng Wu 吴晟 Twitter, wusheng1108 5. Provide Acceptance test(maybe e2e tests of Skywalking?).[TODO] This would be helpful. You are more familiar with the e2e than her. — Daming(@dmsolr) Apache SkyWalking 在 2020年2月17日，上午9:58，Juan Pan 写道： That exactly what i concerned, moreover thanks for your joining. Could you tell me which part you are interested in, so that we can share the task list and work together. :-) Juan Pan (Trista) Senior DBA & PPMC of Apache ShardingSphere(Incubating) E-mail: panj...@apache.org On 02/17/2020 00:34，吴晟 Sheng Wu wrote： Hi Which part or both? ShengWu吴晟 FromWuSheng'sphone. -- Original -- From: apmail-wusheng-owner https://shardingsphere.apache.org/ gt; [2] gt; https://shardingsphere.apache.org/document/current/en/quick-start/sharding-proxy-quick-start/ gt; [3] https://github.com/apache/skywalking/tree/master/install gt; [4 ] gt; https://github.com/apache/skywalking/blob/master/docs/en/setup/backend/backend-storage.md gt; [5] https://github.com/apache/skywalking/issues/4131 gt; gt; gt; gt; gt; gt;nbsp; Juan Pan (Trista) gt; gt; Senior DBA amp; PPMC of Apache ShardingSphere(Incubating) gt; E-mail: panj...@apache.org gt; gt; gt; gt;
Re: [DISCUSS] Add Apache ShardingSphere as the server storage for Apache SkyWalking
Hi Daming, I am glad you can take part in, of course ,you could. Therefore i will take care of sub-task 1to 3. BTW, do you know something about Sharding Proxy[1]? To finish sub-task 4 and 5, you have to use Sharding Proxy to shard DBs and tables, so as to make e2e tests run well. Don’t worry, i will work with you together, and to be frank, it is a good chance to learn two of Apache projects and coordinate them together for a better solution! [1] https://shardingsphere.apache.org/document/current/en/quick-start/sharding-proxy-quick-start/ Juan Pan (Trista) Senior DBA & PPMC of Apache ShardingSphere(Incubating) E-mail: panj...@apache.org On 02/17/2020 10:33，Daming wrote： Could I pick up this two sub-tasks? 4. Provide configuration of Sharding Proxy[2] for Skywalking.[TODO] 5. Provide Acceptance test(maybe e2e tests of Skywalking?).[TODO] — Daming(@dmsolr) Apache SkyWalking 在 2020年2月17日，上午9:58，Juan Pan 写道： That exactly what i concerned, moreover thanks for your joining. Could you tell me which part you are interested in, so that we can share the task list and work together. :-) Juan Pan (Trista) Senior DBA & PPMC of Apache ShardingSphere(Incubating) E-mail: panj...@apache.org On 02/17/2020 00:34，吴晟 Sheng Wu wrote： Hi Which part or both? ShengWu吴晟 FromWuSheng'sphone. -- Original -- From: apmail-wusheng-owner https://shardingsphere.apache.org/ gt; [2] gt; https://shardingsphere.apache.org/document/current/en/quick-start/sharding-proxy-quick-start/ gt; [3] https://github.com/apache/skywalking/tree/master/install gt; [4 ] gt; https://github.com/apache/skywalking/blob/master/docs/en/setup/backend/backend-storage.md gt; [5] https://github.com/apache/skywalking/issues/4131 gt; gt; gt; gt; gt; gt;nbsp; Juan Pan (Trista) gt; gt; Senior DBA amp; PPMC of Apache ShardingSphere(Incubating) gt; E-mail: panj...@apache.org gt; gt; gt; gt;
Re: [DISCUSS] Add Apache ShardingSphere as the server storage for Apache SkyWalking
Daming 于2020年2月17日周一 上午10:33写道： > > Could I pick up this two sub-tasks? > > 4. Provide configuration of Sharding Proxy[2] for Skywalking.[TODO] > Hi Juan I think you have a draft about this already? Sheng Wu 吴晟 Twitter, wusheng1108 > 5. Provide Acceptance test(maybe e2e tests of Skywalking?).[TODO] > This would be helpful. You are more familiar with the e2e than her. > > > — > Daming(@dmsolr) > Apache SkyWalking > > > > 在 2020年2月17日，上午9:58，Juan Pan 写道： > > > > That exactly what i concerned, moreover thanks for your joining. > > Could you tell me which part you are interested in, so that we can share > the task list and work together. :-) > > > > > > Juan Pan (Trista) > > > > Senior DBA & PPMC of Apache ShardingSphere(Incubating) > > E-mail: panj...@apache.org > > > > > > > > > > On 02/17/2020 00:34，吴晟 Sheng Wu wrote： > > Hi > > > > > > Which part or both? > > > > > > > > > > ShengWu吴晟 > > > > FromWuSheng'sphone. > > > > > > > > -- Original -- > > From: apmail-wusheng-owner > Date: Mon,Feb 17,2020 0:26 AM > > To: dev > Subject: Re: [DISCUSS] Add Apache ShardingSphere as the server storage > for Apache SkyWalking > > > > > > > > hinbsp; > > I am glad to take part in. > > > > > > 发自我的iPhone > > > > -- Original -- > > From: Sheng Wu > Date: Sun,Feb 16,2020 10:20 PM > > To: dev > Subject: Re: [DISCUSS] Add Apache ShardingSphere as the server storage > for Apache SkyWalking > > > > > > > > Juan Pan > > > gt; Hi fellows, > > gt; > > gt; > > gt; I am working on providing ShardingSphere[1] as one of the > server storages > > gt; for Skywalking. > > gt; ShardingSphere will help shard the data collected by Skywalking > into > > gt; different databases(Popular RDBMS, like MySQL, PostgreSQL and > so on) and > > gt; tables to seek better query performance and do scale-out of > database. > > gt; > > gt; > > gt; Since it is not an easy job, and needs some helps from > community, i put > > gt; task schedule and discussion items here to welcome your > comments and > > gt; suggestions. > > gt; If you are interested in this one, welcome to be my partner! :-) > > gt; > > gt; > > gt; > > gt; Here is the schedule, > > gt; 1. Acquire all required tables and query SQLs from db logs, > like general > > gt; log of MySQL. [DONE] > > gt; 2. Analyse those tables and query SQLs to explore appropriate > sharding > > gt; strategy.[DOING] > > gt; 3. Provide class implementation of `PreciseShardingAlgorithm` > and > > gt; `RangeShardingAlgorithm` of ShardingSphere to do sharding.[TODO] > > gt; 4. Provide configuration of Sharding Proxy[2] for > Skywalking.[TODO] > > > > 5. Provide Acceptance test(maybe e2e tests of Skywalking?).[TODO > > > > > > Yes. We call it e2e. You could find many storage implementation tests > > there. Zhenxu Ke and Haochao Zhuang are the two leading this part. > > > > BTW, your plan looks good to me. > > > > > > Sheng Wu 吴晟 > > Twitter, wusheng1108 > > > > > > gt; ] > > gt; 6. Update readme[3].[TODO] > > gt; 7. Update storage doc[4].[TODO] > > gt; - > > gt; > > gt; > > gt; It seems a long-term work, and i will do it step by step. Any > progress > > gt; will be updated here, also ISSUE [5] will give you thread of > PRs and > > gt; discussion detail. > > gt; > > gt; > > gt; [1] https://shardingsphere.apache.org/ > > gt; [2] > > gt; > https://shardingsphere.apache.org/document/current/en/quick-start/sharding-proxy-quick-start/ > > gt; [3] https://github.com/apache/skywalking/tree/master/install > > gt; [4 ] > > gt; > https://github.com/apache/skywalking/blob/master/docs/en/setup/backend/backend-storage.md > > gt; [5] https://github.com/apache/skywalking/issues/4131 > > gt; > > gt; > > gt; > > gt; > > gt; > > gt;nbsp; Juan Pan (Trista) > > gt; > > gt; Senior DBA amp; PPMC of Apache ShardingSphere(Incubating) > > gt; E-mail: panj...@apache.org > > gt; > > gt; > > gt; > > gt; > >
Re: [DISCUSS] Add Apache ShardingSphere as the server storage for Apache SkyWalking
Could I pick up this two sub-tasks? 4. Provide configuration of Sharding Proxy[2] for Skywalking.[TODO] 5. Provide Acceptance test(maybe e2e tests of Skywalking?).[TODO] — Daming(@dmsolr) Apache SkyWalking > 在 2020年2月17日，上午9:58，Juan Pan 写道： > > That exactly what i concerned, moreover thanks for your joining. > Could you tell me which part you are interested in, so that we can share the > task list and work together. :-) > > > Juan Pan (Trista) > > Senior DBA & PPMC of Apache ShardingSphere(Incubating) > E-mail: panj...@apache.org > > > > > On 02/17/2020 00:34，吴晟 Sheng Wu wrote： > Hi > > > Which part or both? > > > > > ShengWu吴晟 > > FromWuSheng'sphone. > > > > -- Original -------------- > From: apmail-wusheng-owner Date: Mon,Feb 17,2020 0:26 AM > To: dev Subject: Re: [DISCUSS] Add Apache ShardingSphere as the server storage for > Apache SkyWalking > > > > hinbsp; > I am glad to take part in. > > > 发自我的iPhone > > -- Original ------ > From: Sheng Wu Date: Sun,Feb 16,2020 10:20 PM > To: dev Subject: Re: [DISCUSS] Add Apache ShardingSphere as the server storage for > Apache SkyWalking > > > > Juan Pan > gt; Hi fellows, > gt; > gt; > gt; I am working on providing ShardingSphere[1] as one of the server > storages > gt; for Skywalking. > gt; ShardingSphere will help shard the data collected by Skywalking into > gt; different databases(Popular RDBMS, like MySQL, PostgreSQL and so on) > and > gt; tables to seek better query performance and do scale-out of database. > gt; > gt; > gt; Since it is not an easy job, and needs some helps from community, i > put > gt; task schedule and discussion items here to welcome your comments and > gt; suggestions. > gt; If you are interested in this one, welcome to be my partner! :-) > gt; > gt; > gt; > gt; Here is the schedule, > gt; 1. Acquire all required tables and query SQLs from db logs, like > general > gt; log of MySQL. [DONE] > gt; 2. Analyse those tables and query SQLs to explore appropriate > sharding > gt; strategy.[DOING] > gt; 3. Provide class implementation of `PreciseShardingAlgorithm` and > gt; `RangeShardingAlgorithm` of ShardingSphere to do sharding.[TODO] > gt; 4. Provide configuration of Sharding Proxy[2] for Skywalking.[TODO] > > 5. Provide Acceptance test(maybe e2e tests of Skywalking?).[TODO > > > Yes. We call it e2e. You could find many storage implementation tests > there. Zhenxu Ke and Haochao Zhuang are the two leading this part. > > BTW, your plan looks good to me. > > > Sheng Wu 吴晟 > Twitter, wusheng1108 > > > gt; ] > gt; 6. Update readme[3].[TODO] > gt; 7. Update storage doc[4].[TODO] > gt; - > gt; > gt; > gt; It seems a long-term work, and i will do it step by step. Any > progress > gt; will be updated here, also ISSUE [5] will give you thread of PRs and > gt; discussion detail. > gt; > gt; > gt; [1] https://shardingsphere.apache.org/ > gt; [2] > gt; > https://shardingsphere.apache.org/document/current/en/quick-start/sharding-proxy-quick-start/ > gt; [3] https://github.com/apache/skywalking/tree/master/install > gt; [4 ] > gt; > https://github.com/apache/skywalking/blob/master/docs/en/setup/backend/backend-storage.md > gt; [5] https://github.com/apache/skywalking/issues/4131 > gt; > gt; > gt; > gt; > gt; > gt;nbsp; Juan Pan (Trista) > gt; > gt; Senior DBA amp; PPMC of Apache ShardingSphere(Incubating) > gt; E-mail: panj...@apache.org > gt; > gt; > gt; > gt;
Re: [DISCUSS] Add Apache ShardingSphere as the server storage for Apache SkyWalking
That exactly what i concerned, moreover thanks for your joining. Could you tell me which part you are interested in, so that we can share the task list and work together. :-) Juan Pan (Trista) Senior DBA & PPMC of Apache ShardingSphere(Incubating) E-mail: panj...@apache.org On 02/17/2020 00:34?? Sheng Wu wrote?? Hi Which part or both? ShengWu FromWuSheng'sphone. -- Original -- From: apmail-wusheng-owner https://shardingsphere.apache.org/ gt; [2] gt; https://shardingsphere.apache.org/document/current/en/quick-start/sharding-proxy-quick-start/ gt; [3] https://github.com/apache/skywalking/tree/master/install gt; [4 ] gt; https://github.com/apache/skywalking/blob/master/docs/en/setup/backend/backend-storage.md gt; [5] https://github.com/apache/skywalking/issues/4131 gt; gt; gt; gt; gt; gt;nbsp; Juan Pan (Trista) gt; gt; Senior DBA amp; PPMC of Apache ShardingSphere(Incubating) gt; E-mail: panj...@apache.org gt; gt; gt; gt;
Re: [DISCUSS] Add Apache ShardingSphere as the server storage for Apache SkyWalking
Make sense to me, thanks for your reply. Juan Pan (Trista) Senior DBA & PPMC of Apache ShardingSphere(Incubating) E-mail: panj...@apache.org On 02/16/2020 22:20，Sheng Wu wrote： Juan Pan 于2020年2月16日周日 下午6:36写道： Hi fellows, I am working on providing ShardingSphere[1] as one of the server storages for Skywalking. ShardingSphere will help shard the data collected by Skywalking into different databases(Popular RDBMS, like MySQL, PostgreSQL and so on) and tables to seek better query performance and do scale-out of database. Since it is not an easy job, and needs some helps from community, i put task schedule and discussion items here to welcome your comments and suggestions. If you are interested in this one, welcome to be my partner! :-) Here is the schedule, 1. Acquire all required tables and query SQLs from db logs, like general log of MySQL. [DONE] 2. Analyse those tables and query SQLs to explore appropriate sharding strategy.[DOING] 3. Provide class implementation of `PreciseShardingAlgorithm` and `RangeShardingAlgorithm` of ShardingSphere to do sharding.[TODO] 4. Provide configuration of Sharding Proxy[2] for Skywalking.[TODO] 5. Provide Acceptance test(maybe e2e tests of Skywalking?).[TODO Yes. We call it e2e. You could find many storage implementation tests there. Zhenxu Ke and Haochao Zhuang are the two leading this part. BTW, your plan looks good to me. Sheng Wu 吴晟 Twitter, wusheng1108 ] 6. Update readme[3].[TODO] 7. Update storage doc[4].[TODO] - It seems a long-term work, and i will do it step by step. Any progress will be updated here, also ISSUE [5] will give you thread of PRs and discussion detail. [1] https://shardingsphere.apache.org/ [2] https://shardingsphere.apache.org/document/current/en/quick-start/sharding-proxy-quick-start/ [3] https://github.com/apache/skywalking/tree/master/install [4 ] https://github.com/apache/skywalking/blob/master/docs/en/setup/backend/backend-storage.md [5] https://github.com/apache/skywalking/issues/4131 Juan Pan (Trista) Senior DBA & PPMC of Apache ShardingSphere(Incubating) E-mail: panj...@apache.org
Re: [DISCUSS] Add Apache ShardingSphere as the server storage for Apache SkyWalking
Hi Which part or both? ShengWu FromWuSheng'sphone. -- Original -- From: apmail-wusheng-owner https://shardingsphere.apache.org/ gt; [2] gt; https://shardingsphere.apache.org/document/current/en/quick-start/sharding-proxy-quick-start/ gt; [3] https://github.com/apache/skywalking/tree/master/install gt; [4 ] gt; https://github.com/apache/skywalking/blob/master/docs/en/setup/backend/backend-storage.md gt; [5] https://github.com/apache/skywalking/issues/4131 gt; gt; gt; gt; gt; gt;nbsp; Juan Pan (Trista) gt; gt; Senior DBA amp; PPMC of Apache ShardingSphere(Incubating) gt; E-mail: panj...@apache.org gt; gt; gt; gt;
Re: [DISCUSS] Add Apache ShardingSphere as the server storage for Apache SkyWalking
hi I am glad to take part in. iPhone -- Original -- From: Sheng Wu https://shardingsphere.apache.org/ [2] https://shardingsphere.apache.org/document/current/en/quick-start/sharding-proxy-quick-start/ [3] https://github.com/apache/skywalking/tree/master/install [4 ] https://github.com/apache/skywalking/blob/master/docs/en/setup/backend/backend-storage.md [5] https://github.com/apache/skywalking/issues/4131 Juan Pan (Trista) Senior DBA PPMC of Apache ShardingSphere(Incubating) E-mail: panj...@apache.org
Re: [DISCUSS] Add Apache ShardingSphere as the server storage for Apache SkyWalking
Juan Pan 于2020年2月16日周日 下午6:36写道： > Hi fellows, > > > I am working on providing ShardingSphere[1] as one of the server storages > for Skywalking. > ShardingSphere will help shard the data collected by Skywalking into > different databases(Popular RDBMS, like MySQL, PostgreSQL and so on) and > tables to seek better query performance and do scale-out of database. > > > Since it is not an easy job, and needs some helps from community, i put > task schedule and discussion items here to welcome your comments and > suggestions. > If you are interested in this one, welcome to be my partner! :-) > > > > Here is the schedule, > 1. Acquire all required tables and query SQLs from db logs, like general > log of MySQL. [DONE] > 2. Analyse those tables and query SQLs to explore appropriate sharding > strategy.[DOING] > 3. Provide class implementation of `PreciseShardingAlgorithm` and > `RangeShardingAlgorithm` of ShardingSphere to do sharding.[TODO] > 4. Provide configuration of Sharding Proxy[2] for Skywalking.[TODO] 5. Provide Acceptance test(maybe e2e tests of Skywalking?).[TODO Yes. We call it e2e. You could find many storage implementation tests there. Zhenxu Ke and Haochao Zhuang are the two leading this part. BTW, your plan looks good to me. Sheng Wu 吴晟 Twitter, wusheng1108 > ] > 6. Update readme[3].[TODO] > 7. Update storage doc[4].[TODO] > - > > > It seems a long-term work, and i will do it step by step. Any progress > will be updated here, also ISSUE [5] will give you thread of PRs and > discussion detail. > > > [1] https://shardingsphere.apache.org/ > [2] > https://shardingsphere.apache.org/document/current/en/quick-start/sharding-proxy-quick-start/ > [3] https://github.com/apache/skywalking/tree/master/install > [4 ] > https://github.com/apache/skywalking/blob/master/docs/en/setup/backend/backend-storage.md > [5] https://github.com/apache/skywalking/issues/4131 > > > > > > Juan Pan (Trista) > > Senior DBA & PPMC of Apache ShardingSphere(Incubating) > E-mail: panj...@apache.org > > > >
