Re: [Proposal] Pegasus - proposal for Apache Incubation
Welcome, Liang! Thank you for offering to help. Now we have enough mentors. Liang Chen 于2020年6月10日周三 下午10:46写道： > Hi Tao Wu > > I am willing to be a mentor. I am familiar with big data , can give some > help for incubating. > > > Regards > Liang > > Justin Mclean wrote > > Hi, > > > >> Thank you for your interest, Kevin. Since now we have 2 mentors, shall > we > >> call for a vote? > > > > Given Kevin said he only 1/2 mentor it would be preferable if you had one > > more mentor. Anyone willing to help out? > > > > Thanks, > > Justin > > - > > To unsubscribe, e-mail: > > > general-unsubscribe@.apache > > > For additional commands, e-mail: > > > general-help@.apache > > > > > > -- > Sent from: http://apache-incubator-general.996316.n3.nabble.com/ > > - > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > >
Re: [Proposal] Pegasus - proposal for Apache Incubation
Hi Tao Wu I am willing to be a mentor. I am familiar with big data , can give some help for incubating. Regards Liang Justin Mclean wrote > Hi, > >> Thank you for your interest, Kevin. Since now we have 2 mentors, shall we >> call for a vote? > > Given Kevin said he only 1/2 mentor it would be preferable if you had one > more mentor. Anyone willing to help out? > > Thanks, > Justin > - > To unsubscribe, e-mail: > general-unsubscribe@.apache > For additional commands, e-mail: > general-help@.apache -- Sent from: http://apache-incubator-general.996316.n3.nabble.com/ - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [Proposal] Pegasus - proposal for Apache Incubation
Hi, > Thank you for your interest, Kevin. Since now we have 2 mentors, shall we > call for a vote? Given Kevin said he only 1/2 mentor it would be preferable if you had one more mentor. Anyone willing to help out? Thanks, Justin - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [Proposal] Pegasus - proposal for Apache Incubation
Thank you for your interest, Kevin. Since now we have 2 mentors, shall we call for a vote? @Gosling Von Kevin A. McGrail 于2020年6月10日周三 上午2:41写道： > While I am honored to be asked, I am mentoring actively a number of > projects right now with another one being proposed. > Please count me as a mentor but seek out others and consider me in reality > to be a 1/2 a mentor :-) > > -- > Kevin A. McGrail > Member, Apache Software Foundation > Chair Emeritus Apache SpamAssassin Project > https://www.linkedin.com/in/kmcgrail - 703.798.0171 > > > On Mon, Jun 8, 2020 at 4:13 AM 吴涛 wrote: > > > Thanks, Kevin. I'd very much appreciate it if you can become one of our > > mentors. > > > > Regards > > > > Tao. > > > > - > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > > For additional commands, e-mail: general-h...@incubator.apache.org > > > > >
Re: [Proposal] Pegasus - proposal for Apache Incubation
While I am honored to be asked, I am mentoring actively a number of projects right now with another one being proposed. Please count me as a mentor but seek out others and consider me in reality to be a 1/2 a mentor :-) -- Kevin A. McGrail Member, Apache Software Foundation Chair Emeritus Apache SpamAssassin Project https://www.linkedin.com/in/kmcgrail - 703.798.0171 On Mon, Jun 8, 2020 at 4:13 AM 吴涛 wrote: > Thanks, Kevin. I'd very much appreciate it if you can become one of our > mentors. > > Regards > > Tao. > > - > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > >
Re: [Proposal] Pegasus - proposal for Apache Incubation
Hi, The ASF allows 3rd party code to be included in releases. It wouldn’t be part of the grant and still remain the original headers and copyright. The ASF also doesn’t fork other communities code, but given this is unmaintained and no longer in active development [1] from what I can see I don’t any issues here. Thanks, Justin 1. https://github.com/microsoft/rDSN - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [Proposal] Pegasus - proposal for Apache Incubation
Sure, Sheng. We certainly don't want to steal any IP from anyone. Sorry if I didn't make it clear. My question is, are we legal enough to paste copyright of Microsoft on every file that originates from them, as we always do? As far as I know, it suffices to prevent legal problems. But I'm not familiar that if ASF has any restriction that requires all the code except 3rd-parties are owned by ASF.
Re: [Proposal] Pegasus - proposal for Apache Incubation
Since MIT is compatible with Apache 2.0 so I do not think this is a blocker issue? We can even bundle the rdsn code into the pegasus main repository I think. We can solve this problem during the incubating time. Of course it will be good if microsoft would like to donate the code. Sheng Wu 于2020年6月8日周一 下午6:00写道： > 吴涛 于2020年6月8日周一 下午5:22写道： > > > Hi, Willem, > > > > This is the historical issue we are going to resolve recently. > > > > We've reduced the incompatible modifications on facebook/rocksdb [1] > where > > pegasus-rocksdb was forked, and I believe we will soon no longer need to > > maintain > > this repo. It will be completely used as an external dependency. > > > > As for rdsn, since the original repo (fully MIT-licensed) > > microsoft/rDSN [2] has been > > unmaintained for a long time, we plan to merge the two repo together. > We've > > endeavored a lot > > to improve and refactor rdsn. I'm not sure if we should ask microsoft for > > any CLA for the > > donation of our code. > > > > I am afraid you need to donate this, you have to. Unmaintained can't change > the fact, its IP belongs to the original team. > > Sheng Wu 吴晟 > Twitter, wusheng1108 > > > > > > [1] https://github.com/facebook/rocksdb > > [2] https://github.com/XiaoMi/rdsn > > > > Willem Jiang 于2020年6月8日周一 下午4:12写道： > > > > > Hi, > > > > > > I just went through the proposal and found there are two source code > > > repos[1][2] which are forked. > > > > > > Are you planning to donate these two repos into Apache as a part of > > > Pegasus project? > > > It makes the donation of the Pegasus complicated as we need to address > > > this two code repo belonging issue first. > > > Normally we just contribute the patch to the upstream as it consume > > > lots of resources if we maintain the forked repo. > > > > > > [1] https://github.com/XiaoMi/rdsn > > > [2] https://github.com/XiaoMi/pegasus-rocksdb > > > > > > > > > Willem Jiang > > > > > > Twitter: willemjiang > > > Weibo: 姜宁willem > > > > > > On Tue, Jun 2, 2020 at 3:49 PM 吴涛 > > wrote: > > > > > > > > Dear Apache Incubator Community, > > > > > > > > I'd like to open up a discussion about incubating Pegasus at Apache. > > Our > > > proposal can be found at > https://pegasus-kv.github.io/community/proposal > > > and is also included below. > > > > > > > > We are looking for possible Champion if anyone would like to > volunteer. > > > Thanks a lot! > > > > > > > > Best regards > > > > Tao Wu > > > > > > > > Pegasus Proposal > > > > > > > > == Abstract == > > > > > > > > Pegasus is a distributed key-value storage system that is designed to > > be > > > horizontally scalable, strongly consistent and high-performance. > > > > > > > > - Pegasus codebase: https://github.com/XiaoMi/pegasus > > > > - Website: https://pegasus-kv.github.io > > > > > > > > == Proposal == > > > > > > > > Pegasus is a key-value database that delivers low-latency data access > > > together with horizontal scalability, using hash-based partitioning. > > > Pegasus uses PacificA protocol for strong consistency and RocksDB as > the > > > underlying storage engine. > > > > > > > > We propose to contribute the Pegasus codebase and associated > artifacts > > > (e.g., documentation, website content, etc.) to the Apache Software > > > Foundation, and aim to build an open community around Pegasus’s > continued > > > development in the ‘Apache Way’. > > > > > > > > == Background == > > > > > > > > Apache HBase was recognized as mostly the only large-scale KV store > > > solution in XiaoMi Corp until Pegasus came out in 2015. The original > > > purpose of Pegasus was to solve the problems caused by HBase’s > two-level > > > architecture and implementation, including high latency because of Java > > GC > > > and RPC overhead of the underlying distributed filesystem, and long > > > failover time because of single point of RegionServer and recovery > > overhead > > > of splitting and replaying the HLog files. > > > > > > > > Pegasus aims to fill the gap between Redis and HBase. As the former > is > > > in-memory, low latency, but does not provide a strong-consistency > > > guarantee. And unlike the latter, Pegasus server is entirely written in > > C++ > > > and its read-write path relies merely on the local filesystem. > > > > > > > > Apart from performance requirements, we also need a storage system to > > > ensure multiple-level data safety and support fast data migration among > > > data centers, automatic load balancing, and online partition splitting. > > > > > > > > After investigating lots of existing storage systems in the open > source > > > world, we could hardly find a suitable solution to satisfy all the > > > requirements. So the journey of Pegasus begins. > > > > > > > > === Rationale === > > > > > > > > Pegasus is a mature and active project which has been widely adopted > in > > > XiaoMi. After the initial release of open source project in 2017, we > have > > > seen a great amount of interest across a diverse
Re: [Proposal] Pegasus - proposal for Apache Incubation
吴涛 于2020年6月8日周一 下午5:22写道： > Hi, Willem, > > This is the historical issue we are going to resolve recently. > > We've reduced the incompatible modifications on facebook/rocksdb [1] where > pegasus-rocksdb was forked, and I believe we will soon no longer need to > maintain > this repo. It will be completely used as an external dependency. > > As for rdsn, since the original repo (fully MIT-licensed) > microsoft/rDSN [2] has been > unmaintained for a long time, we plan to merge the two repo together. We've > endeavored a lot > to improve and refactor rdsn. I'm not sure if we should ask microsoft for > any CLA for the > donation of our code. > I am afraid you need to donate this, you have to. Unmaintained can't change the fact, its IP belongs to the original team. Sheng Wu 吴晟 Twitter, wusheng1108 > > [1] https://github.com/facebook/rocksdb > [2] https://github.com/XiaoMi/rdsn > > Willem Jiang 于2020年6月8日周一 下午4:12写道： > > > Hi, > > > > I just went through the proposal and found there are two source code > > repos[1][2] which are forked. > > > > Are you planning to donate these two repos into Apache as a part of > > Pegasus project? > > It makes the donation of the Pegasus complicated as we need to address > > this two code repo belonging issue first. > > Normally we just contribute the patch to the upstream as it consume > > lots of resources if we maintain the forked repo. > > > > [1] https://github.com/XiaoMi/rdsn > > [2] https://github.com/XiaoMi/pegasus-rocksdb > > > > > > Willem Jiang > > > > Twitter: willemjiang > > Weibo: 姜宁willem > > > > On Tue, Jun 2, 2020 at 3:49 PM 吴涛 > wrote: > > > > > > Dear Apache Incubator Community, > > > > > > I'd like to open up a discussion about incubating Pegasus at Apache. > Our > > proposal can be found at https://pegasus-kv.github.io/community/proposal > > and is also included below. > > > > > > We are looking for possible Champion if anyone would like to volunteer. > > Thanks a lot! > > > > > > Best regards > > > Tao Wu > > > > > > Pegasus Proposal > > > > > > == Abstract == > > > > > > Pegasus is a distributed key-value storage system that is designed to > be > > horizontally scalable, strongly consistent and high-performance. > > > > > > - Pegasus codebase: https://github.com/XiaoMi/pegasus > > > - Website: https://pegasus-kv.github.io > > > > > > == Proposal == > > > > > > Pegasus is a key-value database that delivers low-latency data access > > together with horizontal scalability, using hash-based partitioning. > > Pegasus uses PacificA protocol for strong consistency and RocksDB as the > > underlying storage engine. > > > > > > We propose to contribute the Pegasus codebase and associated artifacts > > (e.g., documentation, website content, etc.) to the Apache Software > > Foundation, and aim to build an open community around Pegasus’s continued > > development in the ‘Apache Way’. > > > > > > == Background == > > > > > > Apache HBase was recognized as mostly the only large-scale KV store > > solution in XiaoMi Corp until Pegasus came out in 2015. The original > > purpose of Pegasus was to solve the problems caused by HBase’s two-level > > architecture and implementation, including high latency because of Java > GC > > and RPC overhead of the underlying distributed filesystem, and long > > failover time because of single point of RegionServer and recovery > overhead > > of splitting and replaying the HLog files. > > > > > > Pegasus aims to fill the gap between Redis and HBase. As the former is > > in-memory, low latency, but does not provide a strong-consistency > > guarantee. And unlike the latter, Pegasus server is entirely written in > C++ > > and its read-write path relies merely on the local filesystem. > > > > > > Apart from performance requirements, we also need a storage system to > > ensure multiple-level data safety and support fast data migration among > > data centers, automatic load balancing, and online partition splitting. > > > > > > After investigating lots of existing storage systems in the open source > > world, we could hardly find a suitable solution to satisfy all the > > requirements. So the journey of Pegasus begins. > > > > > > === Rationale === > > > > > > Pegasus is a mature and active project which has been widely adopted in > > XiaoMi. After the initial release of open source project in 2017, we have > > seen a great amount of interest across a diverse set of users and > companies. > > > > > > Our experiences at committers and PMC members on other Apache projects > > have convinced us that having a long-term home at Apache foundation would > > be a great fit for the project, to ensure that processes and procedures > are > > in place to keep project and community ‘healthy’ and free of any > > commercial, political or legal faults. > > > > > > === Initial Goal === > > > > > > Move the existing codebase, website, documentation, and mailing lists > to > > Apache-hosted infrastructure. > > > Work with the infrastructure team to
Re: [Proposal] Pegasus - proposal for Apache Incubation
Hi, Willem, This is the historical issue we are going to resolve recently. We've reduced the incompatible modifications on facebook/rocksdb [1] where pegasus-rocksdb was forked, and I believe we will soon no longer need to maintain this repo. It will be completely used as an external dependency. As for rdsn, since the original repo (fully MIT-licensed) microsoft/rDSN [2] has been unmaintained for a long time, we plan to merge the two repo together. We've endeavored a lot to improve and refactor rdsn. I'm not sure if we should ask microsoft for any CLA for the donation of our code. [1] https://github.com/facebook/rocksdb [2] https://github.com/XiaoMi/rdsn Willem Jiang 于2020年6月8日周一 下午4:12写道： > Hi, > > I just went through the proposal and found there are two source code > repos[1][2] which are forked. > > Are you planning to donate these two repos into Apache as a part of > Pegasus project? > It makes the donation of the Pegasus complicated as we need to address > this two code repo belonging issue first. > Normally we just contribute the patch to the upstream as it consume > lots of resources if we maintain the forked repo. > > [1] https://github.com/XiaoMi/rdsn > [2] https://github.com/XiaoMi/pegasus-rocksdb > > > Willem Jiang > > Twitter: willemjiang > Weibo: 姜宁willem > > On Tue, Jun 2, 2020 at 3:49 PM 吴涛 wrote: > > > > Dear Apache Incubator Community, > > > > I'd like to open up a discussion about incubating Pegasus at Apache. Our > proposal can be found at https://pegasus-kv.github.io/community/proposal > and is also included below. > > > > We are looking for possible Champion if anyone would like to volunteer. > Thanks a lot! > > > > Best regards > > Tao Wu > > > > Pegasus Proposal > > > > == Abstract == > > > > Pegasus is a distributed key-value storage system that is designed to be > horizontally scalable, strongly consistent and high-performance. > > > > - Pegasus codebase: https://github.com/XiaoMi/pegasus > > - Website: https://pegasus-kv.github.io > > > > == Proposal == > > > > Pegasus is a key-value database that delivers low-latency data access > together with horizontal scalability, using hash-based partitioning. > Pegasus uses PacificA protocol for strong consistency and RocksDB as the > underlying storage engine. > > > > We propose to contribute the Pegasus codebase and associated artifacts > (e.g., documentation, website content, etc.) to the Apache Software > Foundation, and aim to build an open community around Pegasus’s continued > development in the ‘Apache Way’. > > > > == Background == > > > > Apache HBase was recognized as mostly the only large-scale KV store > solution in XiaoMi Corp until Pegasus came out in 2015. The original > purpose of Pegasus was to solve the problems caused by HBase’s two-level > architecture and implementation, including high latency because of Java GC > and RPC overhead of the underlying distributed filesystem, and long > failover time because of single point of RegionServer and recovery overhead > of splitting and replaying the HLog files. > > > > Pegasus aims to fill the gap between Redis and HBase. As the former is > in-memory, low latency, but does not provide a strong-consistency > guarantee. And unlike the latter, Pegasus server is entirely written in C++ > and its read-write path relies merely on the local filesystem. > > > > Apart from performance requirements, we also need a storage system to > ensure multiple-level data safety and support fast data migration among > data centers, automatic load balancing, and online partition splitting. > > > > After investigating lots of existing storage systems in the open source > world, we could hardly find a suitable solution to satisfy all the > requirements. So the journey of Pegasus begins. > > > > === Rationale === > > > > Pegasus is a mature and active project which has been widely adopted in > XiaoMi. After the initial release of open source project in 2017, we have > seen a great amount of interest across a diverse set of users and companies. > > > > Our experiences at committers and PMC members on other Apache projects > have convinced us that having a long-term home at Apache foundation would > be a great fit for the project, to ensure that processes and procedures are > in place to keep project and community ‘healthy’ and free of any > commercial, political or legal faults. > > > > === Initial Goal === > > > > Move the existing codebase, website, documentation, and mailing lists to > Apache-hosted infrastructure. > > Work with the infrastructure team to implement and approve our code > review, build, and testing workflows in the context of the ASF. > > Incremental development and releases along with Apache guidelines. > > > > == Current Status == > > > > Pegasus has been an open-source project on GitHub > https://github.com/XiaoMi/pegasus since October 2017. > > > > === Meritocracy === > > > > The intent of this proposal is to start building a diverse developer and > user community around
Re: [Proposal] Pegasus - proposal for Apache Incubation
Thanks, Kevin. I'd very much appreciate it if you can become one of our mentors. Regards Tao. - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [Proposal] Pegasus - proposal for Apache Incubation
Hi, I just went through the proposal and found there are two source code repos[1][2] which are forked. Are you planning to donate these two repos into Apache as a part of Pegasus project? It makes the donation of the Pegasus complicated as we need to address this two code repo belonging issue first. Normally we just contribute the patch to the upstream as it consume lots of resources if we maintain the forked repo. [1] https://github.com/XiaoMi/rdsn [2] https://github.com/XiaoMi/pegasus-rocksdb Willem Jiang Twitter: willemjiang Weibo: 姜宁willem On Tue, Jun 2, 2020 at 3:49 PM 吴涛 wrote: > > Dear Apache Incubator Community, > > I'd like to open up a discussion about incubating Pegasus at Apache. Our > proposal can be found at https://pegasus-kv.github.io/community/proposal and > is also included below. > > We are looking for possible Champion if anyone would like to volunteer. > Thanks a lot! > > Best regards > Tao Wu > > Pegasus Proposal > > == Abstract == > > Pegasus is a distributed key-value storage system that is designed to be > horizontally scalable, strongly consistent and high-performance. > > - Pegasus codebase: https://github.com/XiaoMi/pegasus > - Website: https://pegasus-kv.github.io > > == Proposal == > > Pegasus is a key-value database that delivers low-latency data access > together with horizontal scalability, using hash-based partitioning. Pegasus > uses PacificA protocol for strong consistency and RocksDB as the underlying > storage engine. > > We propose to contribute the Pegasus codebase and associated artifacts (e.g., > documentation, website content, etc.) to the Apache Software Foundation, and > aim to build an open community around Pegasus’s continued development in the > ‘Apache Way’. > > == Background == > > Apache HBase was recognized as mostly the only large-scale KV store solution > in XiaoMi Corp until Pegasus came out in 2015. The original purpose of > Pegasus was to solve the problems caused by HBase’s two-level architecture > and implementation, including high latency because of Java GC and RPC > overhead of the underlying distributed filesystem, and long failover time > because of single point of RegionServer and recovery overhead of splitting > and replaying the HLog files. > > Pegasus aims to fill the gap between Redis and HBase. As the former is > in-memory, low latency, but does not provide a strong-consistency guarantee. > And unlike the latter, Pegasus server is entirely written in C++ and its > read-write path relies merely on the local filesystem. > > Apart from performance requirements, we also need a storage system to ensure > multiple-level data safety and support fast data migration among data > centers, automatic load balancing, and online partition splitting. > > After investigating lots of existing storage systems in the open source > world, we could hardly find a suitable solution to satisfy all the > requirements. So the journey of Pegasus begins. > > === Rationale === > > Pegasus is a mature and active project which has been widely adopted in > XiaoMi. After the initial release of open source project in 2017, we have > seen a great amount of interest across a diverse set of users and companies. > > Our experiences at committers and PMC members on other Apache projects have > convinced us that having a long-term home at Apache foundation would be a > great fit for the project, to ensure that processes and procedures are in > place to keep project and community ‘healthy’ and free of any commercial, > political or legal faults. > > === Initial Goal === > > Move the existing codebase, website, documentation, and mailing lists to > Apache-hosted infrastructure. > Work with the infrastructure team to implement and approve our code review, > build, and testing workflows in the context of the ASF. > Incremental development and releases along with Apache guidelines. > > == Current Status == > > Pegasus has been an open-source project on GitHub > https://github.com/XiaoMi/pegasus since October 2017. > > === Meritocracy === > > The intent of this proposal is to start building a diverse developer and user > community around Pegasus following the ASF meritocracy model. We plan to > invite more people as committers if they contribute to this project. > > === Releases === > > Pegasus has undergone multiple public releases, listed here: > https://github.com/XiaoMi/pegasus/releases. > > These old releases were not performed in the typical ASF fashion. We will > adopt the ASF source release process upon joining the incubator. > > === Code Reviews === > > Pegasus’s code reviews are currently public on Github > https://github.com/XiaoMi/pegasus/pulls. > > === Community === > > Pegasus seeks to develop developer and user communities during incubation. > > === Core Developers === > > Currently most of the core developers of Pegasus are working in the > KV-Storage Team of Xiaomi. Yingchun Lai is one of the Apache Kudu PMC >
Re: [Proposal] Pegasus - proposal for Apache Incubation
This looks very interesting. I've used Redis a long time and Pegasus looks very interesting. I'd like to see a champion and some mentors but otherwise I really like what I see here. Regards, KAM -- Kevin A. McGrail Member, Apache Software Foundation Chair Emeritus Apache SpamAssassin Project https://www.linkedin.com/in/kmcgrail - 703.798.0171 On Tue, Jun 2, 2020 at 3:49 AM 吴涛 wrote: > Dear Apache Incubator Community, > > I'd like to open up a discussion about incubating Pegasus at Apache. Our > proposal can be found at https://pegasus-kv.github.io/community/proposal > and is also included below. > > We are looking for possible Champion if anyone would like to volunteer. > Thanks a lot! > > Best regards > Tao Wu > > Pegasus Proposal > > == Abstract == > > Pegasus is a distributed key-value storage system that is designed to be > horizontally scalable, strongly consistent and high-performance. > > - Pegasus codebase: https://github.com/XiaoMi/pegasus > - Website: https://pegasus-kv.github.io > > == Proposal == > > Pegasus is a key-value database that delivers low-latency data access > together with horizontal scalability, using hash-based partitioning. > Pegasus uses PacificA protocol for strong consistency and RocksDB as the > underlying storage engine. > > We propose to contribute the Pegasus codebase and associated artifacts > (e.g., documentation, website content, etc.) to the Apache Software > Foundation, and aim to build an open community around Pegasus’s continued > development in the ‘Apache Way’. > > == Background == > > Apache HBase was recognized as mostly the only large-scale KV store > solution in XiaoMi Corp until Pegasus came out in 2015. The original > purpose of Pegasus was to solve the problems caused by HBase’s two-level > architecture and implementation, including high latency because of Java GC > and RPC overhead of the underlying distributed filesystem, and long > failover time because of single point of RegionServer and recovery overhead > of splitting and replaying the HLog files. > > Pegasus aims to fill the gap between Redis and HBase. As the former is > in-memory, low latency, but does not provide a strong-consistency > guarantee. And unlike the latter, Pegasus server is entirely written in C++ > and its read-write path relies merely on the local filesystem. > > Apart from performance requirements, we also need a storage system to > ensure multiple-level data safety and support fast data migration among > data centers, automatic load balancing, and online partition splitting. > > After investigating lots of existing storage systems in the open source > world, we could hardly find a suitable solution to satisfy all the > requirements. So the journey of Pegasus begins. > > === Rationale === > > Pegasus is a mature and active project which has been widely adopted in > XiaoMi. After the initial release of open source project in 2017, we have > seen a great amount of interest across a diverse set of users and companies. > > Our experiences at committers and PMC members on other Apache projects > have convinced us that having a long-term home at Apache foundation would > be a great fit for the project, to ensure that processes and procedures are > in place to keep project and community ‘healthy’ and free of any > commercial, political or legal faults. > > === Initial Goal === > > Move the existing codebase, website, documentation, and mailing lists to > Apache-hosted infrastructure. > Work with the infrastructure team to implement and approve our code > review, build, and testing workflows in the context of the ASF. > Incremental development and releases along with Apache guidelines. > > == Current Status == > > Pegasus has been an open-source project on GitHub > https://github.com/XiaoMi/pegasus since October 2017. > > === Meritocracy === > > The intent of this proposal is to start building a diverse developer and > user community around Pegasus following the ASF meritocracy model. We plan > to invite more people as committers if they contribute to this project. > > === Releases === > > Pegasus has undergone multiple public releases, listed here: > https://github.com/XiaoMi/pegasus/releases. > > These old releases were not performed in the typical ASF fashion. We will > adopt the ASF source release process upon joining the incubator. > > === Code Reviews === > > Pegasus’s code reviews are currently public on Github > https://github.com/XiaoMi/pegasus/pulls. > > === Community === > > Pegasus seeks to develop developer and user communities during incubation. > > === Core Developers === > > Currently most of the core developers of Pegasus are working in the > KV-Storage Team of Xiaomi. Yingchun Lai is one of the Apache Kudu PMC > members. Zuoyan Qin is an experienced open-source developer who created > sofa-pbrpc in his last job in Baidu. Wei Huang is also an active > contributor of Apache Doris (Incubating). > > - Zuoyan Qin (https://github.com/qinzuoyan) > - Yuchen He
[Proposal] Pegasus - proposal for Apache Incubation
Dear Apache Incubator Community, I'd like to open up a discussion about incubating Pegasus at Apache. Our proposal can be found at https://pegasus-kv.github.io/community/proposal and is also included below. We are looking for possible Champion if anyone would like to volunteer. Thanks a lot! Best regards Tao Wu Pegasus Proposal == Abstract == Pegasus is a distributed key-value storage system that is designed to be horizontally scalable, strongly consistent and high-performance. - Pegasus codebase: https://github.com/XiaoMi/pegasus - Website: https://pegasus-kv.github.io == Proposal == Pegasus is a key-value database that delivers low-latency data access together with horizontal scalability, using hash-based partitioning. Pegasus uses PacificA protocol for strong consistency and RocksDB as the underlying storage engine. We propose to contribute the Pegasus codebase and associated artifacts (e.g., documentation, website content, etc.) to the Apache Software Foundation, and aim to build an open community around Pegasus’s continued development in the ‘Apache Way’. == Background == Apache HBase was recognized as mostly the only large-scale KV store solution in XiaoMi Corp until Pegasus came out in 2015. The original purpose of Pegasus was to solve the problems caused by HBase’s two-level architecture and implementation, including high latency because of Java GC and RPC overhead of the underlying distributed filesystem, and long failover time because of single point of RegionServer and recovery overhead of splitting and replaying the HLog files. Pegasus aims to fill the gap between Redis and HBase. As the former is in-memory, low latency, but does not provide a strong-consistency guarantee. And unlike the latter, Pegasus server is entirely written in C++ and its read-write path relies merely on the local filesystem. Apart from performance requirements, we also need a storage system to ensure multiple-level data safety and support fast data migration among data centers, automatic load balancing, and online partition splitting. After investigating lots of existing storage systems in the open source world, we could hardly find a suitable solution to satisfy all the requirements. So the journey of Pegasus begins. === Rationale === Pegasus is a mature and active project which has been widely adopted in XiaoMi. After the initial release of open source project in 2017, we have seen a great amount of interest across a diverse set of users and companies. Our experiences at committers and PMC members on other Apache projects have convinced us that having a long-term home at Apache foundation would be a great fit for the project, to ensure that processes and procedures are in place to keep project and community ‘healthy’ and free of any commercial, political or legal faults. === Initial Goal === Move the existing codebase, website, documentation, and mailing lists to Apache-hosted infrastructure. Work with the infrastructure team to implement and approve our code review, build, and testing workflows in the context of the ASF. Incremental development and releases along with Apache guidelines. == Current Status == Pegasus has been an open-source project on GitHub https://github.com/XiaoMi/pegasus since October 2017. === Meritocracy === The intent of this proposal is to start building a diverse developer and user community around Pegasus following the ASF meritocracy model. We plan to invite more people as committers if they contribute to this project. === Releases === Pegasus has undergone multiple public releases, listed here: https://github.com/XiaoMi/pegasus/releases. These old releases were not performed in the typical ASF fashion. We will adopt the ASF source release process upon joining the incubator. === Code Reviews === Pegasus’s code reviews are currently public on Github https://github.com/XiaoMi/pegasus/pulls. === Community === Pegasus seeks to develop developer and user communities during incubation. === Core Developers === Currently most of the core developers of Pegasus are working in the KV-Storage Team of Xiaomi. Yingchun Lai is one of the Apache Kudu PMC members. Zuoyan Qin is an experienced open-source developer who created sofa-pbrpc in his last job in Baidu. Wei Huang is also an active contributor of Apache Doris (Incubating). - Zuoyan Qin (https://github.com/qinzuoyan) - Yuchen He (https://github.com/hycdong) - Tao Wu (https://github.com/neverchanje) - Yingchun Lai (https://github.com/acelyc111) - Wei Huang (https://github.com/vagetablechicken) - Shuo Jia (https://github.com/Shuo-Jia) - Liwei Zhao (https://github.com/levy5307) === Alignment === Pegasus is aligned with several other ASF projects. We are working on a new feature to load data from the HDFS filesystem. Pegasus can also generate and store checkpoints to HDFS, for both backup and analysis purpose. We currently support offline analysis on checkpoints powered by
