Re: [VOTE] Merge HDFS-13891(RBF) to trunk
+1(non-binding). Regards, - Takanobu From: Xiaoqiao He Sent: Monday, June 10, 2019 15:47 To: Ranith Sardar Cc: Brahma Reddy Battula; Hadoop Common; Hdfs-dev Subject: Re: [VOTE] Merge HDFS-13891(RBF) to trunk +1 (non-binding) - Try to merge branch HDFS-13891(RBF) to trunk at local and no conflict or failure. - Built from merged sources. - Ran 85 RBF test class at local and result shows: (Tests run: 639, Failures: 2, Errors: 2, Skipped: 2) - failed tests include #TestRouterWithSecureStartup and #TestRouterHttpDelegationToken. I don't think it is blocked issue. Thanks Brahma for organizing the merge. On Mon, Jun 10, 2019 at 12:27 PM Ranith Sardar wrote: > +1 (Non-binding) > > -Original Message- > From: Brahma Reddy Battula [mailto:bra...@apache.org] > Sent: 09 June 2019 19:31 > To: Hadoop Common ; Hdfs-dev < > hdfs-...@hadoop.apache.org> > Subject: [VOTE] Merge HDFS-13891(RBF) to trunk > > Updated mail... > > -- Forwarded message - > From: Brahma Reddy Battula > Date: Sun, Jun 9, 2019 at 7:26 PM > Subject: Re: [VOTE] Merge HDFS-13891(RBF) to trunk > To: Xiaoqiao He > Cc: Akira Ajisaka , Chittaranjan Hota < > chitts.h...@gmail.com>, Giovanni Matteo Fumarola < > giovanni.fumar...@gmail.com>, Hadoop Common , > Hdfs-dev , Iñigo Goiri > > > Hi All, > > Given the positive response to the discussion thread [1], here is the > formal vote thread to merge HDFS-13891 in to trunk. > > Summary of code changes: > 1. Code changes for this branch are done in the hadoop-hdfs-rbf > subproject, there is no impact to hadoop-hdfs and hadoop-common. > 2. Added Security support for RBF > 3. Added Missing Client Protocol API's > 4. Bug Fixes/ Improvments > > > The vote will run for 7 days, ending Sat June 15th. I will start this > vote with my +1. > > Regards, > Brahma Reddy Battula > > 1). > > https://lists.apache.org/thread.html/cdc2e084874b30bf6af2dd827bcbdba4ab8d3d983a8b9796e61608e6@%3Ccommon-dev.hadoop.apache.org%3E > > > On Wed, Jun 5, 2019 at 3:44 PM Xiaoqiao He wrote: > > > Thanks Brahma for starting the thread. > > +1 for merging. > > > > He Xiaoqiao > > > > On Tue, Jun 4, 2019 at 1:53 PM Chittaranjan Hota > > > > wrote: > > > >> Thanks Brahma initiating this. > >> +1(non-binding) for merge. > >> > >> @Uber we have almost all changes specially rbf security in production > >> for a while now without issues. > >> > >> On Mon, Jun 3, 2019 at 12:56 PM Giovanni Matteo Fumarola < > >> giovanni.fumar...@gmail.com> wrote: > >> > >> > +1 on merging. > >> > > >> > Thanks Brahma for starting the thread. > >> > > >> > On Mon, Jun 3, 2019 at 10:00 AM Iñigo Goiri > wrote: > >> > > >> > > Thank you Brahma for pushing this. > >> > > > >> > > As you mentioned, we have already taken most of the changes into > >> > > production. > >> > > I want to highlight that the main contribution is the addition of > >> > security. > >> > > We have been able to test this at a smaller scale (~500 servers > >> > > and 4 > >> > > subclusters) and the performance is great with our current > >> > > ZooKeeper deployment. > >> > > I would also like to highlight that all the changes are > >> > > constrained to hadoop-hdfs-rbf and there is no differences in > commons or HDFS. > >> > > > >> > > +1 on merging > >> > > > >> > > Inigo > >> > > > >> > > On Sun, Jun 2, 2019 at 10:19 PM Akira Ajisaka > >> > > > >> > wrote: > >> > > > >> > > > Thanks Brahma for starting the discussion. > >> > > > I'm +1 for merging this. > >> > > > > >> > > > FYI: In Yahoo! JAPAN, deployed all these changes in 20 nodes > >> > > > cluster with 2 routers (not in production) and running several > tests. > >> > > > > >> > > > Regards, > >> > > > Akira > >> > > > > >> > > > On Sun, Jun 2, 2019 at 12:40 PM Brahma Reddy Battula < > >> > bra...@apache.org> > >> > > > wrote: > >> > > > > > >> > > > > Dear Hadoop Developers > >> > > > > > >> > > > > I would like to propose RBF Branch
Re: [VOTE] Merge HDFS-13891(RBF) to trunk
+1 (non-binding) - Try to merge branch HDFS-13891(RBF) to trunk at local and no conflict or failure. - Built from merged sources. - Ran 85 RBF test class at local and result shows: (Tests run: 639, Failures: 2, Errors: 2, Skipped: 2) - failed tests include #TestRouterWithSecureStartup and #TestRouterHttpDelegationToken. I don't think it is blocked issue. Thanks Brahma for organizing the merge. On Mon, Jun 10, 2019 at 12:27 PM Ranith Sardar wrote: > +1 (Non-binding) > > -Original Message- > From: Brahma Reddy Battula [mailto:bra...@apache.org] > Sent: 09 June 2019 19:31 > To: Hadoop Common ; Hdfs-dev < > hdfs-...@hadoop.apache.org> > Subject: [VOTE] Merge HDFS-13891(RBF) to trunk > > Updated mail... > > -- Forwarded message - > From: Brahma Reddy Battula > Date: Sun, Jun 9, 2019 at 7:26 PM > Subject: Re: [VOTE] Merge HDFS-13891(RBF) to trunk > To: Xiaoqiao He > Cc: Akira Ajisaka , Chittaranjan Hota < > chitts.h...@gmail.com>, Giovanni Matteo Fumarola < > giovanni.fumar...@gmail.com>, Hadoop Common , > Hdfs-dev , Iñigo Goiri > > > Hi All, > > Given the positive response to the discussion thread [1], here is the > formal vote thread to merge HDFS-13891 in to trunk. > > Summary of code changes: > 1. Code changes for this branch are done in the hadoop-hdfs-rbf > subproject, there is no impact to hadoop-hdfs and hadoop-common. > 2. Added Security support for RBF > 3. Added Missing Client Protocol API's > 4. Bug Fixes/ Improvments > > > The vote will run for 7 days, ending Sat June 15th. I will start this > vote with my +1. > > Regards, > Brahma Reddy Battula > > 1). > > https://lists.apache.org/thread.html/cdc2e084874b30bf6af2dd827bcbdba4ab8d3d983a8b9796e61608e6@%3Ccommon-dev.hadoop.apache.org%3E > > > On Wed, Jun 5, 2019 at 3:44 PM Xiaoqiao He wrote: > > > Thanks Brahma for starting the thread. > > +1 for merging. > > > > He Xiaoqiao > > > > On Tue, Jun 4, 2019 at 1:53 PM Chittaranjan Hota > > > > wrote: > > > >> Thanks Brahma initiating this. > >> +1(non-binding) for merge. > >> > >> @Uber we have almost all changes specially rbf security in production > >> for a while now without issues. > >> > >> On Mon, Jun 3, 2019 at 12:56 PM Giovanni Matteo Fumarola < > >> giovanni.fumar...@gmail.com> wrote: > >> > >> > +1 on merging. > >> > > >> > Thanks Brahma for starting the thread. > >> > > >> > On Mon, Jun 3, 2019 at 10:00 AM Iñigo Goiri > wrote: > >> > > >> > > Thank you Brahma for pushing this. > >> > > > >> > > As you mentioned, we have already taken most of the changes into > >> > > production. > >> > > I want to highlight that the main contribution is the addition of > >> > security. > >> > > We have been able to test this at a smaller scale (~500 servers > >> > > and 4 > >> > > subclusters) and the performance is great with our current > >> > > ZooKeeper deployment. > >> > > I would also like to highlight that all the changes are > >> > > constrained to hadoop-hdfs-rbf and there is no differences in > commons or HDFS. > >> > > > >> > > +1 on merging > >> > > > >> > > Inigo > >> > > > >> > > On Sun, Jun 2, 2019 at 10:19 PM Akira Ajisaka > >> > > > >> > wrote: > >> > > > >> > > > Thanks Brahma for starting the discussion. > >> > > > I'm +1 for merging this. > >> > > > > >> > > > FYI: In Yahoo! JAPAN, deployed all these changes in 20 nodes > >> > > > cluster with 2 routers (not in production) and running several > tests. > >> > > > > >> > > > Regards, > >> > > > Akira > >> > > > > >> > > > On Sun, Jun 2, 2019 at 12:40 PM Brahma Reddy Battula < > >> > bra...@apache.org> > >> > > > wrote: > >> > > > > > >> > > > > Dear Hadoop Developers > >> > > > > > >> > > > > I would like to propose RBF Branch (HDFS-13891) merge into > trunk. > >> We > >> > > have > >> > > > > been working on this feature from last several months. > >> > > > > This feature work received the contributions from different > &
Re: [VOTE] Merge HDFS-13891(RBF) to trunk
+1 -Akira On Sun, Jun 9, 2019 at 10:56 PM Brahma Reddy Battula wrote: > > Hi All, > > Given the positive response to the discussion thread [1], here is the > formal vote thread to merge HDFS-13891 in to trunk. > > Summary of code changes: > 1. Code changes for this branch are done in the hadoop-hdfs-rbf > subproject, there is no impact to hadoop-hdfs and hadoop-common. > 2. Added Security support for RBF > 3. Added Missing Client Protocol API's > 4. Bug Fixes/ Improvments > > > The vote will run for 7 days, ending Sat June 15th. I will start this > vote with my +1. > > Regards, > Brahma Reddy Battula > > 1). > https://lists.apache.org/thread.html/cdc2e084874b30bf6af2dd827bcbdba4ab8d3d983a8b9796e61608e6@%3Ccommon-dev.hadoop.apache.org%3E > > > On Wed, Jun 5, 2019 at 3:44 PM Xiaoqiao He wrote: > > > Thanks Brahma for starting the thread. > > +1 for merging. > > > > He Xiaoqiao > > > > On Tue, Jun 4, 2019 at 1:53 PM Chittaranjan Hota > > wrote: > > > >> Thanks Brahma initiating this. > >> +1(non-binding) for merge. > >> > >> @Uber we have almost all changes specially rbf security in production for > >> a > >> while now without issues. > >> > >> On Mon, Jun 3, 2019 at 12:56 PM Giovanni Matteo Fumarola < > >> giovanni.fumar...@gmail.com> wrote: > >> > >> > +1 on merging. > >> > > >> > Thanks Brahma for starting the thread. > >> > > >> > On Mon, Jun 3, 2019 at 10:00 AM Iñigo Goiri wrote: > >> > > >> > > Thank you Brahma for pushing this. > >> > > > >> > > As you mentioned, we have already taken most of the changes into > >> > > production. > >> > > I want to highlight that the main contribution is the addition of > >> > security. > >> > > We have been able to test this at a smaller scale (~500 servers and 4 > >> > > subclusters) and the performance is great with our current ZooKeeper > >> > > deployment. > >> > > I would also like to highlight that all the changes are constrained to > >> > > hadoop-hdfs-rbf and there is no differences in commons or HDFS. > >> > > > >> > > +1 on merging > >> > > > >> > > Inigo > >> > > > >> > > On Sun, Jun 2, 2019 at 10:19 PM Akira Ajisaka > >> > wrote: > >> > > > >> > > > Thanks Brahma for starting the discussion. > >> > > > I'm +1 for merging this. > >> > > > > >> > > > FYI: In Yahoo! JAPAN, deployed all these changes in 20 nodes cluster > >> > > > with 2 routers (not in production) and running several tests. > >> > > > > >> > > > Regards, > >> > > > Akira > >> > > > > >> > > > On Sun, Jun 2, 2019 at 12:40 PM Brahma Reddy Battula < > >> > bra...@apache.org> > >> > > > wrote: > >> > > > > > >> > > > > Dear Hadoop Developers > >> > > > > > >> > > > > I would like to propose RBF Branch (HDFS-13891) merge into trunk. > >> We > >> > > have > >> > > > > been working on this feature from last several months. > >> > > > > This feature work received the contributions from different > >> > companies. > >> > > > All > >> > > > > of the feature development happened smoothly and collaboratively > >> in > >> > > > JIRAs. > >> > > > > > >> > > > > Kindly do take a look at the branch and raise issues/concerns that > >> > need > >> > > > to > >> > > > > be addressed before the merge. > >> > > > > > >> > > > > *Highlights of HDFS-13891 Branch:* > >> > > > > = > >> > > > > > >> > > > > Adding Security to RBF(1) > >> > > > > Adding Missing Client API's(2) > >> > > > > Improvements/Bug Fixing > >> > > > > Critical - HDFS-13637, HDFS-13834 > >> > > > > > >> > > > > *Commits:* > >> > > > > > >> > > > > > >> > > > > No of JIRAs Resolved: 72 > >> > > > > > >> > > > > All this commits are in RBF Module. No changes in hdfs/common. > >> > > > > > >> > > > > *Tested Cluster:* > >> > > > > = > >> > > > > > >> > > > > Most of these changes verified at Uber,Microsoft,Huawei and some > >> > other > >> > > > > compaines. > >> > > > > > >> > > > > *Uber*: Most changes are running in production @Uber including the > >> > > > critical > >> > > > > security changes, HDFS Clusters are 4000+ nodes with 8 HDFS > >> Routers. > >> > > > > Zookeeper as a state store to hold delegation tokens were also > >> stress > >> > > > > tested to hold more than 2 Million tokens. --CR Hota > >> > > > > > >> > > > > *MicroSoft*: Most of these changes are currently running in > >> > production > >> > > at > >> > > > > Microsoft.The security has also been tested in a 500 server > >> cluster > >> > > with > >> > > > 4 > >> > > > > subclsuters. --Inigo Goiri > >> > > > > > >> > > > > *Huawei* : Deployed all this changes in 20 node cluster with 3 > >> > > > > routers.Planning deploy 10K production cluster. > >> > > > > > >> > > > > *Contributors:* > >> > > > > === > >> > > > > > >> > > > > Many thanks to Akira Ajisaka,Mohammad Arshad,Takanobu > >> Asanuma,Shubham > >> > > > > Dewan,CR Hota,Fei Hui,Inigo Goiri,Dibyendu Karmakar,Fengna Li,Gang > >> > > > > Li,Surendra Singh Lihore,Ranith Sardar,Ayush Saxena,He > >> > > Xiaoqiao,Sherwood > >> > >
Re: [VOTE] Merge HDFS-13891(RBF) to trunk
Hi All, Given the positive response to the discussion thread [1], here is the formal vote thread to merge HDFS-13891 in to trunk. Summary of code changes: 1. Code changes for this branch are done in the hadoop-hdfs-rbf subproject, there is no impact to hadoop-hdfs and hadoop-common. 2. Added Security support for RBF 3. Added Missing Client Protocol API's 4. Bug Fixes/ Improvments The vote will run for 7 days, ending Sat June 15th. I will start this vote with my +1. Regards, Brahma Reddy Battula 1). https://lists.apache.org/thread.html/cdc2e084874b30bf6af2dd827bcbdba4ab8d3d983a8b9796e61608e6@%3Ccommon-dev.hadoop.apache.org%3E On Wed, Jun 5, 2019 at 3:44 PM Xiaoqiao He wrote: > Thanks Brahma for starting the thread. > +1 for merging. > > He Xiaoqiao > > On Tue, Jun 4, 2019 at 1:53 PM Chittaranjan Hota > wrote: > >> Thanks Brahma initiating this. >> +1(non-binding) for merge. >> >> @Uber we have almost all changes specially rbf security in production for >> a >> while now without issues. >> >> On Mon, Jun 3, 2019 at 12:56 PM Giovanni Matteo Fumarola < >> giovanni.fumar...@gmail.com> wrote: >> >> > +1 on merging. >> > >> > Thanks Brahma for starting the thread. >> > >> > On Mon, Jun 3, 2019 at 10:00 AM Iñigo Goiri wrote: >> > >> > > Thank you Brahma for pushing this. >> > > >> > > As you mentioned, we have already taken most of the changes into >> > > production. >> > > I want to highlight that the main contribution is the addition of >> > security. >> > > We have been able to test this at a smaller scale (~500 servers and 4 >> > > subclusters) and the performance is great with our current ZooKeeper >> > > deployment. >> > > I would also like to highlight that all the changes are constrained to >> > > hadoop-hdfs-rbf and there is no differences in commons or HDFS. >> > > >> > > +1 on merging >> > > >> > > Inigo >> > > >> > > On Sun, Jun 2, 2019 at 10:19 PM Akira Ajisaka >> > wrote: >> > > >> > > > Thanks Brahma for starting the discussion. >> > > > I'm +1 for merging this. >> > > > >> > > > FYI: In Yahoo! JAPAN, deployed all these changes in 20 nodes cluster >> > > > with 2 routers (not in production) and running several tests. >> > > > >> > > > Regards, >> > > > Akira >> > > > >> > > > On Sun, Jun 2, 2019 at 12:40 PM Brahma Reddy Battula < >> > bra...@apache.org> >> > > > wrote: >> > > > > >> > > > > Dear Hadoop Developers >> > > > > >> > > > > I would like to propose RBF Branch (HDFS-13891) merge into trunk. >> We >> > > have >> > > > > been working on this feature from last several months. >> > > > > This feature work received the contributions from different >> > companies. >> > > > All >> > > > > of the feature development happened smoothly and collaboratively >> in >> > > > JIRAs. >> > > > > >> > > > > Kindly do take a look at the branch and raise issues/concerns that >> > need >> > > > to >> > > > > be addressed before the merge. >> > > > > >> > > > > *Highlights of HDFS-13891 Branch:* >> > > > > = >> > > > > >> > > > > Adding Security to RBF(1) >> > > > > Adding Missing Client API's(2) >> > > > > Improvements/Bug Fixing >> > > > > Critical - HDFS-13637, HDFS-13834 >> > > > > >> > > > > *Commits:* >> > > > > >> > > > > >> > > > > No of JIRAs Resolved: 72 >> > > > > >> > > > > All this commits are in RBF Module. No changes in hdfs/common. >> > > > > >> > > > > *Tested Cluster:* >> > > > > = >> > > > > >> > > > > Most of these changes verified at Uber,Microsoft,Huawei and some >> > other >> > > > > compaines. >> > > > > >> > > > > *Uber*: Most changes are running in production @Uber including the >> > > > critical >> > > > > security changes, HDFS Clusters are 4000+ nodes with 8 HDFS >> Routers. >> > > > > Zookeeper as a state store to hold delegation tokens were also >> stress >> > > > > tested to hold more than 2 Million tokens. --CR Hota >> > > > > >> > > > > *MicroSoft*: Most of these changes are currently running in >> > production >> > > at >> > > > > Microsoft.The security has also been tested in a 500 server >> cluster >> > > with >> > > > 4 >> > > > > subclsuters. --Inigo Goiri >> > > > > >> > > > > *Huawei* : Deployed all this changes in 20 node cluster with 3 >> > > > > routers.Planning deploy 10K production cluster. >> > > > > >> > > > > *Contributors:* >> > > > > === >> > > > > >> > > > > Many thanks to Akira Ajisaka,Mohammad Arshad,Takanobu >> Asanuma,Shubham >> > > > > Dewan,CR Hota,Fei Hui,Inigo Goiri,Dibyendu Karmakar,Fengna Li,Gang >> > > > > Li,Surendra Singh Lihore,Ranith Sardar,Ayush Saxena,He >> > > Xiaoqiao,Sherwood >> > > > > Zheng,Daryn Sharp,VinayaKumar B,Anu Engineer for invloving >> > discussions >> > > > and >> > > > > contributing to this. >> > > > > >> > > > > *Future Tasks:* >> > > > > >> > > > > >> > > > > will cleanup the jira's under this umbrella and contiue to work. >> > > > > >> > > > > Reference: >> > > > > 1)