Thanks Rohith for that update! Sounds good. Let us know when that testing is complete too.
On Sat, Aug 19, 2017 at 3:34 AM, Rohith Sharma K S < rohithsharm...@apache.org> wrote: > Hi Sangjin > > Thanks for bringing this point. > We did similar exercise with current YARN-5355 branch today. Tests are > validated against default configuration and timeline server v.1.5 as well. > > All YARN major features such as RM HA/Restart/work-preserving-restart, NM > restart, scheduling, sample MR/distributes shell jobs run are validated. > Everything looks fine. > > This testing will also be done once YARN-5355 branch code freezed > completely as well in couple of days. > > Thanks & Regards > Rohith Sharma K S > > On 18 August 2017 at 22:56, Sangjin Lee <sj...@apache.org> wrote: > > > Kudos to Vrushali and the team for getting ready for this large and > > important feature! I know a huge team effort went into this. I look > forward > > to seeing this merged. > > > > I'd like to ask one piece of due diligence. Could you please inspect > > rigorously to ensure that when disabled Timeline Service v.2 does not > > impact other features in any way? We did a similar exercise when we had > the > > first drop, and it would be good to repeat that... Thanks! > > > > Sangjin > > > > On Wed, Aug 16, 2017 at 11:44 AM, Andrew Wang <andrew.w...@cloudera.com> > > wrote: > > > > > Great, thanks Vrushali! Sounds good to me. > > > > > > I have a few procedural release notes comments I'll put on YARN-5355, > to > > > make sure we advertise this to our users appropriately. > > > > > > On Wed, Aug 16, 2017 at 11:32 AM, Vrushali Channapattan < > > > vrushal...@gmail.com> wrote: > > > > > > > Hi Andrew, > > > > > > > > Thanks for your response! > > > > > > > > There have been no changes to existing APIs since alpha1. > > > > > > > > We at Twitter have tested the feature to demonstrate it works at what > > we > > > > consider moderate scale but this did not include the security related > > > > testing. The security testing is in progress at present by Timeline > > > Service > > > > V2 team in the community and we think we will have more details on > this > > > > very soon. > > > > > > > > About the jiras under YARN-5355: Only 3 of those sub-tasks are what > we > > > > think of as "merge-blockers". The issues being targeted for merge are > > in > > > > [link1] below. There are about 59 jiras of which 56 are completed. > > > > > > > > We plan to make a new umbrella jira after the merge to trunk. We will > > > then > > > > create a new branch with the new jira name and move these open jiras > > > under > > > > YARN-5355 as subtasks of that new umbrella jira. > > > > > > > > thanks > > > > Vrushali > > > > [link1] https://issues.apache.org/jira/projects/YARN/versions/ > 12337991 > > > > > > > > > > > > On Wed, Aug 16, 2017 at 10:47 AM, Andrew Wang < > > andrew.w...@cloudera.com> > > > > wrote: > > > > > > > >> Hi Vrushali, > > > >> > > > >> Glad to hear this major dev milestone is nearing completion! > > > >> > > > >> Repeating my request on other merge [DISCUSS] threads, could you > > comment > > > >> on testing and API stability of this merge? Our timeline for beta1 > is > > > about > > > >> a month out, so there's not much time to fix things beforehand. > > > >> > > > >> Looking at YARN-5355 there are also many unresolved subtasks. Should > > > most > > > >> of these be moved out to a new umbrella? I'm wondering what needs to > > be > > > >> completed before sending the merge vote. > > > >> > > > >> Given that TSv2 is committed for 3.0.0 GA, I'm more willing to flex > > the > > > >> beta1 release date for this feature than others. Hopefully that > won't > > be > > > >> necessary though :) > > > >> > > > >> Best, > > > >> Andrew > > > >> > > > >> On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan < > > > >> vrushalic2...@gmail.com> wrote: > > > >> > > > >>> Looks like some of the hyperlinks appear messed up, my apologies, > > > >>> resending > > > >>> the same email with hopefully better looking content: > > > >>> > > > >>> Hi All, > > > >>> > > > >>> I'd like to open a discussion for merging Timeline Service v2 > > > (YARN-5355) > > > >>> to trunk in a few weeks. > > > >>> > > > >>> We have previously completed one merge onto trunk [1] and Timeline > > > >>> Service > > > >>> v2 has been part of Hadoop release 3.0.0-alpha1. > > > >>> > > > >>> Since then, we have been working on extending the capabilities of > > > >>> Timeline > > > >>> Service v2 in a feature branch [2]. There are a few related issues > > > >>> pending > > > >>> that are being actively worked upon and tested. As soon as they are > > > >>> resolved, we plan on starting a merge vote within the next two > weeks. > > > The > > > >>> goal is to get this into hadoop3 beta. > > > >>> > > > >>> We have paid close attention to ensure that once disabled Timeline > > > >>> Service > > > >>> v2 does not impact existing functionality when disabled (by > default). > > > >>> > > > >>> At a high level, following are the key features that have been > > > >>> implemented > > > >>> since 3.0.0-alpha1: > > > >>> - Security (via Kerberos Authentication & delegation tokens) > > > [YARN-3053] > > > >>> - Timeline server usability improvements [timeline-server > > > >>> <https://issues.apache.org/jira/browse/YARN-5715?jql=project > > > >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c > > > >>> omponent%20%3D%20timelineserver%20AND%20labels%20!%3D%20atsv > > > >>> 2-hbase%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC% > > > >>> 2C%20created%20ASC> > > > >>> ] > > > >>> - HBase specific improvements [atsv2-hbase > > > >>> <https://issues.apache.org/jira/browse/YARN-6604?jql=project > > > >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20l > > > >>> abels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C% > > > >>> 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC> > > > >>> ] > > > >>> - REST API additions and improvements [timeline-reader > > > >>> <https://issues.apache.org/jira/browse/YARN-5739?jql=project > > > >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c > > > >>> omponent%20in%20(timelineclient%2C%20timelinereader)% > > > >>> 20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C% > > 20created%20ASC> > > > >>> ] > > > >>> - Reader side simple authorization via whitelist [YARN-6820] > > > >>> > > > >>> We would love to get your thoughts on this before we open a real > > voting > > > >>> thread. > > > >>> > > > >>> Special thanks to a team of folks who worked hard and contributed > > > towards > > > >>> this effort via patches, reviews and guidance: Rohith Sharma K S, > > Varun > > > >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, > Joep > > > >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack. > > > >>> > > > >>> Thanks > > > >>> Vrushali > > > >>> [1] Merge to trunk: http://www.mail-archive.com/ > > yarn-dev@hadoop.apache > > > . > > > >>> org/msg23897.html > > > >>> <http://www.mail-archive.com/yarn-dev@hadoop.apache.org/ > > msg23897.html> > > > >>> [2] feature branch YARN-5355 commits: https://github.com/ap > > > >>> ache/hadoop/commits/YARN-5355 > > > >>> > > > >>> <https://github.com/apache/hadoop/commits/YARN-5355> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan < > > > >>> vrushal...@gmail.com> wrote: > > > >>> > > > >>> > Hi All > > > >>> > > > > >>> > I’d like to open a discussion for merging Timeline Service v.2 > > > >>> (YARN-5355) > > > >>> > to trunk in a few weeks. > > > >>> > > > > >>> > We have previously completed one merge onto trunk [1] and > Timeline > > > >>> Service > > > >>> > v2 has been part of Hadoop release 3.0.0-alpha1. > > > >>> > > > > >>> > Since then, we have been working on extending the capabilities of > > > >>> Timeline > > > >>> > Service v2 in a feature branch [2]. There are a few related > issues > > > >>> pending > > > >>> > that are being actively worked upon & tested. As soon as they are > > > >>> resolved, > > > >>> > we plan on starting a merge vote within the next 2 weeks. The > goal > > is > > > >>> to > > > >>> > get this in for Hadoop3 beta. > > > >>> > > > > >>> > We have paid close attention to ensure that once disabled > Timeline > > > >>> Service > > > >>> > v.2 does not impact existing functionality when disabled (by > > > default). > > > >>> > > > > >>> > At a high level, following are the key features that have been > > > >>> implemented > > > >>> > since 3.0.0-alpha1: > > > >>> > - Security (via Kerberos Authentication & delegation tokens) at > the > > > >>> writer > > > >>> > [YARN-3053] > > > >>> > - Timeline server usability improvements [timeline-server > > > >>> > <https://issues.apache.org/jira/browse/YARN-5715?jql= > > > >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN- > > > >>> > 5355%20AND%20component%20%3D%20timelineserver%20AND% > > > >>> > 20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated% > > > >>> > 20ASC%2C%20priority%20DESC%2C%20created%20ASC> > > > >>> > ] > > > >>> > - HBase specific improvements [atsv2-hbase > > > >>> > <https://issues.apache.org/jira/browse/YARN-6604?jql= > > > >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN- > > > >>> > 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updat > > > >>> ed%20DESC%2C% > > > >>> > 20affectedVersion%20DESC%2C%20priority%20DESC%2C% > 20created%20ASC> > > > >>> > ] > > > >>> > - REST API additions & improvements [timeline-reader > > > >>> > <https://issues.apache.org/jira/browse/YARN-5739?jql= > > > >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN- > > > >>> > 5355%20AND%20component%20in%20%28timelineclient%2C% > > > >>> > 20timelinereader%29%20ORDER%20BY%20updated%20ASC%2C%20priori > > > >>> ty%20DESC%2C% > > > >>> > 20created%20ASC> > > > >>> > ] > > > >>> > - Reader side simple authorization via whitelist [YARN-6820] > > > >>> > > > > >>> > We would love to get your thoughts on these and more before we > > open a > > > >>> real > > > >>> > voting thread. > > > >>> > > > > >>> > Special thanks to a team of folks who worked hard and contributed > > > >>> towards > > > >>> > this effort with patches, reviews and guidance: Rohith Sharma K > S, > > > >>> Varun > > > >>> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, > > Joep > > > >>> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack. > > > >>> > > > > >>> > Thanks! > > > >>> > Vrushali > > > >>> > > > > >>> > [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop. > > > >>> > apache.org/msg23897.html > > > >>> > [2] feature branch YARN-5355 commits: https://github.com/ > > > >>> > apache/hadoop/commits/YARN-5355 > > > >>> > > > > >>> > > > >> > > > >> > > > > > > > > > >
