Hi Naga Thank you for the kind words and appreciate your vote very much. Yes, among everyone who has worked on contributing to the TSv2 code, there has been a strong sense of camaraderie.
Let me try to address your questions one by one. bq. Though i have not tested, just went through the jiras and features added. Thank you. Testing for TSv2 as such has been done at Twitter, Hortonworks, Huawei and Cloudera. bq. whether there was any other key out standing features left after phase 2 and their impacts ? Yes, we have some features that we would like to build into TSv2 and we have included those as part of "Current Status and Future Plans" in the documentation. Some of the future plans include: - Clustering of the readers - Migration and compatibility with v.1 - Timeline collectors as separate instances from node managers - Support for off-cluster clients - More robust storage fault tolerance - Better support for long-running apps - Support for ACLs - Offline (time-based periodic) aggregation for flows, users, and queues for reporting and analysis bq. it would interesting to share the plans (if available) of upstream projects like Tez, Spark to incorporate ATSv2. We've had an initial discussion with some of the Tez community members a few months back. Folks from Twitter, Yahoo!, Hortonworks and Huawei were part of the discussion. It was a productive session and we came away with a set of things to do right away, of which we have already added in features like support for sub-application entities and simple whitelisting of users so that Tez community can start experimenting with it. I believe some of the Tez Yahoo! folks are in the very early stages of exploring TSv2. Rohith Sharma from Hortonworks has done some more groundwork and has some advanced thoughts around the TSv2-Tez integration considerations. We also have had an initial discussion with the Federation team (Subru, Carlo) a while back. Accordingly, we have already updated some aspects of our schema to accommodate things like the application to cluster mappings. We have an open jira for Federation integration (YARN-5357) which is not included in this version. We have not yet reached out to Spark folks but would love to do so. thanks Vrushali On Wed, Aug 23, 2017 at 5:49 PM, Naganarasimha Garla < naganarasimha...@apache.org> wrote: > +1 (Binding), Great work guys, unwavering dedication and team effort. Kudos > to the whole team. > Though i have not tested, just went through the jiras and features added. > > Just wanted to know whether there was any other key out standing features > left after phase 2 and their impacts ? > And it would interesting to share the plans (if available) of upstream > projects like Tez, Spark to incorporate ATSv2. > > Regards, > + Naga > > > > On Thu, Aug 24, 2017 at 12:28 AM, varunsax...@apache.org < > varun.saxena.apa...@gmail.com> wrote: > > > Hi All, > > > > Just to update. > > Folks who may be interested in going through the documentation for the > > feature can refer to current documentation attached in pdf format, on the > > umbrella JIRA i.e. YARN-5355. > > > > Furthermore, we have run the YARN-5355 patch(with all the changes made in > > the branch) against trunk and the build is almost green barring a few > > checkstyle issues. The test failures which come up in the build are > > outstanding issues on trunk. > > Refer to https://issues.apache.org/jira/browse/YARN-5355?focusedCo > > mmentId=16138266&page=com.atlassian.jira.plugin.system. > > issuetabpanels:comment-tabpanel#comment-16138266 > > > > Thanks, > > Varun Saxena. > > > > On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan < > > vrushalic2...@gmail.com> wrote: > > > > > Hi folks, > > > > > > Per earlier discussion [1], I'd like to start a formal vote to merge > > > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote > > will > > > run for 7 days, and will end August 29 11:00 PM PDT. > > > > > > We have previously completed one merge onto trunk [3] and Timeline > > Service > > > v2 has been part of Hadoop release 3.0.0-alpha1. > > > > > > Since then, we have been working on extending the capabilities of > > Timeline > > > Service v2 in a feature branch [2] for a while, and we are reasonably > > > confident that the state of the feature meets the criteria to be merged > > > onto trunk and we'd love folks to get their hands on it in a test > > capacity > > > and provide valuable feedback so that we can make it production-ready. > > > > > > In a nutshell, Timeline Service v.2 delivers significant scalability > and > > > usability improvements based on a new architecture. What we would like > to > > > merge to trunk is termed "alpha 2" (milestone 2). The feature has a > > > complete end-to-end read/write flow with security and read level > > > authorization via whitelists. You should be able to start setting it up > > and > > > testing it. > > > > > > At a high level, the following are the key features that have been > > > implemented since alpha1: > > > - Security via Kerberos Authentication and delegation tokens > > > - Read side simple authorization via whitelist > > > - Client configurable entity sort ordering > > > - Richer REST APIs for apps, app attempts, containers, fetching metrics > > by > > > timerange, pagination, sub-app entities > > > - Support for storing sub-application entities (entities that exist > > outside > > > the scope of an application) > > > - Configurable TTLs (time-to-live) for tables, configurable table > > prefixes, > > > configurable hbase cluster > > > - Flow level aggregations done as dynamic (table level) coprocessors > > > - Uses latest stable HBase release 1.2.6 > > > > > > There are a total of 82 subtasks that were completed as part of this > > > effort. > > > > > > We paid close attention to ensure that once disabled Timeline Service > v.2 > > > does not impact existing functionality when disabled (by default). > > > > > > Special thanks to a team of folks who worked hard and contributed > towards > > > this effort with patches, reviews and guidance: Rohith Sharma K S, > Varun > > > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep > > > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack. > > > > > > Regards, > > > Vrushali > > > > > > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/ > msg27383.html > > > [2] https://issues.apache.org/jira/browse/YARN-5355 > > > [3] https://issues.apache.org/jira/browse/YARN-2928 > > > [4] https://github.com/apache/hadoop/commits/YARN-5355 > > > > > >
