Zhijie, Sounds good.
On the last part of my previous email, lets wait till YARN-321 is updated to latest trunk. > I would like to see #1 and #2 taken care before making a decision. > The reason for this is that if the source changes outside of the AHS > are too pervasive, then we may end up be making difficult backports > from trunk to branch-2 and release branches because of the delta. > > Can we get a rebase of the YARN-321 to the head of trunk to see if > my concerns are valid or not? Thanks. On Mon, Jan 6, 2014 at 8:21 PM, Zhijie Shen <zs...@hortonworks.com> wrote: > Hi folks, > > Thanks for replying. Please see the response bellow. > > bq. 2* The last update of YARN-321 was done in NOV10, this was done > from branch-2 > (that seems a NIT as it should be against trunk) > > Basically it's a discussion thread. I'm already in the process of updating > the branch. As I mentioned before, one of the motivation of merge branch > YARN-321 is to prevent it from further falling behind. > > bq. 1* The merge must be done in trunk first (then, ideally, from trunk > into branch-2) > bq. 3* IMO, until we don’t have security, we should merge into trunk only > > Yes, we plan to merge to trunk, then to branch-2, but I agree, let's have > security ready before going to branch-2. We can continue with security on > trunk. > > bq. Regarding doc, while we don't necessarily need full documentation > before merging, my feeling is that we should at least have a page on it > that will allow give cluster operators a sketch of its role, APIs, and > implications. > > Agree, let's prepare a doc as well. > > bq. My opinion is that we should mark each API @Stable unless there is a > strong reason for it not to be. When I say APIs, I am thinking of the REST > APIs, RPC interface, and RM<->AHS shared-bus. > > I'm afraid it's still too early to confirm which APIs can be marked > @Stable. How about not doing this until security is ready and refactoring > duplicate code is done? At that time, we should have a clearer picture > about it. > > Thanks, > Zhijie > > > On Mon, Jan 6, 2014 at 3:02 PM, Alejandro Abdelnur <t...@cloudera.com > >wrote: > > > This is great news. A few things to consider before doing the merge: > > > > 1* The merge must be done in trunk first (then, ideally, from trunk into > > branch-2) > > 2* The last update of YARN-321 was done in NOV10, this was done from > > branch-2 (that seems a NIT as it should be against trunk) > > 3* IMO, until we don’t have security, we should merge into trunk only > > > > I would like to see #1 and #2 taken care before making a decision. The > > reason for this is that if the source changes outside of the AHS are too > > pervasive, then we may end up be making difficult backports from trunk to > > branch-2 and release branches because of the delta. > > > > Can we get a rebase of the YARN-321 to the head of trunk to see if my > > concerns are valid or not? > > > > Thanks. > > > > Alejandro > > > > > > > > On Mon, Jan 6, 2014 at 1:11 AM, Sandy Ryza <sandy.r...@cloudera.com> > > wrote: > > > > > Very excited for this feature and appreciative of all the work put into > > > this. Reviewed the JIRA and my only two remaining concerns are about > > > documentation and API stability. > > > > > > Regarding doc, while we don't necessarily need full documentation > before > > > merging, my feeling is that we should at least have a page on it that > > will > > > allow give cluster operators a sketch of its role, APIs, and > > implications. > > > If this doesn't exist yet, would be happy to help with reviewing. > > > > > > Regarding APIs, I imagine some will jump on this as soon as it becomes > > part > > > of a release as it's a fairly essential feature for non-MR apps. My > > > opinion is that we should mark each API @Stable unless there is a > strong > > > reason for it not to be. When I say APIs, I am thinking of the REST > > APIs, > > > RPC interface, and RM<->AHS shared-bus. > > > > > > -Sandy > > > > > > > > > On Sun, Jan 5, 2014 at 8:33 PM, lohit <lohit.vijayar...@gmail.com> > > wrote: > > > > > > > +1 to merge into branch2 now. > > > > On Jan 5, 2014 6:22 PM, "Zhijie Shen" <zs...@hortonworks.com> wrote: > > > > > > > > > Hi folks, > > > > > > > > > > Majority of the functionality of Application History Server has > been > > > > > completed on branch YARN-321. AHS can now work end-to-end. > > > > ResourceManager > > > > > records the historical information of the application, the > > application > > > > > attempt and the container in terms of events via a history writer > on > > a > > > > > separate thread. The historical information is going to be > persisted > > on > > > > > HDFS. On the other side, an application history server runs as a > > > separate > > > > > process, and it allows users to access the historical information > via > > > RPC > > > > > interface, web UI and REST APIs. > > > > > > > > > > According to the proposal, the only major missing piece is > security. > > > > Except > > > > > it, YARN-321 should be pretty much ready to be merged to Branch-2. > We > > > > > propose to merge YARN-321 to Branch-2 now, such that we can prevent > > > > > YARN-321 from falling behind too much, and reduce the effort of > > editing > > > > > merge conflicts. After merge, we can continue on security, > > refactoring > > > > > duplicate code, fixing bugs and other improvements. > > > > > > > > > > Anyone who is interested in looking at AHS can review/play with > > > YARN-321 > > > > > branch: > > http://svn.apache.org/viewvc/hadoop/common/branches/YARN-321/ > > > > > > > > > > You can also have a look at the latest design doc: > > > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/secure/attachment/12619638/Generic%20Application%20History%20-%20Design-20131219.pdf > > > > > > > > > > If there are no objections, we'll push towards updating the branch, > > > > running > > > > > the patch through Jenkins and getting ready for merge vote by the > end > > > of > > > > > this week. > > > > > > > > > > Thanks, > > > > > Zhijie > > > > > > > > > > -- > > > > > Zhijie Shen > > > > > Hortonworks Inc. > > > > > http://hortonworks.com/ > > > > > > > > > > -- > > > > > CONFIDENTIALITY NOTICE > > > > > NOTICE: This message is intended for the use of the individual or > > > entity > > > > to > > > > > which it is addressed and may contain information that is > > confidential, > > > > > privileged and exempt from disclosure under applicable law. If the > > > reader > > > > > of this message is not the intended recipient, you are hereby > > notified > > > > that > > > > > any printing, copying, dissemination, distribution, disclosure or > > > > > forwarding of this communication is strictly prohibited. If you > have > > > > > received this communication in error, please contact the sender > > > > immediately > > > > > and delete it from your system. Thank You. > > > > > > > > > > > > > > > > > > > > -- > > Alejandro > > > > > > -- > Zhijie Shen > Hortonworks Inc. > http://hortonworks.com/ > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. > -- Alejandro
