Re: [Discuss] Merge YARN-3368 to trunk

[Rohith Sharma K S]

Thanks Wangda and all other contributors to new YarnWeb UI work.

I am able to built and deploy the Yarn WebUI from branch YARN-3368. The
look and feel of UI is pretty much impressive. The new web UI hosted along
with same http port of existing web UI. I could able to navigate all the
tabs such as Cluster Overview, Queues, Applications, Nodes pages and
verified information displayed.

As mentioned by Wangda, Timelinev2 integration is next phase of work to be
done. Merging will definitely help for moving forward for Atsv2
integration.

I am +1 for merging to trunk.

Thanks & Regards
Rohith Sharma K S

On 19 October 2016 at 05:16, Wangda Tan  wrote:

> YARN Devs,
>
> *Updates since Sep 22, 2016:*
>
> We sent a merge YARN-3368 discussion email on Sep 22, 2016 and received
> some valuable feedbacks, since then, we have updated:
>
> *- The biggest change* is configuration model, the latest UI project
> doesn't require to configure RM/Timeline-server's address in a separate
> config (configs.env) any more, it will directly get timeline server address
> from REST endpoint of RM, more details please refer to (YARN-5145)
>
> And some minor fixes:
> - Moved all temporary ember/npm build stuffs to target/ like all the other
> projects.
> - Removed unnecessary files in source tree and ASF license exclusions.
>
> *Main content:*
>
> We propose to merge YARN-3368 (YARN next generation web UI) development
> branch into trunk for better development, would like to hear your thoughts
> before sending out vote mail.
>
> The new UI will co-exist with the old YARN UI, by default it is disabled.
> Please refer to User documentation of the new YARN UI
>  yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnUI2.md>
> for
> more details.
>
> In addition, There’re two working-in-progress features need the new UI to
> be merged to trunk for further development.
>
>   1) UI of YARN Timeline Server v2 (YARN-2928)
>   2) UI of YARN ResourceManager Federation (YARN-2915).
>
> *Status of YARN next generation web UI*
>
> Completed features
>
>- Cluster Overview Page
>- Scheduler page
>- Applications / Application / Application-attempts pages
>- Nodes / Node page
>
> Integration to YARN
>
>- Hosts new web UI in RM
>- Integrates to maven build / package
>
> Miscs:
>
>- Added dependencies to LICENSE.txt/NOTICE.txt
>- Documentated how to use it. (In hadoop-yarn-project/hadoop-
> yarn/hadoop-
>yarn-site/src/site/markdown/YarnUI2.md)
>
> Major items will finish on trunk:
>
>- Security support
>
> We have run the new UI in our internal cluster for more than 3 months, lots
> of people have tried the new UI and gave lots of valuable feedbacks and
> reported suggestions / issues to us. We fixed many of them so now we
> believe it is more ready for wider folks to try.
>
> Merge JIRA for Jenkins is: https://issues.apache.org/jira/browse/YARN-4734
> .
> The latest Jenkins run
>  focusedCommentId=15583884=com.atlassian.jira.
> plugin.system.issuetabpanels:comment-tabpanel#comment-15583884>
> gave
> +1.
>
> Please share your thoughts about this.
>
> Thanks,
> Wangda
>

Re: [Discuss] Merge YARN-3368 to trunk

[Sreenath Somarajapuram]

Tried the UI.
+1 LGTM

Thanks,
- Sreenath


On 10/24/16, 11:19 AM, "Sunil Govind"  wrote:

>Thanks Wangda for starting the thread.
>
>Tested latest patch against trunk. UI looks good. +1 for the merge.
>
>+ Sunil
>
>On Wed, Oct 19, 2016 at 5:16 AM Wangda Tan  wrote:
>
>> YARN Devs,
>>
>> *Updates since Sep 22, 2016:*
>>
>> We sent a merge YARN-3368 discussion email on Sep 22, 2016 and received
>> some valuable feedbacks, since then, we have updated:
>>
>> *- The biggest change* is configuration model, the latest UI project
>> doesn't require to configure RM/Timeline-server's address in a separate
>> config (configs.env) any more, it will directly get timeline server
>>address
>> from REST endpoint of RM, more details please refer to (YARN-5145)
>>
>> And some minor fixes:
>> - Moved all temporary ember/npm build stuffs to target/ like all the
>>other
>> projects.
>> - Removed unnecessary files in source tree and ASF license exclusions.
>>
>> *Main content:*
>>
>> We propose to merge YARN-3368 (YARN next generation web UI) development
>> branch into trunk for better development, would like to hear your
>>thoughts
>> before sending out vote mail.
>>
>> The new UI will co-exist with the old YARN UI, by default it is
>>disabled.
>> Please refer to User documentation of the new YARN UI
>> <
>> 
>>https://github.com/apache/hadoop/blob/YARN-3368/hadoop-yarn-project/hadoo
>>p-yarn/hadoop-yarn-site/src/site/markdown/YarnUI2.md
>> >
>> for
>> more details.
>>
>> In addition, There¹re two working-in-progress features need the new UI
>>to
>> be merged to trunk for further development.
>>
>>   1) UI of YARN Timeline Server v2 (YARN-2928)
>>   2) UI of YARN ResourceManager Federation (YARN-2915).
>>
>> *Status of YARN next generation web UI*
>>
>> Completed features
>>
>>- Cluster Overview Page
>>- Scheduler page
>>- Applications / Application / Application-attempts pages
>>- Nodes / Node page
>>
>> Integration to YARN
>>
>>- Hosts new web UI in RM
>>- Integrates to maven build / package
>>
>> Miscs:
>>
>>- Added dependencies to LICENSE.txt/NOTICE.txt
>>- Documentated how to use it. (In
>> hadoop-yarn-project/hadoop-yarn/hadoop-
>>yarn-site/src/site/markdown/YarnUI2.md)
>>
>> Major items will finish on trunk:
>>
>>- Security support
>>
>> We have run the new UI in our internal cluster for more than 3 months,
>>lots
>> of people have tried the new UI and gave lots of valuable feedbacks and
>> reported suggestions / issues to us. We fixed many of them so now we
>> believe it is more ready for wider folks to try.
>>
>> Merge JIRA for Jenkins is:
>>https://issues.apache.org/jira/browse/YARN-4734
>> .
>> The latest Jenkins run
>> <
>> 
>>https://issues.apache.org/jira/browse/YARN-4734?focusedCommentId=15583884
>>=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#co
>>mment-15583884
>> >
>> gave
>> +1.
>>
>> Please share your thoughts about this.
>>
>> Thanks,
>> Wangda
>>


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [Discuss] Merge YARN-3368 to trunk

[Sunil Govind]

Thanks Wangda for starting the thread.

Tested latest patch against trunk. UI looks good. +1 for the merge.

+ Sunil

On Wed, Oct 19, 2016 at 5:16 AM Wangda Tan  wrote:

> YARN Devs,
>
> *Updates since Sep 22, 2016:*
>
> We sent a merge YARN-3368 discussion email on Sep 22, 2016 and received
> some valuable feedbacks, since then, we have updated:
>
> *- The biggest change* is configuration model, the latest UI project
> doesn't require to configure RM/Timeline-server's address in a separate
> config (configs.env) any more, it will directly get timeline server address
> from REST endpoint of RM, more details please refer to (YARN-5145)
>
> And some minor fixes:
> - Moved all temporary ember/npm build stuffs to target/ like all the other
> projects.
> - Removed unnecessary files in source tree and ASF license exclusions.
>
> *Main content:*
>
> We propose to merge YARN-3368 (YARN next generation web UI) development
> branch into trunk for better development, would like to hear your thoughts
> before sending out vote mail.
>
> The new UI will co-exist with the old YARN UI, by default it is disabled.
> Please refer to User documentation of the new YARN UI
> <
> https://github.com/apache/hadoop/blob/YARN-3368/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnUI2.md
> >
> for
> more details.
>
> In addition, There’re two working-in-progress features need the new UI to
> be merged to trunk for further development.
>
>   1) UI of YARN Timeline Server v2 (YARN-2928)
>   2) UI of YARN ResourceManager Federation (YARN-2915).
>
> *Status of YARN next generation web UI*
>
> Completed features
>
>- Cluster Overview Page
>- Scheduler page
>- Applications / Application / Application-attempts pages
>- Nodes / Node page
>
> Integration to YARN
>
>- Hosts new web UI in RM
>- Integrates to maven build / package
>
> Miscs:
>
>- Added dependencies to LICENSE.txt/NOTICE.txt
>- Documentated how to use it. (In
> hadoop-yarn-project/hadoop-yarn/hadoop-
>yarn-site/src/site/markdown/YarnUI2.md)
>
> Major items will finish on trunk:
>
>- Security support
>
> We have run the new UI in our internal cluster for more than 3 months, lots
> of people have tried the new UI and gave lots of valuable feedbacks and
> reported suggestions / issues to us. We fixed many of them so now we
> believe it is more ready for wider folks to try.
>
> Merge JIRA for Jenkins is: https://issues.apache.org/jira/browse/YARN-4734
> .
> The latest Jenkins run
> <
> https://issues.apache.org/jira/browse/YARN-4734?focusedCommentId=15583884=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15583884
> >
> gave
> +1.
>
> Please share your thoughts about this.
>
> Thanks,
> Wangda
>

Re: [Discuss] Merge YARN-3368 to trunk

[Li Lu]

Thanks Wangda and all contributors to the new UI work. This is great! As Wangda 
mentioned in the mail, the ongoing work of timeline service v2 works with the 
new UI framework. Besides other significant benefits, merging YARN-3368 back to 
trunk can greatly simplify the development process of downstream features like 
timeline v2. In this mail it looks like we’ve addressed community concerns that 
previously block this from merging back, so shall we plan to merge this to 
trunk soon? Thanks! 

Li Lu

> On Oct 18, 2016, at 16:46, Wangda Tan  wrote:
> 
> YARN Devs,
> 
> *Updates since Sep 22, 2016:*
> 
> We sent a merge YARN-3368 discussion email on Sep 22, 2016 and received
> some valuable feedbacks, since then, we have updated:
> 
> *- The biggest change* is configuration model, the latest UI project
> doesn't require to configure RM/Timeline-server's address in a separate
> config (configs.env) any more, it will directly get timeline server address
> from REST endpoint of RM, more details please refer to (YARN-5145)
> 
> And some minor fixes:
> - Moved all temporary ember/npm build stuffs to target/ like all the other
> projects.
> - Removed unnecessary files in source tree and ASF license exclusions.
> 
> *Main content:*
> 
> We propose to merge YARN-3368 (YARN next generation web UI) development
> branch into trunk for better development, would like to hear your thoughts
> before sending out vote mail.
> 
> The new UI will co-exist with the old YARN UI, by default it is disabled.
> Please refer to User documentation of the new YARN UI
> 
> for
> more details.
> 
> In addition, There’re two working-in-progress features need the new UI to
> be merged to trunk for further development.
> 
>  1) UI of YARN Timeline Server v2 (YARN-2928)
>  2) UI of YARN ResourceManager Federation (YARN-2915).
> 
> *Status of YARN next generation web UI*
> 
> Completed features
> 
>   - Cluster Overview Page
>   - Scheduler page
>   - Applications / Application / Application-attempts pages
>   - Nodes / Node page
> 
> Integration to YARN
> 
>   - Hosts new web UI in RM
>   - Integrates to maven build / package
> 
> Miscs:
> 
>   - Added dependencies to LICENSE.txt/NOTICE.txt
>   - Documentated how to use it. (In hadoop-yarn-project/hadoop-yarn/hadoop-
>   yarn-site/src/site/markdown/YarnUI2.md)
> 
> Major items will finish on trunk:
> 
>   - Security support
> 
> We have run the new UI in our internal cluster for more than 3 months, lots
> of people have tried the new UI and gave lots of valuable feedbacks and
> reported suggestions / issues to us. We fixed many of them so now we
> believe it is more ready for wider folks to try.
> 
> Merge JIRA for Jenkins is: https://issues.apache.org/jira/browse/YARN-4734.
> The latest Jenkins run
> 
> gave
> +1.
> 
> Please share your thoughts about this.
> 
> Thanks,
> Wangda


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[Discuss] Merge YARN-3368 to trunk

[Wangda Tan]

YARN Devs,

*Updates since Sep 22, 2016:*

We sent a merge YARN-3368 discussion email on Sep 22, 2016 and received
some valuable feedbacks, since then, we have updated:

*- The biggest change* is configuration model, the latest UI project
doesn't require to configure RM/Timeline-server's address in a separate
config (configs.env) any more, it will directly get timeline server address
from REST endpoint of RM, more details please refer to (YARN-5145)

And some minor fixes:
- Moved all temporary ember/npm build stuffs to target/ like all the other
projects.
- Removed unnecessary files in source tree and ASF license exclusions.

*Main content:*

We propose to merge YARN-3368 (YARN next generation web UI) development
branch into trunk for better development, would like to hear your thoughts
before sending out vote mail.

The new UI will co-exist with the old YARN UI, by default it is disabled.
Please refer to User documentation of the new YARN UI

for
more details.

In addition, There’re two working-in-progress features need the new UI to
be merged to trunk for further development.

  1) UI of YARN Timeline Server v2 (YARN-2928)
  2) UI of YARN ResourceManager Federation (YARN-2915).

*Status of YARN next generation web UI*

Completed features

   - Cluster Overview Page
   - Scheduler page
   - Applications / Application / Application-attempts pages
   - Nodes / Node page

Integration to YARN

   - Hosts new web UI in RM
   - Integrates to maven build / package

Miscs:

   - Added dependencies to LICENSE.txt/NOTICE.txt
   - Documentated how to use it. (In hadoop-yarn-project/hadoop-yarn/hadoop-
   yarn-site/src/site/markdown/YarnUI2.md)

Major items will finish on trunk:

   - Security support

We have run the new UI in our internal cluster for more than 3 months, lots
of people have tried the new UI and gave lots of valuable feedbacks and
reported suggestions / issues to us. We fixed many of them so now we
believe it is more ready for wider folks to try.

Merge JIRA for Jenkins is: https://issues.apache.org/jira/browse/YARN-4734.
The latest Jenkins run

gave
+1.

Please share your thoughts about this.

Thanks,
Wangda

Re: [Discuss] Merge YARN-3368 to trunk

[Wangda Tan]

On Thu, Sep 22, 2016 at 2:01 PM, Allen Wittenauer 
wrote:

>
> > On Sep 22, 2016, at 1:24 PM, Wangda Tan  wrote:
> > Actually I'm not trying to debate about if it is necessary to make the
> UI project following Hadoop's rules. The answer is always yes to me: we
> should follow Hadoop's guide for any of sub project. We're trying to make
> it follow Hadoop code style for the new sub project. For example, we use
> the same indent rules (2 spaces) in all JS files.
> >
> > But what I want to discuss is, does these files violate existing Hadoop
> code style guide? I'm not be able to find rules states that dot-files or
> files which are used by developing purpose should not be added to source
> tree. Could you please share the link to the guide.
>
>
> We're talking past each other.
>
> If these files enforce the style guidelines, then pick one:
> * move to the root of the source tree
>
> If they don't enforce the style guidelines, then pick one:
> * remove
>
> or
>
> * fix so they do and then moved to the root of the source
> tree
>

Oh OK I realized you were specifically taking about the editor style file
inside UI project, I'm OK to remove it.


>
> > I will file a Yetus issue later, hopefully it should be a pretty simple
> change. Could we enable profile for only specific Hadoop branch?
>
> Nope.  It's all or nothing.


Since Maven won't fail if specified profile doesn't exist, so it should be
safe to enable it for all.

Re: [Discuss] Merge YARN-3368 to trunk

[Allen Wittenauer]

> On Sep 22, 2016, at 1:24 PM, Wangda Tan  wrote:
> Actually I'm not trying to debate about if it is necessary to make the UI 
> project following Hadoop's rules. The answer is always yes to me: we should 
> follow Hadoop's guide for any of sub project. We're trying to make it follow 
> Hadoop code style for the new sub project. For example, we use the same 
> indent rules (2 spaces) in all JS files.
> 
> But what I want to discuss is, does these files violate existing Hadoop code 
> style guide? I'm not be able to find rules states that dot-files or files 
> which are used by developing purpose should not be added to source tree. 
> Could you please share the link to the guide.


We're talking past each other.

If these files enforce the style guidelines, then pick one:
* move to the root of the source tree

If they don't enforce the style guidelines, then pick one:
* remove

or

* fix so they do and then moved to the root of the source tree

> I will file a Yetus issue later, hopefully it should be a pretty simple 
> change. Could we enable profile for only specific Hadoop branch?

Nope.  It's all or nothing.
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [Discuss] Merge YARN-3368 to trunk

[Wangda Tan]

Hi Allen,

Thanks again for replying, see replies inline.

On Thu, Sep 22, 2016 at 11:52 AM, Allen Wittenauer  wrote:

>
> > On Sep 22, 2016, at 11:13 AM, Wangda Tan  wrote:
> > >* why are there editor settings and other superfluous things there?  do
> these settings comply with the PMC's style guide?  why would they be here
> and not in the base of the source tree?
>
>
> > Actually many front-end projects are include these files in their source
> tree. I think the major reason is front end development needs different
> toolchains comparing to Java development.
>
> That's fine for those projects, but we're talking about this
> project--Apache Hadoop. We strive (and even sometimes achieve) code
> consistency throughout the entire project.  It's so important that it's
> even codified in the committer guidelines.  Why should one chunk of
> code--in a sub-project, and a currently optional component at that--have
> settings different (?) than the rest of the code base?   Having potentially
> different formatting rules is a direct hinderance to others looking at and
> contributing to the code.
>
> In order to respect the guidelines established by the PMC, it
> seems these either need to get moved to the root of the source tree and/or
> made consistent with the rest of the code or deleted.
>

Actually I'm not trying to debate about if it is necessary to make the UI
project following Hadoop's rules. The answer is always yes to me: we should
follow Hadoop's guide for any of sub project. We're trying to make it
follow Hadoop code style for the new sub project. For example, we use the
same indent rules (2 spaces) in all JS files.

But what I want to discuss is, does these files violate existing Hadoop
code style guide? I'm not be able to find rules states that dot-files or
files which are used by developing purpose should not be added to source
tree. Could you please share the link to the guide.


>
> > >* I'm still not sure why this is being wrapped in a maven profile:
> > >* create-release is activating the profile, so why not enable it
> for everyone?
> >
> > The major reason is, create-release run using docker image, all
> necessary dependencies are already included. But we don't expect every
> developer install npm and all the other dependencies to build the UI code.
> >
> > This is similar to native code, user who is interested to modify and
> repackage it can enable the profile and build, but I don't expect everybody
> is required to do that.
>
> This is a good point. However,  if the intent is to make this the
> default UI, then it's better to bite the bullet and put these requirements
> in place now while 3.x is in alpha rather than flip the switch later.
>

I would rather do this till we think it is ready to replace the entire
existing UI, otherwise it brings extra effort for existing contributors
building the code which they're not intended to use.


>
> >  >* precommit won't test it with that profile in place; it'd be
> useful to see a *real* run of yetus against this branch without the maven
> profile protecting it
> >
> > Actually this is also what I want to do, I'm not quite sure what need to
> do to enable profile while running the precommit, if it is possible, could
> you point me which code I need to update?
>
> I grouped these two bullet points intentionally: with the
> exception of some documentation bits, create-release is specifically coded
> currently to build what precommit tests. (See, for example, ISA-L support).
> Turning this on for create-release without precommit support is premature.
> You need to get a Yetus issue filed to turn on that profile first.
>
> It's worthwhile pointing out that without this support in place,
> this is basically a discussion about merging a branch that has very little
> precommit testing in place specifically because of the maven profile.
>

I will file a Yetus issue later, hopefully it should be a pretty simple
change. Could we enable profile for only specific Hadoop branch?


>
> > >* this doesn't appear to actually build in target/, which is very
> counter to maven. is there a valid reason for that?
> >
> > I will double check it, I just found bower supports to build in a
> separate folder, but not quite sure about npm. The bottom line is we should
> be able to copy the whole folder to target and build it.
>
> Awesome.  I recognize this is a pain, but maven really really
> really really really doesn't like things outside of target/.  We've got
> some other things like this to fix, but we really need to work on making
> more of them.
>

Yeah I agree :)

Thanks,


>
> Thanks.

Re: [Discuss] Merge YARN-3368 to trunk

[Allen Wittenauer]

> On Sep 22, 2016, at 11:13 AM, Wangda Tan  wrote:
> >* why are there editor settings and other superfluous things there?  do 
> >these settings comply with the PMC's style guide?  why would they be here 
> >and not in the base of the source tree?


> Actually many front-end projects are include these files in their source 
> tree. I think the major reason is front end development needs different 
> toolchains comparing to Java development.

That's fine for those projects, but we're talking about this 
project--Apache Hadoop. We strive (and even sometimes achieve) code consistency 
throughout the entire project.  It's so important that it's even codified in 
the committer guidelines.  Why should one chunk of code--in a sub-project, and 
a currently optional component at that--have settings different (?) than the 
rest of the code base?   Having potentially different formatting rules is a 
direct hinderance to others looking at and contributing to the code.

In order to respect the guidelines established by the PMC, it seems 
these either need to get moved to the root of the source tree and/or made 
consistent with the rest of the code or deleted.

> >* I'm still not sure why this is being wrapped in a maven profile:
> >* create-release is activating the profile, so why not enable it for 
> > everyone?
> 
> The major reason is, create-release run using docker image, all necessary 
> dependencies are already included. But we don't expect every developer 
> install npm and all the other dependencies to build the UI code.
> 
> This is similar to native code, user who is interested to modify and 
> repackage it can enable the profile and build, but I don't expect everybody 
> is required to do that.

This is a good point. However,  if the intent is to make this the 
default UI, then it's better to bite the bullet and put these requirements in 
place now while 3.x is in alpha rather than flip the switch later.

>  >* precommit won't test it with that profile in place; it'd be useful to 
> see a *real* run of yetus against this branch without the maven profile 
> protecting it
> 
> Actually this is also what I want to do, I'm not quite sure what need to do 
> to enable profile while running the precommit, if it is possible, could you 
> point me which code I need to update?

I grouped these two bullet points intentionally: with the exception of 
some documentation bits, create-release is specifically coded currently to 
build what precommit tests. (See, for example, ISA-L support). Turning this on 
for create-release without precommit support is premature.  You need to get a 
Yetus issue filed to turn on that profile first.

It's worthwhile pointing out that without this support in place, this 
is basically a discussion about merging a branch that has very little precommit 
testing in place specifically because of the maven profile.

> >* this doesn't appear to actually build in target/, which is very counter to 
> >maven. is there a valid reason for that?
> 
> I will double check it, I just found bower supports to build in a separate 
> folder, but not quite sure about npm. The bottom line is we should be able to 
> copy the whole folder to target and build it.

Awesome.  I recognize this is a pain, but maven really really really 
really really doesn't like things outside of target/.  We've got some other 
things like this to fix, but we really need to work on making more of them.

Thanks.
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [Discuss] Merge YARN-3368 to trunk

[Wangda Tan]

Hi Allen,

Thanks for your valuable comments, (BTW your new email address is
automatically categorized to spam by Gmail. Gmail noticed the email address
cannot be verified.)

Please see my responses inline:


On Tue, Sep 20, 2016 at 2:56 PM, Allen Wittenauer 
wrote:

> On 2016-09-09 15:54 (-0700), Wangda Tan  wrote:
> > We propose to merge YARN-3368 (YARN next generation web UI) development
> > branch into trunk for better development, would like to hear your
> thoughts
> > before sending out vote mail.
> >
>
> A quick pass through the diff:
>
> * why are there editor settings and other superfluous things there?  do
> these settings comply with the PMC's style guide?  why would they be here
> and not in the base of the source tree?
>

Actually many front-end projects are include these files in their source
tree. I think the major reason is front end development needs different
toolchains comparing to Java development.

For example .editorconfig is used for make code style consistent across
different IDEs/envs, which is pretty important for the code style.

And .watchmanconfig is used to do monitoring code changes while doing
development.

Personally they're not redundant files to me and they can make development
environment can be setup easier.

If you have any specific files want to remove from the source tree, I can
double check it.


>
> * versions of things be set in a higher order pom rather than children if
> possible
>
> * RAT exclusions for files where the license header can be added and files
> that don't actually exist
>
> * documentation references deprecated variables
>
> * documentation top and bottom are basically the same information
>

Will check above 4 items.


>
> * why isn't configuration in HADOOP_CONF_DIR?  That seems like a major
> blocking issue, especially from a downstream packaging front.  users are
> never going to find that because we've trained them to look in H_C_D.
>

Yeah I agree it will be better to put configs into HADOOP_CONF_DIR,
YARN-5145 is originally opened for this, we have some discussions
on YARN-5145 for this, it seems there's no easy solution for this.

We will try to see if this can be done in a reasonable way.


> * I'm still not sure why this is being wrapped in a maven profile:
> * create-release is activating the profile, so why not enable it for
> everyone?
>

The major reason is, create-release run using docker image, all necessary
dependencies are already included. But we don't expect every developer
install npm and all the other dependencies to build the UI code.

This is similar to native code, user who is interested to modify and
repackage it can enable the profile and build, but I don't expect everybody
is required to do that.


> * precommit won't test it with that profile in place; it'd be useful
> to see a *real* run of yetus against this branch without the maven profile
> protecting it
>

Actually this is also what I want to do, I'm not quite sure what need to do
to enable profile while running the precommit, if it is possible, could you
point me which code I need to update?


>
> * this doesn't appear to actually build in target/, which is very counter
> to maven. is there a valid reason for that?
>

I will double check it, I just found bower supports to build in a separate
folder, but not quite sure about npm. The bottom line is we should be able
to copy the whole folder to target and build it.


>
> * for an optional component, why is it first in the main pom.xml's module
> list?  shouldn't it be last to verify that the other stuff gets built first
> and therefore truly optional?
>

Will update


>
> * isn't this bundling the test code together with the actual application
> code?  why? also, is there any concern about the test code being used as a
> backdoor?
>

Will look into if there's any security concern for this. And will remove
test code from packaging if it is not necessary.


>
> * filenames with spaces and mixed case are going to cause all sorts of
> unexpected problems
>

If you're taking about "Sorting icons.psd", this part of the dependency of
datatables, will check if it possible to modify it, but since datatables is
wildly used, I think it should be safe in general.



>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

Re: [Discuss] Merge YARN-3368 to trunk

[Allen Wittenauer]

On 2016-09-09 15:54 (-0700), Wangda Tan  wrote: 
> We propose to merge YARN-3368 (YARN next generation web UI) development
> branch into trunk for better development, would like to hear your thoughts
> before sending out vote mail.
> 

A quick pass through the diff:

* why are there editor settings and other superfluous things there?  do these 
settings comply with the PMC's style guide?  why would they be here and not in 
the base of the source tree?

* versions of things be set in a higher order pom rather than children if 
possible

* RAT exclusions for files where the license header can be added and files that 
don't actually exist

* documentation references deprecated variables

* documentation top and bottom are basically the same information

* why isn't configuration in HADOOP_CONF_DIR?  That seems like a major blocking 
issue, especially from a downstream packaging front.  users are never going to 
find that because we've trained them to look in H_C_D.

* I'm still not sure why this is being wrapped in a maven profile:
* create-release is activating the profile, so why not enable it for 
everyone?
* precommit won't test it with that profile in place; it'd be useful to see 
a *real* run of yetus against this branch without the maven profile protecting 
it

* this doesn't appear to actually build in target/, which is very counter to 
maven. is there a valid reason for that?

* for an optional component, why is it first in the main pom.xml's module list? 
 shouldn't it be last to verify that the other stuff gets built first and 
therefore truly optional?

* isn't this bundling the test code together with the actual application code?  
why? also, is there any concern about the test code being used as a backdoor?

* filenames with spaces and mixed case are going to cause all sorts of 
unexpected problems


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [Discuss] Merge YARN-3368 to trunk

[Sunil Govind]

Thank You Wangda for driving the proposal.

I had taken the branch code and applied against trunk. Also verified basic
operations in all pages. Looks fine to me for now.
I am fine in merging the new UI base to trunk.

Thanks,
Sunil

On Sat, Sep 10, 2016 at 4:25 AM Wangda Tan  wrote:

>  YARN devs,
>
> We propose to merge YARN-3368 (YARN next generation web UI) development
> branch into trunk for better development, would like to hear your thoughts
> before sending out vote mail.
>
> The new UI will co-exist with the old YARN UI, by default it is disabled.
> Please refer to User documentation of the new YARN UI
> <
> https://github.com/apache/hadoop/blob/YARN-3368/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnUI2.md
> >
> for more details.
>
> In addition, There’re two working-in-progress features need the new UI to
> be merged to trunk for further development.
>
>   1) UI of YARN Timeline Server v2 (YARN-2928)
>   2) UI of YARN ResourceManager Federation (YARN-2915).
>
> *Status of YARN next gen web UI*
>
> Compeleted features
>
>- Cluster Overview Page
>- Scheduler page
>- Applications / Application / Application-attempts pages
>- Nodes / Node page
>
> Integration to YARN
>
>- Hosts new web UI in RM
>- Integrates to maven build / package
>
> Miscs:
>
>- Added dependencies to LICENSE.txt/NOTICE.txt
>- Documentated how to use it. (In
>
>  
> hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnUI2.md)
>
> Major items will finish on trunk:
>
>- Security support
>- RM HA support
>
> We have run the new UI in our internal cluster for more than 2 months, lots
> of people have tried the new UI and gave lots of valuable feedbacks and
> reported suggestions / issues to us. We fixed many of them so now we
> believe it is more ready for wider folks to try.
>
> Merge JIRA for Jenkins is: https://issues.apache.org/jira/browse/YARN-4734
> .
> The latest Jenkins run
> <
> https://issues.apache.org/jira/browse/YARN-4734?focusedCommentId=15478100=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15478100
> >
> gave +1.
>
> Please share your thoughts about this.
>
> Thanks,
> Wangda
>

[Discuss] Merge YARN-3368 to trunk

[Wangda Tan]

 YARN devs,

We propose to merge YARN-3368 (YARN next generation web UI) development
branch into trunk for better development, would like to hear your thoughts
before sending out vote mail.

The new UI will co-exist with the old YARN UI, by default it is disabled.
Please refer to User documentation of the new YARN UI

for more details.

In addition, There’re two working-in-progress features need the new UI to
be merged to trunk for further development.

  1) UI of YARN Timeline Server v2 (YARN-2928)
  2) UI of YARN ResourceManager Federation (YARN-2915).

*Status of YARN next gen web UI*

Compeleted features

   - Cluster Overview Page
   - Scheduler page
   - Applications / Application / Application-attempts pages
   - Nodes / Node page

Integration to YARN

   - Hosts new web UI in RM
   - Integrates to maven build / package

Miscs:

   - Added dependencies to LICENSE.txt/NOTICE.txt
   - Documentated how to use it. (In
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnUI2.md)

Major items will finish on trunk:

   - Security support
   - RM HA support

We have run the new UI in our internal cluster for more than 2 months, lots
of people have tried the new UI and gave lots of valuable feedbacks and
reported suggestions / issues to us. We fixed many of them so now we
believe it is more ready for wider folks to try.

Merge JIRA for Jenkins is: https://issues.apache.org/jira/browse/YARN-4734.
The latest Jenkins run

gave +1.

Please share your thoughts about this.

Thanks,
Wangda