[jira] [Commented] (YARN-2985) YARN should support to delete the aggregated logs for Non-MapReduce applications

2017-04-14 Thread Jason Lowe (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969088#comment-15969088
 ] 

Jason Lowe commented on YARN-2985:
--

Doing a config for branch-2 seems reasonable.

bq. my understanding is that the timeline server is supposed to replace the 
JHS, even for deployments that only run MR jobs

This is news to me.  The timeline server has no UI, just REST APIs, so there 
minimally needs to be something that provides the javascript necessary for the 
client browser to render a MapReduce-aware UI.  I haven't seen that in trunk, 
and without it the MapReduce JHS must still be running if there's going to be a 
MapReduce UI for completed jobs.

Even without the timeline server completely replacing the MR JHS, the deletion 
service can still be moved in trunk without a config under the following 
conditions:
* The timeline server is considered a critical server that always needs to be 
running (or we simply document that it must be used when log aggregation is 
enabled)
* There's an equivalent way to refresh the config options like can be done with 
the deletion service in the MR JHS today.

> YARN should support to delete the aggregated logs for Non-MapReduce 
> applications
> 
>
> Key: YARN-2985
> URL: https://issues.apache.org/jira/browse/YARN-2985
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: log-aggregation, nodemanager
>Affects Versions: 2.8.0
>Reporter: Xu Yang
>Assignee: Steven Rand
> Attachments: YARN-2985-branch-2-001.patch
>
>
> Before Hadoop 2.6, the LogAggregationService is started in NodeManager. But 
> the AggregatedLogDeletionService is started in mapreduce`s JobHistoryServer. 
> Therefore, the Non-MapReduce application can aggregate their logs to HDFS, 
> but can not delete those logs. Need the NodeManager take over the function of 
> aggregated log deletion.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-2985) YARN should support to delete the aggregated logs for Non-MapReduce applications

2017-04-13 Thread Steven Rand (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968609#comment-15968609
 ] 

Steven Rand commented on YARN-2985:
---

[~jlowe], thanks for the thoughtful response. Based on that information, it 
seems like the most straightforward way to proceed, at least for branch-2, is 
to add a configuration option for running the deletion service in only the 
timeline server, and not the JHS. Something like 
{{yarn.log-aggregation.run-in-timeline-server}} that defaults to {{false}} for 
backcompat, but when set to {{true}}, prevents the JHS from performing 
retention, and tells the timeline server to do it instead. Does that seem 
reasonable? If so I'll update the patch to do that, but certainly open to 
alternatives if there's a better way.

For trunk, I imagine it might be worth just removing retention from the JHS and 
moving it to the timeline server entirely, since my understanding is that the 
timeline server is supposed to replace the JHS, even for deployments that only 
run MR jobs, and 3.0 seems like a reasonable enough point at which to require 
the switch from JHS to timeline server. I might be misunderstanding the 
relationship between the two though, so please correct me if that doesn't make 
sense.

> YARN should support to delete the aggregated logs for Non-MapReduce 
> applications
> 
>
> Key: YARN-2985
> URL: https://issues.apache.org/jira/browse/YARN-2985
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: log-aggregation, nodemanager
>Affects Versions: 2.8.0
>Reporter: Xu Yang
>Assignee: Steven Rand
> Attachments: YARN-2985-branch-2-001.patch
>
>
> Before Hadoop 2.6, the LogAggregationService is started in NodeManager. But 
> the AggregatedLogDeletionService is started in mapreduce`s JobHistoryServer. 
> Therefore, the Non-MapReduce application can aggregate their logs to HDFS, 
> but can not delete those logs. Need the NodeManager take over the function of 
> aggregated log deletion.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-2985) YARN should support to delete the aggregated logs for Non-MapReduce applications

2017-04-13 Thread Jason Lowe (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15967748#comment-15967748
 ] 

Jason Lowe commented on YARN-2985:
--

Based on the description of this JIRA, I think there's some confusion here.  
Aggregated logs are deleted for non-MapReduce applications as long as the 
deletion service is running, whether that deletion service is hosted by the 
MapReduce job history server or somewhere else.  That's why the proposed patch 
is so small -- it's simply reusing the same code the JHS is already running.  
The log deletion service looks at the remote log directory in HDFS.  It doesn't 
filter the list of application logs it finds there based on whether it thinks 
the app is MapReduce or not, rather it just treats them as generic 
applications.  It happens to run in the MapReduce history server, but it is 
_not_ MapReduce-specific.  If users don't want to run MapReduce applications 
but want to do log aggregtion then they just need to run the MapReduce history 
server.  They won't use it for MapReduce job history since there are no 
MapReduce jobs, but that server will perform aggregated log retention for *all* 
applications.

Therefore this JIRA is really about adding the ability to relocate the 
aggregated log deletion service from the MapReduce job history server to the 
YARN timeline server.  We don't want two of these things running in the cluster 
if someone has deployed the MapReduce history server and the YARN timeline 
server.  That could lead to error messages in the logs as one of them goes to 
traverse/delete the logs just as the other is already deleting them.  However 
we also don't want to just rip it out of the MapReduce history server and move 
it to the timeline server because the timeline server is still an optional 
server in YARN.

So we either need a way for the user to specify where they want the deletion 
service to run, whether that's the legacy location in the MapReduce history 
server (since they aren't going to run a timeline server which is still an 
optional YARN server) or in the timeline server.  Or we need to just declare 
the timeline server a mandatory server to run (at least for log aggregation 
support) and move it from one to the other.

In addition the MapReduce history server supports dynamic refresh of the log 
deletion service configs, and it would be nice not to lose that ability when it 
is hosted in the timeline server.  That could be a separate JIRA unless we're 
ripping it out of the JHS.  If it can only run in the timeline server then we 
would lose refresh functionality unless that JIRA was completed.

As for unit tests, I agree the existing tests for the deletion service cover 
the correctness of the service itself, so we just need unit tests for the 
timeline server and MapReduce JHS to verify each is starting the deletion 
service or not starting the service based on how the cluster is configured.

> YARN should support to delete the aggregated logs for Non-MapReduce 
> applications
> 
>
> Key: YARN-2985
> URL: https://issues.apache.org/jira/browse/YARN-2985
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: log-aggregation, nodemanager
>Affects Versions: 2.8.0
>Reporter: Xu Yang
>Assignee: Steven Rand
> Attachments: YARN-2985-branch-2-001.patch
>
>
> Before Hadoop 2.6, the LogAggregationService is started in NodeManager. But 
> the AggregatedLogDeletionService is started in mapreduce`s JobHistoryServer. 
> Therefore, the Non-MapReduce application can aggregate their logs to HDFS, 
> but can not delete those logs. Need the NodeManager take over the function of 
> aggregated log deletion.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-2985) YARN should support to delete the aggregated logs for Non-MapReduce applications

2017-03-30 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15950287#comment-15950287
 ] 

Hadoop QA commented on YARN-2985:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
25s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m 
58s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
28s{color} | {color:green} branch-2 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
26s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
19s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
33s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
22s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
49s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
20s{color} | {color:green} branch-2 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
22s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
16s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
33s{color} | {color:green} hadoop-yarn-server-applicationhistoryservice in the 
patch passed with JDK v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
25s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 33s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:b59b8b7 |
| JIRA Issue | YARN-2985 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12861373/YARN-2985-branch-2-001.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 178f2753b888 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | branch-2 / 9e7f8cb |
| Default Java | 1.7.0_121 |
| Multi-JDK versions |

[jira] [Commented] (YARN-2985) YARN should support to delete the aggregated logs for Non-MapReduce applications

2017-01-16 Thread Andrew Ash (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824558#comment-15824558
 ] 

Andrew Ash commented on YARN-2985:
--

[~Steven Rand] I'm unable to attach patch files to this ticket either.  Maybe 
you can post a link to public github gist instead?

> YARN should support to delete the aggregated logs for Non-MapReduce 
> applications
> 
>
> Key: YARN-2985
> URL: https://issues.apache.org/jira/browse/YARN-2985
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: log-aggregation, nodemanager
>Reporter: Xu Yang
>
> Before Hadoop 2.6, the LogAggregationService is started in NodeManager. But 
> the AggregatedLogDeletionService is started in mapreduce`s JobHistoryServer. 
> Therefore, the Non-MapReduce application can aggregate their logs to HDFS, 
> but can not delete those logs. Need the NodeManager take over the function of 
> aggregated log deletion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-2985) YARN should support to delete the aggregated logs for Non-MapReduce applications

2016-12-25 Thread Steven Rand (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15777023#comment-15777023
 ] 

Steven Rand commented on YARN-2985:
---

I would like to submit a patch for this issue, but I don't seem to be able to. 
Do I need to be the assignee, and if so, would someone please assign this issue 
to me? If not, please let me know what I need to do. Thanks.

> YARN should support to delete the aggregated logs for Non-MapReduce 
> applications
> 
>
> Key: YARN-2985
> URL: https://issues.apache.org/jira/browse/YARN-2985
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: log-aggregation, nodemanager
>Reporter: Xu Yang
>
> Before Hadoop 2.6, the LogAggregationService is started in NodeManager. But 
> the AggregatedLogDeletionService is started in mapreduce`s JobHistoryServer. 
> Therefore, the Non-MapReduce application can aggregate their logs to HDFS, 
> but can not delete those logs. Need the NodeManager take over the function of 
> aggregated log deletion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-2985) YARN should support to delete the aggregated logs for Non-MapReduce applications

2014-12-23 Thread Naganarasimha G R (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14256880#comment-14256880
 ] 

Naganarasimha G R commented on YARN-2985:
-

Hi [~yxls123123]
bq. The Non-MapReduce application can aggregate their logs to HDFS, but can not 
delete those logs
You mean in secure environment Non-MapReduce application cannot delete the logs 
? basically why not, application specific History server handle this deletion ? 
or you have Non-MapReduce application which doesnt have a daemon service (like 
history server) running ?

 YARN should support to delete the aggregated logs for Non-MapReduce 
 applications
 

 Key: YARN-2985
 URL: https://issues.apache.org/jira/browse/YARN-2985
 Project: Hadoop YARN
  Issue Type: New Feature
  Components: log-aggregation, nodemanager
Reporter: Xu Yang

 Before Hadoop 2.6, the LogAggregationService is started in NodeManager. But 
 the AggregatedLogDeletionService is started in mapreduce`s JobHistoryServer. 
 Therefore, the Non-MapReduce application can aggregate their logs to HDFS, 
 but can not delete those logs. Need the NodeManager take over the function of 
 aggregated log deletion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2985) YARN should support to delete the aggregated logs for Non-MapReduce applications

2014-12-23 Thread Xu Yang (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257808#comment-14257808
 ] 

Xu Yang commented on YARN-2985:
---

Hi [~Naganarasimha]. Thank you for your question. I do not have a clear 
statement. I mean that if YARN need a feature to delete the aggregated logs. 
When I submit a patch job, if its ApplicationMaster has no 
AggregatedLogDeletionService, or before the time to delete overtime logs it`s 
over, its aggregated logs (HDFS) will never be cleared.

 YARN should support to delete the aggregated logs for Non-MapReduce 
 applications
 

 Key: YARN-2985
 URL: https://issues.apache.org/jira/browse/YARN-2985
 Project: Hadoop YARN
  Issue Type: New Feature
  Components: log-aggregation, nodemanager
Reporter: Xu Yang

 Before Hadoop 2.6, the LogAggregationService is started in NodeManager. But 
 the AggregatedLogDeletionService is started in mapreduce`s JobHistoryServer. 
 Therefore, the Non-MapReduce application can aggregate their logs to HDFS, 
 but can not delete those logs. Need the NodeManager take over the function of 
 aggregated log deletion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2985) YARN should support to delete the aggregated logs for Non-MapReduce applications

2014-12-23 Thread Zhijie Shen (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257837#comment-14257837
 ] 

Zhijie Shen commented on YARN-2985:
---

When we were working on YARN-2443, we came across this question too. At that 
moment, we thought of moving the aggregated log retention service to the 
timeline server, which is the central point of serving history information of 
different YARN apps.

 YARN should support to delete the aggregated logs for Non-MapReduce 
 applications
 

 Key: YARN-2985
 URL: https://issues.apache.org/jira/browse/YARN-2985
 Project: Hadoop YARN
  Issue Type: New Feature
  Components: log-aggregation, nodemanager
Reporter: Xu Yang

 Before Hadoop 2.6, the LogAggregationService is started in NodeManager. But 
 the AggregatedLogDeletionService is started in mapreduce`s JobHistoryServer. 
 Therefore, the Non-MapReduce application can aggregate their logs to HDFS, 
 but can not delete those logs. Need the NodeManager take over the function of 
 aggregated log deletion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)