[jira] [Commented] (YARN-3869) Add app name to RM audit log

[Dan Shechter (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14968549#comment-14968549
 ] 

Dan Shechter commented on YARN-3869:


By a stroke of luck I found this issue and I am actually toy no achieve pretty 
much the same thing with yarn + spark

+1 from me about this, I would also like to see something like the application 
summary log message as the application is accepted / started... 

Thanks! 

> Add app name to RM audit log
> 
>
> Key: YARN-3869
> URL: https://issues.apache.org/jira/browse/YARN-3869
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Shay Rojansky
>Priority: Minor
>
> The YARN resource manager audit log currently includes useful info such as 
> APPID, USER, etc. One crucial piece of information missing is the 
> user-supplied application name.
> Users are familiar with their application name as shown in the YARN UI, etc. 
> It's vital for something like logstash to be able to associate logs with the 
> application name for later searching in something like kibana.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3869) Add app name to RM audit log

[Shay Rojansky (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14966673#comment-14966673
 ] 

Shay Rojansky commented on YARN-3869:
-

Any chance we can advance on this?

Note that the resource manager emits a summary message with lots of info, 
including the application name, at the very end of the application (see below). 
A message like this would be ideal at the *beginning* of the application 
lifecycle.

2015-10-21 08:52:12,727 INFO 
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary: 
appId=application_1444927016569_0035,name=Dummy,user=roji,queue=root.roji,state=FINISHED,trackingUrl=http://**/proxy/application_1444927016569_0035/,appMasterHost=,startTime=1445417521080,finishTime=1445417531344,finalStatus=SUCCEEDED,memorySeconds=19316,vcoreSeconds=18,preemptedAMContainers=0,preemptedNonAMContainers=0,preemptedResources=,applicationType=SPARK


> Add app name to RM audit log
> 
>
> Key: YARN-3869
> URL: https://issues.apache.org/jira/browse/YARN-3869
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Shay Rojansky
>Priority: Minor
>
> The YARN resource manager audit log currently includes useful info such as 
> APPID, USER, etc. One crucial piece of information missing is the 
> user-supplied application name.
> Users are familiar with their application name as shown in the YARN UI, etc. 
> It's vital for something like logstash to be able to associate logs with the 
> application name for later searching in something like kibana.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3869) Add app name to RM audit log

[Shay Rojansky (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14616323#comment-14616323
 ] 

Shay Rojansky commented on YARN-3869:
-

Wasn't aware of this case...

How is this handled within the Yarn wen UI, which also displays app names?

One thought is to simply truncate to some max char count when outputting to the 
logs...

 Add app name to RM audit log
 

 Key: YARN-3869
 URL: https://issues.apache.org/jira/browse/YARN-3869
 Project: Hadoop YARN
  Issue Type: Improvement
Reporter: Shay Rojansky
Assignee: nijel
Priority: Minor

 The YARN resource manager audit log currently includes useful info such as 
 APPID, USER, etc. One crucial piece of information missing is the 
 user-supplied application name.
 Users are familiar with their application name as shown in the YARN UI, etc. 
 It's vital for something like logstash to be able to associate logs with the 
 application name for later searching in something like kibana.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3869) Add app name to RM audit log

[nijel (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14616363#comment-14616363
 ] 

nijel commented on YARN-3869:
-

In Web it is shown as truncate.
But if the names are similar, will it serve the purpose ? 

Let us wait for few other opinion as well :) 

 Add app name to RM audit log
 

 Key: YARN-3869
 URL: https://issues.apache.org/jira/browse/YARN-3869
 Project: Hadoop YARN
  Issue Type: Improvement
Reporter: Shay Rojansky
Assignee: nijel
Priority: Minor

 The YARN resource manager audit log currently includes useful info such as 
 APPID, USER, etc. One crucial piece of information missing is the 
 user-supplied application name.
 Users are familiar with their application name as shown in the YARN UI, etc. 
 It's vital for something like logstash to be able to associate logs with the 
 application name for later searching in something like kibana.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3869) Add app name to RM audit log

[nijel (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14616282#comment-14616282
 ] 

nijel commented on YARN-3869:
-

hi
i started working on this. One observation is in some cases the application 
name will not be in readable format.
Like in HIVE, the name will the complete query string. In this case it will not 
be good to print this information in logs !
Any thoughts ? 

 Add app name to RM audit log
 

 Key: YARN-3869
 URL: https://issues.apache.org/jira/browse/YARN-3869
 Project: Hadoop YARN
  Issue Type: Improvement
Reporter: Shay Rojansky
Assignee: nijel
Priority: Minor

 The YARN resource manager audit log currently includes useful info such as 
 APPID, USER, etc. One crucial piece of information missing is the 
 user-supplied application name.
 Users are familiar with their application name as shown in the YARN UI, etc. 
 It's vital for something like logstash to be able to associate logs with the 
 application name for later searching in something like kibana.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3869) Add app name to RM audit log

[Shay Rojansky (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14616521#comment-14616521
 ] 

Shay Rojansky commented on YARN-3869:
-

That's a good point, I'm not sure. Not that there's no requirement for appnames 
to be unique - I can submit two apps with the same name - so that's a problem 
I'd have to deal with anyway in logstash or whatever. In any case I wouldn't 
want to be penalized and lose appnames in logs just because Hive does something 
its own way.

Regardless, this does make me think that the appname should probably be 
surrounded by quotes (may contain spaces etc) and possibly also escaped.

As you say, let's wait for some other opinions.


 Add app name to RM audit log
 

 Key: YARN-3869
 URL: https://issues.apache.org/jira/browse/YARN-3869
 Project: Hadoop YARN
  Issue Type: Improvement
Reporter: Shay Rojansky
Assignee: nijel
Priority: Minor

 The YARN resource manager audit log currently includes useful info such as 
 APPID, USER, etc. One crucial piece of information missing is the 
 user-supplied application name.
 Users are familiar with their application name as shown in the YARN UI, etc. 
 It's vital for something like logstash to be able to associate logs with the 
 application name for later searching in something like kibana.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3869) Add app name to RM audit log

[nijel (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14609812#comment-14609812
 ] 

nijel commented on YARN-3869:
-

hi [~roji]
i would like to work on this improvement.
Please let me know is you already started the work

 Add app name to RM audit log
 

 Key: YARN-3869
 URL: https://issues.apache.org/jira/browse/YARN-3869
 Project: Hadoop YARN
  Issue Type: Improvement
Reporter: Shay Rojansky
Priority: Minor

 The YARN resource manager audit log currently includes useful info such as 
 APPID, USER, etc. One crucial piece of information missing is the 
 user-supplied application name.
 Users are familiar with their application name as shown in the YARN UI, etc. 
 It's vital for something like logstash to be able to associated logs with the 
 application name for later searching in something like kibana.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)