from:"Abhishek Kapoor \(JIRA\)"

[jira] [Commented] (YARN-1124) By default yarn application -list should display all the applications in a state other than FINISHED / FAILED

2013-08-29 Thread Abhishek Kapoor (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-1124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13754398#comment-13754398
 ] 

Abhishek Kapoor commented on YARN-1124:
---

Pardon my ignorance.

I don't see why we have to list submitted/accepted/running application. Isn't 
that a job of job-History server ?

> By default yarn application -list should display all the applications in a 
> state other than FINISHED / FAILED
> -
>
> Key: YARN-1124
> URL: https://issues.apache.org/jira/browse/YARN-1124
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Omkar Vinit Joshi
>Assignee: Xuan Gong
>Priority: Blocker
> Fix For: 2.1.1-beta
>
> Attachments: YARN-1124.1.patch
>
>
> Today we are just listing application in RUNNING state by default for "yarn 
> application -list". Instead we should show all the applications which are 
> either submitted/accepted/running.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-816) Implement AM recovery for distributed shell

2013-07-11 Thread Abhishek Kapoor (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13706704#comment-13706704
 ] 

Abhishek Kapoor commented on YARN-816:
--

Hey [~vinodkv]

I was going through jira YARN-321, can't we make use of proposed AHS for the 
above requirements ?

> Implement AM recovery for distributed shell
> ---
>
> Key: YARN-816
> URL: https://issues.apache.org/jira/browse/YARN-816
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: applications/distributed-shell
>Reporter: Vinod Kumar Vavilapalli
>
> Simple recovery to just continue from where it left off is a good start.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-816) Implement AM recovery for distributed shell

2013-07-11 Thread Abhishek Kapoor (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13706667#comment-13706667
 ] 

Abhishek Kapoor commented on YARN-816:
--

Couldn't agree more [~vinodkv]

We can have state of AM communicated to RM.
When AM boots up, the state from RM should be communicated to AM for example 
whether its a fresh start or a recovery and if its a recovery the state of the 
nodes app was running on, should be communicated to AM by RM. 

The above use case might require communication protocol change between AM and 
RM ..


> Implement AM recovery for distributed shell
> ---
>
> Key: YARN-816
> URL: https://issues.apache.org/jira/browse/YARN-816
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: applications/distributed-shell
>Reporter: Vinod Kumar Vavilapalli
>
> Simple recovery to just continue from where it left off is a good start.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-816) Implement AM recovery for distributed shell

2013-07-11 Thread Abhishek Kapoor (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13706653#comment-13706653
 ] 

Abhishek Kapoor commented on YARN-816:
--

Preemption is one of the case where container can be killed while application 
is still running.
We can take inspiration from CPU scheduling algorithms done in OS.
Also if application is preempted we can provide a way to let app know that if 
it is going to get preempted and during recovery we aware app then it was bring 
preempted.
Probably a event fired to app letting it know what is going(preempt) to happen 
and what has happened(preempted).

Sorry if it sounds confusing
I am open for discussion


> Implement AM recovery for distributed shell
> ---
>
> Key: YARN-816
> URL: https://issues.apache.org/jira/browse/YARN-816
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: applications/distributed-shell
>Reporter: Vinod Kumar Vavilapalli
>
> Simple recovery to just continue from where it left off is a good start.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (YARN-815) Add container failure handling to distributed-shell

2013-07-11 Thread Abhishek Kapoor (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Kapoor reassigned YARN-815:


Assignee: Abhishek Kapoor

> Add container failure handling to distributed-shell
> ---
>
> Key: YARN-815
> URL: https://issues.apache.org/jira/browse/YARN-815
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: applications/distributed-shell
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Abhishek Kapoor
>
> Today if any container fails because of whatever reason, the app simply 
> ignores them. We should handle retries, improve error reporting etc.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-816) Implement AM recovery for distributed shell

2013-07-11 Thread Abhishek Kapoor (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705929#comment-13705929
 ] 

Abhishek Kapoor commented on YARN-816:
--

Please correct me if I am wrong.
Are you suggesting a use case where job if fails will start from where it dies 
? If yes, then i think we need to maintain a sate of user application running 
on container allocated. Isn't it a user application's responsibility to figure 
it out whether its a fresh start of app or a recovery ? 

> Implement AM recovery for distributed shell
> ---
>
> Key: YARN-816
> URL: https://issues.apache.org/jira/browse/YARN-816
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: applications/distributed-shell
>Reporter: Vinod Kumar Vavilapalli
>
> Simple recovery to just continue from where it left off is a good start.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-855) YarnClient.init should ensure that yarn parameters are present

2013-07-11 Thread Abhishek Kapoor (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705919#comment-13705919
 ] 

Abhishek Kapoor commented on YARN-855:
--

I wish to take it over
Any suggestions/update would like to share  [~sseth] ?


> YarnClient.init should ensure that yarn parameters are present
> --
>
> Key: YARN-855
> URL: https://issues.apache.org/jira/browse/YARN-855
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.0.5-alpha
>Reporter: Siddharth Seth
>Assignee: Abhishek Kapoor
>
> It currently accepts a Configuration object in init and doesn't check whether 
> it contains yarn parameters or is a YarnConfiguration. Should either accept 
> YarnConfiguration, check existence of parameters or create a 
> YarnConfiguration based on the configuration passed to it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-903) DistributedShell throwing Errors in logs after successfull completion

2013-07-07 Thread Abhishek Kapoor (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13701554#comment-13701554
 ] 

Abhishek Kapoor commented on YARN-903:
--

On further investigation it was found out the container is successfully 
completed with EXITED_WITH_SUCCESS to DONE and is being removed.
There is no harm because application executed successfully. But the exception 
can be mis-leading. 

> DistributedShell throwing Errors in logs after successfull completion
> -
>
> Key: YARN-903
> URL: https://issues.apache.org/jira/browse/YARN-903
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: applications/distributed-shell
>Affects Versions: 2.0.4-alpha
> Environment: Ununtu 11.10
>Reporter: Abhishek Kapoor
> Attachments: AppMaster.stderr, 
> yarn-sunny-nodemanager-sunny-Inspiron.log
>
>
> I have tried running DistributedShell and also used ApplicationMaster of the 
> same for my test.
> The application is successfully running through logging some errors which 
> would be useful to fix.
> Below are the logs from NodeManager and ApplicationMasterode
> Log Snippet for NodeManager
> =
> 2013-07-07 13:39:18,787 INFO 
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Connecting 
> to ResourceManager at localhost/127.0.0.1:9990. current no. of attempts is 1
> 2013-07-07 13:39:19,050 INFO 
> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
>  Rolling master-key for container-tokens, got key with id -325382586
> 2013-07-07 13:39:19,052 INFO 
> org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM: 
> Rolling master-key for nm-tokens, got key with id :1005046570
> 2013-07-07 13:39:19,053 INFO 
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered 
> with ResourceManager as sunny-Inspiron:9993 with total resource of 
> 
> 2013-07-07 13:39:19,053 INFO 
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Notifying 
> ContainerManager to unblock new container-requests
> 2013-07-07 13:39:35,256 INFO SecurityLogger.org.apache.hadoop.ipc.Server: 
> Auth successful for appattempt_1373184544832_0001_01 (auth:SIMPLE)
> 2013-07-07 13:39:35,492 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>  Start request for container_1373184544832_0001_01_01 by user sunny
> 2013-07-07 13:39:35,507 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>  Creating a new application reference for app application_1373184544832_0001
> 2013-07-07 13:39:35,511 INFO 
> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=sunny  
> IP=127.0.0.1OPERATION=Start Container Request   
> TARGET=ContainerManageImpl  RESULT=SUCCESS  
> APPID=application_1373184544832_0001
> CONTAINERID=container_1373184544832_0001_01_01
> 2013-07-07 13:39:35,511 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>  Application application_1373184544832_0001 transitioned from NEW to INITING
> 2013-07-07 13:39:35,512 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>  Adding container_1373184544832_0001_01_01 to application 
> application_1373184544832_0001
> 2013-07-07 13:39:35,518 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>  Application application_1373184544832_0001 transitioned from INITING to 
> RUNNING
> 2013-07-07 13:39:35,528 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>  Container container_1373184544832_0001_01_01 transitioned from NEW to 
> LOCALIZING
> 2013-07-07 13:39:35,540 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>  Resource hdfs://localhost:9000/application/test.jar transitioned from INIT 
> to DOWNLOADING
> 2013-07-07 13:39:35,540 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>  Created localizer for container_1373184544832_0001_01_01
> 2013-07-07 13:39:35,675 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>  Writing credentials to the nmPrivate file 
> /home/sunny/Hadoop2/hadoopdata/nodemanagerdata/nmPrivate/container_1373184544832_0001_01_01.tokens.
>  Credentials list: 
> 2013-07-07 13:39:35,694 INFO 
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: 
> Initializing user sunny
> 2013-07-07 13:39:35,803 INFO 
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying 
> from 
> /home/sunny/Hadoop2/hadoopdata/nodemanagerdata/nmPrivate/cont

[jira] [Commented] (YARN-903) DistributedShell throwing Errors in logs after successfull completion

2013-07-07 Thread Abhishek Kapoor (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13701548#comment-13701548
 ] 

Abhishek Kapoor commented on YARN-903:
--

Some useful log observation from NodeManager

2013-07-07 13:39:40,158 INFO 
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed 
completed container container_1373184544832_0001_01_02
2013-07-07 13:39:40,683 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
 Getting container-status for container_1373184544832_0001_01_02 

Seems like after container is removed/dis-owned from NodeManager, 
ContainerManagerImpl tries to fetch the status for the same. Hence resulting in 
exception.  

> DistributedShell throwing Errors in logs after successfull completion
> -
>
> Key: YARN-903
> URL: https://issues.apache.org/jira/browse/YARN-903
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: applications/distributed-shell
>Affects Versions: 2.0.4-alpha
> Environment: Ununtu 11.10
>Reporter: Abhishek Kapoor
> Attachments: AppMaster.stderr, 
> yarn-sunny-nodemanager-sunny-Inspiron.log
>
>
> I have tried running DistributedShell and also used ApplicationMaster of the 
> same for my test.
> The application is successfully running through logging some errors which 
> would be useful to fix.
> Below are the logs from NodeManager and ApplicationMasterode
> Log Snippet for NodeManager
> =
> 2013-07-07 13:39:18,787 INFO 
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Connecting 
> to ResourceManager at localhost/127.0.0.1:9990. current no. of attempts is 1
> 2013-07-07 13:39:19,050 INFO 
> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
>  Rolling master-key for container-tokens, got key with id -325382586
> 2013-07-07 13:39:19,052 INFO 
> org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM: 
> Rolling master-key for nm-tokens, got key with id :1005046570
> 2013-07-07 13:39:19,053 INFO 
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered 
> with ResourceManager as sunny-Inspiron:9993 with total resource of 
> 
> 2013-07-07 13:39:19,053 INFO 
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Notifying 
> ContainerManager to unblock new container-requests
> 2013-07-07 13:39:35,256 INFO SecurityLogger.org.apache.hadoop.ipc.Server: 
> Auth successful for appattempt_1373184544832_0001_01 (auth:SIMPLE)
> 2013-07-07 13:39:35,492 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>  Start request for container_1373184544832_0001_01_01 by user sunny
> 2013-07-07 13:39:35,507 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>  Creating a new application reference for app application_1373184544832_0001
> 2013-07-07 13:39:35,511 INFO 
> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=sunny  
> IP=127.0.0.1OPERATION=Start Container Request   
> TARGET=ContainerManageImpl  RESULT=SUCCESS  
> APPID=application_1373184544832_0001
> CONTAINERID=container_1373184544832_0001_01_01
> 2013-07-07 13:39:35,511 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>  Application application_1373184544832_0001 transitioned from NEW to INITING
> 2013-07-07 13:39:35,512 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>  Adding container_1373184544832_0001_01_01 to application 
> application_1373184544832_0001
> 2013-07-07 13:39:35,518 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>  Application application_1373184544832_0001 transitioned from INITING to 
> RUNNING
> 2013-07-07 13:39:35,528 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>  Container container_1373184544832_0001_01_01 transitioned from NEW to 
> LOCALIZING
> 2013-07-07 13:39:35,540 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>  Resource hdfs://localhost:9000/application/test.jar transitioned from INIT 
> to DOWNLOADING
> 2013-07-07 13:39:35,540 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>  Created localizer for container_1373184544832_0001_01_01
> 2013-07-07 13:39:35,675 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>  Writing credentials to the nmPrivate file 
> /home/sunny/Hadoop2/hadoopdata/nodemanagerdata/nmPrivate/container_1373184544832_0001_01_01.tokens.
>  Credentials list

[jira] [Updated] (YARN-903) DistributedShell throwing Errors in logs after successfull completion

2013-07-07 Thread Abhishek Kapoor (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Kapoor updated YARN-903:
-

Attachment: yarn-sunny-nodemanager-sunny-Inspiron.log
AppMaster.stderr

NodeManager and ApplicationMaster logs attached 

> DistributedShell throwing Errors in logs after successfull completion
> -
>
> Key: YARN-903
> URL: https://issues.apache.org/jira/browse/YARN-903
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: applications/distributed-shell
>Affects Versions: 2.0.4-alpha
> Environment: Ununtu 11.10
>Reporter: Abhishek Kapoor
> Attachments: AppMaster.stderr, 
> yarn-sunny-nodemanager-sunny-Inspiron.log
>
>
> I have tried running DistributedShell and also used ApplicationMaster of the 
> same for my test.
> The application is successfully running through logging some errors which 
> would be useful to fix.
> Below are the logs from NodeManager and ApplicationMasterode
> Log Snippet for NodeManager
> =
> 2013-07-07 13:39:18,787 INFO 
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Connecting 
> to ResourceManager at localhost/127.0.0.1:9990. current no. of attempts is 1
> 2013-07-07 13:39:19,050 INFO 
> org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
>  Rolling master-key for container-tokens, got key with id -325382586
> 2013-07-07 13:39:19,052 INFO 
> org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM: 
> Rolling master-key for nm-tokens, got key with id :1005046570
> 2013-07-07 13:39:19,053 INFO 
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered 
> with ResourceManager as sunny-Inspiron:9993 with total resource of 
> 
> 2013-07-07 13:39:19,053 INFO 
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Notifying 
> ContainerManager to unblock new container-requests
> 2013-07-07 13:39:35,256 INFO SecurityLogger.org.apache.hadoop.ipc.Server: 
> Auth successful for appattempt_1373184544832_0001_01 (auth:SIMPLE)
> 2013-07-07 13:39:35,492 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>  Start request for container_1373184544832_0001_01_01 by user sunny
> 2013-07-07 13:39:35,507 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>  Creating a new application reference for app application_1373184544832_0001
> 2013-07-07 13:39:35,511 INFO 
> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=sunny  
> IP=127.0.0.1OPERATION=Start Container Request   
> TARGET=ContainerManageImpl  RESULT=SUCCESS  
> APPID=application_1373184544832_0001
> CONTAINERID=container_1373184544832_0001_01_01
> 2013-07-07 13:39:35,511 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>  Application application_1373184544832_0001 transitioned from NEW to INITING
> 2013-07-07 13:39:35,512 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>  Adding container_1373184544832_0001_01_01 to application 
> application_1373184544832_0001
> 2013-07-07 13:39:35,518 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
>  Application application_1373184544832_0001 transitioned from INITING to 
> RUNNING
> 2013-07-07 13:39:35,528 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>  Container container_1373184544832_0001_01_01 transitioned from NEW to 
> LOCALIZING
> 2013-07-07 13:39:35,540 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>  Resource hdfs://localhost:9000/application/test.jar transitioned from INIT 
> to DOWNLOADING
> 2013-07-07 13:39:35,540 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>  Created localizer for container_1373184544832_0001_01_01
> 2013-07-07 13:39:35,675 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>  Writing credentials to the nmPrivate file 
> /home/sunny/Hadoop2/hadoopdata/nodemanagerdata/nmPrivate/container_1373184544832_0001_01_01.tokens.
>  Credentials list: 
> 2013-07-07 13:39:35,694 INFO 
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: 
> Initializing user sunny
> 2013-07-07 13:39:35,803 INFO 
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying 
> from 
> /home/sunny/Hadoop2/hadoopdata/nodemanagerdata/nmPrivate/container_1373184544832_0001_01_01.tokens
>  to 
> /home/sunny/Hadoop2/hadoopdata/nodemanagerdata/usercache/sunny/appcache/application_1373184544832_0001/

[jira] [Created] (YARN-903) DistributedShell throwing Errors in logs after successfull completion

2013-07-07 Thread Abhishek Kapoor (JIRA)

Abhishek Kapoor created YARN-903:


 Summary: DistributedShell throwing Errors in logs after 
successfull completion
 Key: YARN-903
 URL: https://issues.apache.org/jira/browse/YARN-903
 Project: Hadoop YARN
  Issue Type: Bug
  Components: applications/distributed-shell
Affects Versions: 2.0.4-alpha
 Environment: Ununtu 11.10
Reporter: Abhishek Kapoor


I have tried running DistributedShell and also used ApplicationMaster of the 
same for my test.
The application is successfully running through logging some errors which be 
useful to fix.
Below are the logs from NodeManager and ApplicationMasterode

Log Snippet for NodeManager
=
2013-07-07 13:39:18,787 INFO 
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Connecting to 
ResourceManager at localhost/127.0.0.1:9990. current no. of attempts is 1
2013-07-07 13:39:19,050 INFO 
org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
 Rolling master-key for container-tokens, got key with id -325382586
2013-07-07 13:39:19,052 INFO 
org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM: 
Rolling master-key for nm-tokens, got key with id :1005046570
2013-07-07 13:39:19,053 INFO 
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered 
with ResourceManager as sunny-Inspiron:9993 with total resource of 

2013-07-07 13:39:19,053 INFO 
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Notifying 
ContainerManager to unblock new container-requests
2013-07-07 13:39:35,256 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth 
successful for appattempt_1373184544832_0001_01 (auth:SIMPLE)
2013-07-07 13:39:35,492 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
 Start request for container_1373184544832_0001_01_01 by user sunny
2013-07-07 13:39:35,507 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
 Creating a new application reference for app application_1373184544832_0001
2013-07-07 13:39:35,511 INFO 
org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=sunny
IP=127.0.0.1OPERATION=Start Container Request   
TARGET=ContainerManageImpl  RESULT=SUCCESS  
APPID=application_1373184544832_0001
CONTAINERID=container_1373184544832_0001_01_01
2013-07-07 13:39:35,511 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
 Application application_1373184544832_0001 transitioned from NEW to INITING
2013-07-07 13:39:35,512 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
 Adding container_1373184544832_0001_01_01 to application 
application_1373184544832_0001
2013-07-07 13:39:35,518 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
 Application application_1373184544832_0001 transitioned from INITING to RUNNING
2013-07-07 13:39:35,528 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: 
Container container_1373184544832_0001_01_01 transitioned from NEW to 
LOCALIZING
2013-07-07 13:39:35,540 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
 Resource hdfs://localhost:9000/application/test.jar transitioned from INIT to 
DOWNLOADING
2013-07-07 13:39:35,540 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
 Created localizer for container_1373184544832_0001_01_01
2013-07-07 13:39:35,675 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
 Writing credentials to the nmPrivate file 
/home/sunny/Hadoop2/hadoopdata/nodemanagerdata/nmPrivate/container_1373184544832_0001_01_01.tokens.
 Credentials list: 
2013-07-07 13:39:35,694 INFO 
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: 
Initializing user sunny
2013-07-07 13:39:35,803 INFO 
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying 
from 
/home/sunny/Hadoop2/hadoopdata/nodemanagerdata/nmPrivate/container_1373184544832_0001_01_01.tokens
 to 
/home/sunny/Hadoop2/hadoopdata/nodemanagerdata/usercache/sunny/appcache/application_1373184544832_0001/container_1373184544832_0001_01_01.tokens
2013-07-07 13:39:35,803 INFO 
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: CWD set to 
/home/sunny/Hadoop2/hadoopdata/nodemanagerdata/usercache/sunny/appcache/application_1373184544832_0001
 = 
file:/home/sunny/Hadoop2/hadoopdata/nodemanagerdata/usercache/sunny/appcache/application_1373184544832_0001
2013-07-07 13:39:36,136 INFO 
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending out 
status for container: container_id {, app_attempt_id {, application_id {, id: 
1, cluster_timestamp:

[jira] [Commented] (YARN-69) RM should throw different exceptions for while querying app/node/queue

2013-07-05 Thread Abhishek Kapoor (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-69?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13700534#comment-13700534
 ] 

Abhishek Kapoor commented on YARN-69:
-

Sorry If I am little behind, but it looks like in release 2.0.5 alpha new 
exceptions are introduced YarnException and IOException. Are still thinking to 
introduce more exceptions as described above ?

> RM should throw different exceptions for while querying app/node/queue
> --
>
> Key: YARN-69
> URL: https://issues.apache.org/jira/browse/YARN-69
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Vinod Kumar Vavilapalli
>
> We should distinguish the exceptions for absent app/node/queue, illegally 
> accessed app/node/queue etc. Today everything is a {{YarnRemoteException}}. 
> We should extend {{YarnRemoteException}} to add {{NotFoundException}}, 
> {{AccessControlException}} etc. Today, {{AccessControlException}} exists but 
> not as part of the protocol descriptions (i.e. only available to Java).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-855) YarnClient.init should ensure that yarn parameters are present

2013-07-05 Thread Abhishek Kapoor (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13700519#comment-13700519
 ] 

Abhishek Kapoor commented on YARN-855:
--

Agree with you Siddharth.
Just a thought, wouldn't it be more logical to enforce YarnClient to accept 
YarnConfiguration as a parameter, rather then checking existence of all the 
parameters in YarnConfiguration, since there is no way to know what user will 
override in yarn-site.xml
Please feel free to correct me.
Thanks
Abhi

> YarnClient.init should ensure that yarn parameters are present
> --
>
> Key: YARN-855
> URL: https://issues.apache.org/jira/browse/YARN-855
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.0.5-alpha
>Reporter: Siddharth Seth
>Assignee: Abhishek Kapoor
>
> It currently accepts a Configuration object in init and doesn't check whether 
> it contains yarn parameters or is a YarnConfiguration. Should either accept 
> YarnConfiguration, check existence of parameters or create a 
> YarnConfiguration based on the configuration passed to it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (YARN-855) YarnClient.init should ensure that yarn parameters are present

2013-07-05 Thread Abhishek Kapoor (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Kapoor reassigned YARN-855:


Assignee: Abhishek Kapoor

> YarnClient.init should ensure that yarn parameters are present
> --
>
> Key: YARN-855
> URL: https://issues.apache.org/jira/browse/YARN-855
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.0.5-alpha
>Reporter: Siddharth Seth
>Assignee: Abhishek Kapoor
>
> It currently accepts a Configuration object in init and doesn't check whether 
> it contains yarn parameters or is a YarnConfiguration. Should either accept 
> YarnConfiguration, check existence of parameters or create a 
> YarnConfiguration based on the configuration passed to it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-369) Handle ( or throw a proper error when receiving) status updates from application masters that have not registered

2013-03-12 Thread Abhishek Kapoor (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13599981#comment-13599981
 ] 

Abhishek Kapoor commented on YARN-369:
--

Do we have any other way to test the reboot of old AM other then by allocate 
call ? 

> Handle ( or throw a proper error when receiving) status updates from 
> application masters that have not registered
> -
>
> Key: YARN-369
> URL: https://issues.apache.org/jira/browse/YARN-369
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 2.0.3-alpha, trunk-win
>Reporter: Hitesh Shah
>Assignee: Abhishek Kapoor
> Attachments: YARN-369.patch
>
>
> Currently, an allocate call from an unregistered application is allowed and 
> the status update for it throws a statemachine error that is silently dropped.
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: 
> STATUS_UPDATE at LAUNCHED
>at 
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
>at 
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
>at 
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:445)
>at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:588)
>at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:99)
>at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:471)
>at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:452)
>at 
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:130)
>at 
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:77)
>at java.lang.Thread.run(Thread.java:680)
> ApplicationMasterService should likely throw an appropriate error for 
> applications' requests that should not be handled in such cases.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-379) yarn [node,application] command print logger info messages

2013-03-12 Thread Abhishek Kapoor (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13599985#comment-13599985
 ] 

Abhishek Kapoor commented on YARN-379:
--

Are we okay with the fix ? 

Please suggest 

Thanks
Abhi

> yarn [node,application] command print logger info messages
> --
>
> Key: YARN-379
> URL: https://issues.apache.org/jira/browse/YARN-379
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: client
>Affects Versions: 2.0.3-alpha
>Reporter: Thomas Graves
>Assignee: Abhishek Kapoor
>  Labels: usability
> Attachments: YARN-379.patch
>
>
> Running the yarn node and yarn applications command results in annoying log 
> info messages being printed:
> $ yarn node -list
> 13/02/06 02:36:50 INFO service.AbstractService: 
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
> 13/02/06 02:36:50 INFO service.AbstractService: 
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
> Total Nodes:1
>  Node-IdNode-State  Node-Http-Address   
> Health-Status(isNodeHealthy)Running-Containers
> foo:8041RUNNING  foo:8042   true  
>  0
> 13/02/06 02:36:50 INFO service.AbstractService: 
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is stopped.
> $ yarn application
> 13/02/06 02:38:47 INFO service.AbstractService: 
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
> 13/02/06 02:38:47 INFO service.AbstractService: 
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
> Invalid Command Usage : 
> usage: application
>  -kill  Kills the application.
>  -list   Lists all the Applications from RM.
>  -statusPrints the status of the application.
> 13/02/06 02:38:47 INFO service.AbstractService: 
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is stopped.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-307) NodeManager should log container launch command.

2013-03-07 Thread Abhishek Kapoor (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595744#comment-13595744
 ] 

Abhishek Kapoor commented on YARN-307:
--

Do we still need this jira to remain open or suggested approach by Tom works 
for you [~lohit] ?

> NodeManager should log container launch command.
> 
>
> Key: YARN-307
> URL: https://issues.apache.org/jira/browse/YARN-307
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.0.3-alpha
>Reporter: Lohit Vijayarenu
>
> NodeManager's DefaultContainerExecutor seems to log only path of default 
> container executor script instead of contents of script. It would be good to 
> log the execution command so that one could see what is being launched.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-69) RM should throw different exceptions for while querying app/node/queue

2013-03-06 Thread Abhishek Kapoor (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-69?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595615#comment-13595615
 ] 

Abhishek Kapoor commented on YARN-69:
-

+1

As discribed about, this will require protocol amendment 

> RM should throw different exceptions for while querying app/node/queue
> --
>
> Key: YARN-69
> URL: https://issues.apache.org/jira/browse/YARN-69
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Vinod Kumar Vavilapalli
>
> We should distinguish the exceptions for absent app/node/queue, illegally 
> accessed app/node/queue etc. Today everything is a {{YarnRemoteException}}. 
> We should extend {{YarnRemoteException}} to add {{NotFoundException}}, 
> {{AccessControlException}} etc. Today, {{AccessControlException}} exists but 
> not as part of the protocol descriptions (i.e. only available to Java).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-369) Handle ( or throw a proper error when receiving) status updates from application masters that have not registered

2013-03-06 Thread Abhishek Kapoor (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595569#comment-13595569
 ] 

Abhishek Kapoor commented on YARN-369:
--

Put a check on NPE
public boolean hasApplicationMaster(ApplicationAttemptId appAttemptId){
boolean hasAM=false;
AMResponse lastResponse = responseMap.get(appAttemptId);
if (lastResponse !=null && lastResponse.getResponseId() >= 0) {
 hasAM=true;
  }
return hasAM;
  }


But the code is acutally failing because of the below mentioned flows at the 
moment.

Class: TestRMRestart.java

AMResponse amResponse = am1.allocate(new ArrayList(),
new ArrayList());
Assert.assertTrue(amResponse.getReboot());


call flow structure :

am1.allocate -> MockAM#allocate -> amRMProtocol#allocate

where amRMProtocol is actually ApplicationMasterService class  (Note: holds a 
private final map[responseMap] of )
above mentioned responseMap is used to find out the responseid from AMResponse.

As per the modification allocate function will now check for the repsonse id  
>=0 .
and if registerApplicationMaster is not being called then responseId will not 
be increamented and responseMap will not be updated.

MockAM does have registerAppAttempt() but since responseMap in 
ApplicationMasterService is private final therefore it does not have access to 
it.







> Handle ( or throw a proper error when receiving) status updates from 
> application masters that have not registered
> -
>
> Key: YARN-369
> URL: https://issues.apache.org/jira/browse/YARN-369
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 2.0.3-alpha, trunk-win
>Reporter: Hitesh Shah
>Assignee: Abhishek Kapoor
> Attachments: YARN-369.patch
>
>
> Currently, an allocate call from an unregistered application is allowed and 
> the status update for it throws a statemachine error that is silently dropped.
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: 
> STATUS_UPDATE at LAUNCHED
>at 
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
>at 
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
>at 
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:445)
>at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:588)
>at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:99)
>at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:471)
>at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:452)
>at 
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:130)
>at 
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:77)
>at java.lang.Thread.run(Thread.java:680)
> ApplicationMasterService should likely throw an appropriate error for 
> applications' requests that should not be handled in such cases.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-369) Handle ( or throw a proper error when receiving) status updates from application masters that have not registered

2013-03-06 Thread Abhishek Kapoor (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594827#comment-13594827
 ] 

Abhishek Kapoor commented on YARN-369:
--

On further analysis it looks like allocate call is used to reboot the AM for 
being out-of-sync with RM.
And if allocate is called then that request should have AM and hence the test 
case is failing

Open for suggestions :) 

> Handle ( or throw a proper error when receiving) status updates from 
> application masters that have not registered
> -
>
> Key: YARN-369
> URL: https://issues.apache.org/jira/browse/YARN-369
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 2.0.3-alpha, trunk-win
>Reporter: Hitesh Shah
>Assignee: Abhishek Kapoor
> Attachments: YARN-369.patch
>
>
> Currently, an allocate call from an unregistered application is allowed and 
> the status update for it throws a statemachine error that is silently dropped.
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: 
> STATUS_UPDATE at LAUNCHED
>at 
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
>at 
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
>at 
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:445)
>at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:588)
>at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:99)
>at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:471)
>at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:452)
>at 
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:130)
>at 
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:77)
>at java.lang.Thread.run(Thread.java:680)
> ApplicationMasterService should likely throw an appropriate error for 
> applications' requests that should not be handled in such cases.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-369) Handle ( or throw a proper error when receiving) status updates from application masters that have not registered

2013-03-06 Thread Abhishek Kapoor (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594702#comment-13594702
 ] 

Abhishek Kapoor commented on YARN-369:
--

It looks like the respective change to fix the issue will effect unit test 
cases where allocate call is made prior to registerApplicationMaster. That 
include below mentioned Test case 

TestRMRestart. 

AMResponse amResponse = am1.allocate(new ArrayList(),
new ArrayList());
Assert.assertTrue(amResponse.getReboot());


Any suggestion is welcome.


> Handle ( or throw a proper error when receiving) status updates from 
> application masters that have not registered
> -
>
> Key: YARN-369
> URL: https://issues.apache.org/jira/browse/YARN-369
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 2.0.3-alpha, trunk-win
>Reporter: Hitesh Shah
>Assignee: Abhishek Kapoor
> Attachments: YARN-369.patch
>
>
> Currently, an allocate call from an unregistered application is allowed and 
> the status update for it throws a statemachine error that is silently dropped.
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: 
> STATUS_UPDATE at LAUNCHED
>at 
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
>at 
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
>at 
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:445)
>at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:588)
>at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:99)
>at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:471)
>at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:452)
>at 
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:130)
>at 
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:77)
>at java.lang.Thread.run(Thread.java:680)
> ApplicationMasterService should likely throw an appropriate error for 
> applications' requests that should not be handled in such cases.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-369) Handle ( or throw a proper error when receiving) status updates from application masters that have not registered

2013-03-06 Thread Abhishek Kapoor (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594632#comment-13594632
 ] 

Abhishek Kapoor commented on YARN-369:
--

Am working on it.

> Handle ( or throw a proper error when receiving) status updates from 
> application masters that have not registered
> -
>
> Key: YARN-369
> URL: https://issues.apache.org/jira/browse/YARN-369
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 2.0.3-alpha, trunk-win
>Reporter: Hitesh Shah
>Assignee: Abhishek Kapoor
> Attachments: YARN-369.patch
>
>
> Currently, an allocate call from an unregistered application is allowed and 
> the status update for it throws a statemachine error that is silently dropped.
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: 
> STATUS_UPDATE at LAUNCHED
>at 
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
>at 
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
>at 
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:445)
>at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:588)
>at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:99)
>at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:471)
>at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:452)
>at 
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:130)
>at 
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:77)
>at java.lang.Thread.run(Thread.java:680)
> ApplicationMasterService should likely throw an appropriate error for 
> applications' requests that should not be handled in such cases.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (YARN-369) Handle ( or throw a proper error when receiving) status updates from application masters that have not registered

2013-03-06 Thread Abhishek Kapoor (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Kapoor updated YARN-369:
-

Attachment: YARN-369.patch

Please code review the patch.

> Handle ( or throw a proper error when receiving) status updates from 
> application masters that have not registered
> -
>
> Key: YARN-369
> URL: https://issues.apache.org/jira/browse/YARN-369
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Reporter: Hitesh Shah
>Assignee: Abhishek Kapoor
> Attachments: YARN-369.patch
>
>
> Currently, an allocate call from an unregistered application is allowed and 
> the status update for it throws a statemachine error that is silently dropped.
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: 
> STATUS_UPDATE at LAUNCHED
>at 
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
>at 
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
>at 
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:445)
>at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:588)
>at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:99)
>at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:471)
>at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:452)
>at 
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:130)
>at 
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:77)
>at java.lang.Thread.run(Thread.java:680)
> ApplicationMasterService should likely throw an appropriate error for 
> applications' requests that should not be handled in such cases.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-238) ClientRMProtocol needs to allow the specification of a ResourceRequest so that the Application Master's Container can be placed on the specified host

2013-03-03 Thread Abhishek Kapoor (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13592008#comment-13592008
 ] 

Abhishek Kapoor commented on YARN-238:
--

I think it would be nice to have the respective feature where client is able to 
specify where ApplicationMaster should run. 

> ClientRMProtocol needs to allow the specification of a ResourceRequest so 
> that the Application Master's Container can be placed on the specified host
> -
>
> Key: YARN-238
> URL: https://issues.apache.org/jira/browse/YARN-238
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager
>Reporter: Vinayak Borkar
>
> Currently a client is able to specify only resource requirements in terms of 
> amount of memory required while launching an ApplicationMaster. There needs 
> to be a way to ask for resources using a ResourceRequest so that a host name 
> could be specified in addition to the amount of memory required.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-369) Handle ( or throw a proper error when receiving) status updates from application masters that have not registered

2013-03-03 Thread Abhishek Kapoor (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13591687#comment-13591687
 ] 

Abhishek Kapoor commented on YARN-369:
--

I was investigating the above stackTrace and was able to replicate the issue.

As per my understanding(relatively new) below is the flow.

1) Application masters needs to register to RM before making an allocate 
request (which is not enforced, AM can call allocate without registering to RM).

2) If an allocate is called without registering Application Master to RM we 
will get the above stacktrace.

Possible solution

1) ApplicationMasterService needs to identify whether Application attempt ID 
has registered Application master or not. 
Example: public boolean hasApplicationMaster(ApplicationAttemptId appAttemptId)

2) we can have map 
appIDtoAMResponse which 
would let us know whether Application master is registered or not

3) The above map will be populated when registerApplicationMaster is being 
called and map entry will be removed when finishApplicationMaster is being 
called.

Below is the algo code

public AllocateResponse allocate(AllocateRequest request)
  throws YarnRemoteException {

ApplicationAttemptId appAttemptId = request.getApplicationAttemptId();
authorizeRequest(appAttemptId);
   if(!hasApplicationMaster(appAttemptId)){
String message = "Application Master does not exist for: " + 
appAttemptId.getApplicationId();
LOG.error(message);

this.rmContext.getRMApps().get(appAttemptId.getApplicationId()).getDiagnostics().append("Application
 Master does not exist ");
  
RMAuditLogger.logFailure(this.rmContext.getRMApps().get(appAttemptId.getApplicationId()).getUser(),
  AuditConstants.REGISTER_AM, message, 
"ApplicationMasterService",
  "Application master does not exist", 
appAttemptId.getApplicationId(), appAttemptId);
  throw RPCUtil.getRemoteException(message);
}



Please suggest, if there is better solution for the same.

> Handle ( or throw a proper error when receiving) status updates from 
> application masters that have not registered
> -
>
> Key: YARN-369
> URL: https://issues.apache.org/jira/browse/YARN-369
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Reporter: Hitesh Shah
>Assignee: Abhishek Kapoor
>
> Currently, an allocate call from an unregistered application is allowed and 
> the status update for it throws a statemachine error that is silently dropped.
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: 
> STATUS_UPDATE at LAUNCHED
>at 
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
>at 
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
>at 
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:445)
>at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:588)
>at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:99)
>at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:471)
>at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:452)
>at 
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:130)
>at 
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:77)
>at java.lang.Thread.run(Thread.java:680)
> ApplicationMasterService should likely throw an appropriate error for 
> applications' requests that should not be handled in such cases.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (YARN-369) Handle ( or throw a proper error when receiving) status updates from application masters that have not registered

2013-03-01 Thread Abhishek Kapoor (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Kapoor reassigned YARN-369:


Assignee: Abhishek Kapoor

> Handle ( or throw a proper error when receiving) status updates from 
> application masters that have not registered
> -
>
> Key: YARN-369
> URL: https://issues.apache.org/jira/browse/YARN-369
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Reporter: Hitesh Shah
>Assignee: Abhishek Kapoor
>
> Currently, an allocate call from an unregistered application is allowed and 
> the status update for it throws a statemachine error that is silently dropped.
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: 
> STATUS_UPDATE at LAUNCHED
>at 
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
>at 
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
>at 
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:445)
>at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:588)
>at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:99)
>at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:471)
>at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:452)
>at 
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:130)
>at 
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:77)
>at java.lang.Thread.run(Thread.java:680)
> ApplicationMasterService should likely throw an appropriate error for 
> applications' requests that should not be handled in such cases.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-369) Handle ( or throw a proper error when receiving) status updates from applications that have not registered

2013-03-01 Thread Abhishek Kapoor (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13590552#comment-13590552
 ] 

Abhishek Kapoor commented on YARN-369:
--

[~itwasnoteasy] could you please explain what do you mean by "unregistered 
application".



> Handle ( or throw a proper error when receiving) status updates from 
> applications that have not registered
> --
>
> Key: YARN-369
> URL: https://issues.apache.org/jira/browse/YARN-369
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Reporter: Hitesh Shah
>
> Currently, an allocate call from an unregistered application is allowed and 
> the status update for it throws a statemachine error that is silently dropped.
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: 
> STATUS_UPDATE at LAUNCHED
>at 
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
>at 
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
>at 
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:445)
>at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:588)
>at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:99)
>at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:471)
>at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:452)
>at 
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:130)
>at 
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:77)
>at java.lang.Thread.run(Thread.java:680)
> ApplicationMasterService should likely throw an appropriate error for 
> applications' requests that should not be handled in such cases.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-299) Node Manager throws org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: RESOURCE_FAILED at DONE

2013-03-01 Thread Abhishek Kapoor (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13590504#comment-13590504
 ] 

Abhishek Kapoor commented on YARN-299:
--

Please describe the steps to replicate the issue..


> Node Manager throws 
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: 
> RESOURCE_FAILED at DONE
> ---
>
> Key: YARN-299
> URL: https://issues.apache.org/jira/browse/YARN-299
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.0.1-alpha, 2.0.0-alpha
>Reporter: Devaraj K
>
> {code:xml}
> 2012-12-31 10:36:27,844 WARN 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>  Can't handle this event at current state: Current: [DONE], eventType: 
> [RESOURCE_FAILED]
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: 
> RESOURCE_FAILED at DONE
>   at 
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
>   at 
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
>   at 
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl.handle(ContainerImpl.java:819)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl.handle(ContainerImpl.java:71)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:504)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:497)
>   at 
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
>   at 
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
>   at java.lang.Thread.run(Thread.java:662)
> 2012-12-31 10:36:27,845 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>  Container container_1356792558130_0002_01_01 transitioned from DONE to 
> null
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (YARN-323) Yarn CLI commands prints classpath

2013-02-27 Thread Abhishek Kapoor (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Kapoor updated YARN-323:
-

Priority: Trivial  (was: Minor)

> Yarn CLI commands prints classpath
> --
>
> Key: YARN-323
> URL: https://issues.apache.org/jira/browse/YARN-323
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.0.1-alpha
>Reporter: Nishan Shetty
>Priority: Trivial
>
> Execute ./yarn commands. It will print classpath in console

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-323) Yarn CLI commands prints classpath

2013-02-27 Thread Abhishek Kapoor (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13589183#comment-13589183
 ] 

Abhishek Kapoor commented on YARN-323:
--

I dont see the classpath being printed on console.
Please confirm, or close the issue.

Thanks
Abhishek

> Yarn CLI commands prints classpath
> --
>
> Key: YARN-323
> URL: https://issues.apache.org/jira/browse/YARN-323
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.0.1-alpha
>Reporter: Nishan Shetty
>Priority: Minor
>
> Execute ./yarn commands. It will print classpath in console

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-379) yarn [node,application] command print logger info messages

2013-02-26 Thread Abhishek Kapoor (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13587230#comment-13587230
 ] 

Abhishek Kapoor commented on YARN-379:
--

The NodeCLI utility does not use log4j instead it uses sysout. 

Changing the above messages from INFO to DEBUG would change the logging level 
in AbstractService class which is super class for many services.

If that is okay, then I can go for it and change it.






> yarn [node,application] command print logger info messages
> --
>
> Key: YARN-379
> URL: https://issues.apache.org/jira/browse/YARN-379
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: client
>Affects Versions: 2.0.3-alpha
>Reporter: Thomas Graves
>Assignee: Abhishek Kapoor
>  Labels: usability
> Attachments: YARN-379.patch
>
>
> Running the yarn node and yarn applications command results in annoying log 
> info messages being printed:
> $ yarn node -list
> 13/02/06 02:36:50 INFO service.AbstractService: 
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
> 13/02/06 02:36:50 INFO service.AbstractService: 
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
> Total Nodes:1
>  Node-IdNode-State  Node-Http-Address   
> Health-Status(isNodeHealthy)Running-Containers
> foo:8041RUNNING  foo:8042   true  
>  0
> 13/02/06 02:36:50 INFO service.AbstractService: 
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is stopped.
> $ yarn application
> 13/02/06 02:38:47 INFO service.AbstractService: 
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
> 13/02/06 02:38:47 INFO service.AbstractService: 
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
> Invalid Command Usage : 
> usage: application
>  -kill  Kills the application.
>  -list   Lists all the Applications from RM.
>  -statusPrints the status of the application.
> 13/02/06 02:38:47 INFO service.AbstractService: 
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is stopped.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-380) yarn node -status prints Last-Last-Node-Status

2013-02-25 Thread Abhishek Kapoor (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13586845#comment-13586845
 ] 

Abhishek Kapoor commented on YARN-380:
--

Also I am proposing to use DateFormat.LONG which will be for example (26 
February, 2013 11:51:35 AM IST)

> yarn node -status prints Last-Last-Node-Status
> --
>
> Key: YARN-380
> URL: https://issues.apache.org/jira/browse/YARN-380
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: client
>Affects Versions: 2.0.3-alpha
>Reporter: Thomas Graves
>Assignee: Abhishek Kapoor
>  Labels: usability
>
> I assume the Last-Last-NodeStatus is a typo and it should just be 
> Last-Node-Status.
> $ yarn node -status foo.com:8041
> Node Report : 
> Node-Id : foo.com:8041
> Rack : /10.10.10.0
> Node-State : RUNNING
> Node-Http-Address : foo.com:8042
> Health-Status(isNodeHealthy) : true
> Last-Last-Health-Update : 1360118400219
> Health-Report : 
> Containers : 0
> Memory-Used : 0M
> Memory-Capacity : 24576

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-380) yarn node -status prints Last-Last-Node-Status

2013-02-25 Thread Abhishek Kapoor (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13586843#comment-13586843
 ] 

Abhishek Kapoor commented on YARN-380:
--

I guess Thomas you mean instead of "Last-Last-Health-Update" it should be 
"Last-Node-Status" with a human readable date format ?
Please clarify.
I have looked into the issue and will be patching if after the update from you.

Thanks & Regards
Abhishek

> yarn node -status prints Last-Last-Node-Status
> --
>
> Key: YARN-380
> URL: https://issues.apache.org/jira/browse/YARN-380
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: client
>Affects Versions: 2.0.3-alpha
>Reporter: Thomas Graves
>Assignee: Abhishek Kapoor
>  Labels: usability
>
> I assume the Last-Last-NodeStatus is a typo and it should just be 
> Last-Node-Status.
> $ yarn node -status foo.com:8041
> Node Report : 
> Node-Id : foo.com:8041
> Rack : /10.10.10.0
> Node-State : RUNNING
> Node-Http-Address : foo.com:8042
> Health-Status(isNodeHealthy) : true
> Last-Last-Health-Update : 1360118400219
> Health-Report : 
> Containers : 0
> Memory-Used : 0M
> Memory-Capacity : 24576

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (YARN-380) yarn node -status prints Last-Last-Node-Status

2013-02-25 Thread Abhishek Kapoor (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Kapoor reassigned YARN-380:


Assignee: Abhishek Kapoor

> yarn node -status prints Last-Last-Node-Status
> --
>
> Key: YARN-380
> URL: https://issues.apache.org/jira/browse/YARN-380
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: client
>Affects Versions: 2.0.3-alpha
>Reporter: Thomas Graves
>Assignee: Abhishek Kapoor
>  Labels: usability
>
> I assume the Last-Last-NodeStatus is a typo and it should just be 
> Last-Node-Status.
> $ yarn node -status foo.com:8041
> Node Report : 
> Node-Id : foo.com:8041
> Rack : /10.10.10.0
> Node-State : RUNNING
> Node-Http-Address : foo.com:8042
> Health-Status(isNodeHealthy) : true
> Last-Last-Health-Update : 1360118400219
> Health-Report : 
> Containers : 0
> Memory-Used : 0M
> Memory-Capacity : 24576

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (YARN-379) yarn [node,application] command print logger info messages

2013-02-25 Thread Abhishek Kapoor (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Kapoor updated YARN-379:
-

Attachment: YARN-379.patch

Please review the code.

Many Thanks

> yarn [node,application] command print logger info messages
> --
>
> Key: YARN-379
> URL: https://issues.apache.org/jira/browse/YARN-379
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: client
>Affects Versions: 2.0.3-alpha
>Reporter: Thomas Graves
>Assignee: Abhishek Kapoor
>  Labels: usability
> Fix For: trunk-win
>
> Attachments: YARN-379.patch
>
>
> Running the yarn node and yarn applications command results in annoying log 
> info messages being printed:
> $ yarn node -list
> 13/02/06 02:36:50 INFO service.AbstractService: 
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
> 13/02/06 02:36:50 INFO service.AbstractService: 
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
> Total Nodes:1
>  Node-IdNode-State  Node-Http-Address   
> Health-Status(isNodeHealthy)Running-Containers
> foo:8041RUNNING  foo:8042   true  
>  0
> 13/02/06 02:36:50 INFO service.AbstractService: 
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is stopped.
> $ yarn application
> 13/02/06 02:38:47 INFO service.AbstractService: 
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
> 13/02/06 02:38:47 INFO service.AbstractService: 
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
> Invalid Command Usage : 
> usage: application
>  -kill  Kills the application.
>  -list   Lists all the Applications from RM.
>  -statusPrints the status of the application.
> 13/02/06 02:38:47 INFO service.AbstractService: 
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is stopped.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-379) yarn [node,application] command print logger info messages

2013-02-25 Thread Abhishek Kapoor (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13586086#comment-13586086
 ] 

Abhishek Kapoor commented on YARN-379:
--

Do I need any special permission to attach a patch file for this jira ?

Sorry this is my first time.

Appreciated 

> yarn [node,application] command print logger info messages
> --
>
> Key: YARN-379
> URL: https://issues.apache.org/jira/browse/YARN-379
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: client
>Affects Versions: 2.0.3-alpha
>Reporter: Thomas Graves
>Assignee: Abhishek Kapoor
>  Labels: usability
> Fix For: trunk-win
>
>
> Running the yarn node and yarn applications command results in annoying log 
> info messages being printed:
> $ yarn node -list
> 13/02/06 02:36:50 INFO service.AbstractService: 
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
> 13/02/06 02:36:50 INFO service.AbstractService: 
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
> Total Nodes:1
>  Node-IdNode-State  Node-Http-Address   
> Health-Status(isNodeHealthy)Running-Containers
> foo:8041RUNNING  foo:8042   true  
>  0
> 13/02/06 02:36:50 INFO service.AbstractService: 
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is stopped.
> $ yarn application
> 13/02/06 02:38:47 INFO service.AbstractService: 
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
> 13/02/06 02:38:47 INFO service.AbstractService: 
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
> Invalid Command Usage : 
> usage: application
>  -kill  Kills the application.
>  -list   Lists all the Applications from RM.
>  -statusPrints the status of the application.
> 13/02/06 02:38:47 INFO service.AbstractService: 
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is stopped.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-379) yarn [node,application] command print logger info messages

2013-02-25 Thread Abhishek Kapoor (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13586002#comment-13586002
 ] 

Abhishek Kapoor commented on YARN-379:
--

It order to fix this request, two files will be tweaked

File name:

1) yarn
2) yarn-env.sh

I have added a new OPTS called "COMMON_LOGGING_OPTS" in yarn-env.sh to turn off 
the unnecessary log info messages.

COMMON_LOGGING_OPTS is used into the yarn shell script to turn the log message.

Suggestion are welcome



> yarn [node,application] command print logger info messages
> --
>
> Key: YARN-379
> URL: https://issues.apache.org/jira/browse/YARN-379
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: client
>Affects Versions: 2.0.3-alpha
>Reporter: Thomas Graves
>Assignee: Abhishek Kapoor
>  Labels: usability
>
> Running the yarn node and yarn applications command results in annoying log 
> info messages being printed:
> $ yarn node -list
> 13/02/06 02:36:50 INFO service.AbstractService: 
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
> 13/02/06 02:36:50 INFO service.AbstractService: 
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
> Total Nodes:1
>  Node-IdNode-State  Node-Http-Address   
> Health-Status(isNodeHealthy)Running-Containers
> foo:8041RUNNING  foo:8042   true  
>  0
> 13/02/06 02:36:50 INFO service.AbstractService: 
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is stopped.
> $ yarn application
> 13/02/06 02:38:47 INFO service.AbstractService: 
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
> 13/02/06 02:38:47 INFO service.AbstractService: 
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
> Invalid Command Usage : 
> usage: application
>  -kill  Kills the application.
>  -list   Lists all the Applications from RM.
>  -statusPrints the status of the application.
> 13/02/06 02:38:47 INFO service.AbstractService: 
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is stopped.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-379) yarn [node,application] command print logger info messages

2013-02-25 Thread Abhishek Kapoor (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13585938#comment-13585938
 ] 

Abhishek Kapoor commented on YARN-379:
--

I would like to fix it, could someone help me out in assigning the bug to 
me..Seems like I could not assign the issue to me .

Thanks

> yarn [node,application] command print logger info messages
> --
>
> Key: YARN-379
> URL: https://issues.apache.org/jira/browse/YARN-379
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: client
>Affects Versions: 2.0.3-alpha
>Reporter: Thomas Graves
>  Labels: usability
>
> Running the yarn node and yarn applications command results in annoying log 
> info messages being printed:
> $ yarn node -list
> 13/02/06 02:36:50 INFO service.AbstractService: 
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
> 13/02/06 02:36:50 INFO service.AbstractService: 
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
> Total Nodes:1
>  Node-IdNode-State  Node-Http-Address   
> Health-Status(isNodeHealthy)Running-Containers
> foo:8041RUNNING  foo:8042   true  
>  0
> 13/02/06 02:36:50 INFO service.AbstractService: 
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is stopped.
> $ yarn application
> 13/02/06 02:38:47 INFO service.AbstractService: 
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
> 13/02/06 02:38:47 INFO service.AbstractService: 
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
> Invalid Command Usage : 
> usage: application
>  -kill  Kills the application.
>  -list   Lists all the Applications from RM.
>  -statusPrints the status of the application.
> 13/02/06 02:38:47 INFO service.AbstractService: 
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is stopped.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

38 matches

Mail list logo