[jira] [Updated] (YARN-2790) NM can't aggregate logs past HDFS delegation token expiry.
[ https://issues.apache.org/jira/browse/YARN-2790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-2790: -- Priority: Critical (was: Major) Target Version/s: 2.6.0 NM can't aggregate logs past HDFS delegation token expiry. -- Key: YARN-2790 URL: https://issues.apache.org/jira/browse/YARN-2790 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 2.6.0 Reporter: Tassapol Athiapinya Assignee: Jian He Priority: Critical Fix For: 2.6.0 Attachments: YARN-2790.1.patch We shorten hdfs delegation token lifetime, set RM to act as as a proxy user in order to be able to renew the token on behalf of a user submitting the application. Still we see NM log aggregation fail due to token expiry error. {code} 2014-10-31 00:11:56,579 INFO logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:finishLogAggregation(481)) - Application just finished : application_1414705229376_0004 2014-10-31 00:11:56,588 WARN ipc.Client (Client.java:run(675)) - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 75 for user1) is expired 2014-10-31 00:11:56,589 ERROR logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:uploadLogsForContainers(233)) - Cannot create writer for app application_1414705229376_0004. Skip log upload this time. {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2790) NM can't aggregate logs past HDFS delegation token expiry.
[ https://issues.apache.org/jira/browse/YARN-2790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tassapol Athiapinya updated YARN-2790: -- Description: We shorten hdfs delegation token lifetime, set RM to act as as a proxy user in order to be able to renew the token on behalf of a user submitting the application. Still we see NM log aggregation fail due to token expiry error. {code} 2014-10-31 00:11:56,579 INFO logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:finishLogAggregation(481)) - Application just finished : application_1414705229376_0004 2014-10-31 00:11:56,588 WARN ipc.Client (Client.java:run(675)) - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 75 for user1) is expired 2014-10-31 00:11:56,589 ERROR logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:uploadLogsForContainers(233)) - Cannot create writer for app application_1414705229376_0004. Skip log upload this time. {code} was: We shorten hdfs delegation token lifetime, set RM to act as as a proxy user in order to be able to renew the token on behalf of a user submitting the application. Still we see NM log aggregation fail due to token expiry error. {code} 2014-10-31 00:11:56,579 INFO logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:finishLogAggregation(481)) - Application just finished : application_1414705229376_0004 2014-10-31 00:11:56,588 WARN ipc.Client (Client.java:run(675)) - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 75 for hrt_qa) is expired 2014-10-31 00:11:56,589 ERROR logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:uploadLogsForContainers(233)) - Cannot create writer for app application_1414705229376_0004. Skip log upload this time. {code} NM can't aggregate logs past HDFS delegation token expiry. -- Key: YARN-2790 URL: https://issues.apache.org/jira/browse/YARN-2790 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 2.6.0 Reporter: Tassapol Athiapinya We shorten hdfs delegation token lifetime, set RM to act as as a proxy user in order to be able to renew the token on behalf of a user submitting the application. Still we see NM log aggregation fail due to token expiry error. {code} 2014-10-31 00:11:56,579 INFO logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:finishLogAggregation(481)) - Application just finished : application_1414705229376_0004 2014-10-31 00:11:56,588 WARN ipc.Client (Client.java:run(675)) - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 75 for user1) is expired 2014-10-31 00:11:56,589 ERROR logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:uploadLogsForContainers(233)) - Cannot create writer for app application_1414705229376_0004. Skip log upload this time. {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2790) NM can't aggregate logs past HDFS delegation token expiry.
[ https://issues.apache.org/jira/browse/YARN-2790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-2790: -- Attachment: YARN-2790.1.patch Upload a patch. Move the adding new token to right before uploading the logs. NM can't aggregate logs past HDFS delegation token expiry. -- Key: YARN-2790 URL: https://issues.apache.org/jira/browse/YARN-2790 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 2.6.0 Reporter: Tassapol Athiapinya Assignee: Jian He Attachments: YARN-2790.1.patch We shorten hdfs delegation token lifetime, set RM to act as as a proxy user in order to be able to renew the token on behalf of a user submitting the application. Still we see NM log aggregation fail due to token expiry error. {code} 2014-10-31 00:11:56,579 INFO logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:finishLogAggregation(481)) - Application just finished : application_1414705229376_0004 2014-10-31 00:11:56,588 WARN ipc.Client (Client.java:run(675)) - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 75 for user1) is expired 2014-10-31 00:11:56,589 ERROR logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:uploadLogsForContainers(233)) - Cannot create writer for app application_1414705229376_0004. Skip log upload this time. {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)