[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[ https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258184#comment-14258184 ] Hudson commented on YARN-2837: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #51 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/51/]) YARN-2837. Support TimeLine server to recover delegation token when restarting. Contributed by Zhijie Shen (jianhe: rev 149512a83743715bb100feb6747b7e88301018ab) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/TimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/MemoryTimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryServer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/util/LeveldbUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/LeveldbTimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineDelegationTokenSecretManagerService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/recovery/TestLeveldbTimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java * hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/records/TimelineDelegationTokenIdentifierData.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/proto/yarn_server_timelineserver_recovery.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml Timeline server needs to recover the timeline DT when restarting Key: YARN-2837 URL: https://issues.apache.org/jira/browse/YARN-2837 Project: Hadoop YARN Issue Type: New Feature Components: timelineserver Reporter: Zhijie Shen Assignee: Zhijie Shen Priority: Blocker Fix For: 2.7.0 Attachments: YARN-2837.1.patch, YARN-2837.2.patch, YARN-2837.3.patch, YARN-2837.4.patch, YARN-2837.5.patch, YARN-2837.6.patch, YARN-2837.7.patch Timeline server needs to recover the stateful information when restarting as RM/NM/JHS does now. So far the stateful information only includes the timeline DT. Without recovery, the timeline DT of the existing YARN apps is not long valid, and cannot be renewed any more after the timeline server is restarted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[ https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258191#comment-14258191 ] Hudson commented on YARN-2837: -- FAILURE: Integrated in Hadoop-Yarn-trunk #785 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/785/]) YARN-2837. Support TimeLine server to recover delegation token when restarting. Contributed by Zhijie Shen (jianhe: rev 149512a83743715bb100feb6747b7e88301018ab) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/util/LeveldbUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryServer.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/records/TimelineDelegationTokenIdentifierData.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineDelegationTokenSecretManagerService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/LeveldbTimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/proto/yarn_server_timelineserver_recovery.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/TimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/recovery/TestLeveldbTimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/MemoryTimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java Timeline server needs to recover the timeline DT when restarting Key: YARN-2837 URL: https://issues.apache.org/jira/browse/YARN-2837 Project: Hadoop YARN Issue Type: New Feature Components: timelineserver Reporter: Zhijie Shen Assignee: Zhijie Shen Priority: Blocker Fix For: 2.7.0 Attachments: YARN-2837.1.patch, YARN-2837.2.patch, YARN-2837.3.patch, YARN-2837.4.patch, YARN-2837.5.patch, YARN-2837.6.patch, YARN-2837.7.patch Timeline server needs to recover the stateful information when restarting as RM/NM/JHS does now. So far the stateful information only includes the timeline DT. Without recovery, the timeline DT of the existing YARN apps is not long valid, and cannot be renewed any more after the timeline server is restarted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[ https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258271#comment-14258271 ] Hudson commented on YARN-2837: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1983 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1983/]) YARN-2837. Support TimeLine server to recover delegation token when restarting. Contributed by Zhijie Shen (jianhe: rev 149512a83743715bb100feb6747b7e88301018ab) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/TimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/LeveldbTimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/records/TimelineDelegationTokenIdentifierData.java * hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/recovery/TestLeveldbTimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/util/LeveldbUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/proto/yarn_server_timelineserver_recovery.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/MemoryTimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryServer.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineDelegationTokenSecretManagerService.java Timeline server needs to recover the timeline DT when restarting Key: YARN-2837 URL: https://issues.apache.org/jira/browse/YARN-2837 Project: Hadoop YARN Issue Type: New Feature Components: timelineserver Reporter: Zhijie Shen Assignee: Zhijie Shen Priority: Blocker Fix For: 2.7.0 Attachments: YARN-2837.1.patch, YARN-2837.2.patch, YARN-2837.3.patch, YARN-2837.4.patch, YARN-2837.5.patch, YARN-2837.6.patch, YARN-2837.7.patch Timeline server needs to recover the stateful information when restarting as RM/NM/JHS does now. So far the stateful information only includes the timeline DT. Without recovery, the timeline DT of the existing YARN apps is not long valid, and cannot be renewed any more after the timeline server is restarted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[ https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258285#comment-14258285 ] Hudson commented on YARN-2837: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #48 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/48/]) YARN-2837. Support TimeLine server to recover delegation token when restarting. Contributed by Zhijie Shen (jianhe: rev 149512a83743715bb100feb6747b7e88301018ab) * hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/util/LeveldbUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/TimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/recovery/TestLeveldbTimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineDelegationTokenSecretManagerService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryServer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/LeveldbTimelineStateStore.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/MemoryTimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/records/TimelineDelegationTokenIdentifierData.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/proto/yarn_server_timelineserver_recovery.proto Timeline server needs to recover the timeline DT when restarting Key: YARN-2837 URL: https://issues.apache.org/jira/browse/YARN-2837 Project: Hadoop YARN Issue Type: New Feature Components: timelineserver Reporter: Zhijie Shen Assignee: Zhijie Shen Priority: Blocker Fix For: 2.7.0 Attachments: YARN-2837.1.patch, YARN-2837.2.patch, YARN-2837.3.patch, YARN-2837.4.patch, YARN-2837.5.patch, YARN-2837.6.patch, YARN-2837.7.patch Timeline server needs to recover the stateful information when restarting as RM/NM/JHS does now. So far the stateful information only includes the timeline DT. Without recovery, the timeline DT of the existing YARN apps is not long valid, and cannot be renewed any more after the timeline server is restarted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[ https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258314#comment-14258314 ] Hudson commented on YARN-2837: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #52 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/52/]) YARN-2837. Support TimeLine server to recover delegation token when restarting. Contributed by Zhijie Shen (jianhe: rev 149512a83743715bb100feb6747b7e88301018ab) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/LeveldbTimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineDelegationTokenSecretManagerService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/util/LeveldbUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/recovery/TestLeveldbTimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/records/TimelineDelegationTokenIdentifierData.java * hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryServer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/MemoryTimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/proto/yarn_server_timelineserver_recovery.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/TimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml Timeline server needs to recover the timeline DT when restarting Key: YARN-2837 URL: https://issues.apache.org/jira/browse/YARN-2837 Project: Hadoop YARN Issue Type: New Feature Components: timelineserver Reporter: Zhijie Shen Assignee: Zhijie Shen Priority: Blocker Fix For: 2.7.0 Attachments: YARN-2837.1.patch, YARN-2837.2.patch, YARN-2837.3.patch, YARN-2837.4.patch, YARN-2837.5.patch, YARN-2837.6.patch, YARN-2837.7.patch Timeline server needs to recover the stateful information when restarting as RM/NM/JHS does now. So far the stateful information only includes the timeline DT. Without recovery, the timeline DT of the existing YARN apps is not long valid, and cannot be renewed any more after the timeline server is restarted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[ https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258332#comment-14258332 ] Hudson commented on YARN-2837: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2002 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2002/]) YARN-2837. Support TimeLine server to recover delegation token when restarting. Contributed by Zhijie Shen (jianhe: rev 149512a83743715bb100feb6747b7e88301018ab) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/proto/yarn_server_timelineserver_recovery.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/util/LeveldbUtils.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineDelegationTokenSecretManagerService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/records/TimelineDelegationTokenIdentifierData.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryServer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/TimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/LeveldbTimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/MemoryTimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/recovery/TestLeveldbTimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java Timeline server needs to recover the timeline DT when restarting Key: YARN-2837 URL: https://issues.apache.org/jira/browse/YARN-2837 Project: Hadoop YARN Issue Type: New Feature Components: timelineserver Reporter: Zhijie Shen Assignee: Zhijie Shen Priority: Blocker Fix For: 2.7.0 Attachments: YARN-2837.1.patch, YARN-2837.2.patch, YARN-2837.3.patch, YARN-2837.4.patch, YARN-2837.5.patch, YARN-2837.6.patch, YARN-2837.7.patch Timeline server needs to recover the stateful information when restarting as RM/NM/JHS does now. So far the stateful information only includes the timeline DT. Without recovery, the timeline DT of the existing YARN apps is not long valid, and cannot be renewed any more after the timeline server is restarted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[
https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257754#comment-14257754
]
Hadoop QA commented on YARN-2837:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12688940/YARN-2837.7.patch
against trunk revision 66cfe1d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 24 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests:
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6182//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6182//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-applicationhistoryservice.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6182//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6182//console
This message is automatically generated.
Timeline server needs to recover the timeline DT when restarting
Key: YARN-2837
URL: https://issues.apache.org/jira/browse/YARN-2837
Project: Hadoop YARN
Issue Type: New Feature
Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Priority: Blocker
Fix For: 2.7.0
Attachments: YARN-2837.1.patch, YARN-2837.2.patch, YARN-2837.3.patch,
YARN-2837.4.patch, YARN-2837.5.patch, YARN-2837.6.patch, YARN-2837.7.patch
Timeline server needs to recover the stateful information when restarting as
RM/NM/JHS does now. So far the stateful information only includes the
timeline DT. Without recovery, the timeline DT of the existing YARN apps is
not long valid, and cannot be renewed any more after the timeline server is
restarted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[ https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257770#comment-14257770 ] Zhijie Shen commented on YARN-2837: --- The findbug warnings are not related to this patch. The test failure is also not related. File a separate Jira for the test failure as it turns out to be failed once before too. Timeline server needs to recover the timeline DT when restarting Key: YARN-2837 URL: https://issues.apache.org/jira/browse/YARN-2837 Project: Hadoop YARN Issue Type: New Feature Components: timelineserver Reporter: Zhijie Shen Assignee: Zhijie Shen Priority: Blocker Fix For: 2.7.0 Attachments: YARN-2837.1.patch, YARN-2837.2.patch, YARN-2837.3.patch, YARN-2837.4.patch, YARN-2837.5.patch, YARN-2837.6.patch, YARN-2837.7.patch Timeline server needs to recover the stateful information when restarting as RM/NM/JHS does now. So far the stateful information only includes the timeline DT. Without recovery, the timeline DT of the existing YARN apps is not long valid, and cannot be renewed any more after the timeline server is restarted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[ https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257785#comment-14257785 ] Jian He commented on YARN-2837: --- looks good, +1 Timeline server needs to recover the timeline DT when restarting Key: YARN-2837 URL: https://issues.apache.org/jira/browse/YARN-2837 Project: Hadoop YARN Issue Type: New Feature Components: timelineserver Reporter: Zhijie Shen Assignee: Zhijie Shen Priority: Blocker Fix For: 2.7.0 Attachments: YARN-2837.1.patch, YARN-2837.2.patch, YARN-2837.3.patch, YARN-2837.4.patch, YARN-2837.5.patch, YARN-2837.6.patch, YARN-2837.7.patch Timeline server needs to recover the stateful information when restarting as RM/NM/JHS does now. So far the stateful information only includes the timeline DT. Without recovery, the timeline DT of the existing YARN apps is not long valid, and cannot be renewed any more after the timeline server is restarted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[ https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257854#comment-14257854 ] Hudson commented on YARN-2837: -- FAILURE: Integrated in Hadoop-trunk-Commit #6782 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6782/]) YARN-2837. Support TimeLine server to recover delegation token when restarting. Contributed by Zhijie Shen (jianhe: rev 149512a83743715bb100feb6747b7e88301018ab) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineDelegationTokenSecretManagerService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/TimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryServer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/MemoryTimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/records/TimelineDelegationTokenIdentifierData.java * hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/recovery/LeveldbTimelineStateStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/proto/yarn_server_timelineserver_recovery.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/util/LeveldbUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/recovery/TestLeveldbTimelineStateStore.java Timeline server needs to recover the timeline DT when restarting Key: YARN-2837 URL: https://issues.apache.org/jira/browse/YARN-2837 Project: Hadoop YARN Issue Type: New Feature Components: timelineserver Reporter: Zhijie Shen Assignee: Zhijie Shen Priority: Blocker Fix For: 2.7.0 Attachments: YARN-2837.1.patch, YARN-2837.2.patch, YARN-2837.3.patch, YARN-2837.4.patch, YARN-2837.5.patch, YARN-2837.6.patch, YARN-2837.7.patch Timeline server needs to recover the stateful information when restarting as RM/NM/JHS does now. So far the stateful information only includes the timeline DT. Without recovery, the timeline DT of the existing YARN apps is not long valid, and cannot be renewed any more after the timeline server is restarted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[ https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258023#comment-14258023 ] Jian He commented on YARN-2837: --- Thanks [~gtCarrera9] for reviewing the patch ! Timeline server needs to recover the timeline DT when restarting Key: YARN-2837 URL: https://issues.apache.org/jira/browse/YARN-2837 Project: Hadoop YARN Issue Type: New Feature Components: timelineserver Reporter: Zhijie Shen Assignee: Zhijie Shen Priority: Blocker Fix For: 2.7.0 Attachments: YARN-2837.1.patch, YARN-2837.2.patch, YARN-2837.3.patch, YARN-2837.4.patch, YARN-2837.5.patch, YARN-2837.6.patch, YARN-2837.7.patch Timeline server needs to recover the stateful information when restarting as RM/NM/JHS does now. So far the stateful information only includes the timeline DT. Without recovery, the timeline DT of the existing YARN apps is not long valid, and cannot be renewed any more after the timeline server is restarted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[ https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245035#comment-14245035 ] Zhijie Shen commented on YARN-2837: --- bq. we need to keep track of the latest sequenceNumber so that it can be recovered. For the reason of YARN-2958, I don't think it's proper to following the way of RMStateStore. Please let me know if I missed something. Timeline server needs to recover the timeline DT when restarting Key: YARN-2837 URL: https://issues.apache.org/jira/browse/YARN-2837 Project: Hadoop YARN Issue Type: New Feature Components: timelineserver Reporter: Zhijie Shen Assignee: Zhijie Shen Priority: Blocker Fix For: 2.7.0 Attachments: YARN-2837.1.patch, YARN-2837.2.patch, YARN-2837.3.patch, YARN-2837.4.patch, YARN-2837.5.patch, YARN-2837.6.patch Timeline server needs to recover the stateful information when restarting as RM/NM/JHS does now. So far the stateful information only includes the timeline DT. Without recovery, the timeline DT of the existing YARN apps is not long valid, and cannot be renewed any more after the timeline server is restarted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[
https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14241678#comment-14241678
]
Hadoop QA commented on YARN-2837:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12686346/YARN-2837.6.patch
against trunk revision 2e98ad3.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 111 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6073//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6073//artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6073//console
This message is automatically generated.
Timeline server needs to recover the timeline DT when restarting
Key: YARN-2837
URL: https://issues.apache.org/jira/browse/YARN-2837
Project: Hadoop YARN
Issue Type: New Feature
Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Priority: Blocker
Fix For: 2.7.0
Attachments: YARN-2837.1.patch, YARN-2837.2.patch, YARN-2837.3.patch,
YARN-2837.4.patch, YARN-2837.5.patch, YARN-2837.6.patch
Timeline server needs to recover the stateful information when restarting as
RM/NM/JHS does now. So far the stateful information only includes the
timeline DT. Without recovery, the timeline DT of the existing YARN apps is
not long valid, and cannot be renewed any more after the timeline server is
restarted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[
https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14238684#comment-14238684
]
Hadoop QA commented on YARN-2837:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12685889/YARN-2837.5.patch
against trunk revision ddffcd8.
{color:red}-1 patch{color}. Trunk compilation may be broken.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6043//console
This message is automatically generated.
Timeline server needs to recover the timeline DT when restarting
Key: YARN-2837
URL: https://issues.apache.org/jira/browse/YARN-2837
Project: Hadoop YARN
Issue Type: New Feature
Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Priority: Blocker
Fix For: 2.7.0
Attachments: YARN-2837.1.patch, YARN-2837.2.patch, YARN-2837.3.patch,
YARN-2837.4.patch, YARN-2837.5.patch
Timeline server needs to recover the stateful information when restarting as
RM/NM/JHS does now. So far the stateful information only includes the
timeline DT. Without recovery, the timeline DT of the existing YARN apps is
not long valid, and cannot be renewed any more after the timeline server is
restarted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[
https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14238763#comment-14238763
]
Hadoop QA commented on YARN-2837:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12685897/YARN-2837.5.patch
against trunk revision ddffcd8.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 7 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6045//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6045//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6045//console
This message is automatically generated.
Timeline server needs to recover the timeline DT when restarting
Key: YARN-2837
URL: https://issues.apache.org/jira/browse/YARN-2837
Project: Hadoop YARN
Issue Type: New Feature
Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Priority: Blocker
Fix For: 2.7.0
Attachments: YARN-2837.1.patch, YARN-2837.2.patch, YARN-2837.3.patch,
YARN-2837.4.patch, YARN-2837.5.patch
Timeline server needs to recover the stateful information when restarting as
RM/NM/JHS does now. So far the stateful information only includes the
timeline DT. Without recovery, the timeline DT of the existing YARN apps is
not long valid, and cannot be renewed any more after the timeline server is
restarted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[
https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14238966#comment-14238966
]
Zhijie Shen commented on YARN-2837:
---
The findbug warnings seem not to be related and valid. {{System.out}} and
{{System.err}} are challenged to be dereferenced and may result in NPE.
Timeline server needs to recover the timeline DT when restarting
Key: YARN-2837
URL: https://issues.apache.org/jira/browse/YARN-2837
Project: Hadoop YARN
Issue Type: New Feature
Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Priority: Blocker
Fix For: 2.7.0
Attachments: YARN-2837.1.patch, YARN-2837.2.patch, YARN-2837.3.patch,
YARN-2837.4.patch, YARN-2837.5.patch
Timeline server needs to recover the stateful information when restarting as
RM/NM/JHS does now. So far the stateful information only includes the
timeline DT. Without recovery, the timeline DT of the existing YARN apps is
not long valid, and cannot be renewed any more after the timeline server is
restarted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[
https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234801#comment-14234801
]
Jian He commented on YARN-2837:
---
Thanks Zhijie and Li, some comments on my side:
- Logic in {{Version loadedVersion = loadVersion();}} Consider this scenario:
CURRENT_VERSION_INFO = 2.0; there’s no version info currently saved in
store-store. {{loadVersion}} returns 1.0; It’ll throw inCompatible exception,
even though it should not.
- we can probably use protobuf to incorporate both tokenIdentifer and the
renewDate to support better compatibility. e.g. RMDelegationTokenIdentifierData
{code}
renewDate = in.readLong();
tokenId.readFields(in);
{code}
- {{LeveldbTimelineStateStore#updateToken}}, it’s always adding new token, the
old token still remain, we should remove the old token
- {{AbstractDelegationTokenSecretManager#delegationTokenSequenceNumber}} is not
updated on recovery; the implementation seems using sequenceNumber as the key,
we need to keep track of the latest sequenceNumber so that it can be recovered.
- the following looks ok , a simpler way might be to just concatenate two
strings.
{code}
.add(TOKEN_MASTER_KEY_ENTRY_PREFIX).add(Integer.toString(keyId)){code}
- the following log is present in both
{{TimelineDelegationTokenSecretManager#storeNewMasterKey}} and the underlying
state-store implementation; only printing in one place is enough. similar to
other operations.
{code}
if (LOG.isDebugEnabled()) {
LOG.debug(Storing master key + key.getKeyId());
}
{code}
- FILENAME-DB_NAME; leveldb-state-store.ldb - timeline-state-store.ldb
- default path for state store is the same as time-line store for application
data. If apps posts massive data in store, will that also affect system data
seek performance ? If so, we should have a different store path from the one
for apps. Or we could force user to configure the path properly and throw
exception otherwise.
Timeline server needs to recover the timeline DT when restarting
Key: YARN-2837
URL: https://issues.apache.org/jira/browse/YARN-2837
Project: Hadoop YARN
Issue Type: New Feature
Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Priority: Blocker
Fix For: 2.7.0
Attachments: YARN-2837.1.patch, YARN-2837.2.patch, YARN-2837.3.patch
Timeline server needs to recover the stateful information when restarting as
RM/NM/JHS does now. So far the stateful information only includes the
timeline DT. Without recovery, the timeline DT of the existing YARN apps is
not long valid, and cannot be renewed any more after the timeline server is
restarted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[
https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234844#comment-14234844
]
Li Lu commented on YARN-2837:
-
{quote}
Logic in Version loadedVersion = loadVersion(); Consider this scenario:
CURRENT_VERSION_INFO = 2.0; there’s no version info currently saved in
store-store. loadVersion returns 1.0; It’ll throw inCompatible exception, even
though it should not.
{quote}
This looks to be a valid concern, but I noticed similar logic also exist in
LeveldbTimelineStore. We need to be consistent on this logic.
{quote}
default path for state store is the same as time-line store for application
data. If apps posts massive data in store, will that also affect system data
seek performance ?
{quote}
The two leveldb stores are working on different leveldb files, so I think it's
fine.
Timeline server needs to recover the timeline DT when restarting
Key: YARN-2837
URL: https://issues.apache.org/jira/browse/YARN-2837
Project: Hadoop YARN
Issue Type: New Feature
Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Priority: Blocker
Fix For: 2.7.0
Attachments: YARN-2837.1.patch, YARN-2837.2.patch, YARN-2837.3.patch
Timeline server needs to recover the stateful information when restarting as
RM/NM/JHS does now. So far the stateful information only includes the
timeline DT. Without recovery, the timeline DT of the existing YARN apps is
not long valid, and cannot be renewed any more after the timeline server is
restarted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[
https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234847#comment-14234847
]
Zhijie Shen commented on YARN-2837:
---
bq. It’ll throw inCompatible exception, even though it should not.
I guess it's way we want to do. When currently CURRENT_VERSION_INFO = 1.0, we
make it compatible with the existing store without version info. When
CURRENT_VERSION_INFO is upgraded to 2.0, the db schema is no longer compatible
to 1.0 and the exception is thrown. Going directly from no version to 2.0 may
not be a valid use case.
Here I simply reuse the same logic that other leveldb impl is using. If you
have some concern, we can figure out it separately.
bq. RMDelegationTokenIdentifierData
I noticed it before, but I don't think it is really necessary to create another
protobuf obj just because we have one additional long integer to ser/des.
bq. it’s always adding new token, the old token still remain, we should remove
the old token
{{db.put(k, v);}} will update the value if key already exists.
bq. the following looks ok , a simpler way might be to just concatenate two
strings.
That's different. See KeyBuilder and KeyParser for more detail. It's need to
be broken into two section for the convenience of parsing it.
bq. Or we could force user to configure the path properly and throw exception
otherwise.
I prefer to keep state store and data store under the same dir by default. For
advanced deployment, the user is free to config them separately.
bq. we need to keep track of the latest sequenceNumber so that it can be
recovered.
It is updated here.
{code}
public void recover(TimelineServiceState state) throws IOException {
LOG.info(Recovering + getClass().getSimpleName());
for (DelegationKey key : state.getTokenMasterKeyState()) {
addKey(key);
}
for (EntryTimelineDelegationTokenIdentifier, Long entry :
state.getTokenState().entrySet()) {
addPersistedDelegationToken(entry.getKey(), entry.getValue());
}
}
{code}
Otherwise, I updated the patch accordingly. BTW, I've removed the cache config,
as it is not so important to the state store.
Timeline server needs to recover the timeline DT when restarting
Key: YARN-2837
URL: https://issues.apache.org/jira/browse/YARN-2837
Project: Hadoop YARN
Issue Type: New Feature
Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Priority: Blocker
Fix For: 2.7.0
Attachments: YARN-2837.1.patch, YARN-2837.2.patch, YARN-2837.3.patch
Timeline server needs to recover the stateful information when restarting as
RM/NM/JHS does now. So far the stateful information only includes the
timeline DT. Without recovery, the timeline DT of the existing YARN apps is
not long valid, and cannot be renewed any more after the timeline server is
restarted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[
https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14235055#comment-14235055
]
Hadoop QA commented on YARN-2837:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12685212/YARN-2837.4.patch
against trunk revision 7896815.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5997//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5997//console
This message is automatically generated.
Timeline server needs to recover the timeline DT when restarting
Key: YARN-2837
URL: https://issues.apache.org/jira/browse/YARN-2837
Project: Hadoop YARN
Issue Type: New Feature
Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Priority: Blocker
Fix For: 2.7.0
Attachments: YARN-2837.1.patch, YARN-2837.2.patch, YARN-2837.3.patch,
YARN-2837.4.patch
Timeline server needs to recover the stateful information when restarting as
RM/NM/JHS does now. So far the stateful information only includes the
timeline DT. Without recovery, the timeline DT of the existing YARN apps is
not long valid, and cannot be renewed any more after the timeline server is
restarted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[ https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14233234#comment-14233234 ] Zhijie Shen commented on YARN-2837: --- bq. Maybe we want to make the version control logic a unified interface in future? I think it's a good suggestion, but how about leaving code refactoring separately? In addition to the timeline server, other components have the state store that is built on top of leveldb, and have the similar version related code. We can do one-pass refactoring, make all leveldb store impls share the common code. Let's file a Jira for it. Timeline server needs to recover the timeline DT when restarting Key: YARN-2837 URL: https://issues.apache.org/jira/browse/YARN-2837 Project: Hadoop YARN Issue Type: New Feature Components: timelineserver Reporter: Zhijie Shen Assignee: Zhijie Shen Priority: Blocker Fix For: 2.7.0 Attachments: YARN-2837.1.patch, YARN-2837.2.patch, YARN-2837.3.patch Timeline server needs to recover the stateful information when restarting as RM/NM/JHS does now. So far the stateful information only includes the timeline DT. Without recovery, the timeline DT of the existing YARN apps is not long valid, and cannot be renewed any more after the timeline server is restarted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[ https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14233294#comment-14233294 ] Li Lu commented on YARN-2837: - Agree to [~zjshen]'s suggestion. Let's do that in a separate Jira. I'd +1 this patch, and maybe some committers would like to take a look at it? Timeline server needs to recover the timeline DT when restarting Key: YARN-2837 URL: https://issues.apache.org/jira/browse/YARN-2837 Project: Hadoop YARN Issue Type: New Feature Components: timelineserver Reporter: Zhijie Shen Assignee: Zhijie Shen Priority: Blocker Fix For: 2.7.0 Attachments: YARN-2837.1.patch, YARN-2837.2.patch, YARN-2837.3.patch Timeline server needs to recover the stateful information when restarting as RM/NM/JHS does now. So far the stateful information only includes the timeline DT. Without recovery, the timeline DT of the existing YARN apps is not long valid, and cannot be renewed any more after the timeline server is restarted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[
https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14231953#comment-14231953
]
Hadoop QA commented on YARN-2837:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12684557/YARN-2837.3.patch
against trunk revision 26319ba.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5979//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5979//console
This message is automatically generated.
Timeline server needs to recover the timeline DT when restarting
Key: YARN-2837
URL: https://issues.apache.org/jira/browse/YARN-2837
Project: Hadoop YARN
Issue Type: New Feature
Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Priority: Blocker
Fix For: 2.7.0
Attachments: YARN-2837.1.patch, YARN-2837.2.patch, YARN-2837.3.patch
Timeline server needs to recover the stateful information when restarting as
RM/NM/JHS does now. So far the stateful information only includes the
timeline DT. Without recovery, the timeline DT of the existing YARN apps is
not long valid, and cannot be renewed any more after the timeline server is
restarted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[ https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14232210#comment-14232210 ] Li Lu commented on YARN-2837: - Hi [~zjshen], this version looks good to me (disabling the findbugs warning here looks fine to me). One question is about the version storage: seems like LeveldbTimelineStore also has the same logic on storing/loading versions. Maybe we want to make the version control logic a unified interface in future? Timeline server needs to recover the timeline DT when restarting Key: YARN-2837 URL: https://issues.apache.org/jira/browse/YARN-2837 Project: Hadoop YARN Issue Type: New Feature Components: timelineserver Reporter: Zhijie Shen Assignee: Zhijie Shen Priority: Blocker Fix For: 2.7.0 Attachments: YARN-2837.1.patch, YARN-2837.2.patch, YARN-2837.3.patch Timeline server needs to recover the stateful information when restarting as RM/NM/JHS does now. So far the stateful information only includes the timeline DT. Without recovery, the timeline DT of the existing YARN apps is not long valid, and cannot be renewed any more after the timeline server is restarted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[
https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230882#comment-14230882
]
Zhijie Shen commented on YARN-2837:
---
bq. One question is, are we planning to support more types of status
information in the future?
Timeline server is the stateless machine. Currently, the only state I think of
now is the delegation token. Maybe in future, if we use queue to buffer the
requests to process, we need to store those too when restarting the timeline
server. IAC, let's leave it later.
bq. Shall we further restrict its visibility to YARN only (by
{{@LimitedPrivate}})?
AFAIK, {{@Private}} equals to {{@LimitedPrivate(current project)}}, while
{{@LimitedPrivate}} can mark a class private to others and multiple sub
projects.
Timeline server needs to recover the timeline DT when restarting
Key: YARN-2837
URL: https://issues.apache.org/jira/browse/YARN-2837
Project: Hadoop YARN
Issue Type: New Feature
Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Priority: Blocker
Attachments: YARN-2837.1.patch, YARN-2837.2.patch
Timeline server needs to recover the stateful information when restarting as
RM/NM/JHS does now. So far the stateful information only includes the
timeline DT. Without recovery, the timeline DT of the existing YARN apps is
not long valid, and cannot be renewed any more after the timeline server is
restarted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[
https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230915#comment-14230915
]
Hadoop QA commented on YARN-2837:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12684536/YARN-2837.2.patch
against trunk revision a7fba0b.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5968//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/5968//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-applicationhistoryservice.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5968//console
This message is automatically generated.
Timeline server needs to recover the timeline DT when restarting
Key: YARN-2837
URL: https://issues.apache.org/jira/browse/YARN-2837
Project: Hadoop YARN
Issue Type: New Feature
Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Priority: Blocker
Fix For: 2.7.0
Attachments: YARN-2837.1.patch, YARN-2837.2.patch
Timeline server needs to recover the stateful information when restarting as
RM/NM/JHS does now. So far the stateful information only includes the
timeline DT. Without recovery, the timeline DT of the existing YARN apps is
not long valid, and cannot be renewed any more after the timeline server is
restarted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[
https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225490#comment-14225490
]
Li Lu commented on YARN-2837:
-
Hi [~zjshen], I reviewed your patch. Generally it looks good. I have the
following comments:
1. If I understand this patch correctly, storing and loading timeline server
status is asymmetric: stores happen each time
TimelineDelegationTokenSecretManager's methods are called, while loads only
happen when TimelineDelegationTokenSecretManagerService starts. This may look
counterintuitive, but it makes sense because in this way the state store
service can keep the status updated. One question is, are we planning to
support more types of status information in the future? Though totally fine for
now, having separate state store methods may not be a scalable approach if we
want to support more status info (if we do not, I think it's fine).
2. The name of MemoryTimelineServerStateStoreService.java looks weird. Other
similar files are all named as \*TimelineServiceStateStoreService. Seems like a
typo because the class inside is also named as \*TimelineService\*. (Names are
too long to notice the difference?)
3. Line 244 of LeveldbTimelineServiceStateStoreService.java is too long. It has
86 characters.
4. From Line 291 of LeveldbTimelineServiceStateStoreService.java:
{code}
/**
* Creates a domain entity key with column name suffix, of the form
* TOKEN_MASTER_KEY_ENTRY_PREFIX + sequence number.
*/
{code}
This is inconsistent with the actual case in the code. In the code it shows
that the master key's key in Leveldb is its key id.
5. I noticed you made the KeyBuilder class in LeveldbTimelineStore.java to be
public, and then annotated it as {{@Private}}. Shall we further restrict its
visibility to YARN only (by {{@LimitedPrivate}})? I think we just need
KeyBuilder to be public to be used by
LeveldbTimelineServiceStateStoreService, right?
Timeline server needs to recover the timeline DT when restarting
Key: YARN-2837
URL: https://issues.apache.org/jira/browse/YARN-2837
Project: Hadoop YARN
Issue Type: New Feature
Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Priority: Blocker
Attachments: YARN-2837.1.patch
Timeline server needs to recover the stateful information when restarting as
RM/NM/JHS does now. So far the stateful information only includes the
timeline DT. Without recovery, the timeline DT of the existing YARN apps is
not long valid, and cannot be renewed any more after the timeline server is
restarted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[
https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14204525#comment-14204525
]
Hadoop QA commented on YARN-2837:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12680544/YARN-2837.1.patch
against trunk revision ab30d51.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5808//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5808//console
This message is automatically generated.
Timeline server needs to recover the timeline DT when restarting
Key: YARN-2837
URL: https://issues.apache.org/jira/browse/YARN-2837
Project: Hadoop YARN
Issue Type: New Feature
Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Priority: Blocker
Attachments: YARN-2837.1.patch
Timeline server needs to recover the stateful information when restarting as
RM/NM/JHS does now. So far the stateful information only includes the
timeline DT. Without recovery, the timeline DT of the existing YARN apps is
not long valid, and cannot be renewed any more after the timeline server is
restarted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2837) Timeline server needs to recover the timeline DT when restarting
[ https://issues.apache.org/jira/browse/YARN-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205768#comment-14205768 ] Zhijie Shen commented on YARN-2837: --- Test the patch on a single node secure cluster: 1. Start and restart the timeline server, and the DT information is recovered properly. 2. The DT generated in before the timeline server can be renewed properly afterwards. Some other issues I've observed while doing test: At the very early seconds the http server is started, the MR job, which tries to emit the timeline data, gets a number of 404 error. I guess the server is not fully ready before it taking the incoming requests. Timeline server needs to recover the timeline DT when restarting Key: YARN-2837 URL: https://issues.apache.org/jira/browse/YARN-2837 Project: Hadoop YARN Issue Type: New Feature Components: timelineserver Reporter: Zhijie Shen Assignee: Zhijie Shen Priority: Blocker Attachments: YARN-2837.1.patch Timeline server needs to recover the stateful information when restarting as RM/NM/JHS does now. So far the stateful information only includes the timeline DT. Without recovery, the timeline DT of the existing YARN apps is not long valid, and cannot be renewed any more after the timeline server is restarted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
