[jira] [Updated] (YARN-11383) Workflow priority mappings is case sensitive
[ https://issues.apache.org/jira/browse/YARN-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-11383: Fix Version/s: 3.4.0 3.3.5 > Workflow priority mappings is case sensitive > > > Key: YARN-11383 > URL: https://issues.apache.org/jira/browse/YARN-11383 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Aparajita Choudhary >Assignee: Aparajita Choudhary >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.5 > > > Issue is related to feature - https://issues.apache.org/jira/browse/YARN-9760 > Workflow ID of an application submitted to yarn is passed in application tags > and yarn lowercases it. However, users (responsible for populating workflow > to priority mappings), might not add workflow IDs in lowercase. Hence, this > creates an issue. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11383) Workflow priority mappings is case sensitive
[ https://issues.apache.org/jira/browse/YARN-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena resolved YARN-11383. - Hadoop Flags: Reviewed Resolution: Fixed > Workflow priority mappings is case sensitive > > > Key: YARN-11383 > URL: https://issues.apache.org/jira/browse/YARN-11383 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Aparajita Choudhary >Assignee: Aparajita Choudhary >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.5 > > > Issue is related to feature - https://issues.apache.org/jira/browse/YARN-9760 > Workflow ID of an application submitted to yarn is passed in application tags > and yarn lowercases it. However, users (responsible for populating workflow > to priority mappings), might not add workflow IDs in lowercase. Hence, this > creates an issue. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11383) Workflow priority mappings is case sensitive
[ https://issues.apache.org/jira/browse/YARN-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-11383: Target Version/s: 3.4.0 (was: 2.10.0) > Workflow priority mappings is case sensitive > > > Key: YARN-11383 > URL: https://issues.apache.org/jira/browse/YARN-11383 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Aparajita Choudhary >Assignee: Aparajita Choudhary >Priority: Major > Labels: pull-request-available > > Issue is related to feature - https://issues.apache.org/jira/browse/YARN-9760 > Workflow ID of an application submitted to yarn is passed in application tags > and yarn lowercases it. However, users (responsible for populating workflow > to priority mappings), might not add workflow IDs in lowercase. Hence, this > creates an issue. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-11406) Print Resource Utilization Aggregates of a Container in logs
[ https://issues.apache.org/jira/browse/YARN-11406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena reassigned YARN-11406: --- Assignee: Balaji Kannimandhiri Muniswami > Print Resource Utilization Aggregates of a Container in logs > > > Key: YARN-11406 > URL: https://issues.apache.org/jira/browse/YARN-11406 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Balaji Kannimandhiri Muniswami >Assignee: Balaji Kannimandhiri Muniswami >Priority: Major > Labels: pull-request-available > > Average and Maximum usage of Physical Memory, Virtual Memory & Cpu of RM > Containers provides great level of visibility about container's resource > utilization. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-11406) Print Resource Utilization Aggregates of a Container in logs
[ https://issues.apache.org/jira/browse/YARN-11406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena reassigned YARN-11406: --- Assignee: (was: Varun Saxena) > Print Resource Utilization Aggregates of a Container in logs > > > Key: YARN-11406 > URL: https://issues.apache.org/jira/browse/YARN-11406 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Balaji Kannimandhiri Muniswami >Priority: Major > Labels: pull-request-available > > Average and Maximum usage of Physical Memory, Virtual Memory & Cpu of RM > Containers provides great level of visibility about container's resource > utilization. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-11406) Print Resource Utilization Aggregates of a Container in logs
[ https://issues.apache.org/jira/browse/YARN-11406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena reassigned YARN-11406: --- Assignee: Varun Saxena > Print Resource Utilization Aggregates of a Container in logs > > > Key: YARN-11406 > URL: https://issues.apache.org/jira/browse/YARN-11406 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Balaji Kannimandhiri Muniswami >Assignee: Varun Saxena >Priority: Major > Labels: pull-request-available > > Average and Maximum usage of Physical Memory, Virtual Memory & Cpu of RM > Containers provides great level of visibility about container's resource > utilization. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-3008) FairScheduler: Use lock for queuemanager instead of synchronized on FairScheduler
[ https://issues.apache.org/jira/browse/YARN-3008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena reassigned YARN-3008: -- Assignee: (was: Varun Saxena) > FairScheduler: Use lock for queuemanager instead of synchronized on > FairScheduler > - > > Key: YARN-3008 > URL: https://issues.apache.org/jira/browse/YARN-3008 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Reporter: Anubhav Dhoot >Priority: Major > > Instead of a big monolithic lock on FairScheduler, we can have an explicit > lock on queuemanager and revisit all synchronized methods in FairScheduler -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9760) Support configuring application priorities on a workflow level
[ https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16947034#comment-16947034 ] Varun Saxena commented on YARN-9760: branch-2 and branch-3.1 patches LGTM. The test failures are the same ones that have been discussed for the trunk patch. > Support configuring application priorities on a workflow level > -- > > Key: YARN-9760 > URL: https://issues.apache.org/jira/browse/YARN-9760 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Jonathan Hung >Assignee: Varun Saxena >Priority: Major > Labels: release-blocker > Attachments: YARN-9760-branch-2.001.patch, > YARN-9760-branch-3.1.001.patch, YARN-9760-test.patch, YARN-9760.01.patch, > YARN-9760.02.patch, YARN-9760.03.patch, YARN-9760.04.patch, > YARN-9760.05.patch, YARN-9760.06.patch > > > Currently priorities are submitted on an application level, but for end users > it's common to submit workloads to YARN at a workflow level. This jira > proposes a feature to store workflow id + priority mappings on RM (similar to > queue mappings). If app is submitted with a certain workflow id (as set in > application submission context) RM will override this app's priority with the > one defined in the mapping. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9760) Support configuring application priorities on a workflow level
[ https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16947034#comment-16947034 ] Varun Saxena edited comment on YARN-9760 at 10/8/19 4:31 PM: - +1. branch-2 and branch-3.1 patches LGTM. The test failures are the same ones that have been discussed for the trunk patch. was (Author: varun_saxena): branch-2 and branch-3.1 patches LGTM. The test failures are the same ones that have been discussed for the trunk patch. > Support configuring application priorities on a workflow level > -- > > Key: YARN-9760 > URL: https://issues.apache.org/jira/browse/YARN-9760 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Jonathan Hung >Assignee: Varun Saxena >Priority: Major > Labels: release-blocker > Attachments: YARN-9760-branch-2.001.patch, > YARN-9760-branch-3.1.001.patch, YARN-9760-test.patch, YARN-9760.01.patch, > YARN-9760.02.patch, YARN-9760.03.patch, YARN-9760.04.patch, > YARN-9760.05.patch, YARN-9760.06.patch > > > Currently priorities are submitted on an application level, but for end users > it's common to submit workloads to YARN at a workflow level. This jira > proposes a feature to store workflow id + priority mappings on RM (similar to > queue mappings). If app is submitted with a certain workflow id (as set in > application submission context) RM will override this app's priority with the > one defined in the mapping. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9760) Support configuring application priorities on a workflow level
[ https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-9760: --- Attachment: YARN-9760.04.patch > Support configuring application priorities on a workflow level > -- > > Key: YARN-9760 > URL: https://issues.apache.org/jira/browse/YARN-9760 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Jonathan Hung >Assignee: Varun Saxena >Priority: Major > Labels: release-blocker > Attachments: YARN-9760.01.patch, YARN-9760.02.patch, > YARN-9760.03.patch, YARN-9760.04.patch > > > Currently priorities are submitted on an application level, but for end users > it's common to submit workloads to YARN at a workflow level. This jira > proposes a feature to store workflow id + priority mappings on RM (similar to > queue mappings). If app is submitted with a certain workflow id (as set in > application submission context) RM will override this app's priority with the > one defined in the mapping. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9760) Support configuring application priorities on a workflow level
[ https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-9760: --- Attachment: YARN-9760.03.patch > Support configuring application priorities on a workflow level > -- > > Key: YARN-9760 > URL: https://issues.apache.org/jira/browse/YARN-9760 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Jonathan Hung >Assignee: Varun Saxena >Priority: Major > Labels: release-blocker > Attachments: YARN-9760.01.patch, YARN-9760.02.patch, > YARN-9760.03.patch > > > Currently priorities are submitted on an application level, but for end users > it's common to submit workloads to YARN at a workflow level. This jira > proposes a feature to store workflow id + priority mappings on RM (similar to > queue mappings). If app is submitted with a certain workflow id (as set in > application submission context) RM will override this app's priority with the > one defined in the mapping. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9760) Support configuring application priorities on a workflow level
[ https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16938635#comment-16938635 ] Varun Saxena commented on YARN-9760: MAPREDUCE-7231 tracks hadoop-mapreduce-client-jobclient timeout. Related to TestPipeApplication. Going by MAPREDUCE-7036, the ASF license error may also be related. TestFairSchedulerPreemption failure is tracked by YARN-9333. Will fix some of the checkstyle issues post the review. > Support configuring application priorities on a workflow level > -- > > Key: YARN-9760 > URL: https://issues.apache.org/jira/browse/YARN-9760 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Jonathan Hung >Assignee: Varun Saxena >Priority: Major > Labels: release-blocker > Attachments: YARN-9760.01.patch, YARN-9760.02.patch > > > Currently priorities are submitted on an application level, but for end users > it's common to submit workloads to YARN at a workflow level. This jira > proposes a feature to store workflow id + priority mappings on RM (similar to > queue mappings). If app is submitted with a certain workflow id (as set in > application submission context) RM will override this app's priority with the > one defined in the mapping. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9760) Support configuring application priorities on a workflow level
[ https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-9760: --- Attachment: YARN-9760.02.patch > Support configuring application priorities on a workflow level > -- > > Key: YARN-9760 > URL: https://issues.apache.org/jira/browse/YARN-9760 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Jonathan Hung >Assignee: Varun Saxena >Priority: Major > Labels: release-blocker > Attachments: YARN-9760.01.patch, YARN-9760.02.patch > > > Currently priorities are submitted on an application level, but for end users > it's common to submit workloads to YARN at a workflow level. This jira > proposes a feature to store workflow id + priority mappings on RM (similar to > queue mappings). If app is submitted with a certain workflow id (as set in > application submission context) RM will override this app's priority with the > one defined in the mapping. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9760) Support configuring application priorities on a workflow level
[ https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-9760: --- Attachment: YARN-9760.01.patch > Support configuring application priorities on a workflow level > -- > > Key: YARN-9760 > URL: https://issues.apache.org/jira/browse/YARN-9760 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Jonathan Hung >Assignee: Varun Saxena >Priority: Major > Labels: release-blocker > Attachments: YARN-9760.01.patch > > > Currently priorities are submitted on an application level, but for end users > it's common to submit workloads to YARN at a workflow level. This jira > proposes a feature to store workflow id + priority mappings on RM (similar to > queue mappings). If app is submitted with a certain workflow id (as set in > application submission context) RM will override this app's priority with the > one defined in the mapping. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9760) Support configuring application priorities on a workflow level
[ https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929815#comment-16929815 ] Varun Saxena edited comment on YARN-9760 at 9/14/19 8:20 PM: - Attached a patch as a reference for what we are planning to do was (Author: varun_saxena): Attached a patch as a reference to what we are planning to do > Support configuring application priorities on a workflow level > -- > > Key: YARN-9760 > URL: https://issues.apache.org/jira/browse/YARN-9760 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Jonathan Hung >Assignee: Varun Saxena >Priority: Major > Labels: release-blocker > Attachments: YARN-9760.01.patch > > > Currently priorities are submitted on an application level, but for end users > it's common to submit workloads to YARN at a workflow level. This jira > proposes a feature to store workflow id + priority mappings on RM (similar to > queue mappings). If app is submitted with a certain workflow id (as set in > application submission context) RM will override this app's priority with the > one defined in the mapping. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9760) Support configuring application priorities on a workflow level
[ https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929815#comment-16929815 ] Varun Saxena commented on YARN-9760: Attached a patch as a reference to what we are planning to do > Support configuring application priorities on a workflow level > -- > > Key: YARN-9760 > URL: https://issues.apache.org/jira/browse/YARN-9760 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Jonathan Hung >Assignee: Varun Saxena >Priority: Major > Labels: release-blocker > Attachments: YARN-9760.01.patch > > > Currently priorities are submitted on an application level, but for end users > it's common to submit workloads to YARN at a workflow level. This jira > proposes a feature to store workflow id + priority mappings on RM (similar to > queue mappings). If app is submitted with a certain workflow id (as set in > application submission context) RM will override this app's priority with the > one defined in the mapping. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9760) Support configuring application priorities on a workflow level
[ https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929810#comment-16929810 ] Varun Saxena commented on YARN-9760: Sorry, I missed the comments here. [~eepayne], as Jonathan explained we wanted cluster admins to control the priority, hence the feature. [~jhung], we can use YARN tags to get workflow information as we do in timeline service. By leveraging queue mapping interface, I guess you mean the PlacementRule interface. That though is meant primarily for queue placement, the way it is defined right now. We can potentially have a new interface, once we have more other use cases for setting priority i.e. based on something other than workflows? > Support configuring application priorities on a workflow level > -- > > Key: YARN-9760 > URL: https://issues.apache.org/jira/browse/YARN-9760 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Jonathan Hung >Assignee: Varun Saxena >Priority: Major > Labels: release-blocker > > Currently priorities are submitted on an application level, but for end users > it's common to submit workloads to YARN at a workflow level. This jira > proposes a feature to store workflow id + priority mappings on RM (similar to > queue mappings). If app is submitted with a certain workflow id (as set in > application submission context) RM will override this app's priority with the > one defined in the mapping. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9825) Changes for initializing placement rules with ResourceScheduler in branch-2
[ https://issues.apache.org/jira/browse/YARN-9825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929134#comment-16929134 ] Varun Saxena commented on YARN-9825: Committed to branch-2. Thanks [~jhung] for your contribution > Changes for initializing placement rules with ResourceScheduler in branch-2 > --- > > Key: YARN-9825 > URL: https://issues.apache.org/jira/browse/YARN-9825 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Labels: release-blocker > Attachments: YARN-9825-branch-2.001.patch > > > YARN-8016 and YARN-8948 add functionality to initialize placement rules with > ResourceScheduler. We need this in branch-2, but it doesn't apply cleanly. > Hence we just port the initialization logic. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9825) Changes for initializing placement rules with ResourceScheduler in branch-2
[ https://issues.apache.org/jira/browse/YARN-9825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928569#comment-16928569 ] Varun Saxena commented on YARN-9825: +1. Will commit it in a few hours. > Changes for initializing placement rules with ResourceScheduler in branch-2 > --- > > Key: YARN-9825 > URL: https://issues.apache.org/jira/browse/YARN-9825 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Labels: release-blocker > Attachments: YARN-9825-branch-2.001.patch > > > YARN-8016 and YARN-8948 add functionality to initialize placement rules with > ResourceScheduler. We need this in branch-2, but it doesn't apply cleanly. > Hence we just port the initialization logic. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9764) Print application submission context label in application summary
[ https://issues.apache.org/jira/browse/YARN-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena reassigned YARN-9764: -- Assignee: Manoj Kumar > Print application submission context label in application summary > - > > Key: YARN-9764 > URL: https://issues.apache.org/jira/browse/YARN-9764 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Manoj Kumar >Priority: Major > Labels: release-blocker > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9762) Add submission context label to audit logs
[ https://issues.apache.org/jira/browse/YARN-9762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena reassigned YARN-9762: -- Assignee: Manoj Kumar > Add submission context label to audit logs > -- > > Key: YARN-9762 > URL: https://issues.apache.org/jira/browse/YARN-9762 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Manoj Kumar >Priority: Major > Labels: release-blocker > > Currently we log NODELABEL in container allocation/release audit logs, we > should also log NODELABEL of application submission context on app submission. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9763) Print application tags in application summary
[ https://issues.apache.org/jira/browse/YARN-9763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena reassigned YARN-9763: -- Assignee: Manoj Kumar > Print application tags in application summary > - > > Key: YARN-9763 > URL: https://issues.apache.org/jira/browse/YARN-9763 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Manoj Kumar >Priority: Major > Labels: release-blocker > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9810) Add queue capacity/maxcapacity percentage metrics
[ https://issues.apache.org/jira/browse/YARN-9810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena reassigned YARN-9810: -- Assignee: Shubham Gupta (was: Jonathan Hung) > Add queue capacity/maxcapacity percentage metrics > - > > Key: YARN-9810 > URL: https://issues.apache.org/jira/browse/YARN-9810 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Shubham Gupta >Priority: Major > Labels: release-blocker > Attachments: YARN-9810.01.patch > > > Similar to YARN-9085, it'd be good to have queue (absolute) capacity / > (absolute) max capacity metrics in CSQueueMetrics. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9761) Allow overriding application submissions based on server side configs
[ https://issues.apache.org/jira/browse/YARN-9761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena reassigned YARN-9761: -- Assignee: pralabhkumar > Allow overriding application submissions based on server side configs > - > > Key: YARN-9761 > URL: https://issues.apache.org/jira/browse/YARN-9761 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Jonathan Hung >Assignee: pralabhkumar >Priority: Major > Labels: release-blocker > > Create a preprocessor/interceptor which takes each app submitted to RM and > overrides the submission context based on server side configs. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9760) Support configuring application priorities on a workflow level
[ https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena reassigned YARN-9760: -- Assignee: Varun Saxena > Support configuring application priorities on a workflow level > -- > > Key: YARN-9760 > URL: https://issues.apache.org/jira/browse/YARN-9760 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Jonathan Hung >Assignee: Varun Saxena >Priority: Major > > Currently priorities are submitted on an application level, but for end users > it's common to submit workloads to YARN at a workflow level. This jira > proposes a feature to store workflow id + priority mappings on RM (similar to > queue mappings). If app is submitted with a certain workflow id (as set in > application submission context) RM will override this app's priority with the > one defined in the mapping. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6149) Allow port range to be specified while starting NM Timeline collector manager.
[ https://issues.apache.org/jira/browse/YARN-6149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena reassigned YARN-6149: -- Assignee: Abhishek Modi (was: Varun Saxena) > Allow port range to be specified while starting NM Timeline collector manager. > -- > > Key: YARN-6149 > URL: https://issues.apache.org/jira/browse/YARN-6149 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Abhishek Modi >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6149) Allow port range to be specified while starting NM Timeline collector manager.
[ https://issues.apache.org/jira/browse/YARN-6149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651159#comment-16651159 ] Varun Saxena commented on YARN-6149: Sure [~abmodi], go ahead. I will help with the reviews. > Allow port range to be specified while starting NM Timeline collector manager. > -- > > Key: YARN-6149 > URL: https://issues.apache.org/jira/browse/YARN-6149 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3879) [Storage implementation] Create HDFS backing storage implementation for ATS reads
[ https://issues.apache.org/jira/browse/YARN-3879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328321#comment-16328321 ] Varun Saxena commented on YARN-3879: cc [~ozawa], hope you are fine with Abhishek picking this up. > [Storage implementation] Create HDFS backing storage implementation for ATS > reads > - > > Key: YARN-3879 > URL: https://issues.apache.org/jira/browse/YARN-3879 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Tsuyoshi Ozawa >Assignee: Abhishek Modi >Priority: Major > Labels: YARN-5355 > > Reader version of YARN-3841 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3879) [Storage implementation] Create HDFS backing storage implementation for ATS reads
[ https://issues.apache.org/jira/browse/YARN-3879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328311#comment-16328311 ] Varun Saxena commented on YARN-3879: [~abmodi], added you to the list of contributors and assigned the Jira to you > [Storage implementation] Create HDFS backing storage implementation for ATS > reads > - > > Key: YARN-3879 > URL: https://issues.apache.org/jira/browse/YARN-3879 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Tsuyoshi Ozawa >Assignee: Abhishek Modi >Priority: Major > Labels: YARN-5355 > > Reader version of YARN-3841 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-3879) [Storage implementation] Create HDFS backing storage implementation for ATS reads
[ https://issues.apache.org/jira/browse/YARN-3879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena reassigned YARN-3879: -- Assignee: Abhishek Modi > [Storage implementation] Create HDFS backing storage implementation for ATS > reads > - > > Key: YARN-3879 > URL: https://issues.apache.org/jira/browse/YARN-3879 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Tsuyoshi Ozawa >Assignee: Abhishek Modi >Priority: Major > Labels: YARN-5355 > > Reader version of YARN-3841 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7714) YARN_TIMELINESERVER_OPTS is not valid env variable for TimelineReaderServer
[
https://issues.apache.org/jira/browse/YARN-7714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318319#comment-16318319
]
Varun Saxena commented on YARN-7714:
Yes, this environment variable is not used by ATSv2.
> YARN_TIMELINESERVER_OPTS is not valid env variable for TimelineReaderServer
> ---
>
> Key: YARN-7714
> URL: https://issues.apache.org/jira/browse/YARN-7714
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelinereader
>Affects Versions: 3.0.0
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>
> From hadoop-yarn-project/hadoop-yarn/conf/yarn-env.sh,
> {code}
> # Specify the max heapsize for the timelineserver. If no units are
> # given, it will be assumed to be in MB.
> # This value will be overridden by an Xmx setting specified in either
> # HADOOP_OPTS and/or YARN_TIMELINESERVER_OPTS.
> # Default is the same as HADOOP_HEAPSIZE_MAX.
> #export YARN_TIMELINE_HEAPSIZE=
> # Specify the JVM options to be used when starting the TimeLineServer.
> # These options will be appended to the options specified as HADOOP_OPTS
> # and therefore may override any similar flags set in HADOOP_OPTS
> #
> # See ResourceManager for some examples
> #
> #export YARN_TIMELINESERVER_OPTS=
> {code}
> However, YARN_TIMELINESERVER_OPTS does not work. The correct one to set is
> YARN_TIMELINEREADER_OPTS instead.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7662) [Atsv2] Define new set of configurations for reader and collectors to bind.
[ https://issues.apache.org/jira/browse/YARN-7662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297130#comment-16297130 ] Varun Saxena commented on YARN-7662: Minor changes made in description of reader bind host configuration in yarn-default.xml before committing. Also removed a class member variable webAppURLWithoutScheme from NodeTimelineCollectorManager as it was used inside only one method. So converted it into a local variable within the method. > [Atsv2] Define new set of configurations for reader and collectors to bind. > --- > > Key: YARN-7662 > URL: https://issues.apache.org/jira/browse/YARN-7662 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-7662.01.patch, YARN-7662.01.patch, > YARN-7662.02.patch, YARN-7662.03.patch, YARN-7662.04.patch, YARN-7662.05.patch > > > While starting Timeline Reader in secure mode, login happens using > timeline.service.address even though timeline.service.bindhost is configured > with 0.0.0.0. This causes exact principal name that matches address name to > be present in keytabs. > It is always better to login using getLocalHost that gives machine hostname > which is configured in /etc/hosts unlike NodeManager does in serviceStart. > And timeline.service.address is not required in non-secure mode, so its > better to keep consistent in secure and non-secure mode -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7662) [Atsv2] Define new set of configurations for reader and collectors to bind.
[ https://issues.apache.org/jira/browse/YARN-7662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297132#comment-16297132 ] Varun Saxena commented on YARN-7662: Committed to trunk, branch-3.0 and branch-2. Does this need to go into branch-2.9? > [Atsv2] Define new set of configurations for reader and collectors to bind. > --- > > Key: YARN-7662 > URL: https://issues.apache.org/jira/browse/YARN-7662 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-7662.01.patch, YARN-7662.01.patch, > YARN-7662.02.patch, YARN-7662.03.patch, YARN-7662.04.patch, YARN-7662.05.patch > > > While starting Timeline Reader in secure mode, login happens using > timeline.service.address even though timeline.service.bindhost is configured > with 0.0.0.0. This causes exact principal name that matches address name to > be present in keytabs. > It is always better to login using getLocalHost that gives machine hostname > which is configured in /etc/hosts unlike NodeManager does in serviceStart. > And timeline.service.address is not required in non-secure mode, so its > better to keep consistent in secure and non-secure mode -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7662) [Atsv2] Define new set of configurations for reader and collectors to bind.
[ https://issues.apache.org/jira/browse/YARN-7662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297008#comment-16297008 ] Varun Saxena commented on YARN-7662: Will commit it shortly > [Atsv2] Define new set of configurations for reader and collectors to bind. > --- > > Key: YARN-7662 > URL: https://issues.apache.org/jira/browse/YARN-7662 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-7662.01.patch, YARN-7662.01.patch, > YARN-7662.02.patch, YARN-7662.03.patch, YARN-7662.04.patch, YARN-7662.05.patch > > > While starting Timeline Reader in secure mode, login happens using > timeline.service.address even though timeline.service.bindhost is configured > with 0.0.0.0. This causes exact principal name that matches address name to > be present in keytabs. > It is always better to login using getLocalHost that gives machine hostname > which is configured in /etc/hosts unlike NodeManager does in serviceStart. > And timeline.service.address is not required in non-secure mode, so its > better to keep consistent in secure and non-secure mode -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7662) [Atsv2] Define new set of configurations for reader and collectors to bind.
[ https://issues.apache.org/jira/browse/YARN-7662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295818#comment-16295818 ] Varun Saxena commented on YARN-7662: Latest patch looks good. Will wait for a few hours before committing to give others a chance to review. By the way in yarn-default.xml, description of yarn reader bind host config can say " The actual address the reader will bind to. " instead of saying "The actual address the server will bind to. ". No need to give a patch for it though. Can change this while committing. > [Atsv2] Define new set of configurations for reader and collectors to bind. > --- > > Key: YARN-7662 > URL: https://issues.apache.org/jira/browse/YARN-7662 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-7662.01.patch, YARN-7662.01.patch, > YARN-7662.02.patch, YARN-7662.03.patch, YARN-7662.04.patch, YARN-7662.05.patch > > > While starting Timeline Reader in secure mode, login happens using > timeline.service.address even though timeline.service.bindhost is configured > with 0.0.0.0. This causes exact principal name that matches address name to > be present in keytabs. > It is always better to login using getLocalHost that gives machine hostname > which is configured in /etc/hosts unlike NodeManager does in serviceStart. > And timeline.service.address is not required in non-secure mode, so its > better to keep consistent in secure and non-secure mode -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7662) [Atsv2] Define new set of configurations for reader and collectors to bind.
[ https://issues.apache.org/jira/browse/YARN-7662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295155#comment-16295155 ] Varun Saxena commented on YARN-7662: [~rohithsharma], thanks for the latest patch. As we have already released ATSv2 and the addresses depend on yarn.timeline-service.bind-host, to maintain backward compatibility, we should fall back to it if yarn.timeline-service.reader/collector.bind-host is not set > [Atsv2] Define new set of configurations for reader and collectors to bind. > --- > > Key: YARN-7662 > URL: https://issues.apache.org/jira/browse/YARN-7662 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-7662.01.patch, YARN-7662.01.patch, > YARN-7662.02.patch, YARN-7662.03.patch, YARN-7662.04.patch > > > While starting Timeline Reader in secure mode, login happens using > timeline.service.address even though timeline.service.bindhost is configured > with 0.0.0.0. This causes exact principal name that matches address name to > be present in keytabs. > It is always better to login using getLocalHost that gives machine hostname > which is configured in /etc/hosts unlike NodeManager does in serviceStart. > And timeline.service.address is not required in non-secure mode, so its > better to keep consistent in secure and non-secure mode -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7662) [Atsv2] Remove timeline.service.address dependency for starting Timeline Reader in secure mode
[ https://issues.apache.org/jira/browse/YARN-7662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16293744#comment-16293744 ] Varun Saxena commented on YARN-7662: Have we looked at YARN-1590? > [Atsv2] Remove timeline.service.address dependency for starting Timeline > Reader in secure mode > --- > > Key: YARN-7662 > URL: https://issues.apache.org/jira/browse/YARN-7662 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-7662.01.patch, YARN-7662.01.patch > > > While starting Timeline Reader in secure mode, login happens using > timeline.service.address even though timeline.service.bindhost is configured > with 0.0.0.0. This causes exact principal name that matches address name to > be present in keytabs. > It is always better to login using getLocalHost that gives machine hostname > which is configured in /etc/hosts unlike NodeManager does in serviceStart. > And timeline.service.address is not required in non-secure mode, so its > better to keep consistent in secure and non-secure mode -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7190) Ensure only NM classpath in 2.x gets TSv2 related hbase jars, not the user classpath
[
https://issues.apache.org/jira/browse/YARN-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16291537#comment-16291537
]
Varun Saxena commented on YARN-7190:
[~rohithsharma],
bq. I build and compared with branch-2, I see that jsr311-api-1.1.1 and
netty-all-4.0.23.Final jars are missing in trunk. Is it expected or missed?
In trunk, dev-support/bin/dist-layout-stitching script does not copy jars which
already exist somewhere inside share folder (say, inside hdfs, common, etc.).
As netty-all and jsr311-api jars exist in share/hadoop/hdfs/lib folder, they
are not copied to share/hadoop/yarn/timelineservice/lib. As
share/hadoop/hdfs/lib is part of YARN classpath, this should be fine.
The said behavior is not due to this patch. If we have to override this
behavior of dist-layout-stitching and decide on this behavior based on the
component in which duplicate jar was found, we may have to look at other jars
too and that wont be in the scope of this JIRA.
By the way, are you +1 on the patch otherwise? As the original patch has been
given by me, I would suggest either you or Vrushali review and commit it.
> Ensure only NM classpath in 2.x gets TSv2 related hbase jars, not the user
> classpath
>
>
> Key: YARN-7190
> URL: https://issues.apache.org/jira/browse/YARN-7190
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineclient, timelinereader, timelineserver
>Reporter: Vrushali C
>Assignee: Varun Saxena
> Fix For: YARN-5355_branch2, 2.9.1
>
> Attachments: YARN-7190-YARN-5355_branch2.01.patch,
> YARN-7190-YARN-5355_branch2.02.patch, YARN-7190-YARN-5355_branch2.03.patch,
> YARN-7190.01.patch, YARN-7190.02.patch
>
>
> [~jlowe] had a good observation about the user classpath getting extra jars
> in hadoop 2.x brought in with TSv2. If users start picking up Hadoop 2,x's
> version of HBase jars instead of the ones they shipped with their job, it
> could be a problem.
> So when TSv2 is to be used in 2,x, the hbase related jars should come into
> only the NM classpath not the user classpath.
> Here is a list of some jars
> {code}
> commons-csv-1.0.jar
> commons-el-1.0.jar
> commons-httpclient-3.1.jar
> disruptor-3.3.0.jar
> findbugs-annotations-1.3.9-1.jar
> hbase-annotations-1.2.6.jar
> hbase-client-1.2.6.jar
> hbase-common-1.2.6.jar
> hbase-hadoop2-compat-1.2.6.jar
> hbase-hadoop-compat-1.2.6.jar
> hbase-prefix-tree-1.2.6.jar
> hbase-procedure-1.2.6.jar
> hbase-protocol-1.2.6.jar
> hbase-server-1.2.6.jar
> htrace-core-3.1.0-incubating.jar
> jamon-runtime-2.4.1.jar
> jasper-compiler-5.5.23.jar
> jasper-runtime-5.5.23.jar
> jcodings-1.0.8.jar
> joni-2.1.2.jar
> jsp-2.1-6.1.14.jar
> jsp-api-2.1-6.1.14.jar
> jsr311-api-1.1.1.jar
> metrics-core-2.2.0.jar
> servlet-api-2.5-6.1.14.jar
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7190) Ensure only NM classpath in 2.x gets TSv2 related hbase jars, not the user classpath
[
https://issues.apache.org/jira/browse/YARN-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16291050#comment-16291050
]
Varun Saxena commented on YARN-7190:
[~rohithsharma], thanks for rebasing the patch. Sorry I had missed the comment.
Let me see whether netty jar is required or not
> Ensure only NM classpath in 2.x gets TSv2 related hbase jars, not the user
> classpath
>
>
> Key: YARN-7190
> URL: https://issues.apache.org/jira/browse/YARN-7190
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineclient, timelinereader, timelineserver
>Reporter: Vrushali C
>Assignee: Varun Saxena
> Fix For: YARN-5355_branch2, 2.9.1
>
> Attachments: YARN-7190-YARN-5355_branch2.01.patch,
> YARN-7190-YARN-5355_branch2.02.patch, YARN-7190-YARN-5355_branch2.03.patch,
> YARN-7190.01.patch, YARN-7190.02.patch
>
>
> [~jlowe] had a good observation about the user classpath getting extra jars
> in hadoop 2.x brought in with TSv2. If users start picking up Hadoop 2,x's
> version of HBase jars instead of the ones they shipped with their job, it
> could be a problem.
> So when TSv2 is to be used in 2,x, the hbase related jars should come into
> only the NM classpath not the user classpath.
> Here is a list of some jars
> {code}
> commons-csv-1.0.jar
> commons-el-1.0.jar
> commons-httpclient-3.1.jar
> disruptor-3.3.0.jar
> findbugs-annotations-1.3.9-1.jar
> hbase-annotations-1.2.6.jar
> hbase-client-1.2.6.jar
> hbase-common-1.2.6.jar
> hbase-hadoop2-compat-1.2.6.jar
> hbase-hadoop-compat-1.2.6.jar
> hbase-prefix-tree-1.2.6.jar
> hbase-procedure-1.2.6.jar
> hbase-protocol-1.2.6.jar
> hbase-server-1.2.6.jar
> htrace-core-3.1.0-incubating.jar
> jamon-runtime-2.4.1.jar
> jasper-compiler-5.5.23.jar
> jasper-runtime-5.5.23.jar
> jcodings-1.0.8.jar
> joni-2.1.2.jar
> jsp-2.1-6.1.14.jar
> jsp-api-2.1-6.1.14.jar
> jsr311-api-1.1.1.jar
> metrics-core-2.2.0.jar
> servlet-api-2.5-6.1.14.jar
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7476) Fix miscellaneous issues in ATSv2 after merge to branch-2
[ https://issues.apache.org/jira/browse/YARN-7476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16248814#comment-16248814 ] Varun Saxena commented on YARN-7476: Test failures are unrelated to the patch. However, the test failure can be simulated and happens only on branch-2. It passes on trunk. > Fix miscellaneous issues in ATSv2 after merge to branch-2 > - > > Key: YARN-7476 > URL: https://issues.apache.org/jira/browse/YARN-7476 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: YARN-7476-branch-2.01.patch > > > a) We are still using Resource#getMemory in > NMTimelinePublisher#publishContainerCreatedEvent. This has been deprecated > since YARN-4844. Better to use getMemorySize instead. > b) Post YARN-5865, application priority should be fetched from RMAppImpl > instead of app submission context. But we are still fetching it from > submission context while publishing entities to timeline service. This would > mean that if priority is updated, it will not be published to timeline > service. > c) The order of app_collectors in NodeHeartbeatResponseProto is different > from trunk. Better to make it consistent. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7476) Fix miscellaneous issues in ATSv2 after merge to branch-2
[ https://issues.apache.org/jira/browse/YARN-7476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-7476: --- Attachment: YARN-7476-branch-2.01.patch > Fix miscellaneous issues in ATSv2 after merge to branch-2 > - > > Key: YARN-7476 > URL: https://issues.apache.org/jira/browse/YARN-7476 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: YARN-7476-branch-2.01.patch > > > a) We are still using Resource#getMemory in > NMTimelinePublisher#publishContainerCreatedEvent. This has been deprecated > since YARN-4844. Better to use getMemorySize instead. > b) Post YARN-5865, application priority should be fetched from RMAppImpl > instead of app submission context. But we are still fetching it from > submission context while publishing entities to timeline service. This would > mean that if priority is updated, it will not be published to timeline > service. > c) The order of app_collectors in NodeHeartbeatResponseProto is different > from trunk. Better to make it consistent. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7476) Fix miscellaneous issues in ATSv2 after merge to branch-2
Varun Saxena created YARN-7476: -- Summary: Fix miscellaneous issues in ATSv2 after merge to branch-2 Key: YARN-7476 URL: https://issues.apache.org/jira/browse/YARN-7476 Project: Hadoop YARN Issue Type: Bug Reporter: Varun Saxena Assignee: Varun Saxena a) We are still using Resource#getMemory in NMTimelinePublisher#publishContainerCreatedEvent. This has been deprecated since YARN-4844. Better to use getMemorySize instead. b) Post YARN-5865, application priority should be fetched from RMAppImpl instead of app submission context. But we are still fetching it from submission context while publishing entities to timeline service. This would mean that if priority is updated, it will not be published to timeline service. c) The order of app_collectors in NodeHeartbeatResponseProto is different from trunk. Better to make it consistent. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7272) Enable timeline collector fault tolerance
[ https://issues.apache.org/jira/browse/YARN-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240754#comment-16240754 ] Varun Saxena edited comment on YARN-7272 at 11/6/17 7:34 PM: - Sorry for coming in a little late on this discussion, although we did discuss it during the call. The primary objective of fault tolerance is to ensure that the entities which are guaranteed to be written by timeline service v2 are not lost. But writing every entity to some sort of WAL implementation would be expensive. Now, we have 2 kinds of entity writes, sync and async. Sync entities are guaranteed to be written to the backend via collector and an exception, even for server-side failures, is returned i.e. we indicate to the client that an entity could not be written all the way to the backend so that it can retry or take some other suitable action. Async entities, as the name suggests are written asynchronously. They are not guaranteed to be written to the backend, by design. We initially cache them at the client side for some time or till a sync entity arrives, combine them and then send them to collector. Moreover, if any exception occurs in writing to the backend, the result is not propagated back to the client. We only throw exceptions for client-side failures. Async entities are later cached in HBase writer implementation too, inside collector, before being flushed to HBase. Sync writes hence should be used for publishing important events, while async writes should be used for not so important events, losing which should not be a big deal in case of a failure. For instance, publishing metric values every N seconds can be an asynchronous write, unless the metric is very important, say, used for billing. Keeping this in mind, a client can potentially do synchronous writes if it cares about durability of entity data. Furthermore, asynchronous writes can have other points of failure too. For instance, the collector can crash while writing the async entity to WAL. In this case, we currently do not propagate this error to timeline client i.e. client would not know which entity writes have failed. Another possible case to handle is the case where storage is down i.e. instead of waiting for sync entity call to wait, it can be potentially committed to WAL till backend is unavailable. We can potentially explore this option. Say, in cases where HBase cluster runs separately from the cluster where ATS is running. For HBase, would HBaseAdmin#checkHBaseAvailable be sufficient to check if HBase storage is down? Thoughts? was (Author: varun_saxena): Sorry for coming in a little late on this discussion, although we did discuss it during the call. The primary objective of fault tolerance is to ensure that the entities which are guaranteed to be written by timeline service v2 are not lost. But writing every entity to some sort of WAL implementation would be expensive. Now, we have 2 kinds of entity writes, sync and async. Sync entities are guaranteed to be written to the backend via collector or an exception, even for server-side failures, is returned i.e. we indicate to the client that an entity could not be written all the way to the backend so that it can retry or take some other suitable action. Async entities, as the name suggests are written asynchronously. They are not guaranteed to be written to the backend, by design. We initially cache them at the client side for some time or till a sync entity arrives, combine them and then send them to collector. Moreover, if any exception occurs in writing to the backend, the result is not propagated back to the client. We only throw exceptions for client-side failures. Async entities are later cached in HBase writer implementation too, inside collector, before being flushed to HBase. Sync writes hence should be used for publishing important events, while async writes should be used for not so important events, losing which should not be a big deal in case of a failure. For instance, publishing metric values every N seconds can be an asynchronous write, unless the metric is very important, say, used for billing. Keeping this in mind, a client can potentially do synchronous writes if it cares about durability of entity data. Furthermore, asynchronous writes can have other points of failure too. For instance, the collector can crash while writing the async entity to WAL. In this case, we currently do not propagate this error to timeline client i.e. client would not know which entity writes have failed. Another possible case to handle is the case where storage is down i.e. instead of waiting for sync entity call to wait, it can be potentially committed to WAL till backend is unavailable. We can potentially explore this option. Say, in cases where HBase cluster runs separately from the cluster where ATS is running. For HBase, would
[jira] [Commented] (YARN-7272) Enable timeline collector fault tolerance
[ https://issues.apache.org/jira/browse/YARN-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240754#comment-16240754 ] Varun Saxena commented on YARN-7272: Sorry for coming in a little late on this discussion, although we did discuss it during the call. The primary objective of fault tolerance is to ensure that the entities which are guaranteed to be written by timeline service v2 are not lost. But writing every entity to some sort of WAL implementation would be expensive. Now, we have 2 kinds of entity writes, sync and async. Sync entities are guaranteed to be written to the backend via collector or an exception, even for server-side failures, is returned i.e. we indicate to the client that an entity could not be written all the way to the backend so that it can retry or take some other suitable action. Async entities, as the name suggests are written asynchronously. They are not guaranteed to be written to the backend, by design. We initially cache them at the client side for some time or till a sync entity arrives, combine them and then send them to collector. Moreover, if any exception occurs in writing to the backend, the result is not propagated back to the client. We only throw exceptions for client-side failures. Async entities are later cached in HBase writer implementation too, inside collector, before being flushed to HBase. Sync writes hence should be used for publishing important events, while async writes should be used for not so important events, losing which should not be a big deal in case of a failure. For instance, publishing metric values every N seconds can be an asynchronous write, unless the metric is very important, say, used for billing. Keeping this in mind, a client can potentially do synchronous writes if it cares about durability of entity data. Furthermore, asynchronous writes can have other points of failure too. For instance, the collector can crash while writing the async entity to WAL. In this case, we currently do not propagate this error to timeline client i.e. client would not know which entity writes have failed. Another possible case to handle is the case where storage is down i.e. instead of waiting for sync entity call to wait, it can be potentially committed to WAL till backend is unavailable. We can potentially explore this option. Say, in cases where HBase cluster runs separately from the cluster where ATS is running. For HBase, would HBaseAdmin#checkHBaseAvailable be sufficient to check if HBase storage is down? Thoughts? > Enable timeline collector fault tolerance > - > > Key: YARN-7272 > URL: https://issues.apache.org/jira/browse/YARN-7272 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineclient, timelinereader, timelineserver >Reporter: Vrushali C >Assignee: Rohith Sharma K S > Attachments: YARN-7272-wip.patch > > > If a NM goes down and along with it the timeline collector aux service for a > running yarn app, we would like that yarn app to re-establish connection with > a new timeline collector. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7398) LICENSE.txt is broken in branch-2 by YARN-4849 merge
[ https://issues.apache.org/jira/browse/YARN-7398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-7398: --- Attachment: YARN-7398.branch-2.03.patch Mentioned that D3 license would be used for hadoop-yarn-ui as well. Currently, D3 license though is placed inside SLS folder which is a little weird. Maybe we can move it to another folder and this can be handled under a new JIRA for trunk. > LICENSE.txt is broken in branch-2 by YARN-4849 merge > > > Key: YARN-7398 > URL: https://issues.apache.org/jira/browse/YARN-7398 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.9.0 >Reporter: Subru Krishnan >Assignee: Varun Saxena >Priority: Blocker > Attachments: YARN-7398.branch-2.01.patch, > YARN-7398.branch-2.02.patch, YARN-7398.branch-2.03.patch > > > YARN-4849 (commit sha id 56654d8820f345fdefd6a3f81836125aa67adbae) seems to > have been based out of stale version of LICENSE.txt, for e.g: HSQLDB, gtest > etc, so I have reverted it. > [~leftnoteasy]/[~sunilg], can you guys take a look and fix the UI v2 licenses > asap. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7398) LICENSE.txt is broken in branch-2 by YARN-4849 merge
[ https://issues.apache.org/jira/browse/YARN-7398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16226502#comment-16226502 ] Varun Saxena commented on YARN-7398: Thanks Sunil for the latest patch including missing dependencies. +1 LGTM. > LICENSE.txt is broken in branch-2 by YARN-4849 merge > > > Key: YARN-7398 > URL: https://issues.apache.org/jira/browse/YARN-7398 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.9.0 >Reporter: Subru Krishnan >Assignee: Varun Saxena >Priority: Blocker > Attachments: YARN-7398.branch-2.01.patch, YARN-7398.branch-2.02.patch > > > YARN-4849 (commit sha id 56654d8820f345fdefd6a3f81836125aa67adbae) seems to > have been based out of stale version of LICENSE.txt, for e.g: HSQLDB, gtest > etc, so I have reverted it. > [~leftnoteasy]/[~sunilg], can you guys take a look and fix the UI v2 licenses > asap. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7378) Documentation changes post branch-2 merge
[ https://issues.apache.org/jira/browse/YARN-7378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16225648#comment-16225648 ] Varun Saxena commented on YARN-7378: +1 LGTM > Documentation changes post branch-2 merge > - > > Key: YARN-7378 > URL: https://issues.apache.org/jira/browse/YARN-7378 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineclient, timelinereader, timelineserver >Reporter: Varun Saxena >Assignee: Vrushali C > Attachments: YARN-7378-branch-2.0001.patch, > YARN-7378-branch-2.0002.patch, YARN-7378-branch-2.0003.patch, > YARN-7378-branch-2.0004.patch, YARN-7378-branch-2.0005.patch, schema creation > documentation.png > > > Need to update the documentation for the schema creator command. It should > include the timeline-service-hbase jar as well as hbase-server jar in > classpath when the command is to be run. Due to YARN-7190 classpath changes. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7380) Fix findbugs warning in timeline service on branch-2
[ https://issues.apache.org/jira/browse/YARN-7380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-7380: --- Fix Version/s: 2.9.0 > Fix findbugs warning in timeline service on branch-2 > > > Key: YARN-7380 > URL: https://issues.apache.org/jira/browse/YARN-7380 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineclient, timelinereader, timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Fix For: 2.9.0 > > Attachments: YARN-7380-branch-2.0001.patch > > > Some findbugs warnings have been noticed in the branch-2 nightly builds. I > haven't investigated them yet but filing to confirm/fix if possible. > I recollect some known findbugs issues with the webservices function calls > with number of parameters which we could not fix. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7380) Fix findbugs warning in timeline service on branch-2
[ https://issues.apache.org/jira/browse/YARN-7380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-7380: --- Summary: Fix findbugs warning in timeline service on branch-2 (was: Check findbugs in timeline service branch-2) > Fix findbugs warning in timeline service on branch-2 > > > Key: YARN-7380 > URL: https://issues.apache.org/jira/browse/YARN-7380 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineclient, timelinereader, timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Attachments: YARN-7380-branch-2.0001.patch > > > Some findbugs warnings have been noticed in the branch-2 nightly builds. I > haven't investigated them yet but filing to confirm/fix if possible. > I recollect some known findbugs issues with the webservices function calls > with number of parameters which we could not fix. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7380) Check findbugs in timeline service branch-2
[ https://issues.apache.org/jira/browse/YARN-7380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223930#comment-16223930 ] Varun Saxena commented on YARN-7380: Thanks [~vrushalic] for the fix. Changes LGTM. Will commit it shortly. ASF License warning is unrelated. cc [~subru], for your notice. > Check findbugs in timeline service branch-2 > --- > > Key: YARN-7380 > URL: https://issues.apache.org/jira/browse/YARN-7380 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineclient, timelinereader, timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Attachments: YARN-7380-branch-2.0001.patch > > > Some findbugs warnings have been noticed in the branch-2 nightly builds. I > haven't investigated them yet but filing to confirm/fix if possible. > I recollect some known findbugs issues with the webservices function calls > with number of parameters which we could not fix. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7378) Documentation changes post branch-2 merge
[
https://issues.apache.org/jira/browse/YARN-7378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223924#comment-16223924
]
Varun Saxena edited comment on YARN-7378 at 10/29/17 9:48 AM:
--
[~vrushalic], one more comment.
In the documentation, we are referring to version 3.0.0-alpha1. Atleast for the
jars, this is incorrect.
We can change the file extension from {{md}} to {{md.vm}} and refer to
{{$\{project.version\}}} instead of specific version inside the documentation.
Ideally this change should be made for trunk as well.
was (Author: varun_saxena):
[~vrushalic], one more comment.
In the documentation, we are referring to version 3.0.0-alpha1. Atleast for the
jars, this is incorrect.
We can rename the change the file extension from {{md}} to {{md.vm}} and refer
to {{$\{project.version\}}} instead of specific version inside the
documentation. Ideally this change should be made for trunk as well.
> Documentation changes post branch-2 merge
> -
>
> Key: YARN-7378
> URL: https://issues.apache.org/jira/browse/YARN-7378
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineclient, timelinereader, timelineserver
>Reporter: Varun Saxena
>Assignee: Vrushali C
> Attachments: YARN-7378-branch-2.0001.patch,
> YARN-7378-branch-2.0002.patch, YARN-7378-branch-2.0003.patch, schema creation
> documentation.png
>
>
> Need to update the documentation for the schema creator command. It should
> include the timeline-service-hbase jar as well as hbase-server jar in
> classpath when the command is to be run. Due to YARN-7190 classpath changes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7378) Documentation changes post branch-2 merge
[
https://issues.apache.org/jira/browse/YARN-7378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223924#comment-16223924
]
Varun Saxena commented on YARN-7378:
[~vrushalic], one more comment.
In the documentation, we are referring to version 3.0.0-alpha1. Atleast for the
jars, this is incorrect.
We can rename the change the file extension from {{md}} to {{md.vm}} and refer
to {{$\{project.version\}}} instead of specific version inside the
documentation. Ideally this change should be made for trunk as well.
> Documentation changes post branch-2 merge
> -
>
> Key: YARN-7378
> URL: https://issues.apache.org/jira/browse/YARN-7378
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineclient, timelinereader, timelineserver
>Reporter: Varun Saxena
>Assignee: Vrushali C
> Attachments: YARN-7378-branch-2.0001.patch,
> YARN-7378-branch-2.0002.patch, YARN-7378-branch-2.0003.patch, schema creation
> documentation.png
>
>
> Need to update the documentation for the schema creator command. It should
> include the timeline-service-hbase jar as well as hbase-server jar in
> classpath when the command is to be run. Due to YARN-7190 classpath changes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7378) Documentation changes post branch-2 merge
[
https://issues.apache.org/jira/browse/YARN-7378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222681#comment-16222681
]
Varun Saxena commented on YARN-7378:
[~vrushalic],
Following line
{noformat}
export
HBASE_CLASSPATH=$HBASE_CLASSPATH:/hadoop-yarn-server-timelineservice-hbase.jar
{noformat}
should be
{noformat}
export
HBASE_CLASSPATH=$HBASE_CLASSPATH:/hadoop-yarn-server-timelineservice.jar
{noformat}
> Documentation changes post branch-2 merge
> -
>
> Key: YARN-7378
> URL: https://issues.apache.org/jira/browse/YARN-7378
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineclient, timelinereader, timelineserver
>Reporter: Varun Saxena
>Assignee: Vrushali C
> Attachments: YARN-7378-branch-2.0001.patch,
> YARN-7378-branch-2.0002.patch, schema creation documentation.png
>
>
> Need to update the documentation for the schema creator command. It should
> include the timeline-service-hbase jar as well as hbase-server jar in
> classpath when the command is to be run. Due to YARN-7190 classpath changes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7398) LICENSE.txt is broken in branch-2 by YARN-4849 merge
[
https://issues.apache.org/jira/browse/YARN-7398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222626#comment-16222626
]
Varun Saxena commented on YARN-7398:
The said section was removed in HADOOP-13780 so I did not keep it here as well.
What we have is a section like this in the LICENSE file, both in trunk and
branch-2:
{noformat}
D3 is available under a 3-clause BSD license. For details, see:
hadoop-tools/hadoop-sls/src/main/html/js/thirdparty/d3-LICENSE
{noformat}
However, as I mentioned in one of the comments above, we are not drawing a
direct linkage with web UI here even though we say D3 is under 3-clause BSD
license. The d3-LICENSE is placed under SLS folder as well. If this is an issue
(seems to be), it needs to be fixed in trunk as well, so possibly in another
JIRA.
> LICENSE.txt is broken in branch-2 by YARN-4849 merge
>
>
> Key: YARN-7398
> URL: https://issues.apache.org/jira/browse/YARN-7398
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 2.9.0
>Reporter: Subru Krishnan
>Assignee: Varun Saxena
>Priority: Blocker
> Attachments: YARN-7398.branch-2.01.patch
>
>
> YARN-4849 (commit sha id 56654d8820f345fdefd6a3f81836125aa67adbae) seems to
> have been based out of stale version of LICENSE.txt, for e.g: HSQLDB, gtest
> etc, so I have reverted it.
> [~leftnoteasy]/[~sunilg], can you guys take a look and fix the UI v2 licenses
> asap.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7398) LICENSE.txt is broken in branch-2 by YARN-4849 merge
[ https://issues.apache.org/jira/browse/YARN-7398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222593#comment-16222593 ] Varun Saxena commented on YARN-7398: Yes, unless there are further comments. However, I am not sure if web UI needs to explicitly mentioned for D3 as per one of my comments. > LICENSE.txt is broken in branch-2 by YARN-4849 merge > > > Key: YARN-7398 > URL: https://issues.apache.org/jira/browse/YARN-7398 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.9.0 >Reporter: Subru Krishnan >Assignee: Varun Saxena >Priority: Blocker > Attachments: YARN-7398.branch-2.01.patch > > > YARN-4849 (commit sha id 56654d8820f345fdefd6a3f81836125aa67adbae) seems to > have been based out of stale version of LICENSE.txt, for e.g: HSQLDB, gtest > etc, so I have reverted it. > [~leftnoteasy]/[~sunilg], can you guys take a look and fix the UI v2 licenses > asap. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7398) LICENSE.txt is broken in branch-2 by YARN-4849 merge
[ https://issues.apache.org/jira/browse/YARN-7398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222301#comment-16222301 ] Varun Saxena edited comment on YARN-7398 at 10/27/17 12:47 PM: --- [~subru], but the changes brought in for UI merge have already been reverted from branch-2. So, LICENSE file on branch-2 currently is effectively equivalent to what it was before UI merge, barring a couple of recent commits. The changes pointed out above were incorrectly brought in at the time of UI merge, I presume due to a conflict which may have arisen during cherry-picking. But these changes are not required and the current patch is on top of current state of branch-2 and contains only the required changes in LICENSE for Web UI merge. was (Author: varun_saxena): [~subru], but the changes brought in for UI merge have already been reverted from branch-2. So, LICENSE file on branch-2 currently is effectively equivalent to what it was before UI merge, barring a couple of recent commits. The changes pointed out above were incorrectly brought in, I presume due to a conflict which may have arisen during cherry-picking. But these changes are not required and the current patch is on top of current state of branch-2 and contains only the required changes in LICENSE for Web UI merge. > LICENSE.txt is broken in branch-2 by YARN-4849 merge > > > Key: YARN-7398 > URL: https://issues.apache.org/jira/browse/YARN-7398 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.9.0 >Reporter: Subru Krishnan >Assignee: Varun Saxena >Priority: Blocker > Attachments: YARN-7398.branch-2.01.patch > > > YARN-4849 (commit sha id 56654d8820f345fdefd6a3f81836125aa67adbae) seems to > have been based out of stale version of LICENSE.txt, for e.g: HSQLDB, gtest > etc, so I have reverted it. > [~leftnoteasy]/[~sunilg], can you guys take a look and fix the UI v2 licenses > asap. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7398) LICENSE.txt is broken in branch-2 by YARN-4849 merge
[ https://issues.apache.org/jira/browse/YARN-7398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222301#comment-16222301 ] Varun Saxena commented on YARN-7398: [~subru], but the changes brought in for UI merge have already been reverted from branch-2. So, LICENSE file on branch-2 currently is effectively equivalent to what it was before UI merge, barring a couple of recent commits. The changes pointed out above were incorrectly brought in, I presume due to a conflict which may have arisen during cherry-picking. But these changes are not required and the current patch is on top of current state of branch-2 and contains only the required changes in LICENSE for Web UI merge. > LICENSE.txt is broken in branch-2 by YARN-4849 merge > > > Key: YARN-7398 > URL: https://issues.apache.org/jira/browse/YARN-7398 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.9.0 >Reporter: Subru Krishnan >Assignee: Varun Saxena >Priority: Blocker > Attachments: YARN-7398.branch-2.01.patch > > > YARN-4849 (commit sha id 56654d8820f345fdefd6a3f81836125aa67adbae) seems to > have been based out of stale version of LICENSE.txt, for e.g: HSQLDB, gtest > etc, so I have reverted it. > [~leftnoteasy]/[~sunilg], can you guys take a look and fix the UI v2 licenses > asap. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7398) LICENSE.txt is broken in branch-2 by YARN-4849 merge
[ https://issues.apache.org/jira/browse/YARN-7398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220751#comment-16220751 ] Varun Saxena commented on YARN-7398: [~sunilg], this is based on changes made in HADOOP-13780. I have basically made LICENSE.txt consistent with trunk. We already had a host of dependencies covered under the MIT License. These web UI dependencies mentioned in LICENSE were merged together with other dependecies in HADOOP-13780. So, this patch also takes care of that. Another section related to D3 was also removed in HADOOP-13780. This is because we have already mentioned that D3 is available under a 3-clause BSD license elsewhere. But this is mentioned for d3.v3.js under sls. Maybe we should mention that this applies to web UI too and move the d3-LICENSE information from sls project to LICENSE.txt. If we do so, this should be fixed in trunk too. Thoughts? > LICENSE.txt is broken in branch-2 by YARN-4849 merge > > > Key: YARN-7398 > URL: https://issues.apache.org/jira/browse/YARN-7398 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.0 >Reporter: Subru Krishnan >Assignee: Varun Saxena >Priority: Blocker > Attachments: YARN-7398.branch-2.01.patch > > > YARN-4849 (commit sha id 56654d8820f345fdefd6a3f81836125aa67adbae) seems to > have been based out of stale version of LICENSE.txt, for e.g: HSQLDB, gtest > etc, so I have reverted it. > [~leftnoteasy]/[~sunilg], can you guys take a look and fix the UI v2 licenses > asap. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7398) LICENSE.txt is broken in branch-2 by YARN-4849 merge
[ https://issues.apache.org/jira/browse/YARN-7398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-7398: --- Attachment: YARN-7398.branch-2.01.patch [~sunilg], can you have a look? I picked up changes from YARN-4849. > LICENSE.txt is broken in branch-2 by YARN-4849 merge > > > Key: YARN-7398 > URL: https://issues.apache.org/jira/browse/YARN-7398 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.0 >Reporter: Subru Krishnan >Assignee: Varun Saxena >Priority: Blocker > Attachments: YARN-7398.branch-2.01.patch > > > YARN-4849 (commit sha id 56654d8820f345fdefd6a3f81836125aa67adbae) seems to > have been based out of stale version of LICENSE.txt, for e.g: HSQLDB, gtest > etc, so I have reverted it. > [~leftnoteasy]/[~sunilg], can you guys take a look and fix the UI v2 licenses > asap. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7190) Ensure only NM classpath in 2.x gets TSv2 related hbase jars, not the user classpath
[
https://issues.apache.org/jira/browse/YARN-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated YARN-7190:
---
Attachment: YARN-7190.01.patch
> Ensure only NM classpath in 2.x gets TSv2 related hbase jars, not the user
> classpath
>
>
> Key: YARN-7190
> URL: https://issues.apache.org/jira/browse/YARN-7190
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineclient, timelinereader, timelineserver
>Reporter: Vrushali C
>Assignee: Varun Saxena
> Fix For: 2.9.0, YARN-5355_branch2
>
> Attachments: YARN-7190-YARN-5355_branch2.01.patch,
> YARN-7190-YARN-5355_branch2.02.patch, YARN-7190-YARN-5355_branch2.03.patch,
> YARN-7190.01.patch
>
>
> [~jlowe] had a good observation about the user classpath getting extra jars
> in hadoop 2.x brought in with TSv2. If users start picking up Hadoop 2,x's
> version of HBase jars instead of the ones they shipped with their job, it
> could be a problem.
> So when TSv2 is to be used in 2,x, the hbase related jars should come into
> only the NM classpath not the user classpath.
> Here is a list of some jars
> {code}
> commons-csv-1.0.jar
> commons-el-1.0.jar
> commons-httpclient-3.1.jar
> disruptor-3.3.0.jar
> findbugs-annotations-1.3.9-1.jar
> hbase-annotations-1.2.6.jar
> hbase-client-1.2.6.jar
> hbase-common-1.2.6.jar
> hbase-hadoop2-compat-1.2.6.jar
> hbase-hadoop-compat-1.2.6.jar
> hbase-prefix-tree-1.2.6.jar
> hbase-procedure-1.2.6.jar
> hbase-protocol-1.2.6.jar
> hbase-server-1.2.6.jar
> htrace-core-3.1.0-incubating.jar
> jamon-runtime-2.4.1.jar
> jasper-compiler-5.5.23.jar
> jasper-runtime-5.5.23.jar
> jcodings-1.0.8.jar
> joni-2.1.2.jar
> jsp-2.1-6.1.14.jar
> jsp-api-2.1-6.1.14.jar
> jsr311-api-1.1.1.jar
> metrics-core-2.2.0.jar
> servlet-api-2.5-6.1.14.jar
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7190) Ensure only NM classpath in 2.x gets TSv2 related hbase jars, not the user classpath
[
https://issues.apache.org/jira/browse/YARN-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16218118#comment-16218118
]
Varun Saxena commented on YARN-7190:
bq. I am wondering why the commit was not propagated from trunk but instead
done directly in branch-2?
The patch for trunk will be somewhat different owing to the way scripts are
written in trunk.
As this JIRA was blocking ATS branch-2 merge and had to be resolved before it,
we took a call to first get the changes in YARN-5355_branch2. And focus on
merging ATS and UI ASAP as the feature freeze date was nearing and most of us
were on leave from middle of last week due to Diwali. For trunk, we thought of
raising another JIRA.
As I am now back from leave, I can land up a patch here itself in a day or
less, instead of raising a new JIRA for trunk, if that works fine.
> Ensure only NM classpath in 2.x gets TSv2 related hbase jars, not the user
> classpath
>
>
> Key: YARN-7190
> URL: https://issues.apache.org/jira/browse/YARN-7190
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineclient, timelinereader, timelineserver
>Reporter: Vrushali C
>Assignee: Varun Saxena
> Fix For: 2.9.0, YARN-5355_branch2
>
> Attachments: YARN-7190-YARN-5355_branch2.01.patch,
> YARN-7190-YARN-5355_branch2.02.patch, YARN-7190-YARN-5355_branch2.03.patch
>
>
> [~jlowe] had a good observation about the user classpath getting extra jars
> in hadoop 2.x brought in with TSv2. If users start picking up Hadoop 2,x's
> version of HBase jars instead of the ones they shipped with their job, it
> could be a problem.
> So when TSv2 is to be used in 2,x, the hbase related jars should come into
> only the NM classpath not the user classpath.
> Here is a list of some jars
> {code}
> commons-csv-1.0.jar
> commons-el-1.0.jar
> commons-httpclient-3.1.jar
> disruptor-3.3.0.jar
> findbugs-annotations-1.3.9-1.jar
> hbase-annotations-1.2.6.jar
> hbase-client-1.2.6.jar
> hbase-common-1.2.6.jar
> hbase-hadoop2-compat-1.2.6.jar
> hbase-hadoop-compat-1.2.6.jar
> hbase-prefix-tree-1.2.6.jar
> hbase-procedure-1.2.6.jar
> hbase-protocol-1.2.6.jar
> hbase-server-1.2.6.jar
> htrace-core-3.1.0-incubating.jar
> jamon-runtime-2.4.1.jar
> jasper-compiler-5.5.23.jar
> jasper-runtime-5.5.23.jar
> jcodings-1.0.8.jar
> joni-2.1.2.jar
> jsp-2.1-6.1.14.jar
> jsp-api-2.1-6.1.14.jar
> jsr311-api-1.1.1.jar
> metrics-core-2.2.0.jar
> servlet-api-2.5-6.1.14.jar
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7378) Documentation changes post branch-2 merge
Varun Saxena created YARN-7378: -- Summary: Documentation changes post branch-2 merge Key: YARN-7378 URL: https://issues.apache.org/jira/browse/YARN-7378 Project: Hadoop YARN Issue Type: Sub-task Reporter: Varun Saxena -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3993) Change to use the AM flag in ContainerContext determine AM container
[ https://issues.apache.org/jira/browse/YARN-3993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3993: --- Fix Version/s: 2.9.0 > Change to use the AM flag in ContainerContext determine AM container > > > Key: YARN-3993 > URL: https://issues.apache.org/jira/browse/YARN-3993 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Zhijie Shen >Assignee: Sunil G > Labels: newbie > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: 0001-YARN-3993-YARN-2928.patch, > YARN-3993-YARN-2928.0001.patch > > > After YARN-3116, we will have a flag in ContainerContext to determine if the > container is AM or not in aux service. We need to change accordingly to make > use of this feature instead of depending on container ID. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5102) timeline service build fails with java 8
[
https://issues.apache.org/jira/browse/YARN-5102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated YARN-5102:
---
Fix Version/s: 2.9.0
> timeline service build fails with java 8
>
>
> Key: YARN-5102
> URL: https://issues.apache.org/jira/browse/YARN-5102
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Blocker
> Labels: yarn-2928-1st-milestone
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-5102-YARN-2928.01.patch
>
>
> The build fails with java 8:
> {noformat}
> [WARNING]
> Dependency convergence error for jdk.tools:jdk.tools:1.8 paths to dependency
> are:
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
> +-org.apache.hadoop:hadoop-annotations:3.0.0-SNAPSHOT
> +-jdk.tools:jdk.tools:1.8
> and
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
> +-org.apache.hbase:hbase-common:1.0.1
> +-org.apache.hbase:hbase-annotations:1.0.1
> +-jdk.tools:jdk.tools:1.7
> [WARNING] Rule 0: org.apache.maven.plugins.enforcer.DependencyConvergence
> failed with message:
> Failed while enforcing releasability the error(s) are [
> Dependency convergence error for jdk.tools:jdk.tools:1.8 paths to dependency
> are:
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
> +-org.apache.hadoop:hadoop-annotations:3.0.0-SNAPSHOT
> +-jdk.tools:jdk.tools:1.8
> and
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
> +-org.apache.hbase:hbase-common:1.0.1
> +-org.apache.hbase:hbase-annotations:1.0.1
> +-jdk.tools:jdk.tools:1.7
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3862) Support for fetching specific configs and metrics based on prefixes
[ https://issues.apache.org/jira/browse/YARN-3862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3862: --- Fix Version/s: 2.9.0 > Support for fetching specific configs and metrics based on prefixes > --- > > Key: YARN-3862 > URL: https://issues.apache.org/jira/browse/YARN-3862 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-2928-1st-milestone > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-3862-YARN-2928.wip.01.patch, > YARN-3862-YARN-2928.wip.02.patch, YARN-3862-feature-YARN-2928.005.patch, > YARN-3862-feature-YARN-2928.04.patch, YARN-3862-feature-YARN-2928.wip.03.patch > > > Currently, we will retrieve all the contents of the field if that field is > specified in the query API. In case of configs and metrics, this can become a > lot of data even though the user doesn't need it. So we need to provide a way > to query only a set of configs or metrics. > As a comma spearated list of configs/metrics to be returned will be quite > cumbersome to specify, we have to support either of the following options : > # Prefix match > # Regex > # Group the configs/metrics and query that group. > We also need a facility to specify a metric time window to return metrics in > a that window. This may be useful in plotting graphs -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3461) Consolidate flow name/version/run defaults
[ https://issues.apache.org/jira/browse/YARN-3461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3461: --- Fix Version/s: 2.9.0 > Consolidate flow name/version/run defaults > -- > > Key: YARN-3461 > URL: https://issues.apache.org/jira/browse/YARN-3461 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Sangjin Lee > Labels: yarn-2928-1st-milestone > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-3461-YARN-2928.01.patch, > YARN-3461-YARN-2928.02.patch, YARN-3461-YARN-2928.03.patch > > > In YARN-3391, it's not resolved what should be the defaults for flow > name/version/run. Let's continue the discussion here and unblock YARN-3391 > from moving forward. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4711) NM is going down with NPE's due to single thread processing of events by Timeline client
[
https://issues.apache.org/jira/browse/YARN-4711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated YARN-4711:
---
Fix Version/s: 2.9.0
> NM is going down with NPE's due to single thread processing of events by
> Timeline client
>
>
> Key: YARN-4711
> URL: https://issues.apache.org/jira/browse/YARN-4711
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>Priority: Critical
> Labels: yarn-2928-1st-milestone
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: 4711Analysis.txt, YARN-4711-YARN-2928.v1.001.patch,
> YARN-4711-YARN-2928.v1.002.patch
>
>
> After YARN-3367, while testing the latest 2928 branch came across few NPEs
> due to which NM is shutting down.
> {code}
> 2016-02-21 23:19:54,078 FATAL org.apache.hadoop.yarn.event.AsyncDispatcher:
> Error in dispatcher thread
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher$ContainerEventHandler.handle(NMTimelinePublisher.java:306)
> at
> org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher$ContainerEventHandler.handle(NMTimelinePublisher.java:296)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:183)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:109)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher.putEntity(NMTimelinePublisher.java:213)
> at
> org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher.publishContainerFinishedEvent(NMTimelinePublisher.java:192)
> at
> org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher.access$400(NMTimelinePublisher.java:63)
> at
> org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher$ApplicationEventHandler.handle(NMTimelinePublisher.java:289)
> at
> org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher$ApplicationEventHandler.handle(NMTimelinePublisher.java:280)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:183)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:109)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> On analysis found that the there was delay in processing of events, as after
> YARN-3367 all the events were getting processed by a single thread inside the
> timeline client.
> Additionally found one scenario where there is possibility of NPE:
> * TimelineEntity.toString() when {{real}} is not null
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7038) [Atsv2 Security] CollectorNodemanagerProtocol RPC interface doesn't work when service authorization is enabled
[
https://issues.apache.org/jira/browse/YARN-7038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated YARN-7038:
---
Fix Version/s: 2.9.0
> [Atsv2 Security] CollectorNodemanagerProtocol RPC interface doesn't work when
> service authorization is enabled
> --
>
> Key: YARN-7038
> URL: https://issues.apache.org/jira/browse/YARN-7038
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: YARN-5355
>Reporter: Rohith Sharma K S
>Assignee: Varun Saxena
> Labels: yarn-5355-merge-blocker
> Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355-branch-2
>
> Attachments: YARN-7038-YARN-5355.01.patch,
> YARN-7038-YARN-5355.02.patch, YARN-7038-YARN-5355.03.patch
>
>
> Below error appears in the log when authorization is enabled.
> {noformat}
> 2017-08-17 11:16:40,664 ERROR collector.NodeTimelineCollectorManager
> (NodeTimelineCollectorManager.java:doPostPut(227)) - Failed to communicate
> with NM Collector Service for application_1502964541476_0001
> 2017-08-17 11:16:40,665 WARN containermanager.AuxServices
> (AuxServices.java:logWarningWhenAuxServiceThrowExceptions(283)) - The
> auxService name is timeline_collector and it got an error at event:
> CONTAINER_INIT
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> org.apache.hadoop.security.authorize.AuthorizationException: Protocol
> interface org.apache.hadoop.yarn.server.api.CollectorNodemanagerProtocolPB is
> not known.
> at
> org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollectorManager.putIfAbsent(TimelineCollectorManager.java:146)
> at
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5109) timestamps are stored unencoded causing parse errors
[
https://issues.apache.org/jira/browse/YARN-5109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated YARN-5109:
---
Fix Version/s: 2.9.0
> timestamps are stored unencoded causing parse errors
>
>
> Key: YARN-5109
> URL: https://issues.apache.org/jira/browse/YARN-5109
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
>Priority: Blocker
> Labels: yarn-2928-1st-milestone
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-5109-YARN-2928.003.patch,
> YARN-5109-YARN-2928.01.patch, YARN-5109-YARN-2928.02.patch,
> YARN-5109-YARN-2928.03.patch, YARN-5109-YARN-2928.04.patch,
> YARN-5109-YARN-2928.05.patch, YARN-5109-YARN-2928.06.patch,
> YARN-5109-YARN-2928.07.patch, YARN-5109-YARN-2928.08.patch
>
>
> When we store timestamps (for example as part of the row key or part of the
> column name for an event), the bytes are used as is without any encoding. If
> the byte value happens to contain a separator character we use (e.g. "!" or
> "="), it causes a parse failure when we read it.
> I came across this while looking into this error in the timeline reader:
> {noformat}
> 2016-05-17 21:28:38,643 WARN
> org.apache.hadoop.yarn.server.timelineservice.storage.common.TimelineStorageUtils:
> incorrectly formatted column name: it will be discarded
> {noformat}
> I traced the data that was causing this, and the column name (for the event)
> was the following:
> {noformat}
> i:e!YARN_RM_CONTAINER_CREATED=\x7F\xFF\xFE\xABDY=\x99=YARN_CONTAINER_ALLOCATED_HOST
> {noformat}
> Note that the column name is supposed to be of the format (event
> id)=(timestamp)=(event info key). However, observe the timestamp portion:
> {noformat}
> \x7F\xFF\xFE\xABDY=\x99
> {noformat}
> The presence of the separator ("=") causes the parse error.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5142) fix findbugs warnings/errors for hadoop-yarn-server-timelineservice-hbase-tests
[ https://issues.apache.org/jira/browse/YARN-5142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-5142: --- Fix Version/s: 2.9.0 > fix findbugs warnings/errors for > hadoop-yarn-server-timelineservice-hbase-tests > --- > > Key: YARN-5142 > URL: https://issues.apache.org/jira/browse/YARN-5142 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Labels: yarn-2928-1st-milestone > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-5142-YARN-2928.01.patch > > > Fix the errors/warnings reported for > hadoop-yarn-server-timelineservice-hbase-tests once YARN-5138 is in -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6146) Add Builder methods for TimelineEntityFilters
[ https://issues.apache.org/jira/browse/YARN-6146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6146: --- Fix Version/s: 2.9.0 > Add Builder methods for TimelineEntityFilters > - > > Key: YARN-6146 > URL: https://issues.apache.org/jira/browse/YARN-6146 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Rohith Sharma K S >Assignee: Haibo Chen > Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355-branch-2 > > Attachments: YARN-6146-YARN-5355.01.patch, > YARN-6146-YARN-5355.02.patch, YARN-6146-YARN-5355.03.patch, > YARN-6146.01.patch, YARN-6146.02.patch, YARN-6146.03.patch > > > The timeline filters are evolving and can be add more and more filters. It is > better to start using Builder methods rather than changing constructor every > time for adding new filters. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6555) Store application flow context in NM state store for work-preserving restart
[
https://issues.apache.org/jira/browse/YARN-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated YARN-6555:
---
Fix Version/s: 2.9.0
> Store application flow context in NM state store for work-preserving restart
>
>
> Key: YARN-6555
> URL: https://issues.apache.org/jira/browse/YARN-6555
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-5355, YARN-5355-branch-2, 3.0.0-alpha4
>Reporter: Vrushali C
>Assignee: Rohith Sharma K S
> Labels: yarn-5355-merge-blocker
> Fix For: 2.9.0, YARN-5355, YARN-5355-branch-2, 3.0.0-alpha4
>
> Attachments: YARN-6555.001.patch, YARN-6555.002.patch,
> YARN-6555.003.patch
>
>
> If timeline service v2 is enabled and NM is restarted with recovery enabled,
> then NM fails to start and throws an error as "flow context can't be null".
> This is happening because the flow context did not exist before but now that
> timeline service v2 is enabled, ApplicationImpl expects it to exist.
> This would also happen even if flow context existed before but since we are
> not persisting it / reading it during
> ContainerManagerImpl#recoverApplication, it does not get passed in to
> ApplicationImpl.
> full stack trace
> {code}
> 2017-05-03 21:51:52,178 FATAL
> org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting
> NodeManager
> java.lang.IllegalArgumentException: flow context cannot be null
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.(ApplicationImpl.java:104)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.(ApplicationImpl.java:90)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recoverApplication(ContainerManagerImpl.java:318)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recover(ContainerManagerImpl.java:280)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:267)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:276)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:588)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:649)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3816) [Aggregation] App-level aggregation and accumulation for YARN system metrics
[ https://issues.apache.org/jira/browse/YARN-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3816: --- Fix Version/s: 2.9.0 > [Aggregation] App-level aggregation and accumulation for YARN system metrics > > > Key: YARN-3816 > URL: https://issues.apache.org/jira/browse/YARN-3816 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Junping Du >Assignee: Li Lu > Labels: yarn-2928-1st-milestone > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: Application Level Aggregation of Timeline Data.pdf, > YARN-3816-YARN-2928-v1.patch, YARN-3816-YARN-2928-v2.1.patch, > YARN-3816-YARN-2928-v2.2.patch, YARN-3816-YARN-2928-v2.3.patch, > YARN-3816-YARN-2928-v2.patch, YARN-3816-YARN-2928-v3.1.patch, > YARN-3816-YARN-2928-v3.patch, YARN-3816-YARN-2928-v4.patch, > YARN-3816-YARN-2928-v5.patch, YARN-3816-YARN-2928-v6.patch, > YARN-3816-YARN-2928-v7.patch, YARN-3816-YARN-2928-v8.patch, > YARN-3816-YARN-2928-v9.patch, YARN-3816-feature-YARN-2928.v4.1.patch, > YARN-3816-poc-v1.patch, YARN-3816-poc-v2.patch > > > We need application level aggregation of Timeline data: > - To present end user aggregated states for each application, include: > resource (CPU, Memory) consumption across all containers, number of > containers launched/completed/failed, etc. We need this for apps while they > are running as well as when they are done. > - Also, framework specific metrics, e.g. HDFS_BYTES_READ, should be > aggregated to show details of states in framework level. > - Other level (Flow/User/Queue) aggregation can be more efficient to be based > on Application-level aggregations rather than raw entity-level data as much > less raws need to scan (with filter out non-aggregated entities, like: > events, configurations, etc.). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6134) [ATSv2 Security] Regenerate delegation token for app just before token expires if app collector is active
[ https://issues.apache.org/jira/browse/YARN-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6134: --- Fix Version/s: 2.9.0 > [ATSv2 Security] Regenerate delegation token for app just before token > expires if app collector is active > - > > Key: YARN-6134 > URL: https://issues.apache.org/jira/browse/YARN-6134 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355-branch-2 > > Attachments: YARN-6134-YARN-5355.01.patch, > YARN-6134-YARN-5355.02.patch, YARN-6134-YARN-5355.03.patch, > YARN-6134-YARN-5355.04.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3125) [Event producers] Change distributed shell to use new timeline service
[ https://issues.apache.org/jira/browse/YARN-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3125: --- Fix Version/s: 2.9.0 > [Event producers] Change distributed shell to use new timeline service > -- > > Key: YARN-3125 > URL: https://issues.apache.org/jira/browse/YARN-3125 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Junping Du > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-3125.patch, YARN-3125_UT-022615.patch, > YARN-3125_UT-022715.patch, YARN-3125v2.patch, YARN-3125v3.patch > > > We can start with changing distributed shell to use new timeline service once > the framework is completed, in which way we can quickly verify the next gen > is working fine end-to-end. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3901) Populate flow run data in the flow_run & flow activity tables
[ https://issues.apache.org/jira/browse/YARN-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3901: --- Fix Version/s: 2.9.0 > Populate flow run data in the flow_run & flow activity tables > - > > Key: YARN-3901 > URL: https://issues.apache.org/jira/browse/YARN-3901 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-3901-YARN-2928.1.patch, > YARN-3901-YARN-2928.10.patch, YARN-3901-YARN-2928.2.patch, > YARN-3901-YARN-2928.3.patch, YARN-3901-YARN-2928.4.patch, > YARN-3901-YARN-2928.5.patch, YARN-3901-YARN-2928.6.patch, > YARN-3901-YARN-2928.7.patch, YARN-3901-YARN-2928.8.patch, > YARN-3901-YARN-2928.9.patch > > > As per the schema proposed in YARN-3815 in > https://issues.apache.org/jira/secure/attachment/12743391/hbase-schema-proposal-for-aggregation.pdf > filing jira to track creation and population of data in the flow run table. > Some points that are being considered: > - Stores per flow run information aggregated across applications, flow version > RM’s collector writes to on app creation and app completion > - Per App collector writes to it for metric updates at a slower frequency > than the metric updates to application table > primary key: cluster ! user ! flow ! flow run id > - Only the latest version of flow-level aggregated metrics will be kept, even > if the entity and application level keep a timeseries. > - The running_apps column will be incremented on app creation, and > decremented on app completion. > - For min_start_time the RM writer will simply write a value with the tag for > the applicationId. A coprocessor will return the min value of all written > values. - > - Upon flush and compactions, the min value between all the cells of this > column will be written to the cell without any tag (empty tag) and all the > other cells will be discarded. > - Ditto for the max_end_time, but then the max will be kept. > - Tags are represented as #type:value. The type can be not set (0), or can > indicate running (1) or complete (2). In those cases (for metrics) only > complete app metrics are collapsed on compaction. > - The m! values are aggregated (summed) upon read. Only when applications are > completed (indicated by tag type 2) can the values be collapsed. > - The application ids that have completed and been aggregated into the flow > numbers are retained in a separate column for historical tracking: we don’t > want to re-aggregate for those upon replay > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4455) Support fetching metrics by time range
[ https://issues.apache.org/jira/browse/YARN-4455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-4455: --- Fix Version/s: 2.9.0 > Support fetching metrics by time range > -- > > Key: YARN-4455 > URL: https://issues.apache.org/jira/browse/YARN-4455 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-5355 >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: YARN-5355, yarn-5355-merge-blocker > Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355-branch-2 > > Attachments: YARN-4455-YARN-5355.01.patch, > YARN-4455-YARN-5355.02.patch, YARN-4455-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4765) Split TestHBaseTimelineStorage into multiple test classes
[ https://issues.apache.org/jira/browse/YARN-4765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-4765: --- Fix Version/s: 2.9.0 > Split TestHBaseTimelineStorage into multiple test classes > - > > Key: YARN-4765 > URL: https://issues.apache.org/jira/browse/YARN-4765 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: YARN-5355, atsv2-hbase, oct16-medium > Fix For: 2.9.0, 3.0.0-alpha2, YARN-5355 > > Attachments: YARN-4765-YARN-5355.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6159) Documentation changes for TimelineV2Client
[
https://issues.apache.org/jira/browse/YARN-6159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated YARN-6159:
---
Fix Version/s: 2.9.0
> Documentation changes for TimelineV2Client
> --
>
> Key: YARN-6159
> URL: https://issues.apache.org/jira/browse/YARN-6159
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: documentation
>Reporter: Varun Saxena
>Assignee: Naganarasimha G R
>Priority: Minor
> Fix For: 2.9.0, YARN-5355, YARN-5355-branch-2, 3.0.0-alpha4
>
> Attachments: TimelineServiceV2.html, YARN-6159.v1.001.patch,
> YARN-6159.v1.002.patch, YARN-6159.v1.003.patch, YARN-6159.v1.004.patch
>
>
> Make documentation changes for TimelineV2Client i.e. to reflect changes made
> in client API in YARN-4675.
> Also in TimelineServiceV2.md, under section Publishing application specific
> data, we have the following code snippet. Here,
> {{timelineClient.putEntitiesAsync(entity);}} should be
> {{client.putEntitiesAsync(entity);}} instead.
> {code}
> // Create and start the Timeline client v.2
> TimelineClient client = TimelineClient.createTimelineClient(appId);
> client.init(conf);
> client.start();
> try {
> TimelineEntity myEntity = new TimelineEntity();
> myEntity.setEntityType("MY_APPLICATION");
> myEntity.setEntityId("MyApp1")
> // Compose other entity info
> // Blocking write
> client.putEntities(entity);
> TimelineEntity myEntity2 = new TimelineEntity();
> // Compose other info
> // Non-blocking write
> timelineClient.putEntitiesAsync(entity);
> } catch (IOException e) {
> // Handle the exception
> } catch (RuntimeException e) {
> {code}
> Below can also be changed to client to keep it consistent.
> {code}
> amRMClient.registerTimelineClient(timelineClient);
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3041) [Data Model] create overall data objects of TS next gen
[ https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3041: --- Fix Version/s: 2.9.0 > [Data Model] create overall data objects of TS next gen > --- > > Key: YARN-3041 > URL: https://issues.apache.org/jira/browse/YARN-3041 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Zhijie Shen > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: Data_model_proposal_v2.pdf, YARN-3041.2.patch, > YARN-3041.3.patch, YARN-3041.4.patch, YARN-3041.5.patch, > YARN-3041.preliminary.001.patch > > > Per design in YARN-2928, create the ATS entity and events API. > Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, > flow, flow run, YARN app, ...). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5156) YARN_CONTAINER_FINISHED of YARN_CONTAINERs will always have running state
[
https://issues.apache.org/jira/browse/YARN-5156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated YARN-5156:
---
Fix Version/s: 2.9.0
> YARN_CONTAINER_FINISHED of YARN_CONTAINERs will always have running state
> -
>
> Key: YARN-5156
> URL: https://issues.apache.org/jira/browse/YARN-5156
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Li Lu
>Assignee: Vrushali C
> Labels: YARN-5355
> Fix For: 2.9.0, YARN-5355, 3.0.0-beta1
>
> Attachments: YARN-5156-YARN-2928.01.patch,
> YARN-5156-YARN-5355.01.patch, YARN-5156-YARN-5355.02.patch
>
>
> On container finished, we're reporting "YARN_CONTAINER_STATE: "RUNNING"". Do
> we design this deliberately or it's a bug?
> {code}
> {
> metrics: [ ],
> events: [
> {
> id: "YARN_CONTAINER_FINISHED",
> timestamp: 1464213765890,
> info: {
> YARN_CONTAINER_EXIT_STATUS: 0,
> YARN_CONTAINER_STATE: "RUNNING",
> YARN_CONTAINER_DIAGNOSTICS_INFO: ""
> }
> },
> {
> id: "YARN_NM_CONTAINER_LOCALIZATION_FINISHED",
> timestamp: 1464213761133,
> info: { }
> },
> {
> id: "YARN_CONTAINER_CREATED",
> timestamp: 1464213761132,
> info: { }
> },
> {
> id: "YARN_NM_CONTAINER_LOCALIZATION_STARTED",
> timestamp: 1464213761132,
> info: { }
> }
> ],
> id: "container_e15_1464213707405_0001_01_18",
> type: "YARN_CONTAINER",
> createdtime: 1464213761132,
> info: {
> YARN_CONTAINER_ALLOCATED_PRIORITY: "20",
> YARN_CONTAINER_ALLOCATED_VCORE: 1,
> YARN_CONTAINER_ALLOCATED_HOST_HTTP_ADDRESS: "10.22.16.164:0",
> UID:
> "yarn_cluster!application_1464213707405_0001!YARN_CONTAINER!container_e15_1464213707405_0001_01_18",
> YARN_CONTAINER_ALLOCATED_HOST: "10.22.16.164",
> YARN_CONTAINER_ALLOCATED_MEMORY: 1024,
> SYSTEM_INFO_PARENT_ENTITY: {
> type: "YARN_APPLICATION_ATTEMPT",
> id: "appattempt_1464213707405_0001_01"
> },
> YARN_CONTAINER_ALLOCATED_PORT: 64694
> },
> configs: { },
> isrelatedto: { },
> relatesto: { }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6237) Move UID constant to TimelineReaderUtils
[ https://issues.apache.org/jira/browse/YARN-6237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6237: --- Fix Version/s: 2.9.0 > Move UID constant to TimelineReaderUtils > > > Key: YARN-6237 > URL: https://issues.apache.org/jira/browse/YARN-6237 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Labels: newbie > Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355-branch-2 > > Attachments: YARN-6237-YARN-5355.0001.patch > > > UID constant is kept in TimelineReader Manager. This can be moved to > TimelineReaderUtils which can keep track of all reader constants. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4409) Fix javadoc and checkstyle issues in timelineservice code
[ https://issues.apache.org/jira/browse/YARN-4409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-4409: --- Fix Version/s: 2.9.0 > Fix javadoc and checkstyle issues in timelineservice code > - > > Key: YARN-4409 > URL: https://issues.apache.org/jira/browse/YARN-4409 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-2928-1st-milestone > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-4409-YARN-2928.01.patch, > YARN-4409-YARN-2928.02.patch, YARN-4409-YARN-2928.03.patch > > > There are a large number of javadoc and checkstyle issues currently open in > timelineservice code. We need to fix them before we merge it into trunk. > Refer to > https://issues.apache.org/jira/browse/YARN-3862?focusedCommentId=15035267=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15035267 > We still have 94 open checkstyle issues and javadocs failing for Java 8. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3276) Refactor and fix null casting in some map cast for TimelineEntity (old and new) and fix findbug warnings
[ https://issues.apache.org/jira/browse/YARN-3276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3276: --- Fix Version/s: 2.9.0 > Refactor and fix null casting in some map cast for TimelineEntity (old and > new) and fix findbug warnings > > > Key: YARN-3276 > URL: https://issues.apache.org/jira/browse/YARN-3276 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Junping Du >Assignee: Junping Du > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-3276-YARN-2928.v3.patch, > YARN-3276-YARN-2928.v4.patch, YARN-3276-YARN-2928.v5-fix-checkstyle.patch, > YARN-3276-YARN-2928.v5.patch, YARN-3276-YARN-2928.v6.patch, > YARN-3276-v2.patch, YARN-3276-v3.patch, YARN-3276.patch > > > Per discussion in YARN-3087, we need to refactor some similar logic to cast > map to hashmap and get rid of NPE issue. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4297) TestJobHistoryEventHandler and TestRMContainerAllocator failing on YARN-2928 branch
[
https://issues.apache.org/jira/browse/YARN-4297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated YARN-4297:
---
Fix Version/s: 2.9.0
> TestJobHistoryEventHandler and TestRMContainerAllocator failing on YARN-2928
> branch
> ---
>
> Key: YARN-4297
> URL: https://issues.apache.org/jira/browse/YARN-4297
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-4297-YARN-2928.01.patch,
> YARN-4297-feature-YARN-2928.02.patch, YARN-4297-feature-YARN-2928.03.patch
>
>
> {noformat}
> Tests run: 13, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 16.09 sec
> <<< FAILURE! - in
> org.apache.hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler
> testTimelineEventHandling(org.apache.hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler)
> Time elapsed: 0.11 sec <<< ERROR!
> java.lang.ClassCastException:
> org.apache.hadoop.mapreduce.v2.app.AppContext$$EnhancerByMockitoWithCGLIB$$95d3ddbe
> cannot be cast to
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$RunningAppContext
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.serviceInit(JobHistoryEventHandler.java:271)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler.testTimelineEventHandling(TestJobHistoryEventHandler.java:495)
> {noformat}
> {noformat}
> testRMContainerAllocatorResendsRequestsOnRMRestart(org.apache.hadoop.mapreduce.v2.app.rm.TestRMContainerAllocator)
> Time elapsed: 2.649 sec <<< ERROR!
> java.lang.ClassCastException:
> org.apache.hadoop.mapreduce.v2.app.AppContext$$EnhancerByMockitoWithCGLIB$$8e08559a
> cannot be cast to
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$RunningAppContext
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.getResources(RMContainerAllocator.java:802)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:269)
> Tests in error:
> TestRMContainerAllocator.testExcessReduceContainerAssign:669 » ClassCast
> org.a...
> TestRMContainerAllocator.testReportedAppProgress:970 » NullPointer
> TestRMContainerAllocator.testBlackListedNodesWithSchedulingToThatNode:1578
> » ClassCast
> TestRMContainerAllocator.testBlackListedNodes:1292 » ClassCast
> org.apache.hado...
> TestRMContainerAllocator.testAMRMTokenUpdate:2691 » ClassCast
> org.apache.hadoo...
> TestRMContainerAllocator.testMapReduceAllocationWithNodeLabelExpression:722
> » ClassCast
> TestRMContainerAllocator.testReducerRampdownDiagnostics:443 » ClassCast
> org.ap...
> TestRMContainerAllocator.testReportedAppProgressWithOnlyMaps:1118 »
> NullPointer
> TestRMContainerAllocator.testMapReduceScheduling:819 » ClassCast
> org.apache.ha...
> TestRMContainerAllocator.testResource:390 » ClassCast
> org.apache.hadoop.mapred...
> TestRMContainerAllocator.testUpdatedNodes:1190 » ClassCast
> org.apache.hadoop.m...
> TestRMContainerAllocator.testCompletedTasksRecalculateSchedule:2249 »
> ClassCast
> TestRMContainerAllocator.testConcurrentTaskLimits:2779 » ClassCast
> org.apache
> TestRMContainerAllocator.testSimple:219 » ClassCast
> org.apache.hadoop.mapreduc...
>
> TestRMContainerAllocator.testIgnoreBlacklisting:1378->getContainerOnHost:1511
> » ClassCast
> TestRMContainerAllocator.testMapNodeLocality:310 » ClassCast
> org.apache.hadoop...
>
> TestRMContainerAllocator.testRMContainerAllocatorResendsRequestsOnRMRestart:2489
> » ClassCast
> Tests run: 26, Failures: 0, Errors: 17, Skipped: 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6424) TimelineCollector is not stopped when an app finishes in RM
[ https://issues.apache.org/jira/browse/YARN-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6424: --- Fix Version/s: 2.9.0 > TimelineCollector is not stopped when an app finishes in RM > --- > > Key: YARN-6424 > URL: https://issues.apache.org/jira/browse/YARN-6424 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: 3.0.0-alpha2 >Reporter: Varun Saxena >Assignee: Varun Saxena >Priority: Critical > Fix For: 2.9.0, YARN-5355, YARN-5355-branch-2, 3.0.0-alpha4 > > Attachments: YARN-6424.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3390) Reuse TimelineCollectorManager for RM
[ https://issues.apache.org/jira/browse/YARN-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3390: --- Fix Version/s: 2.9.0 > Reuse TimelineCollectorManager for RM > - > > Key: YARN-3390 > URL: https://issues.apache.org/jira/browse/YARN-3390 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-3390.1.patch, YARN-3390.2.patch, YARN-3390.3.patch, > YARN-3390.4.patch > > > RMTimelineCollector should have the context info of each app whose entity > has been put -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3908) Bugs in HBaseTimelineWriterImpl
[ https://issues.apache.org/jira/browse/YARN-3908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3908: --- Fix Version/s: 2.9.0 > Bugs in HBaseTimelineWriterImpl > --- > > Key: YARN-3908 > URL: https://issues.apache.org/jira/browse/YARN-3908 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Vrushali C > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-3908-YARN-2928.001.patch, > YARN-3908-YARN-2928.002.patch, YARN-3908-YARN-2928.003.patch, > YARN-3908-YARN-2928.004.patch, YARN-3908-YARN-2928.004.patch, > YARN-3908-YARN-2928.005.patch > > > 1. In HBaseTimelineWriterImpl, the info column family contains the basic > fields of a timeline entity plus events. However, entity#info map is not > stored at all. > 2 event#timestamp is also not persisted. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5378) Accommodate app-id->cluster mapping
[ https://issues.apache.org/jira/browse/YARN-5378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-5378: --- Fix Version/s: 2.9.0 > Accommodate app-id->cluster mapping > --- > > Key: YARN-5378 > URL: https://issues.apache.org/jira/browse/YARN-5378 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Joep Rottinghuis >Assignee: Sangjin Lee > Labels: yarn-5355-merge-blocker > Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355-branch-2 > > Attachments: YARN-5378-YARN-5355.01.patch, > YARN-5378-YARN-5355.02.patch, YARN-5378-YARN-5355.03.patch > > > In discussion with [~sjlee0], [~vrushalic], [~subru], and [~curino] a > use-case came up to be able to map from application-id to cluster-id in > context of federation for Yarn. > What happens is that a "random" cluster in the federation is asked to > generate an app-id and then potentially a different cluster can be the "home" > cluster for the AM. Furthermore, tasks can then run in yet other clusters. > In order to be able to pull up the logical home cluster on which the > application ran, there needs to be a mapping from application-id to > cluster-id. This mapping is available in the federated Yarn case only during > the active live of the application. > A similar situation is common in our larger production environment. Somebody > will complain about a slow job, some failure or whatever. If we're lucky we > have an application-id. When we ask the user which cluster they ran on, > they'll typically answer with the machine from where they launched the job > (many users are unaware of the underlying physical clusters). This leaves us > to spelunk through various RM ui's to find a matching epoch in the > application ID. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5170) Eliminate singleton converters and static method access
[ https://issues.apache.org/jira/browse/YARN-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-5170: --- Fix Version/s: 2.9.0 > Eliminate singleton converters and static method access > --- > > Key: YARN-5170 > URL: https://issues.apache.org/jira/browse/YARN-5170 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Joep Rottinghuis >Assignee: Joep Rottinghuis > Labels: yarn-2928-1st-milestone > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-5170-YARN-2928.01.patch, > YARN-5170-YARN-2928.02.patch, YARN-5170-YARN-2928.03.patch, > YARN-5170-YARN-2928.04.patch, YARN-5170-YARN-2928.05.patch, > YARN-5170-YARN-2928.06.patch, YARN-5170-YARN-2928.07.patch, > YARN-5170-YARN-2928.08.patch, YARN-5170-YARN-2928.09.patch, > YARN-5170-YARN-2928.10.patch, YARN-5170-YARN-2928.11.patch, > YARN-5170-YARN-2928.12.patch, YARN-5170-YARN-2928.13.patch > > > As part of YARN-5109 we introduced several KeyConverter classes. > To stay consistent with the existing LongConverter in the sample patch I > created I made these other converter classes singleton as well. > In conversation with [~sjlee0] who has a general dislike of singletons, we > discussed it is best to get rid of these singletons and make them simply > instance variables. > There are other classes where the keys have static methods referring to a > singleton converter. > Moreover, it turns out that due to code evolution we end up creating the same > keys several times. > So general approach is to not re-instantiate rowkeys, converters when not > needed. > I would like to create the byte[] rowKey in the RowKey classes their > constructor, but that would leak an incomplete object to the converter. > There are a few method in TimelineStorageUtils that are used only once, or > only by one class, as part of this refactor I'll move these to keep the > "Utils" class as small as possible and keep them for truly generally used > utils that don't really belong anywhere else. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4622) TestDistributedShell fails for v2 test cases after modifications for 1.5
[
https://issues.apache.org/jira/browse/YARN-4622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated YARN-4622:
---
Fix Version/s: 2.9.0
> TestDistributedShell fails for v2 test cases after modifications for 1.5
>
>
> Key: YARN-4622
> URL: https://issues.apache.org/jira/browse/YARN-4622
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
> Labels: test
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-4622-YARN-2928.v1.001.patch
>
>
> TestDistributedShell fails for v2 test cases :
> *testDSShellWithoutDomainV2DefaultFlow and
> testDSShellWithoutDomainV2CustomizedFlow* after trunk rebase with
> modifications for 1.5,
> {code}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
> java.lang.NullPointerException
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:187)
> at com.google.common.base.Joiner.toString(Joiner.java:532)
> at com.google.common.base.Joiner.appendTo(Joiner.java:124)
> at com.google.common.base.Joiner.appendTo(Joiner.java:181)
> at com.google.common.base.Joiner.join(Joiner.java:237)
> at com.google.common.base.Joiner.join(Joiner.java:226)
> at com.google.common.base.Joiner.join(Joiner.java:253)
> at
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.constructResURI(TimelineClientImpl.java:726)
> at
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.serviceStart(TimelineClientImpl.java:336)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.createAndStartTimelineClient(ApplicationImpl.java:149)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.(ApplicationImpl.java:113)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainerInternal(ContainerManagerImpl.java:971)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainers(ContainerManagerImpl.java:830)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagementProtocolPBServiceImpl.startContainers(ContainerManagementProtocolPBServiceImpl.java:65)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3264) [Storage implementation] Create backing storage write interface and a POC only file based storage implementation
[ https://issues.apache.org/jira/browse/YARN-3264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3264: --- Fix Version/s: 2.9.0 > [Storage implementation] Create backing storage write interface and a POC > only file based storage implementation > - > > Key: YARN-3264 > URL: https://issues.apache.org/jira/browse/YARN-3264 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-3264.001.patch, YARN-3264.002.patch, > YARN-3264.003.patch, YARN-3264.004.patch, YARN-3264.005.patch, > YARN-3264.006.patch, YARN-3264.007.patch, YARN-3264.008.patch > > > For the PoC, need to create a backend impl for file based storage of entities -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5243) fix several rebase and other miscellaneous issues before merge
[ https://issues.apache.org/jira/browse/YARN-5243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-5243: --- Fix Version/s: 2.9.0 > fix several rebase and other miscellaneous issues before merge > -- > > Key: YARN-5243 > URL: https://issues.apache.org/jira/browse/YARN-5243 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Labels: yarn-2928-1st-milestone > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-5243-YARN-2928.01.patch, > YARN-5243-YARN-2928.02.patch, YARN-5243-YARN-2928.03.patch > > > I have come across a couple of miscellaneous issues while inspecting the > diffs against the trunk. > We also need to review one last time (probably after the final rebase) to > ensure the timeline services v.2 leaves no impact when disabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5052) [Documentation] Update timeline service v2 documentation to capture information about filters
[ https://issues.apache.org/jira/browse/YARN-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-5052: --- Fix Version/s: 2.9.0 > [Documentation] Update timeline service v2 documentation to capture > information about filters > - > > Key: YARN-5052 > URL: https://issues.apache.org/jira/browse/YARN-5052 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-2928-1st-milestone > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: Apache Hadoop 3.0.0-SNAPSHOT – The YARN Timeline Service > v.pdf, Hierarchy.png, The YARN Timeline Service v2.02.pdf, > YARN-5052-YARN-2928.01.patch, YARN-5052-YARN-2928.02.patch > > > Since YARN-4447 has gone in, we can update our documentation to capture > information about usage of filters. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4712) CPU Usage Metric is not captured properly in YARN-2928
[
https://issues.apache.org/jira/browse/YARN-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated YARN-4712:
---
Fix Version/s: 2.9.0
> CPU Usage Metric is not captured properly in YARN-2928
> --
>
> Key: YARN-4712
> URL: https://issues.apache.org/jira/browse/YARN-4712
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
> Labels: yarn-2928-1st-milestone
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-4712-YARN-2928.v1.001.patch,
> YARN-4712-YARN-2928.v1.002.patch, YARN-4712-YARN-2928.v1.003.patch,
> YARN-4712-YARN-2928.v1.004.patch, YARN-4712-YARN-2928.v1.005.patch,
> YARN-4712-YARN-2928.v1.006.patch
>
>
> There are 2 issues with CPU usage collection
> * I was able to observe that that many times CPU usage got from
> {{pTree.getCpuUsagePercent()}} is
> ResourceCalculatorProcessTree.UNAVAILABLE(i.e. -1) but ContainersMonitor do
> the calculation i.e. {{cpuUsageTotalCoresPercentage = cpuUsagePercentPerCore
> /resourceCalculatorPlugin.getNumProcessors()}} because of which UNAVAILABLE
> check in {{NMTimelinePublisher.reportContainerResourceUsage}} is not
> encountered. so proper checks needs to be handled
> * {{EntityColumnPrefix.METRIC}} uses always LongConverter but
> ContainerMonitor is publishing decimal values for the CPU usage.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4102) Add a "skip existing table" mode for timeline schema creator
[ https://issues.apache.org/jira/browse/YARN-4102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-4102: --- Fix Version/s: 2.9.0 > Add a "skip existing table" mode for timeline schema creator > > > Key: YARN-4102 > URL: https://issues.apache.org/jira/browse/YARN-4102 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu >Assignee: Li Lu > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-4102-YARN-2928.001.patch, > YARN-4102-YARN-2928.002.patch, YARN-4102-YARN-2928.003.patch, > YARN-4102-YARN-2928.004.patch > > > When debugging timeline POCs, we may need to create hbase tables that are > added in some ongoing patches. Right now, our schema creator will exit when > it hits one existing table. While this is a correct behavior with end users, > this introduces much trouble in debugging POCs: every time we have to disable > all existing tables, drop them, run the schema creator to generate all > tables, and regenerate all test data. > Maybe we'd like to add an "incremental" mode so that the creator will only > create non-existing tables? This is pretty handy in deploying our POCs. Of > course, consistency has to be kept in mind across tables. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3706) Generalize native HBase writer for additional tables
[ https://issues.apache.org/jira/browse/YARN-3706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3706: --- Fix Version/s: 2.9.0 > Generalize native HBase writer for additional tables > > > Key: YARN-3706 > URL: https://issues.apache.org/jira/browse/YARN-3706 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Joep Rottinghuis >Assignee: Joep Rottinghuis >Priority: Minor > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-3706-YARN-2928.001.patch, > YARN-3706-YARN-2928.010.patch, YARN-3706-YARN-2928.011.patch, > YARN-3706-YARN-2928.012.patch, YARN-3706-YARN-2928.013.patch, > YARN-3706-YARN-2928.014.patch, YARN-3706-YARN-2928.015.patch, > YARN-3726-YARN-2928.002.patch, YARN-3726-YARN-2928.003.patch, > YARN-3726-YARN-2928.004.patch, YARN-3726-YARN-2928.005.patch, > YARN-3726-YARN-2928.006.patch, YARN-3726-YARN-2928.007.patch, > YARN-3726-YARN-2928.008.patch, YARN-3726-YARN-2928.009.patch > > > When reviewing YARN-3411 we noticed that we could change the class hierarchy > a little in order to accommodate additional tables easily. > In order to get ready for benchmark testing we left the original layout in > place, as performance would not be impacted by the code hierarchy. > Here is a separate jira to address the hierarchy. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6064) Support fromId for flowRuns and flow/flowRun apps REST API's
[ https://issues.apache.org/jira/browse/YARN-6064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6064: --- Fix Version/s: 2.9.0 > Support fromId for flowRuns and flow/flowRun apps REST API's > > > Key: YARN-6064 > URL: https://issues.apache.org/jira/browse/YARN-6064 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Labels: yarn-5355-merge-blocker > Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355-branch-2 > > Attachments: YARN-6064-YARN-5355.0001.patch, > YARN-6064-YARN-5355.0002.patch, YARN-6064-YARN-5355.0003.patch, > YARN-6064-YARN-5355.0004.patch, YARN-6064-YARN-5355.addendum.patch > > > Splitting out JIRA YARN-6027 for pagination support for flowRuns, flow apps > and flow run apps. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3034) [Collector wireup] Implement RM starting its timeline collector
[ https://issues.apache.org/jira/browse/YARN-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3034: --- Fix Version/s: 2.9.0 > [Collector wireup] Implement RM starting its timeline collector > --- > > Key: YARN-3034 > URL: https://issues.apache.org/jira/browse/YARN-3034 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Naganarasimha G R > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-3024.20150324-1.patch, YARN-3034-20150312-1.patch, > YARN-3034.20150205-1.patch, YARN-3034.20150316-1.patch, > YARN-3034.20150318-1.patch, YARN-3034.20150320-1.patch > > > Per design in YARN-2928, implement resource managers starting their own ATS > writers. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5095) flow activities and flow runs are populated with wrong timestamp when RM restarts w/ recovery enabled
[
https://issues.apache.org/jira/browse/YARN-5095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated YARN-5095:
---
Fix Version/s: 2.9.0
> flow activities and flow runs are populated with wrong timestamp when RM
> restarts w/ recovery enabled
> -
>
> Key: YARN-5095
> URL: https://issues.apache.org/jira/browse/YARN-5095
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
>Priority: Critical
> Labels: yarn-2928-1st-milestone
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-5095-YARN-2928.01.patch,
> YARN-5095-YARN-2928.02.patch, YARN-5095-YARN-2928.03.patch
>
>
> I have the RM recovery enabled. I see that upon restart the RM populates
> records into flow activity and flow runs but with *wrong* timestamps. What I
> mean by the timestamp is the part of the row key:
> - flow activity: row created with the day of the RM restart
> - flow run: row created with the RM start time as the "run id"
> The following illustrates an example flow run:
> {noformat}
> metrics: [ ],
> events: [ ],
> id: "sjlee@Sleep job/1463433569917",
> type: "YARN_FLOW_RUN",
> createdtime: 1463422860987,
> info: {
> UID: "yarn_cluster!sjlee!Sleep job!1463433569917",
> SYSTEM_INFO_FLOW_RUN_ID: 1463433569917,
> SYSTEM_INFO_FLOW_NAME: "Sleep job",
> SYSTEM_INFO_FLOW_RUN_END_TIME: 1463422865033,
> SYSTEM_INFO_USER: "sjlee"
> },
> isrelatedto: { },
> relatesto: { }
> {noformat}
> The created time and the end time are correct (i.e. original time), whereas
> the timestamp in the row key (= run id: 1463433569917) is actually later than
> the end time and coincides with the RM restart.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
