[jira] [Updated] (YARN-7619) Max AM Resource value in CS UI is different for every user
[
https://issues.apache.org/jira/browse/YARN-7619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated YARN-7619:
-
Attachment: YARN-7619.005.branch-2.8.patch
Attaching version 005 for branch-2.8
> Max AM Resource value in CS UI is different for every user
> --
>
> Key: YARN-7619
> URL: https://issues.apache.org/jira/browse/YARN-7619
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, yarn
>Affects Versions: 2.9.0, 3.0.0-beta1, 2.8.2, 3.1.0
>Reporter: Eric Payne
>Assignee: Eric Payne
> Attachments: Max AM Resources is Different for Each User.png,
> YARN-7619.001.patch, YARN-7619.002.patch, YARN-7619.003.patch,
> YARN-7619.004.branch-2.8.patch, YARN-7619.004.branch-3.0.patch,
> YARN-7619.004.patch, YARN-7619.005.branch-2.8.patch,
> YARN-7619.005.branch-3.0.patch, YARN-7619.005.patch
>
>
> YARN-7245 addressed the problem that the {{Max AM Resource}} in the capacity
> scheduler UI used to contain the queue-level AM limit instead of the
> user-level AM limit. It fixed this by using the user-specific AM limit that
> is calculated in {{LeafQueue#activateApplications}}, stored in each user's
> {{LeafQueue#User}} object, and retrieved via
> {{UserInfo#getResourceUsageInfo}}.
> The problem is that this user-specific AM limit depends on the activity of
> other users and other applications in a queue, and it is only calculated and
> updated when a user's application is activated. So, when
> {{CapacitySchedulerPage}} retrieves the user-specific AM limit, it is a stale
> value unless an application was recently activated for a particular user.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7619) Max AM Resource value in CS UI is different for every user
[
https://issues.apache.org/jira/browse/YARN-7619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated YARN-7619:
-
Attachment: YARN-7619.005.branch-3.0.patch
Attaching 005 branch-3.0
> Max AM Resource value in CS UI is different for every user
> --
>
> Key: YARN-7619
> URL: https://issues.apache.org/jira/browse/YARN-7619
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, yarn
>Affects Versions: 2.9.0, 3.0.0-beta1, 2.8.2, 3.1.0
>Reporter: Eric Payne
>Assignee: Eric Payne
> Attachments: Max AM Resources is Different for Each User.png,
> YARN-7619.001.patch, YARN-7619.002.patch, YARN-7619.003.patch,
> YARN-7619.004.branch-2.8.patch, YARN-7619.004.branch-3.0.patch,
> YARN-7619.004.patch, YARN-7619.005.branch-3.0.patch, YARN-7619.005.patch
>
>
> YARN-7245 addressed the problem that the {{Max AM Resource}} in the capacity
> scheduler UI used to contain the queue-level AM limit instead of the
> user-level AM limit. It fixed this by using the user-specific AM limit that
> is calculated in {{LeafQueue#activateApplications}}, stored in each user's
> {{LeafQueue#User}} object, and retrieved via
> {{UserInfo#getResourceUsageInfo}}.
> The problem is that this user-specific AM limit depends on the activity of
> other users and other applications in a queue, and it is only calculated and
> updated when a user's application is activated. So, when
> {{CapacitySchedulerPage}} retrieves the user-specific AM limit, it is a stale
> value unless an application was recently activated for a particular user.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7619) Max AM Resource value in CS UI is different for every user
[
https://issues.apache.org/jira/browse/YARN-7619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated YARN-7619:
-
Attachment: YARN-7619.005.patch
Thanks [~sunilg]. Attaching patch 005.
bq. I think we can use Resources.min here also.
Yes, I did notice that, but I was reluctant to change the existing code to
minimize the risk of behavior change. However, I think this will be fine, so I
went ahead and made the change.
bq. Its better to use Resources.clone on this userAMLimit object, correct?
I don't think it is actually modified anywhere, but it is better to be safe, so
I made the change.
> Max AM Resource value in CS UI is different for every user
> --
>
> Key: YARN-7619
> URL: https://issues.apache.org/jira/browse/YARN-7619
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, yarn
>Affects Versions: 2.9.0, 3.0.0-beta1, 2.8.2, 3.1.0
>Reporter: Eric Payne
>Assignee: Eric Payne
> Attachments: Max AM Resources is Different for Each User.png,
> YARN-7619.001.patch, YARN-7619.002.patch, YARN-7619.003.patch,
> YARN-7619.004.branch-2.8.patch, YARN-7619.004.branch-3.0.patch,
> YARN-7619.004.patch, YARN-7619.005.patch
>
>
> YARN-7245 addressed the problem that the {{Max AM Resource}} in the capacity
> scheduler UI used to contain the queue-level AM limit instead of the
> user-level AM limit. It fixed this by using the user-specific AM limit that
> is calculated in {{LeafQueue#activateApplications}}, stored in each user's
> {{LeafQueue#User}} object, and retrieved via
> {{UserInfo#getResourceUsageInfo}}.
> The problem is that this user-specific AM limit depends on the activity of
> other users and other applications in a queue, and it is only calculated and
> updated when a user's application is activated. So, when
> {{CapacitySchedulerPage}} retrieves the user-specific AM limit, it is a stale
> value unless an application was recently activated for a particular user.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7619) Max AM Resource value in CS UI is different for every user
[
https://issues.apache.org/jira/browse/YARN-7619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated YARN-7619:
-
Attachment: YARN-7619.004.branch-2.8.patch
And here is the branch 2.8 patch
> Max AM Resource value in CS UI is different for every user
> --
>
> Key: YARN-7619
> URL: https://issues.apache.org/jira/browse/YARN-7619
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, yarn
>Affects Versions: 2.9.0, 3.0.0-beta1, 2.8.2, 3.1.0
>Reporter: Eric Payne
>Assignee: Eric Payne
> Attachments: Max AM Resources is Different for Each User.png,
> YARN-7619.001.patch, YARN-7619.002.patch, YARN-7619.003.patch,
> YARN-7619.004.branch-2.8.patch, YARN-7619.004.branch-3.0.patch,
> YARN-7619.004.patch
>
>
> YARN-7245 addressed the problem that the {{Max AM Resource}} in the capacity
> scheduler UI used to contain the queue-level AM limit instead of the
> user-level AM limit. It fixed this by using the user-specific AM limit that
> is calculated in {{LeafQueue#activateApplications}}, stored in each user's
> {{LeafQueue#User}} object, and retrieved via
> {{UserInfo#getResourceUsageInfo}}.
> The problem is that this user-specific AM limit depends on the activity of
> other users and other applications in a queue, and it is only calculated and
> updated when a user's application is activated. So, when
> {{CapacitySchedulerPage}} retrieves the user-specific AM limit, it is a stale
> value unless an application was recently activated for a particular user.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7619) Max AM Resource value in CS UI is different for every user
[
https://issues.apache.org/jira/browse/YARN-7619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated YARN-7619:
-
Attachment: YARN-7619.004.branch-3.0.patch
{{TestNodeLabelContainerAllocation}} is also failing in trunk and is not
related to this patch.
Attaching branch-3.0 patch.
> Max AM Resource value in CS UI is different for every user
> --
>
> Key: YARN-7619
> URL: https://issues.apache.org/jira/browse/YARN-7619
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, yarn
>Affects Versions: 2.9.0, 3.0.0-beta1, 2.8.2, 3.1.0
>Reporter: Eric Payne
>Assignee: Eric Payne
> Attachments: Max AM Resources is Different for Each User.png,
> YARN-7619.001.patch, YARN-7619.002.patch, YARN-7619.003.patch,
> YARN-7619.004.branch-3.0.patch, YARN-7619.004.patch
>
>
> YARN-7245 addressed the problem that the {{Max AM Resource}} in the capacity
> scheduler UI used to contain the queue-level AM limit instead of the
> user-level AM limit. It fixed this by using the user-specific AM limit that
> is calculated in {{LeafQueue#activateApplications}}, stored in each user's
> {{LeafQueue#User}} object, and retrieved via
> {{UserInfo#getResourceUsageInfo}}.
> The problem is that this user-specific AM limit depends on the activity of
> other users and other applications in a queue, and it is only calculated and
> updated when a user's application is activated. So, when
> {{CapacitySchedulerPage}} retrieves the user-specific AM limit, it is a stale
> value unless an application was recently activated for a particular user.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7619) Max AM Resource value in CS UI is different for every user
[
https://issues.apache.org/jira/browse/YARN-7619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated YARN-7619:
-
Attachment: YARN-7619.004.patch
> Max AM Resource value in CS UI is different for every user
> --
>
> Key: YARN-7619
> URL: https://issues.apache.org/jira/browse/YARN-7619
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, yarn
>Affects Versions: 2.9.0, 3.0.0-beta1, 2.8.2, 3.1.0
>Reporter: Eric Payne
>Assignee: Eric Payne
> Attachments: Max AM Resources is Different for Each User.png,
> YARN-7619.001.patch, YARN-7619.002.patch, YARN-7619.003.patch,
> YARN-7619.004.patch
>
>
> YARN-7245 addressed the problem that the {{Max AM Resource}} in the capacity
> scheduler UI used to contain the queue-level AM limit instead of the
> user-level AM limit. It fixed this by using the user-specific AM limit that
> is calculated in {{LeafQueue#activateApplications}}, stored in each user's
> {{LeafQueue#User}} object, and retrieved via
> {{UserInfo#getResourceUsageInfo}}.
> The problem is that this user-specific AM limit depends on the activity of
> other users and other applications in a queue, and it is only calculated and
> updated when a user's application is activated. So, when
> {{CapacitySchedulerPage}} retrieves the user-specific AM limit, it is a stale
> value unless an application was recently activated for a particular user.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7619) Max AM Resource value in CS UI is different for every user
[
https://issues.apache.org/jira/browse/YARN-7619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated YARN-7619:
-
Attachment: YARN-7619.003.patch
> Max AM Resource value in CS UI is different for every user
> --
>
> Key: YARN-7619
> URL: https://issues.apache.org/jira/browse/YARN-7619
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, yarn
>Affects Versions: 2.9.0, 3.0.0-beta1, 2.8.2, 3.1.0
>Reporter: Eric Payne
>Assignee: Eric Payne
> Attachments: Max AM Resources is Different for Each User.png,
> YARN-7619.001.patch, YARN-7619.002.patch, YARN-7619.003.patch
>
>
> YARN-7245 addressed the problem that the {{Max AM Resource}} in the capacity
> scheduler UI used to contain the queue-level AM limit instead of the
> user-level AM limit. It fixed this by using the user-specific AM limit that
> is calculated in {{LeafQueue#activateApplications}}, stored in each user's
> {{LeafQueue#User}} object, and retrieved via
> {{UserInfo#getResourceUsageInfo}}.
> The problem is that this user-specific AM limit depends on the activity of
> other users and other applications in a queue, and it is only calculated and
> updated when a user's application is activated. So, when
> {{CapacitySchedulerPage}} retrieves the user-specific AM limit, it is a stale
> value unless an application was recently activated for a particular user.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7619) Max AM Resource value in CS UI is different for every user
[
https://issues.apache.org/jira/browse/YARN-7619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated YARN-7619:
-
Attachment: YARN-7619.002.patch
Thanks again [~sunilg]. Uploading patch 002.
> Max AM Resource value in CS UI is different for every user
> --
>
> Key: YARN-7619
> URL: https://issues.apache.org/jira/browse/YARN-7619
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, yarn
>Affects Versions: 2.9.0, 3.0.0-beta1, 2.8.2, 3.1.0
>Reporter: Eric Payne
>Assignee: Eric Payne
> Attachments: Max AM Resources is Different for Each User.png,
> YARN-7619.001.patch, YARN-7619.002.patch
>
>
> YARN-7245 addressed the problem that the {{Max AM Resource}} in the capacity
> scheduler UI used to contain the queue-level AM limit instead of the
> user-level AM limit. It fixed this by using the user-specific AM limit that
> is calculated in {{LeafQueue#activateApplications}}, stored in each user's
> {{LeafQueue#User}} object, and retrieved via
> {{UserInfo#getResourceUsageInfo}}.
> The problem is that this user-specific AM limit depends on the activity of
> other users and other applications in a queue, and it is only calculated and
> updated when a user's application is activated. So, when
> {{CapacitySchedulerPage}} retrieves the user-specific AM limit, it is a stale
> value unless an application was recently activated for a particular user.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7619) Max AM Resource value in CS UI is different for every user
[
https://issues.apache.org/jira/browse/YARN-7619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated YARN-7619:
-
Attachment: YARN-7619.001.patch
Uploading patch 001. This is not a perfect solution, but it's close. The
pre-weighted AM limit for all users in a particular queue is calculated in
{{LeafQueue#getUserAMResourceLimitPerPartition}} and passed to the UI via the
{{UserInfo}} object for each user when the UI is rendered. This is a little
awkward because the AM Limit for users in the queue is a per-queue value, but
when rendering, I wanted to multiply the value by each users' weight.
The value displayed on the UI in the Max AM Resource may not always be valid
for weighted users because it is not normalized, and it may be more than the
queue-level AM limit on the high end if the weight is large. But since this is
only for display purposes, I think it's acceptable.
> Max AM Resource value in CS UI is different for every user
> --
>
> Key: YARN-7619
> URL: https://issues.apache.org/jira/browse/YARN-7619
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, yarn
>Affects Versions: 2.9.0, 3.0.0-beta1, 2.8.2, 3.1.0
>Reporter: Eric Payne
>Assignee: Eric Payne
> Attachments: Max AM Resources is Different for Each User.png,
> YARN-7619.001.patch
>
>
> YARN-7245 addressed the problem that the {{Max AM Resource}} in the capacity
> scheduler UI used to contain the queue-level AM limit instead of the
> user-level AM limit. It fixed this by using the user-specific AM limit that
> is calculated in {{LeafQueue#activateApplications}}, stored in each user's
> {{LeafQueue#User}} object, and retrieved via
> {{UserInfo#getResourceUsageInfo}}.
> The problem is that this user-specific AM limit depends on the activity of
> other users and other applications in a queue, and it is only calculated and
> updated when a user's application is activated. So, when
> {{CapacitySchedulerPage}} retrieves the user-specific AM limit, it is a stale
> value unless an application was recently activated for a particular user.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7619) Max AM Resource value in CS UI is different for every user
[
https://issues.apache.org/jira/browse/YARN-7619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated YARN-7619:
-
Attachment: Max AM Resources is Different for Each User.png
> Max AM Resource value in CS UI is different for every user
> --
>
> Key: YARN-7619
> URL: https://issues.apache.org/jira/browse/YARN-7619
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, yarn
>Affects Versions: 2.9.0, 3.0.0-beta1, 2.8.2, 3.1.0
>Reporter: Eric Payne
>Assignee: Eric Payne
> Attachments: Max AM Resources is Different for Each User.png
>
>
> YARN-7245 addressed the problem that the {{Max AM Resource}} in the capacity
> scheduler UI used to contain the queue-level AM limit instead of the
> user-level AM limit. It fixed this by using the user-specific AM limit that
> is calculated in {{LeafQueue#activateApplications}}, stored in each user's
> {{LeafQueue#User}} object, and retrieved via
> {{UserInfo#getResourceUsageInfo}}.
> The problem is that this user-specific AM limit depends on the activity of
> other users and other applications in a queue, and it is only calculated and
> updated when a user's application is activated. So, when
> {{CapacitySchedulerPage}} retrieves the user-specific AM limit, it is a stale
> value unless an application was recently activated for a particular user.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
