[jira] [Updated] (YARN-3348) Add a 'yarn top' tool to help understand cluster usage
[ https://issues.apache.org/jira/browse/YARN-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-3348: Attachment: apache-yarn-3348.5.patch Uploaded a new patch fixing the scrolling issue in the sort, fields screen. Add a 'yarn top' tool to help understand cluster usage -- Key: YARN-3348 URL: https://issues.apache.org/jira/browse/YARN-3348 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager Reporter: Varun Vasudev Assignee: Varun Vasudev Attachments: apache-yarn-3348.0.patch, apache-yarn-3348.1.patch, apache-yarn-3348.2.patch, apache-yarn-3348.3.patch, apache-yarn-3348.4.patch, apache-yarn-3348.5.patch It would be helpful to have a 'yarn top' tool that would allow administrators to understand which apps are consuming resources. Ideally the tool would allow you to filter by queue, user, maybe labels, etc and show you statistics on container allocation across the cluster to find out which apps are consuming the most resources on the cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3348) Add a 'yarn top' tool to help understand cluster usage
[ https://issues.apache.org/jira/browse/YARN-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-3348: Attachment: apache-yarn-3348.4.patch Uploaded a new patch which fixes an issue with yarn top output not clearing itself correctly. Add a 'yarn top' tool to help understand cluster usage -- Key: YARN-3348 URL: https://issues.apache.org/jira/browse/YARN-3348 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager Reporter: Varun Vasudev Assignee: Varun Vasudev Attachments: apache-yarn-3348.0.patch, apache-yarn-3348.1.patch, apache-yarn-3348.2.patch, apache-yarn-3348.3.patch, apache-yarn-3348.4.patch It would be helpful to have a 'yarn top' tool that would allow administrators to understand which apps are consuming resources. Ideally the tool would allow you to filter by queue, user, maybe labels, etc and show you statistics on container allocation across the cluster to find out which apps are consuming the most resources on the cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3348) Add a 'yarn top' tool to help understand cluster usage
[ https://issues.apache.org/jira/browse/YARN-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-3348: Attachment: apache-yarn-3348.5.patch Add a 'yarn top' tool to help understand cluster usage -- Key: YARN-3348 URL: https://issues.apache.org/jira/browse/YARN-3348 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager Reporter: Varun Vasudev Assignee: Varun Vasudev Attachments: apache-yarn-3348.0.patch, apache-yarn-3348.1.patch, apache-yarn-3348.2.patch, apache-yarn-3348.3.patch, apache-yarn-3348.4.patch, apache-yarn-3348.5.patch It would be helpful to have a 'yarn top' tool that would allow administrators to understand which apps are consuming resources. Ideally the tool would allow you to filter by queue, user, maybe labels, etc and show you statistics on container allocation across the cluster to find out which apps are consuming the most resources on the cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3348) Add a 'yarn top' tool to help understand cluster usage
[ https://issues.apache.org/jira/browse/YARN-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-3348: Attachment: (was: apache-yarn-3348.5.patch) Add a 'yarn top' tool to help understand cluster usage -- Key: YARN-3348 URL: https://issues.apache.org/jira/browse/YARN-3348 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager Reporter: Varun Vasudev Assignee: Varun Vasudev Attachments: apache-yarn-3348.0.patch, apache-yarn-3348.1.patch, apache-yarn-3348.2.patch, apache-yarn-3348.3.patch, apache-yarn-3348.4.patch It would be helpful to have a 'yarn top' tool that would allow administrators to understand which apps are consuming resources. Ideally the tool would allow you to filter by queue, user, maybe labels, etc and show you statistics on container allocation across the cluster to find out which apps are consuming the most resources on the cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3348) Add a 'yarn top' tool to help understand cluster usage
[ https://issues.apache.org/jira/browse/YARN-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-3348: Attachment: apache-yarn-3348.3.patch bq. In YarnClusterMetricsPBImpl, should the default num*NodeManagers return 0 ? Fixed. bq. getApplications in YarnClient.java may be an abstract method? It's a public class. Adding an abstract method will break compatibility. bq. “Queue Applications:” - if it’s aggregated number , maybe Queue(s) ? Fixed. Some other changes: # Based on an offline conversation with Jian, I've moved the app reports cache into TopCLI itself for now. # Improved help # The queue memory statistics are in GB instead of MB. Add a 'yarn top' tool to help understand cluster usage -- Key: YARN-3348 URL: https://issues.apache.org/jira/browse/YARN-3348 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager Reporter: Varun Vasudev Assignee: Varun Vasudev Attachments: apache-yarn-3348.0.patch, apache-yarn-3348.1.patch, apache-yarn-3348.2.patch, apache-yarn-3348.3.patch It would be helpful to have a 'yarn top' tool that would allow administrators to understand which apps are consuming resources. Ideally the tool would allow you to filter by queue, user, maybe labels, etc and show you statistics on container allocation across the cluster to find out which apps are consuming the most resources on the cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3348) Add a 'yarn top' tool to help understand cluster usage
[
https://issues.apache.org/jira/browse/YARN-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Vasudev updated YARN-3348:
Attachment: apache-yarn-3348.2.patch
Thanks for the reviews [~aw] and [~jianhe].
bq. Why are we doing this manipulation here and not in the Java code?
I get different values when I run the command in the yarn script vs spawn it
via Java. From Java, I get lower values - 80x24, whereas the yarn script gives
me 204x44.
bq. backticks are antiquated in modern bash. Use $() construction
Fixed.
bq. What happens if tput gives you zero or an error because you are on a
non-addressable terminal? (You can generally simulate this by unset TERM or
equivalent env var)
Thank you for pointing this out. I hadn't considered it. I've added additional
checks in the script. If the values can't be determined either by the script or
by the Java code, it sets it to 80x24.
bq. “Unable to fetach cluster metrics” - typo
Fixed.
bq. exceeding 80 Column limit,
Fixed.
bq. the -rows, -cols options seems not having effect on my screen when I tried
it, could you double check ?
I found an issue with cols option which I've fixed. Can you please try it again?
bq. the ‘yarn top’ output is repeatedly showing up on terminal every $delay
seconds. it’ll be better to only show that only once.
I didn't understand this - do you mean that it shouldn't auto-refresh?
bq. Does the patch only show root queue info ? should we show all queues info ?
Queues can be specified as a comma seperated string using the -queues option.
By default, it shows information for the root queue.
bq. “F + Enter : Select sort field” ; may be use ’S’ for sorting ?
Fixed.
bq. “Memory seconds(in GBseconds” - missing “)”
Fixed
{quote}
It seems a bit odd to have this method in a public API record. Do you know why
hashcode is not correct without this method ? Or we can just type cast it to
GetApplicationsRequestPBImpl and use the method from there.
// need this otherwise the hashcode doesn't get generated correctly
request.initAllFields();
for the caching in ClientRMService. Do you think we can do the cache on
client side ? that’ll save RPCs, especially if we have many top commands
running on client side.
{quote}
Fixed. Moved the cache to YarnClientImpl where the hashcode doesn't show up. As
to why it wasn't correct - I suspect it might be to do with lazy initialization
but I'm not sure.
Add a 'yarn top' tool to help understand cluster usage
--
Key: YARN-3348
URL: https://issues.apache.org/jira/browse/YARN-3348
Project: Hadoop YARN
Issue Type: Improvement
Components: resourcemanager
Reporter: Varun Vasudev
Assignee: Varun Vasudev
Attachments: apache-yarn-3348.0.patch, apache-yarn-3348.1.patch,
apache-yarn-3348.2.patch
It would be helpful to have a 'yarn top' tool that would allow administrators
to understand which apps are consuming resources.
Ideally the tool would allow you to filter by queue, user, maybe labels, etc
and show you statistics on container allocation across the cluster to find
out which apps are consuming the most resources on the cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3348) Add a 'yarn top' tool to help understand cluster usage
[ https://issues.apache.org/jira/browse/YARN-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-3348: Attachment: apache-yarn-3348.0.patch Uploaded initial version of the patch. Most of the work is in a new TopCLI class. I added an application reports cache in ClientRMService with a timeout of 5 seconds as well as a boolean in GetApplicationsRequest to fetch cached versions of the reports. The tool essentially prints out the application report. The default refresh rate is 3 seconds. Add a 'yarn top' tool to help understand cluster usage -- Key: YARN-3348 URL: https://issues.apache.org/jira/browse/YARN-3348 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager Reporter: Varun Vasudev Assignee: Varun Vasudev Attachments: apache-yarn-3348.0.patch It would be helpful to have a 'yarn top' tool that would allow administrators to understand which apps are consuming resources. Ideally the tool would allow you to filter by queue, user, maybe labels, etc and show you statistics on container allocation across the cluster to find out which apps are consuming the most resources on the cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3348) Add a 'yarn top' tool to help understand cluster usage
[ https://issues.apache.org/jira/browse/YARN-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-3348: Attachment: apache-yarn-3348.1.patch Uploaded a new patch to fix release audit warning and failing test. The findbugs warning is from another test. Add a 'yarn top' tool to help understand cluster usage -- Key: YARN-3348 URL: https://issues.apache.org/jira/browse/YARN-3348 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager Reporter: Varun Vasudev Assignee: Varun Vasudev Attachments: apache-yarn-3348.0.patch, apache-yarn-3348.1.patch It would be helpful to have a 'yarn top' tool that would allow administrators to understand which apps are consuming resources. Ideally the tool would allow you to filter by queue, user, maybe labels, etc and show you statistics on container allocation across the cluster to find out which apps are consuming the most resources on the cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
