[jira] [Commented] (YARN-10061) job historyserver old gen may be 100% when too many jobs load history
[
https://issues.apache.org/jira/browse/YARN-10061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17003561#comment-17003561
]
zhoukang commented on YARN-10061:
-
job below generated 170+ requests, we should add a filter for the same job
which is replaying
{code:java}
/jobhistory/job/job_1576831312050_362973/mapreduce/job/job_1576831312050_362973
{code}
come from the same browser
{code:java}
GET
/jobhistory/job/job_1576831312050_362973/mapreduce/job/job_1576831312050_362973
HTTP/1.1..Connection: upgrade..X-Real-IP: 10.232.22.174..X-Forwarded-For:
10.232.22.174..Host: zjy-hadoop-prc-ct11.bj:20901..User-Agent: Mozilla/5.0
(X11; Ubuntu; Linux x86_64; rv:71.0) Gecko/20100101 Firefox/71.0..Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8..Accept-Language:
en-US,en;q=0.5..Accept-Encoding: gzip, deflate..Referer:
http://zjy-hadoop-prc-ct11.bj:21001/proxy/application_1576831312050_362973/?proxyapproved=true..Upgrade-Insecure-Requests:
1..
{code}
> job historyserver old gen may be 100% when too many jobs load history
> -
>
> Key: YARN-10061
> URL: https://issues.apache.org/jira/browse/YARN-10061
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: zhoukang
>Assignee: zhoukang
>Priority: Major
> Attachments: 001.png, 002.png, 003.png
>
>
> !003.png!
> {code:java}
> [work@zjy-hadoop-prc-ct11 log]$ jstat -gcutil 26774
> S0 S1 E O M CCSYGC YGCTFGCFGCT GCT
> 0.00 99.99 100.00 100.00 98.13 96.35 10999 2786.664 497 989.782
> 3776.446
> {code}
> {code:java}
> hread 1058215567@qtp-1107509430-6121
> Thread Properties
> Object / Stack Frame org.mortbay.thread.QueuedThreadPool$PoolThread @
> 0x7606db678
> Name 1058215567@qtp-1107509430-6121
> Shallow Heap 0.00 MB
> Retained Heap 0.17 MB
> Context Class Loader jobhistory
> Is Daemon true
> Total: 6 entries
> Thread Stack
> 1058215567@qtp-1107509430-6121
> at
> org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager$UserLogDir.scanIfNeeded(Lorg/apache/hadoop/fs/FileStatus;)V
> (HistoryFileManager.java:278)
> at
> org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.scanIntermediateDirectory()V
> (HistoryFileManager.java:798)
> at
> org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.getFileInfo(Lorg/apache/hadoop/mapreduce/v2/api/records/JobId;)Lorg/apache/hadoop/mapreduce/v2/hs/HistoryFileManager$HistoryFileInfo;
> (HistoryFileManager.java:948)
> at
> org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getFullJob(Lorg/apache/hadoop/mapreduce/v2/api/records/JobId;)Lorg/apache/hadoop/mapreduce/v2/app/job/Job;
> (CachedHistoryStorage.java:135)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(Lorg/apache/hadoop/mapreduce/v2/api/records/JobId;)Lorg/apache/hadoop/mapreduce/v2/app/job/Job;
> (JobHistory.java:221)
> at org.apache.hadoop.mapreduce.v2.app.webapp.AppController.requireJob()V
> (AppController.java:382)
> at org.apache.hadoop.mapreduce.v2.app.webapp.AppController.job()V
> (AppController.java:109)
> at org.apache.hadoop.mapreduce.v2.hs.webapp.HsController.job()V
> (HsController.java:104)
> at
> sun.reflect.GeneratedMethodAccessor30.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
> (Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
> (DelegatingMethodAccessorImpl.java:43)
> at
> java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
> (Method.java:498)
> at
> org.apache.hadoop.yarn.webapp.Dispatcher.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V
> (Dispatcher.java:153)
> at
> javax.servlet.http.HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V
> (HttpServlet.java:820)
> at
> com.google.inject.servlet.ServletDefinition.doService(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V
> (ServletDefinition.java:263)
> at
> com.google.inject.servlet.ServletDefinition.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)Z
> (ServletDefinition.java:178)
> at
> com.google.inject.servlet.ManagedServletPipeline.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)Z
>
[jira] [Commented] (YARN-10061) job historyserver old gen may be 100% when too many jobs load history
[
https://issues.apache.org/jira/browse/YARN-10061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17003558#comment-17003558
]
zhoukang commented on YARN-10061:
-
{code:java}
Start Page
Table Of Contents
Thread 1058215567@qtp-1107509430-6121
Thread Properties
Thread Stack
Requests
The thread is executing an HTTP Request to
/jobhistory/job/job_1576831312050_362973/mapreduce/job/job_1576831312050_362973.
Summary
URI
Thread 1333459277@qtp-1107509430-6120
Thread Properties
Thread Stack
Requests
The thread is executing an HTTP Request to
/jobhistory/job/job_1576831312050_362973/mapreduce/job/job_1576831312050_362973.
Summary
URI
Thread 582278496@qtp-1107509430-6119
Thread Properties
Thread Stack
Requests
The thread is executing an HTTP Request to
/jobhistory/attempts/job_1576831312050_347354/r/KILLED.
Summary
URI
Thread 1010032540@qtp-1107509430-6118
Thread Properties
Thread Stack
Requests
The thread is executing an HTTP Request to
/jobhistory/job/job_1576831312050_362973/mapreduce/job/job_1576831312050_362973.
Summary
URI
Thread 2057128499@qtp-1107509430-6117
Thread Properties
Thread Stack
Requests
The thread is executing an HTTP Request to
/jobhistory/job/job_1576831312050_362973/mapreduce/job/job_1576831312050_362973.
Summary
URI
Thread 1230320783@qtp-1107509430-6116
Thread Properties
Thread Stack
Requests
The thread is executing an HTTP Request to
/jobhistory/job/job_1576831312050_362973/mapreduce/job/job_1576831312050_362973.
Summary
URI
Thread 787412968@qtp-1107509430-6115
Thread Properties
Thread Stack
Requests
The thread is executing an HTTP Request to
/jobhistory/job/job_1576831312050_362973/mapreduce/job/job_1576831312050_362973.
Summary
URI
Thread 328094070@qtp-1107509430-6114
Thread Properties
Thread Stack
Requests
The thread is executing an HTTP Request to
/jobhistory/job/job_1576831312050_362973/mapreduce/job/job_1576831312050_362973.
Summary
URI
Thread 284896606@qtp-1107509430-6113
Thread Properties
Thread Stack
Requests
The thread is executing an HTTP Request to
/jobhistory/job/job_1576831312050_362973/mapreduce/job/job_1576831312050_362973.
Summary
URI
Thread 1764513565@qtp-1107509430-6112
Thread Properties
Thread Stack
Requests
The thread is executing an HTTP Request to
/jobhistory/attempts/job_1576831312050_347314/m/KILLED.
Summary
URI
Thread 1013350884@qtp-1107509430-6111
Thread Properties
Thread Stack
Requests
The thread is executing an HTTP Request to
/jobhistory/job/job_1576831312050_362973/mapreduce/job/job_1576831312050_362973.
Summary
URI
Thread 2030348115@qtp-1107509430-6110
Thread Properties
Thread Stack
Requests
The thread is executing an HTTP Request to
/jobhistory/job/job_1576831312050_362973/mapreduce/job/job_1576831312050_362973.
Summary
URI
Thread 1609530906@qtp-1107509430-6109
Thread Properties
Thread Stack
Requests
The thread is executing an HTTP Request to
/jobhistory/job/job_1576831312050_362973/mapreduce/job/job_1576831312050_362973.
Summary
URI
Thread 2112512892@qtp-1107509430-6108
Thread Properties
Thread Stack
Requests
The thread is executing an HTTP Request to
/jobhistory/job/job_1576831312050_362973/mapreduce/job/job_1576831312050_362973.
Summary
URI
Thread 1508380482@qtp-1107509430-6107
Thread Properties
Thread Stack
Requests
The thread is executing an HTTP Request to
/jobhistory/job/job_1576831312050_362973/mapreduce/job/job_1576831312050_362973.
Summary
URI
Thread 2066251373@qtp-1107509430-6106
Thread Properties
Thread Stack
Requests
The thread is executing an HTTP Request to
/jobhistory/job/job_1576831312050_362973/mapreduce/job/job_1576831312050_362973.
Summary
URI
Thread 367949850@qtp-1107509430-6105
Thread Properties
Thread Stack
Requests
The thread is executing an HTTP Request to
/jobhistory/job/job_1576831312050_362973/mapreduce/job/job_1576831312050_362973.
Summary
URI
Thread 626277387@qtp-1107509430-6104
Thread Properties
Thread Stack
Requests
The thread is executing an HTTP Request to
/jobhistory/job/job_1576831312050_362973/mapreduce/job/job_1576831312050_362973.
Summary
URI
Thread 515689957@qtp-1107509430-6103
Thread Properties
Thread Stack
Requests
The thread is executing an HTTP Request to
/jobhistory/attempts/job_1576831312050_347325/r/KILLED.
Summary
URI
Thread 2097370166@qtp-1107509430-6102
Thread Properties
Thread Stack
Requests
The thread is executing an HTTP Request to
/jobhistory/attempts/job_1576831312050_347313/m/KILLED.
Summary
URI
Thread 1680793908@qtp-1107509430-6101
Thread Properties
Thread Stack
Requests
The thread is executing an HTTP Request to
/jobhistory/job/job_1576831312050_362973/mapreduce/job/job_1576831312050_362973.
Summary
URI
Thread 1425331186@qtp-1107509430-6100
Thread Properties
Thread Stack
Requests
The thread is executing an HTTP Request to
/jobhistory/attempts/job_1576831312050_349232/r/KILLED.
Summary
URI
Thread 1797324868@qtp-1107509430-6099
Thread Properties
Thread Stack
Requests
The thread is executing an HTTP Request to
[jira] [Created] (YARN-10061) job historyserver old gen may be 100% when too many jobs load history
zhoukang created YARN-10061:
---
Summary: job historyserver old gen may be 100% when too many jobs
load history
Key: YARN-10061
URL: https://issues.apache.org/jira/browse/YARN-10061
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Reporter: zhoukang
Assignee: zhoukang
Attachments: 001.png, 002.png, 003.png
!003.png!
{code:java}
[work@zjy-hadoop-prc-ct11 log]$ jstat -gcutil 26774
S0 S1 E O M CCSYGC YGCTFGCFGCT GCT
0.00 99.99 100.00 100.00 98.13 96.35 10999 2786.664 497 989.782
3776.446
{code}
{code:java}
hread 1058215567@qtp-1107509430-6121
Thread Properties
Object / Stack Frameorg.mortbay.thread.QueuedThreadPool$PoolThread @
0x7606db678
Name1058215567@qtp-1107509430-6121
Shallow Heap0.00 MB
Retained Heap 0.17 MB
Context Class Loaderjobhistory
Is Daemon true
Total: 6 entries
Thread Stack
1058215567@qtp-1107509430-6121
at
org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager$UserLogDir.scanIfNeeded(Lorg/apache/hadoop/fs/FileStatus;)V
(HistoryFileManager.java:278)
at
org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.scanIntermediateDirectory()V
(HistoryFileManager.java:798)
at
org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.getFileInfo(Lorg/apache/hadoop/mapreduce/v2/api/records/JobId;)Lorg/apache/hadoop/mapreduce/v2/hs/HistoryFileManager$HistoryFileInfo;
(HistoryFileManager.java:948)
at
org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getFullJob(Lorg/apache/hadoop/mapreduce/v2/api/records/JobId;)Lorg/apache/hadoop/mapreduce/v2/app/job/Job;
(CachedHistoryStorage.java:135)
at
org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(Lorg/apache/hadoop/mapreduce/v2/api/records/JobId;)Lorg/apache/hadoop/mapreduce/v2/app/job/Job;
(JobHistory.java:221)
at org.apache.hadoop.mapreduce.v2.app.webapp.AppController.requireJob()V
(AppController.java:382)
at org.apache.hadoop.mapreduce.v2.app.webapp.AppController.job()V
(AppController.java:109)
at org.apache.hadoop.mapreduce.v2.hs.webapp.HsController.job()V
(HsController.java:104)
at
sun.reflect.GeneratedMethodAccessor30.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
(DelegatingMethodAccessorImpl.java:43)
at
java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
(Method.java:498)
at
org.apache.hadoop.yarn.webapp.Dispatcher.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V
(Dispatcher.java:153)
at
javax.servlet.http.HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V
(HttpServlet.java:820)
at
com.google.inject.servlet.ServletDefinition.doService(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V
(ServletDefinition.java:263)
at
com.google.inject.servlet.ServletDefinition.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)Z
(ServletDefinition.java:178)
at
com.google.inject.servlet.ManagedServletPipeline.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)Z
(ManagedServletPipeline.java:91)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V
(FilterChainInvocation.java:62)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Ljavax/servlet/FilterChain;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)V
(ServletContainer.java:900)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Ljavax/servlet/FilterChain;)V
(ServletContainer.java:834)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V
(ServletContainer.java:795)
at
com.google.inject.servlet.FilterDefinition.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Lcom/google/inject/servlet/FilterChainInvocation;)V
(FilterDefinition.java:163)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V
(FilterChainInvocation.java:58)
at
com.google.inject.servlet.ManagedFilterPipeline.dispatch(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V
(ManagedFilterPipeline.java:118)
at
com.google.inject.servlet.GuiceFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V
(GuiceFilter.java:113)
at
