[jira] [Commented] (MAPREDUCE-4522) DBOutputFormat Times out on large batch inserts
[ https://issues.apache.org/jira/browse/MAPREDUCE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179395#comment-15179395 ] Tsuyoshi Ozawa commented on MAPREDUCE-4522: --- In addition to the above comment, MR_DBOUTPUTFORMAT_BATCH_SIZE should be renamed as MR_DB_OUTPUT_FORMAT_BATCH_SIZE. > DBOutputFormat Times out on large batch inserts > --- > > Key: MAPREDUCE-4522 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4522 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task-controller >Affects Versions: 0.20.205.0 >Reporter: Nathan Jarus >Assignee: Shyam Gavulla > Labels: newbie > > In DBRecordWriter#close(), progress is never updated. In large batch inserts, > this can cause the reduce task to time out due to the amount of time it takes > the SQL engine to process that insert. > Potential solutions I can see: > Don't batch inserts; do the insert when DBRecordWriter#write() is called > (awful) > Spin up a thread in DBRecordWriter#close() and update progress in that. > (gross) > I can provide code for either if you're interested. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-4522) DBOutputFormat Times out on large batch inserts
[ https://issues.apache.org/jira/browse/MAPREDUCE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated MAPREDUCE-4522: -- Assignee: Shyam Gavulla (was: Nathan Jarus) > DBOutputFormat Times out on large batch inserts > --- > > Key: MAPREDUCE-4522 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4522 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task-controller >Affects Versions: 0.20.205.0 >Reporter: Nathan Jarus >Assignee: Shyam Gavulla > Labels: newbie > > In DBRecordWriter#close(), progress is never updated. In large batch inserts, > this can cause the reduce task to time out due to the amount of time it takes > the SQL engine to process that insert. > Potential solutions I can see: > Don't batch inserts; do the insert when DBRecordWriter#write() is called > (awful) > Spin up a thread in DBRecordWriter#close() and update progress in that. > (gross) > I can provide code for either if you're interested. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-4045) RM UI -> Applications -> Application Master Link -> Job Link -> New Maps/Reduces leads to circular redirect error
[
https://issues.apache.org/jira/browse/MAPREDUCE-4045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J resolved MAPREDUCE-4045.
Resolution: Duplicate
Fix Version/s: MAPREDUCE-3706
> RM UI -> Applications -> Application Master Link -> Job Link -> New
> Maps/Reduces leads to circular redirect error
> -
>
> Key: MAPREDUCE-4045
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4045
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.2
>Reporter: Devaraj K
> Fix For: MAPREDUCE-3706
>
>
> {code:xml}
> HTTP ERROR 500
> Problem accessing
> /proxy/application_1332261815858_0002/mapreduce/attempts/job_1332261815858_2_2/m/NEW.
> Reason:
> Circular redirect to
> 'http://HOST-192-168-47-207:41992/mapreduce/attempts/job_1332261815858_2_2/m/NEW'
> Caused by:
> org.apache.commons.httpclient.CircularRedirectException: Circular redirect to
> 'http://HOST-192-168-47-207:41992/mapreduce/attempts/job_1332261815858_2_2/m/NEW'
> at
> org.apache.commons.httpclient.HttpMethodDirector.processRedirectResponse(HttpMethodDirector.java:638)
> at
> org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:179)
> at
> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
> at
> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
> at
> org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.proxyLink(WebAppProxyServlet.java:148)
> at
> org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.doGet(WebAppProxyServlet.java:269)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:66)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
> at
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
> at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:940)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
> at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Powered by Jetty://
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-4045) RM UI -> Applications -> Application Master Link -> Job Link -> New Maps/Reduces leads to circular redirect error
[
https://issues.apache.org/jira/browse/MAPREDUCE-4045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated MAPREDUCE-4045:
---
Fix Version/s: (was: MAPREDUCE-3706)
> RM UI -> Applications -> Application Master Link -> Job Link -> New
> Maps/Reduces leads to circular redirect error
> -
>
> Key: MAPREDUCE-4045
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4045
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.2
>Reporter: Devaraj K
>
> {code:xml}
> HTTP ERROR 500
> Problem accessing
> /proxy/application_1332261815858_0002/mapreduce/attempts/job_1332261815858_2_2/m/NEW.
> Reason:
> Circular redirect to
> 'http://HOST-192-168-47-207:41992/mapreduce/attempts/job_1332261815858_2_2/m/NEW'
> Caused by:
> org.apache.commons.httpclient.CircularRedirectException: Circular redirect to
> 'http://HOST-192-168-47-207:41992/mapreduce/attempts/job_1332261815858_2_2/m/NEW'
> at
> org.apache.commons.httpclient.HttpMethodDirector.processRedirectResponse(HttpMethodDirector.java:638)
> at
> org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:179)
> at
> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
> at
> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
> at
> org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.proxyLink(WebAppProxyServlet.java:148)
> at
> org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.doGet(WebAppProxyServlet.java:269)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:66)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
> at
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
> at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:940)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
> at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Powered by Jetty://
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6647) MR usage counters use the resources requested instead of the resources allocated
[
https://issues.apache.org/jira/browse/MAPREDUCE-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179221#comment-15179221
]
Hadoop QA commented on MAPREDUCE-6647:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 9s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
44s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app:
patch generated 0 new + 228 unchanged - 6 fixed = 228 total (was 234) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 39s
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed with
JDK v1.8.0_74. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 41s
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed with
JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 32s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12791371/mapreduce6647.003.patch
|
| JIRA Issue | MAPREDUCE-6647 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 9a4493484bfb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (MAPREDUCE-4522) DBOutputFormat Times out on large batch inserts
[ https://issues.apache.org/jira/browse/MAPREDUCE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179088#comment-15179088 ] Tsuyoshi Ozawa commented on MAPREDUCE-4522: --- [~shyam_gav] thank you for updating. MR_DBOUTPUTFORMAT_BATCH_SIZE is DBInputFormat specific configuration. Please add the configuraition to MR_DBOUTPUTFORMAT_BATCH_SIZE instead of MRJobConfig. After addressing the comment, could you attach the patch to jira? > DBOutputFormat Times out on large batch inserts > --- > > Key: MAPREDUCE-4522 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4522 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task-controller >Affects Versions: 0.20.205.0 >Reporter: Nathan Jarus > Labels: newbie > > In DBRecordWriter#close(), progress is never updated. In large batch inserts, > this can cause the reduce task to time out due to the amount of time it takes > the SQL engine to process that insert. > Potential solutions I can see: > Don't batch inserts; do the insert when DBRecordWriter#write() is called > (awful) > Spin up a thread in DBRecordWriter#close() and update progress in that. > (gross) > I can provide code for either if you're interested. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-4522) DBOutputFormat Times out on large batch inserts
[ https://issues.apache.org/jira/browse/MAPREDUCE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated MAPREDUCE-4522: -- Assignee: Nathan Jarus > DBOutputFormat Times out on large batch inserts > --- > > Key: MAPREDUCE-4522 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4522 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task-controller >Affects Versions: 0.20.205.0 >Reporter: Nathan Jarus >Assignee: Nathan Jarus > Labels: newbie > > In DBRecordWriter#close(), progress is never updated. In large batch inserts, > this can cause the reduce task to time out due to the amount of time it takes > the SQL engine to process that insert. > Potential solutions I can see: > Don't batch inserts; do the insert when DBRecordWriter#write() is called > (awful) > Spin up a thread in DBRecordWriter#close() and update progress in that. > (gross) > I can provide code for either if you're interested. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6622) Add capability to set JHS job cache to a task-based limit
[ https://issues.apache.org/jira/browse/MAPREDUCE-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhihai xu updated MAPREDUCE-6622: - Target Version/s: 2.8.0, 2.7.3, 2.6.5 (was: 2.8.0) > Add capability to set JHS job cache to a task-based limit > - > > Key: MAPREDUCE-6622 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6622 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Affects Versions: 2.7.2 >Reporter: Ray Chiang >Assignee: Ray Chiang >Priority: Critical > Labels: supportability > Fix For: 2.9.0 > > Attachments: MAPREDUCE-6622.001.patch, MAPREDUCE-6622.002.patch, > MAPREDUCE-6622.003.patch, MAPREDUCE-6622.004.patch, MAPREDUCE-6622.005.patch, > MAPREDUCE-6622.006.patch, MAPREDUCE-6622.007.patch, MAPREDUCE-6622.008.patch, > MAPREDUCE-6622.009.patch, MAPREDUCE-6622.010.patch, MAPREDUCE-6622.011.patch, > MAPREDUCE-6622.012.patch, MAPREDUCE-6622.013.patch, MAPREDUCE-6622.014.patch > > > When setting the property mapreduce.jobhistory.loadedjobs.cache.size the jobs > can be of varying size. This is generally not a problem when the jobs sizes > are uniform or small, but when the job sizes can be very large (say greater > than 250k tasks), then the JHS heap size can grow tremendously. > In cases, where multiple jobs are very large, then the JHS can lock up and > spend all its time in GC. However, since the cache is holding on to all the > jobs, not much heap space can be freed up. > By setting a property that sets a cap on the number of tasks allowed in the > cache and since the total number of tasks loaded is directly proportional to > the amount of heap used, this should help prevent the JHS from locking up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6622) Add capability to set JHS job cache to a task-based limit
[ https://issues.apache.org/jira/browse/MAPREDUCE-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179080#comment-15179080 ] zhihai xu commented on MAPREDUCE-6622: -- Thanks for the confirmation [~rchiang]! I will back port the patch to 2.6.5 and 2.7.3 branch after waiting for several days if no one objects. > Add capability to set JHS job cache to a task-based limit > - > > Key: MAPREDUCE-6622 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6622 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Affects Versions: 2.7.2 >Reporter: Ray Chiang >Assignee: Ray Chiang >Priority: Critical > Labels: supportability > Fix For: 2.9.0 > > Attachments: MAPREDUCE-6622.001.patch, MAPREDUCE-6622.002.patch, > MAPREDUCE-6622.003.patch, MAPREDUCE-6622.004.patch, MAPREDUCE-6622.005.patch, > MAPREDUCE-6622.006.patch, MAPREDUCE-6622.007.patch, MAPREDUCE-6622.008.patch, > MAPREDUCE-6622.009.patch, MAPREDUCE-6622.010.patch, MAPREDUCE-6622.011.patch, > MAPREDUCE-6622.012.patch, MAPREDUCE-6622.013.patch, MAPREDUCE-6622.014.patch > > > When setting the property mapreduce.jobhistory.loadedjobs.cache.size the jobs > can be of varying size. This is generally not a problem when the jobs sizes > are uniform or small, but when the job sizes can be very large (say greater > than 250k tasks), then the JHS heap size can grow tremendously. > In cases, where multiple jobs are very large, then the JHS can lock up and > spend all its time in GC. However, since the cache is holding on to all the > jobs, not much heap space can be freed up. > By setting a property that sets a cap on the number of tasks allowed in the > cache and since the total number of tasks loaded is directly proportional to > the amount of heap used, this should help prevent the JHS from locking up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6647) MR usage counters use the resources requested instead of the resources allocated
[
https://issues.apache.org/jira/browse/MAPREDUCE-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibo Chen updated MAPREDUCE-6647:
--
Attachment: mapreduce6647.003.patch
updated to take care of checkstyle issue
> MR usage counters use the resources requested instead of the resources
> allocated
>
>
> Key: MAPREDUCE-6647
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6647
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: mapreduce6647.001.patch, mapreduce6647.002.patch,
> mapreduce6647.003.patch
>
>
> As can be seen in the following snippet, the MR counters for usage use the
> resources requested instead of the resources allocated. The scheduler
> increment-allocation-mb configs could lead to these values not being the
> same. We could change the counters to use the allocated resources in order to
> account for this.
> {code}
> private static void updateMillisCounters(JobCounterUpdateEvent jce,
> TaskAttemptImpl taskAttempt) {
> /***omitted**/
> long duration = (taskAttempt.getFinishTime() -
> taskAttempt.getLaunchTime());
> int mbRequired =
> taskAttempt.getMemoryRequired(taskAttempt.conf, taskType);
> int vcoresRequired = taskAttempt.getCpuRequired(taskAttempt.conf,
> taskType);
> int minSlotMemSize = taskAttempt.conf.getInt(
> YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB,
> YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB);
> int simSlotsRequired =
> minSlotMemSize == 0 ? 0 : (int) Math.ceil((float) mbRequired
> / minSlotMemSize);
> if (taskType == TaskType.MAP) {
> jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_MAPS, simSlotsRequired *
> duration);
> jce.addCounterUpdate(JobCounter.MB_MILLIS_MAPS, duration * mbRequired);
> jce.addCounterUpdate(JobCounter.VCORES_MILLIS_MAPS, duration *
> vcoresRequired);
> jce.addCounterUpdate(JobCounter.MILLIS_MAPS, duration);
> } else {
> jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_REDUCES, simSlotsRequired
> * duration);
> jce.addCounterUpdate(JobCounter.MB_MILLIS_REDUCES, duration *
> mbRequired);
> jce.addCounterUpdate(JobCounter.VCORES_MILLIS_REDUCES, duration *
> vcoresRequired);
> jce.addCounterUpdate(JobCounter.MILLIS_REDUCES, duration);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6647) MR usage counters use the resources requested instead of the resources allocated
[
https://issues.apache.org/jira/browse/MAPREDUCE-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179053#comment-15179053
]
Hadoop QA commented on MAPREDUCE-6647:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
0s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 16s
{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app:
patch generated 5 new + 228 unchanged - 6 fixed = 233 total (was 234) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 59s
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed with
JDK v1.8.0_74. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 37s
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed with
JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 20s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12791360/mapreduce6647.002.patch
|
| JIRA Issue | MAPREDUCE-6647 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 872e058693ba 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (MAPREDUCE-4785) TestMRApp occasionally fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-4785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179040#comment-15179040
]
Hudson commented on MAPREDUCE-4785:
---
FAILURE: Integrated in Hadoop-trunk-Commit #9421 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/9421/])
MAPREDUCE-4785. TestMRApp occasionally fails (haibochen via rkanter) (rkanter:
rev ff0ee84d77d9438f0954ae4e1497d63997bb7347)
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRApp.java
> TestMRApp occasionally fails
>
>
> Key: MAPREDUCE-4785
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4785
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
>Affects Versions: 2.0.3-alpha
>Reporter: Jason Lowe
>Assignee: Haibo Chen
> Fix For: 2.9.0
>
> Attachments: mapreduce4785.001.patch, mapreduce4785.prelim.patch
>
>
> TestMRApp is failing occasionally with this error:
> {noformat}
> testUpdatedNodes(org.apache.hadoop.mapreduce.v2.app.TestMRApp): Expecting 2
> more completion events for killed expected:<4> but was:<2>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-4785) TestMRApp occasionally fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-4785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated MAPREDUCE-4785:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.9.0
Status: Resolved (was: Patch Available)
Thanks Haibo and others for reviews. Committed to trunk and branch-2!
> TestMRApp occasionally fails
>
>
> Key: MAPREDUCE-4785
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4785
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, test
>Affects Versions: 2.0.3-alpha
>Reporter: Jason Lowe
>Assignee: Haibo Chen
> Fix For: 2.9.0
>
> Attachments: mapreduce4785.001.patch, mapreduce4785.prelim.patch
>
>
> TestMRApp is failing occasionally with this error:
> {noformat}
> testUpdatedNodes(org.apache.hadoop.mapreduce.v2.app.TestMRApp): Expecting 2
> more completion events for killed expected:<4> but was:<2>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6647) MR usage counters use the resources requested instead of the resources allocated
[
https://issues.apache.org/jira/browse/MAPREDUCE-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibo Chen updated MAPREDUCE-6647:
--
Attachment: mapreduce6647.002.patch
updated patch to take care of test failures in TestRecovery
> MR usage counters use the resources requested instead of the resources
> allocated
>
>
> Key: MAPREDUCE-6647
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6647
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: mapreduce6647.001.patch, mapreduce6647.002.patch
>
>
> As can be seen in the following snippet, the MR counters for usage use the
> resources requested instead of the resources allocated. The scheduler
> increment-allocation-mb configs could lead to these values not being the
> same. We could change the counters to use the allocated resources in order to
> account for this.
> {code}
> private static void updateMillisCounters(JobCounterUpdateEvent jce,
> TaskAttemptImpl taskAttempt) {
> /***omitted**/
> long duration = (taskAttempt.getFinishTime() -
> taskAttempt.getLaunchTime());
> int mbRequired =
> taskAttempt.getMemoryRequired(taskAttempt.conf, taskType);
> int vcoresRequired = taskAttempt.getCpuRequired(taskAttempt.conf,
> taskType);
> int minSlotMemSize = taskAttempt.conf.getInt(
> YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB,
> YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB);
> int simSlotsRequired =
> minSlotMemSize == 0 ? 0 : (int) Math.ceil((float) mbRequired
> / minSlotMemSize);
> if (taskType == TaskType.MAP) {
> jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_MAPS, simSlotsRequired *
> duration);
> jce.addCounterUpdate(JobCounter.MB_MILLIS_MAPS, duration * mbRequired);
> jce.addCounterUpdate(JobCounter.VCORES_MILLIS_MAPS, duration *
> vcoresRequired);
> jce.addCounterUpdate(JobCounter.MILLIS_MAPS, duration);
> } else {
> jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_REDUCES, simSlotsRequired
> * duration);
> jce.addCounterUpdate(JobCounter.MB_MILLIS_REDUCES, duration *
> mbRequired);
> jce.addCounterUpdate(JobCounter.VCORES_MILLIS_REDUCES, duration *
> vcoresRequired);
> jce.addCounterUpdate(JobCounter.MILLIS_REDUCES, duration);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6527) Data race on field org.apache.hadoop.mapred.JobConf.credentials
[
https://issues.apache.org/jira/browse/MAPREDUCE-6527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15178703#comment-15178703
]
Haibo Chen commented on MAPREDUCE-6527:
---
I believe the expected behavior for accessing the file system without proper
credential is denial of access, so I don't think this bug will allow the thread
to access files that it doesn't have access to. The credentials are used by the
output committer to make commit directories for local jobs, therefore, I think
it could potentially cause directory creations to fail. I am not quite sure
about the security setup of FileSystem in LocalJobRunner, it could totally be
possible that the security is not enabled/enforced by the File System. Anyone
knows some details of this?
> Data race on field org.apache.hadoop.mapred.JobConf.credentials
> ---
>
> Key: MAPREDUCE-6527
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6527
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Ali Kheradmand
>Assignee: Haibo Chen
> Attachments: mapreduce6527.001.patch
>
>
> I am running the test suite against a dynamic race detector called
> RV-Predict. Here is a race report that I got:
> {noformat}
> Data race on field org.apache.hadoop.mapred.JobConf.credentials: {{{
> Concurrent read in thread T327 (locks held: {})
> > at org.apache.hadoop.mapred.JobConf.getCredentials(JobConf.java:505)
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.(JobContextImpl.java:70)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:524)
> T327 is created by T22
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.(LocalJobRunner.java:218)
> Concurrent write in thread T22 (locks held: {Monitor@496c673a,
> Monitor@496319b0})
> > at org.apache.hadoop.mapred.JobConf.setCredentials(JobConf.java:510)
> at
> org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:787)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:241)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
> at
> org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
> locked Monitor@496319b0 at
> org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:n/a)
>
> at
> org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:245)
> locked Monitor@496c673a at
> org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:229)
>
> T22 is created by T1
> at
> org.apache.hadoop.mapred.jobcontrol.TestJobControl.doJobControlTest(TestJobControl.java:111)
> }}}
> {noformat}
> In the source code of org.apache.hadoop.mapreduce.JobStatus.submitJob
> function, we have the following lines:
> {code}
> Job job = new Job(JobID.downgrade(jobid), jobSubmitDir);
> job.job.setCredentials(credentials);
> {code}
> It looks a bit suspicious: Job extends thread and at the end of its
> constructor it starts a new thread which creates a new instance of
> JobContextImpl which reads credentials. However, the first thread
> concurrently sets credentials after a creating the Job instance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6527) Data race on field org.apache.hadoop.mapred.JobConf.credentials
[
https://issues.apache.org/jira/browse/MAPREDUCE-6527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15178358#comment-15178358
]
Edgar Pek commented on MAPREDUCE-6527:
--
Thank you, for the reply.
This also meant that the thread could have accessed the files it does not have
permission for because setCredentials was not set yet? Is that correct?
> Data race on field org.apache.hadoop.mapred.JobConf.credentials
> ---
>
> Key: MAPREDUCE-6527
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6527
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Ali Kheradmand
>Assignee: Haibo Chen
> Attachments: mapreduce6527.001.patch
>
>
> I am running the test suite against a dynamic race detector called
> RV-Predict. Here is a race report that I got:
> {noformat}
> Data race on field org.apache.hadoop.mapred.JobConf.credentials: {{{
> Concurrent read in thread T327 (locks held: {})
> > at org.apache.hadoop.mapred.JobConf.getCredentials(JobConf.java:505)
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.(JobContextImpl.java:70)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:524)
> T327 is created by T22
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.(LocalJobRunner.java:218)
> Concurrent write in thread T22 (locks held: {Monitor@496c673a,
> Monitor@496319b0})
> > at org.apache.hadoop.mapred.JobConf.setCredentials(JobConf.java:510)
> at
> org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:787)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:241)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
> at
> org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
> locked Monitor@496319b0 at
> org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:n/a)
>
> at
> org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:245)
> locked Monitor@496c673a at
> org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:229)
>
> T22 is created by T1
> at
> org.apache.hadoop.mapred.jobcontrol.TestJobControl.doJobControlTest(TestJobControl.java:111)
> }}}
> {noformat}
> In the source code of org.apache.hadoop.mapreduce.JobStatus.submitJob
> function, we have the following lines:
> {code}
> Job job = new Job(JobID.downgrade(jobid), jobSubmitDir);
> job.job.setCredentials(credentials);
> {code}
> It looks a bit suspicious: Job extends thread and at the end of its
> constructor it starts a new thread which creates a new instance of
> JobContextImpl which reads credentials. However, the first thread
> concurrently sets credentials after a creating the Job instance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6527) Data race on field org.apache.hadoop.mapred.JobConf.credentials
[
https://issues.apache.org/jira/browse/MAPREDUCE-6527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15178269#comment-15178269
]
Haibo Chen commented on MAPREDUCE-6527:
---
The job (LocalRunner.Job) is a thread by itself, in which the credential is
accessed by the FileSystem. If the job thread could runs before the
setCredential is called, which could potentially cause issues accessing files.
> Data race on field org.apache.hadoop.mapred.JobConf.credentials
> ---
>
> Key: MAPREDUCE-6527
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6527
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Ali Kheradmand
>Assignee: Haibo Chen
> Attachments: mapreduce6527.001.patch
>
>
> I am running the test suite against a dynamic race detector called
> RV-Predict. Here is a race report that I got:
> {noformat}
> Data race on field org.apache.hadoop.mapred.JobConf.credentials: {{{
> Concurrent read in thread T327 (locks held: {})
> > at org.apache.hadoop.mapred.JobConf.getCredentials(JobConf.java:505)
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.(JobContextImpl.java:70)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:524)
> T327 is created by T22
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.(LocalJobRunner.java:218)
> Concurrent write in thread T22 (locks held: {Monitor@496c673a,
> Monitor@496319b0})
> > at org.apache.hadoop.mapred.JobConf.setCredentials(JobConf.java:510)
> at
> org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:787)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:241)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
> at
> org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
> locked Monitor@496319b0 at
> org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:n/a)
>
> at
> org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:245)
> locked Monitor@496c673a at
> org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:229)
>
> T22 is created by T1
> at
> org.apache.hadoop.mapred.jobcontrol.TestJobControl.doJobControlTest(TestJobControl.java:111)
> }}}
> {noformat}
> In the source code of org.apache.hadoop.mapreduce.JobStatus.submitJob
> function, we have the following lines:
> {code}
> Job job = new Job(JobID.downgrade(jobid), jobSubmitDir);
> job.job.setCredentials(credentials);
> {code}
> It looks a bit suspicious: Job extends thread and at the end of its
> constructor it starts a new thread which creates a new instance of
> JobContextImpl which reads credentials. However, the first thread
> concurrently sets credentials after a creating the Job instance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
