[jira] [Commented] (MAPREDUCE-6499) Add More Informations Of Jobs In JobHistory Main Page
[ https://issues.apache.org/jira/browse/MAPREDUCE-6499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978292#comment-14978292 ] Lin Yiqun commented on MAPREDUCE-6499: -- Thanks [~ajisakaa] advices. So should I update my patch and only leave the things as you said ? > Add More Informations Of Jobs In JobHistory Main Page > - > > Key: MAPREDUCE-6499 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6499 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun >Priority: Trivial > Attachments: HADOOP-MAPREDUCE-6499.2.patch, > HADOOP-MAPREDUCE-6499.3.patch, HADOOP-MAPREDUCE-6499.4.patch, > HADOOP-MAPREDUCE-6499.5.patch, HADOOP-MAPREDUCE-6499.patch > > > Now in JobHistory Main Page show too little information about finished > jobs,even don't have the job's running total time,only > startTime,finishedTime.So,in jobHistory main page,we can add more > informations about jobs, that we can better analyze jobs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6526) Remove usage of metrics v1 from hadoop-mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978349#comment-14978349 ] Tsuyoshi Ozawa commented on MAPREDUCE-6526: --- Do you think LocalJobRunnerMetrics and ShuffleClientMetrics are not useful Metrics? IMHO, they are still useful to analyse performance. I prefer to rewrite them to use Metrics v2. Compatibility itself is second priority. However, I think third party tools use the values of counters. I also prefer to use same counter name at least. > Remove usage of metrics v1 from hadoop-mapreduce > > > Key: MAPREDUCE-6526 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6526 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA >Priority: Blocker > > LocalJobRunnerMetrics and ShuffleClientMetrics are still using metrics v1. We > should remove these metrics or rewrite them to use metrics v2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6499) Add More Informations Of Jobs In JobHistory Main Page
[ https://issues.apache.org/jira/browse/MAPREDUCE-6499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lin Yiqun updated MAPREDUCE-6499: - Attachment: HADOOP-MAPREDUCE-6499.6.patch Simplified the patch, only leave the elapsed time and modfiy some wrong hint text in search input. > Add More Informations Of Jobs In JobHistory Main Page > - > > Key: MAPREDUCE-6499 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6499 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun >Priority: Trivial > Attachments: HADOOP-MAPREDUCE-6499.2.patch, > HADOOP-MAPREDUCE-6499.3.patch, HADOOP-MAPREDUCE-6499.4.patch, > HADOOP-MAPREDUCE-6499.5.patch, HADOOP-MAPREDUCE-6499.6.patch, > HADOOP-MAPREDUCE-6499.patch > > > Now in JobHistory Main Page show too little information about finished > jobs,even don't have the job's running total time,only > startTime,finishedTime.So,in jobHistory main page,we can add more > informations about jobs, that we can better analyze jobs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6525) Fix test failure of TestMiniMRClientCluster.testRestart
[ https://issues.apache.org/jira/browse/MAPREDUCE-6525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977859#comment-14977859 ] Akira AJISAKA commented on MAPREDUCE-6525: -- Triggered Jenkins build: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6095 > Fix test failure of TestMiniMRClientCluster.testRestart > --- > > Key: MAPREDUCE-6525 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6525 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Minor > Attachments: MAPREDUCE-6525.001.patch > > > MiniMRYarnClusterAdapter#restart creates new MiniMRYarnCluster with > configuration of existing MiniMRYarnCluster but the address of HistoryServer > is properly set. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6525) Fix test failure of TestMiniMRClientCluster.testRestart
[ https://issues.apache.org/jira/browse/MAPREDUCE-6525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977857#comment-14977857 ] Akira AJISAKA commented on MAPREDUCE-6525: -- bq. The copy and logging of configuration should be done after all services are started. Make sense to me. +1 for the patch. bq. The error message of TestNonExistentJob is similar to MAPREDUCE-6524 and it implies that the fix is not enough. I could not reproduce the error locally. > Fix test failure of TestMiniMRClientCluster.testRestart > --- > > Key: MAPREDUCE-6525 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6525 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Minor > Attachments: MAPREDUCE-6525.001.patch > > > MiniMRYarnClusterAdapter#restart creates new MiniMRYarnCluster with > configuration of existing MiniMRYarnCluster but the address of HistoryServer > is properly set. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6519) hadoop-mapreduce - Avoid unsafe split and append on fields that might be IPv6 literals
[ https://issues.apache.org/jira/browse/MAPREDUCE-6519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nemanja Matkovic updated MAPREDUCE-6519: Attachment: MAPREDUCE-6519-HADOOP-11890.2.patch Identical code as in HADOOP-11890.1.patch, only difference is that YARN changes are committed. All tests passing locally as with HADOOP-11890.1.patch, re-ran tests that failed with patch 1 and they still passed, let's see if things repro again or not. > hadoop-mapreduce - Avoid unsafe split and append on fields that might be IPv6 > literals > -- > > Key: MAPREDUCE-6519 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6519 > Project: Hadoop Map/Reduce > Issue Type: Task >Reporter: Nemanja Matkovic >Assignee: Nemanja Matkovic > Labels: ipv6 > Attachments: MAPREDUCE-6519-HADOOP-11890.1.patch, > MAPREDUCE-6519-HADOOP-11890.2.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > mapreduce portion of patch in HADOOP-12122 that couldn't run due to number of > components touched. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6527) Data race on field org.apache.hadoop.mapred.JobConf.credentials
Ali Kheradmand created MAPREDUCE-6527:
-
Summary: Data race on field
org.apache.hadoop.mapred.JobConf.credentials
Key: MAPREDUCE-6527
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6527
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Ali Kheradmand
I am running the test suite against a dynamic race detector called RV-Predict.
Here is a race report that I got:
Data race on field org.apache.hadoop.mapred.JobConf.credentials: {{{
Concurrent read in thread T327 (locks held: {})
> at org.apache.hadoop.mapred.JobConf.getCredentials(JobConf.java:505)
at
org.apache.hadoop.mapreduce.task.JobContextImpl.(JobContextImpl.java:70)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:524)
T327 is created by T22
at
org.apache.hadoop.mapred.LocalJobRunner$Job.(LocalJobRunner.java:218)
Concurrent write in thread T22 (locks held: {Monitor@496c673a,
Monitor@496319b0})
> at org.apache.hadoop.mapred.JobConf.setCredentials(JobConf.java:510)
at
org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:787)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:241)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
at
org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
- locked Monitor@496319b0 at
org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:n/a)
at
org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:245)
- locked Monitor@496c673a at
org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:229)
T22 is created by T1
at
org.apache.hadoop.mapred.jobcontrol.TestJobControl.doJobControlTest(TestJobControl.java:111)
}}}
In the source code of org.apache.hadoop.mapreduce.JobStatus.submitJob function,
we have the following lines:
Job job = new Job(JobID.downgrade(jobid), jobSubmitDir);
job.job.setCredentials(credentials);
It looks a bit suspicious: Job extends thread and at the end of its constructor
it starts a new thread which creates a new instance of JobContextImpl which
reads credentials. However, the first thread concurrently sets credentials
after a creating the Job instance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6527) Data race on field org.apache.hadoop.mapred.JobConf.credentials
[
https://issues.apache.org/jira/browse/MAPREDUCE-6527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ali Kheradmand updated MAPREDUCE-6527:
--
Description:
I am running the test suite against a dynamic race detector called RV-Predict.
Here is a race report that I got:
Data race on field org.apache.hadoop.mapred.JobConf.credentials: {{{
Concurrent read in thread T327 (locks held: {})
> at org.apache.hadoop.mapred.JobConf.getCredentials(JobConf.java:505)
at
org.apache.hadoop.mapreduce.task.JobContextImpl.(JobContextImpl.java:70)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:524)
T327 is created by T22
at
org.apache.hadoop.mapred.LocalJobRunner$Job.(LocalJobRunner.java:218)
Concurrent write in thread T22 (locks held: {Monitor@496c673a,
Monitor@496319b0})
> at org.apache.hadoop.mapred.JobConf.setCredentials(JobConf.java:510)
at
org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:787)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:241)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
at
org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
locked Monitor@496319b0 at
org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:n/a)
at
org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:245)
locked Monitor@496c673a at
org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:229)
T22 is created by T1
at
org.apache.hadoop.mapred.jobcontrol.TestJobControl.doJobControlTest(TestJobControl.java:111)
}}}
In the source code of org.apache.hadoop.mapreduce.JobStatus.submitJob function,
we have the following lines:
Job job = new Job(JobID.downgrade(jobid), jobSubmitDir);
job.job.setCredentials(credentials);
It looks a bit suspicious: Job extends thread and at the end of its constructor
it starts a new thread which creates a new instance of JobContextImpl which
reads credentials. However, the first thread concurrently sets credentials
after a creating the Job instance.
was:
I am running the test suite against a dynamic race detector called RV-Predict.
Here is a race report that I got:
Data race on field org.apache.hadoop.mapred.JobConf.credentials: {{{
Concurrent read in thread T327 (locks held: {})
> at org.apache.hadoop.mapred.JobConf.getCredentials(JobConf.java:505)
at
org.apache.hadoop.mapreduce.task.JobContextImpl.(JobContextImpl.java:70)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:524)
T327 is created by T22
at
org.apache.hadoop.mapred.LocalJobRunner$Job.(LocalJobRunner.java:218)
Concurrent write in thread T22 (locks held: {Monitor@496c673a,
Monitor@496319b0})
> at org.apache.hadoop.mapred.JobConf.setCredentials(JobConf.java:510)
at
org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:787)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:241)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
at
org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
- locked Monitor@496319b0 at
org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:n/a)
at
org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:245)
- locked Monitor@496c673a at
org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:229)
T22 is created by T1
at
org.apache.hadoop.mapred.jobcontrol.TestJobControl.doJobControlTest(TestJobControl.java:111)
}}}
In the source code of org.apache.hadoop.mapreduce.JobStatus.submitJob function,
we have the following lines:
Job job = new Job(JobID.downgrade(jobid), jobSubmitDir);
job.job.setCredentials(credentials);
It looks a bit suspicious: Job extends thread and at the end of its constructor
it starts a new thread which creates a new instance of JobContextImpl which
reads credentials. However, the first thread concurrently sets credentials
after a creating the Job instance.
> Data race on field org.apache.hadoop.mapred.JobConf.credentials
> ---
>
> Key: MAPREDUCE-6527
>
[jira] [Commented] (MAPREDUCE-6519) hadoop-mapreduce - Avoid unsafe split and append on fields that might be IPv6 literals
[
https://issues.apache.org/jira/browse/MAPREDUCE-6519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14979490#comment-14979490
]
Hadoop QA commented on MAPREDUCE-6519:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
59s {color} | {color:green} HADOOP-11890 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 55s
{color} | {color:green} HADOOP-11890 passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 54s
{color} | {color:green} HADOOP-11890 passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s {color} | {color:green} HADOOP-11890 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
2s {color} | {color:green} HADOOP-11890 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
57s {color} | {color:green} HADOOP-11890 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 10s
{color} | {color:green} HADOOP-11890 passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s
{color} | {color:green} HADOOP-11890 passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 53s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 48s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 12s {color}
| {color:red} hadoop-mapreduce-client-core in the patch failed with JDK
v1.8.0_60. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m 54s
{color} | {color:green} hadoop-mapreduce-client-hs in the patch passed with JDK
v1.8.0_60. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 118m 9s {color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed with JDK
v1.8.0_60. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 19s {color}
| {color:red} hadoop-mapreduce-client-core in the patch failed with JDK
v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m 31s
{color} | {color:green} hadoop-mapreduce-client-hs in the patch passed with JDK
v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 116m 31s
{color} | {color:red} hadoop-mapreduce-client-jobclient in the patch failed
with JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 31s
{color} | {color:red} Patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 291m 35s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.7.0_79 Failed junit tests |
hadoop.mapreduce.filecache.TestClientDistributedCacheManager |
| |
[jira] [Commented] (MAPREDUCE-6525) Fix test failure of TestMiniMRClientCluster.testRestart
[
https://issues.apache.org/jira/browse/MAPREDUCE-6525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14979525#comment-14979525
]
Masatake Iwasaki commented on MAPREDUCE-6525:
-
bq. .. we should use Configuration to enable timeline server instead of using
enableAHS
I agree but the constructer has already marked as deprecated by YARN-3573. I
think there is nothing we can do more. We should not remove public constructor
of mini cluster because it is possible to be used by downstream projects.
This jira is addressing the issue about not AHS but JHS in MiniMRYarnCluster:-)
Though {{JHAdminConfig}} does not have on/off switch like
{{YarnConfiguration.TIMELINE_SERVICE_ENABLED}}, I don't think we need to add it.
> Fix test failure of TestMiniMRClientCluster.testRestart
> ---
>
> Key: MAPREDUCE-6525
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6525
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Minor
> Attachments: MAPREDUCE-6525.001.patch
>
>
> MiniMRYarnClusterAdapter#restart creates new MiniMRYarnCluster with
> configuration of existing MiniMRYarnCluster but the address of HistoryServer
> is properly set.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6527) Data race on field org.apache.hadoop.mapred.JobConf.credentials
[
https://issues.apache.org/jira/browse/MAPREDUCE-6527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ali Kheradmand updated MAPREDUCE-6527:
--
Description:
I am running the test suite against a dynamic race detector called RV-Predict.
Here is a race report that I got:
{noformat}
Data race on field org.apache.hadoop.mapred.JobConf.credentials: {{{
Concurrent read in thread T327 (locks held: {})
> at org.apache.hadoop.mapred.JobConf.getCredentials(JobConf.java:505)
at
org.apache.hadoop.mapreduce.task.JobContextImpl.(JobContextImpl.java:70)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:524)
T327 is created by T22
at
org.apache.hadoop.mapred.LocalJobRunner$Job.(LocalJobRunner.java:218)
Concurrent write in thread T22 (locks held: {Monitor@496c673a,
Monitor@496319b0})
> at org.apache.hadoop.mapred.JobConf.setCredentials(JobConf.java:510)
at
org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:787)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:241)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
at
org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
locked Monitor@496319b0 at
org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:n/a)
at
org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:245)
locked Monitor@496c673a at
org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:229)
T22 is created by T1
at
org.apache.hadoop.mapred.jobcontrol.TestJobControl.doJobControlTest(TestJobControl.java:111)
}}}
{noformat}
In the source code of org.apache.hadoop.mapreduce.JobStatus.submitJob function,
we have the following lines:
{code}
Job job = new Job(JobID.downgrade(jobid), jobSubmitDir);
job.job.setCredentials(credentials);
{code}
It looks a bit suspicious: Job extends thread and at the end of its constructor
it starts a new thread which creates a new instance of JobContextImpl which
reads credentials. However, the first thread concurrently sets credentials
after a creating the Job instance.
was:
I am running the test suite against a dynamic race detector called RV-Predict.
Here is a race report that I got:
Data race on field org.apache.hadoop.mapred.JobConf.credentials: {{{
Concurrent read in thread T327 (locks held: {})
> at org.apache.hadoop.mapred.JobConf.getCredentials(JobConf.java:505)
at
org.apache.hadoop.mapreduce.task.JobContextImpl.(JobContextImpl.java:70)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:524)
T327 is created by T22
at
org.apache.hadoop.mapred.LocalJobRunner$Job.(LocalJobRunner.java:218)
Concurrent write in thread T22 (locks held: {Monitor@496c673a,
Monitor@496319b0})
> at org.apache.hadoop.mapred.JobConf.setCredentials(JobConf.java:510)
at
org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:787)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:241)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
at
org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
locked Monitor@496319b0 at
org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:n/a)
at
org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:245)
locked Monitor@496c673a at
org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:229)
T22 is created by T1
at
org.apache.hadoop.mapred.jobcontrol.TestJobControl.doJobControlTest(TestJobControl.java:111)
}}}
In the source code of org.apache.hadoop.mapreduce.JobStatus.submitJob function,
we have the following lines:
Job job = new Job(JobID.downgrade(jobid), jobSubmitDir);
job.job.setCredentials(credentials);
It looks a bit suspicious: Job extends thread and at the end of its constructor
it starts a new thread which creates a new instance of JobContextImpl which
reads credentials. However, the first thread concurrently sets credentials
after a creating the Job instance.
> Data race on field org.apache.hadoop.mapred.JobConf.credentials
> ---
>
> Key:
[jira] [Commented] (MAPREDUCE-6525) Fix test failure of TestMiniMRClientCluster.testRestart
[
https://issues.apache.org/jira/browse/MAPREDUCE-6525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977991#comment-14977991
]
Tsuyoshi Ozawa commented on MAPREDUCE-6525:
---
This is not related to this issue directly, but we should use Configuration to
enable timeline server instead of using enableAHS.
{code}
public MiniMRYarnCluster(String testName, int noOfNMs, boolean enableAHS) {
super(testName, 1, noOfNMs, 4, 4, enableAHS);
}
{code}
> Fix test failure of TestMiniMRClientCluster.testRestart
> ---
>
> Key: MAPREDUCE-6525
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6525
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Minor
> Attachments: MAPREDUCE-6525.001.patch
>
>
> MiniMRYarnClusterAdapter#restart creates new MiniMRYarnCluster with
> configuration of existing MiniMRYarnCluster but the address of HistoryServer
> is properly set.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6526) Remove usage of metrics v1 from hadoop-mapreduce
Akira AJISAKA created MAPREDUCE-6526: Summary: Remove usage of metrics v1 from hadoop-mapreduce Key: MAPREDUCE-6526 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6526 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Akira AJISAKA Priority: Blocker LocalJobRunnerMetrics and ShuffleClientMetrics are still using metrics v1. We should remove these metrics or rewrite them to use metrics v2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (MAPREDUCE-6526) Remove usage of metrics v1 from hadoop-mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA reassigned MAPREDUCE-6526: Assignee: Akira AJISAKA > Remove usage of metrics v1 from hadoop-mapreduce > > > Key: MAPREDUCE-6526 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6526 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA >Priority: Blocker > > LocalJobRunnerMetrics and ShuffleClientMetrics are still using metrics v1. We > should remove these metrics or rewrite them to use metrics v2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
