[jira] [Resolved] (MAPREDUCE-6531) CLONE - Mumak: Map-Reduce Simulator
[ https://issues.apache.org/jira/browse/MAPREDUCE-6531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla resolved MAPREDUCE-6531. - Resolution: Won't Fix Resolving as "Won't Fix". > CLONE - Mumak: Map-Reduce Simulator > --- > > Key: MAPREDUCE-6531 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6531 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 0.21.0 >Reporter: GD >Assignee: Hong Tang > Fix For: 0.21.0 > > Attachments: 19-jobs.topology.json.gz, 19-jobs.trace.json.gz, > mapreduce-728-20090917-3.patch, mapreduce-728-20090917-4.patch, > mapreduce-728-20090917.patch, mapreduce-728-20090918-2.patch, > mapreduce-728-20090918-3.patch, mapreduce-728-20090918-5.patch, > mapreduce-728-20090918-6.patch, mapreduce-728-20090918.patch, mumak.png > > > h3. Vision: > We want to build a Simulator to simulate large-scale Hadoop clusters, > applications and workloads. This would be invaluable in furthering Hadoop by > providing a tool for researchers and developers to prototype features (e.g. > pluggable block-placement for HDFS, Map-Reduce schedulers etc.) and predict > their behaviour and performance with reasonable amount of confidence, > there-by aiding rapid innovation. > > h3. First Cut: Simulator for the Map-Reduce Scheduler > The Map-Reduce Scheduler is a fertile area of interest with at least four > schedulers, each with their own set of features, currently in existence: > Default Scheduler, Capacity Scheduler, Fairshare Scheduler & Priority > Scheduler. > Each scheduler's scheduling decisions are driven by many factors, such as > fairness, capacity guarantee, resource availability, data-locality etc. > Given that, it is non-trivial to accurately choose a single scheduler or even > a set of desired features to predict the right scheduler (or features) for a > given workload. Hence a simulator which can predict how well a particular > scheduler works for some specific workload by quickly iterating over > schedulers and/or scheduler features would be quite useful. > So, the first cut is to implement a simulator for the Map-Reduce scheduler > which take as input a job trace derived from production workload and a > cluster definition, and simulates the execution of the jobs in as defined in > the trace in this virtual cluster. As output, the detailed job execution > trace (recorded in relation to virtual simulated time) could then be analyzed > to understand various traits of individual schedulers (individual jobs turn > around time, throughput, faireness, capacity guarantee, etc). To support > this, we would need a simulator which could accurately model the conditions > of the actual system which would affect a schedulers decisions. These include > very large-scale clusters (thousands of nodes), the detailed characteristics > of the workload thrown at the clusters, job or task failures, data locality, > and cluster hardware (cpu, memory, disk i/o, network i/o, network topology) > etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6519) Avoid unsafe split and append on fields that might be IPv6 literals
[ https://issues.apache.org/jira/browse/MAPREDUCE-6519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14990407#comment-14990407 ] Elliott Clark commented on MAPREDUCE-6519: -- +1 > Avoid unsafe split and append on fields that might be IPv6 literals > --- > > Key: MAPREDUCE-6519 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6519 > Project: Hadoop Map/Reduce > Issue Type: Task >Reporter: Nemanja Matkovic >Assignee: Nemanja Matkovic > Labels: ipv6 > Attachments: MAPREDUCE-6519-HADOOP-11890.1.patch, > MAPREDUCE-6519-HADOOP-11890.2.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > mapreduce portion of patch in HADOOP-12122 that couldn't run due to number of > components touched. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6519) Avoid unsafe split and append on fields that might be IPv6 literals
[ https://issues.apache.org/jira/browse/MAPREDUCE-6519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated MAPREDUCE-6519: - Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to branch. Thanks > Avoid unsafe split and append on fields that might be IPv6 literals > --- > > Key: MAPREDUCE-6519 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6519 > Project: Hadoop Map/Reduce > Issue Type: Task >Reporter: Nemanja Matkovic >Assignee: Nemanja Matkovic > Labels: ipv6 > Attachments: MAPREDUCE-6519-HADOOP-11890.1.patch, > MAPREDUCE-6519-HADOOP-11890.2.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > mapreduce portion of patch in HADOOP-12122 that couldn't run due to number of > components touched. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6519) Avoid unsafe split and append on fields that might be IPv6 literals
[ https://issues.apache.org/jira/browse/MAPREDUCE-6519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated MAPREDUCE-6519: - Summary: Avoid unsafe split and append on fields that might be IPv6 literals (was: hadoop-mapreduce - Avoid unsafe split and append on fields that might be IPv6 literals) > Avoid unsafe split and append on fields that might be IPv6 literals > --- > > Key: MAPREDUCE-6519 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6519 > Project: Hadoop Map/Reduce > Issue Type: Task >Reporter: Nemanja Matkovic >Assignee: Nemanja Matkovic > Labels: ipv6 > Attachments: MAPREDUCE-6519-HADOOP-11890.1.patch, > MAPREDUCE-6519-HADOOP-11890.2.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > mapreduce portion of patch in HADOOP-12122 that couldn't run due to number of > components touched. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6535) TaskID default constructor results in NPE on toString()
[
https://issues.apache.org/jira/browse/MAPREDUCE-6535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14990635#comment-14990635
]
Daniel Templeton commented on MAPREDUCE-6535:
-
It's a custom OutputFormat that's using the TaskAttemptID to build the output
path. It works in MR1.
I agree that mucking with {{toString()}} has the potential to be a can of
worms. The other alternative would be to lock down the default constructors in
some way so that users know not to use them.
> TaskID default constructor results in NPE on toString()
> ---
>
> Key: MAPREDUCE-6535
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6535
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.6.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>
> This code will reproduce the issue:
> {code}
> new TaskAttemptID().toString();
> {code}
> The issue is that the default constructor leaves the type {{null}}. The
> {{get()}} in {{CharTaskTypesMaps.getRepresentingCharacter()}} then throws an
> NPE on the null type key.
> The simplest solution would be to only call the {{get()}} on line 288 of
> {{TaskID.java}} if {{type}} is not {{null}} and return some other literal
> otherwise. Since no part of the code is tripping on the NPE, what we choose
> for the literal shouldn't matter. How about "x"?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6535) TaskID default constructor results in NPE on toString()
Daniel Templeton created MAPREDUCE-6535:
---
Summary: TaskID default constructor results in NPE on toString()
Key: MAPREDUCE-6535
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6535
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.6.0
Reporter: Daniel Templeton
Assignee: Daniel Templeton
This code will reproduce the issue:
{code}
new TaskAttemptID().toString();
{code}
The issue is that the default constructor leaves the type {{null}}. The
{{get()}} in {{CharTaskTypesMaps.getRepresentingCharacter()}} then throws an
NPE on the null type key.
The simplest solution would be to only call the {{get()}} on line 288 of
{{TaskID.java}} if {{type}} is not {{null}} and return some other literal
otherwise. Since no part of the code is tripping on the NPE, what we choose
for the literal shouldn't matter. How about "x"?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6533) testDetermineCacheVisibilities of TestClientDistributedCacheManager is broken
[
https://issues.apache.org/jira/browse/MAPREDUCE-6533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14990632#comment-14990632
]
Hadoop QA commented on MAPREDUCE-6533:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
50s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
10s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 51s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.8.0_60. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 8s {color} |
{color:red} hadoop-mapreduce-client-core in the patch failed with JDK
v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 29s
{color} | {color:red} Patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 12s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.7.0_79 Failed junit tests | hadoop.mapred.TestJobEndNotifier |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.7.1 Server=1.7.1
Image:test-patch-base-hadoop-date2015-11-04 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12770659/MAPREDUCE-6533.2.patch
|
| JIRA Issue | MAPREDUCE-6533 |
| Optional Tests | asflicense javac javadoc mvninstall unit findbugs
checkstyle compile |
| uname | Linux 1888ed5ad363 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/patchprocess/apache-yetus-e8bd3ad/precommit/personality/hadoop.sh
|
| git revision | trunk / 5667129 |
| Default Java | 1.7.0_79 |
| Multi-JDK versions |
[jira] [Commented] (MAPREDUCE-6531) CLONE - Mumak: Map-Reduce Simulator
[ https://issues.apache.org/jira/browse/MAPREDUCE-6531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14990337#comment-14990337 ] Karthik Kambatla commented on MAPREDUCE-6531: - Looks like this was committed to 0.21.0. What is the expectation here? > CLONE - Mumak: Map-Reduce Simulator > --- > > Key: MAPREDUCE-6531 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6531 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 0.21.0 >Reporter: GD >Assignee: Hong Tang > Fix For: 0.21.0 > > Attachments: 19-jobs.topology.json.gz, 19-jobs.trace.json.gz, > mapreduce-728-20090917-3.patch, mapreduce-728-20090917-4.patch, > mapreduce-728-20090917.patch, mapreduce-728-20090918-2.patch, > mapreduce-728-20090918-3.patch, mapreduce-728-20090918-5.patch, > mapreduce-728-20090918-6.patch, mapreduce-728-20090918.patch, mumak.png > > > h3. Vision: > We want to build a Simulator to simulate large-scale Hadoop clusters, > applications and workloads. This would be invaluable in furthering Hadoop by > providing a tool for researchers and developers to prototype features (e.g. > pluggable block-placement for HDFS, Map-Reduce schedulers etc.) and predict > their behaviour and performance with reasonable amount of confidence, > there-by aiding rapid innovation. > > h3. First Cut: Simulator for the Map-Reduce Scheduler > The Map-Reduce Scheduler is a fertile area of interest with at least four > schedulers, each with their own set of features, currently in existence: > Default Scheduler, Capacity Scheduler, Fairshare Scheduler & Priority > Scheduler. > Each scheduler's scheduling decisions are driven by many factors, such as > fairness, capacity guarantee, resource availability, data-locality etc. > Given that, it is non-trivial to accurately choose a single scheduler or even > a set of desired features to predict the right scheduler (or features) for a > given workload. Hence a simulator which can predict how well a particular > scheduler works for some specific workload by quickly iterating over > schedulers and/or scheduler features would be quite useful. > So, the first cut is to implement a simulator for the Map-Reduce scheduler > which take as input a job trace derived from production workload and a > cluster definition, and simulates the execution of the jobs in as defined in > the trace in this virtual cluster. As output, the detailed job execution > trace (recorded in relation to virtual simulated time) could then be analyzed > to understand various traits of individual schedulers (individual jobs turn > around time, throughput, faireness, capacity guarantee, etc). To support > this, we would need a simulator which could accurately model the conditions > of the actual system which would affect a schedulers decisions. These include > very large-scale clusters (thousands of nodes), the detailed characteristics > of the workload thrown at the clusters, job or task failures, data locality, > and cluster hardware (cpu, memory, disk i/o, network i/o, network topology) > etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6535) TaskID default constructor results in NPE on toString()
[
https://issues.apache.org/jira/browse/MAPREDUCE-6535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14990621#comment-14990621
]
Jason Lowe commented on MAPREDUCE-6535:
---
Curious, what code is trying to print an invalid task attempt ID? The only
reason there's a default constructor is so something can call readFields on it.
It's not a valid task ID until it's been initialized, and the next step of
fixing toString is that the type converters will then say they can't convert
the string back into a task attempt ID, etc.
> TaskID default constructor results in NPE on toString()
> ---
>
> Key: MAPREDUCE-6535
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6535
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.6.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>
> This code will reproduce the issue:
> {code}
> new TaskAttemptID().toString();
> {code}
> The issue is that the default constructor leaves the type {{null}}. The
> {{get()}} in {{CharTaskTypesMaps.getRepresentingCharacter()}} then throws an
> NPE on the null type key.
> The simplest solution would be to only call the {{get()}} on line 288 of
> {{TaskID.java}} if {{type}} is not {{null}} and return some other literal
> otherwise. Since no part of the code is tripping on the NPE, what we choose
> for the literal shouldn't matter. How about "x"?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6533) testDetermineCacheVisibilities of TestClientDistributedCacheManager is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-6533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chang Li updated MAPREDUCE-6533: Attachment: MAPREDUCE-6533.3.patch .3 patch fix whitespace issue > testDetermineCacheVisibilities of TestClientDistributedCacheManager is broken > - > > Key: MAPREDUCE-6533 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6533 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Chang Li >Assignee: Chang Li > Attachments: MAPREDUCE-6533.2.patch, MAPREDUCE-6533.3.patch, > MAPREDUCE-6533.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6533) testDetermineCacheVisibilities of TestClientDistributedCacheManager is broken
[
https://issues.apache.org/jira/browse/MAPREDUCE-6533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14990852#comment-14990852
]
Hadoop QA commented on MAPREDUCE-6533:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
53s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
10s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 52s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.8.0_60. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 8s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 23s
{color} | {color:red} Patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 24s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.7.1 Server=1.7.1
Image:test-patch-base-hadoop-date2015-11-05 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12770707/MAPREDUCE-6533.3.patch
|
| JIRA Issue | MAPREDUCE-6533 |
| Optional Tests | asflicense javac javadoc mvninstall unit findbugs
checkstyle compile |
| uname | Linux 29af941d9a12 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/patchprocess/apache-yetus-e8bd3ad/precommit/personality/hadoop.sh
|
| git revision | trunk / 5667129 |
| Default Java | 1.7.0_79 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_60
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 |
| findbugs | v3.0.0 |
| JDK v1.7.0_79 Test Results |
[jira] [Commented] (MAPREDUCE-6535) TaskID default constructor results in NPE on toString()
[
https://issues.apache.org/jira/browse/MAPREDUCE-6535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14990686#comment-14990686
]
Jason Lowe commented on MAPREDUCE-6535:
---
bq. It works in MR1.
I was surprised by that, so I went looking at how it would end up working in
MR1. That code basically defaults to a reducer with ID 0 since it has a
boolean for the type rather than an enum. Seems like we could simply do the
same thing and default the task type to TaskType.REDUCE since that's
effectively what the old code did as well.
> TaskID default constructor results in NPE on toString()
> ---
>
> Key: MAPREDUCE-6535
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6535
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.6.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>
> This code will reproduce the issue:
> {code}
> new TaskAttemptID().toString();
> {code}
> The issue is that the default constructor leaves the type {{null}}. The
> {{get()}} in {{CharTaskTypesMaps.getRepresentingCharacter()}} then throws an
> NPE on the null type key.
> The simplest solution would be to only call the {{get()}} on line 288 of
> {{TaskID.java}} if {{type}} is not {{null}} and return some other literal
> otherwise. Since no part of the code is tripping on the NPE, what we choose
> for the literal shouldn't matter. How about "x"?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6535) TaskID default constructor results in NPE on toString()
[
https://issues.apache.org/jira/browse/MAPREDUCE-6535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14990699#comment-14990699
]
Daniel Templeton commented on MAPREDUCE-6535:
-
I had considered that, but I thought it might violate the principle of least
astonishment. If that was the effective behavior in MR1, then that's probably
the best approach.
> TaskID default constructor results in NPE on toString()
> ---
>
> Key: MAPREDUCE-6535
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6535
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.6.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>
> This code will reproduce the issue:
> {code}
> new TaskAttemptID().toString();
> {code}
> The issue is that the default constructor leaves the type {{null}}. The
> {{get()}} in {{CharTaskTypesMaps.getRepresentingCharacter()}} then throws an
> NPE on the null type key.
> The simplest solution would be to only call the {{get()}} on line 288 of
> {{TaskID.java}} if {{type}} is not {{null}} and return some other literal
> otherwise. Since no part of the code is tripping on the NPE, what we choose
> for the literal shouldn't matter. How about "x"?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6535) TaskID default constructor results in NPE on toString()
[
https://issues.apache.org/jira/browse/MAPREDUCE-6535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991244#comment-14991244
]
zhihai xu commented on MAPREDUCE-6535:
--
+1 to use TaskType.REDUCE as the default task type and make it compatible with
MR1.
> TaskID default constructor results in NPE on toString()
> ---
>
> Key: MAPREDUCE-6535
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6535
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.6.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>
> This code will reproduce the issue:
> {code}
> new TaskAttemptID().toString();
> {code}
> The issue is that the default constructor leaves the type {{null}}. The
> {{get()}} in {{CharTaskTypesMaps.getRepresentingCharacter()}} then throws an
> NPE on the null type key.
> The simplest solution would be to only call the {{get()}} on line 288 of
> {{TaskID.java}} if {{type}} is not {{null}} and return some other literal
> otherwise. Since no part of the code is tripping on the NPE, what we choose
> for the literal shouldn't matter. How about "x"?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated MAPREDUCE-5870: --- Attachment: (was: 0009-MAPREDUCE-5870.patch) > Support for passing Job priority through Application Submission Context in > Mapreduce Side > - > > Key: MAPREDUCE-5870 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch, > 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch, > 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch, > 0007-MAPREDUCE-5870.patch, 0008-MAPREDUCE-5870.patch, > 0009-MAPREDUCE-5870.patch, Yarn-2002.1.patch > > > Job Prioirty can be set from client side as below [Configuration and api]. > a. JobConf.getJobPriority() and > Job.setPriority(JobPriority priority) > b. We can also use configuration > "mapreduce.job.priority". > Now this Job priority can be passed in Application Submission > context from Client side. > Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated MAPREDUCE-5870: --- Attachment: 0009-MAPREDUCE-5870.patch Kicking Jenkins. > Support for passing Job priority through Application Submission Context in > Mapreduce Side > - > > Key: MAPREDUCE-5870 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch, > 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch, > 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch, > 0007-MAPREDUCE-5870.patch, 0008-MAPREDUCE-5870.patch, > 0009-MAPREDUCE-5870.patch, Yarn-2002.1.patch > > > Job Prioirty can be set from client side as below [Configuration and api]. > a. JobConf.getJobPriority() and > Job.setPriority(JobPriority priority) > b. We can also use configuration > "mapreduce.job.priority". > Now this Job priority can be passed in Application Submission > context from Client side. > Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5485) Allow repeating job commit by extending OutputCommitter API
[
https://issues.apache.org/jira/browse/MAPREDUCE-5485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14990296#comment-14990296
]
Hadoop QA commented on MAPREDUCE-5485:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
3s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 52s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 44s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 9s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 17s
{color} | {color:red} hadoop-mapreduce-client-app in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 55s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 55s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 57s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 57s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 25s
{color} | {color:red} Patch generated 4 new checkstyle issues in
hadoop-mapreduce-project/hadoop-mapreduce-client (total was 129, now 129).
{color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 6 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 35s
{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
introduced 2 new FindBugs issues. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 2m 47s
{color} | {color:red}
hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_66
with JDK v1.8.0_66 generated 2 new issues (was 100, now 100). {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m 14s
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed with
JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 6s {color} |
{color:red} hadoop-mapreduce-client-core in the patch failed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m 49s
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed with
JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 25s {color}
| {color:red} hadoop-mapreduce-client-core in the patch failed with JDK
v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 31s
{color} | {color:red} Patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 54m 4s {color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
[jira] [Updated] (MAPREDUCE-6530) Jobtracker is slow when more JT UI requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-6530: Affects Version/s: (was: 2.5.1) 1.2.1 > Jobtracker is slow when more JT UI requests > --- > > Key: MAPREDUCE-6530 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6530 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: Prabhu Joseph > > JobTracker is slow when there are huge number of Jobs running and 30 > connections were established to info port to view Job status and counters. > hadoop job -list took 4m22.412s > We took Jstack traces and found most of the server threads waiting on > JobTracker object and the thread which has the lock on JobTracker waits for > ResourceBundle object. > "retireJobs" prio=10 tid=0x7f2345200800 nid=0x11c1 waiting for > monitor entry [0x7f22e3499000] >java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.hadoop.mapreduce.util.ResourceBundles.getValue(ResourceBundles.java:56) > - waiting to lock <0x000197cc6218> (a java.lang.Class for > org.apache.hadoop.mapreduce.util.ResourceBundles) > at > org.apache.hadoop.mapreduce.util.ResourceBundles.getCounterName(ResourceBundles.java:89) > at > org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.localizeCounterName(FrameworkCounterGroup.java:135) > at > org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.access$000(FrameworkCounterGroup.java:47) > at > org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup$FrameworkCounter.getDisplayName(FrameworkCounterGroup.java:75) > at > org.apache.hadoop.mapred.Counters$Counter.getDisplayName(Counters.java:130) > at > org.apache.hadoop.mapred.Counters.incrAllCounters(Counters.java:534) > - locked <0x0007f8411608> (a org.apache.hadoop.mapred.Counters) > at > org.apache.hadoop.mapred.JobInProgress.incrementTaskCounters(JobInProgress.java:1728) > at > org.apache.hadoop.mapred.JobInProgress.getMapCounters(JobInProgress.java:1669) > at > org.apache.hadoop.mapred.JobTracker$RetireJobs.addToCache(JobTracker.java:657) > - locked <0x9644ae08> (a > org.apache.hadoop.mapred.JobTracker$RetireJobs) > at > org.apache.hadoop.mapred.JobTracker$RetireJobs.run(JobTracker.java:769) > - locked <0x964c5550> (a > org.apache.hadoop.mapred.FairScheduler) > - locked <0x9644a9d0> (a > java.util.Collections$SynchronizedMap) > - locked <0x962ac660> (a org.apache.hadoop.mapred.JobTracker) > at java.lang.Thread.run(Thread.java:745) > The ResourceBundle object is locked most of the time by JT GUI jobtracker_jsp > and does getMapCounters(). > "926410165@qtp-1732070199-56" daemon prio=10 tid=0x7f232c4df000 nid=0x27c0 > runnable [0x7f22db7bf000] >java.lang.Thread.State: RUNNABLE > at java.lang.Throwable.fillInStackTrace(Native Method) > at java.lang.Throwable.fillInStackTrace(Throwable.java:783) > - locked <0x00061a49ede0> (a java.util.MissingResourceException) > at java.lang.Throwable.(Throwable.java:287) > at java.lang.Exception.(Exception.java:84) > at java.lang.RuntimeException.(RuntimeException.java:80) > at > java.util.MissingResourceException.(MissingResourceException.java:85) > at > java.util.ResourceBundle.throwMissingResourceException(ResourceBundle.java:1499) > at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1322) > at java.util.ResourceBundle.getBundle(ResourceBundle.java:1028) > at > org.apache.hadoop.mapreduce.util.ResourceBundles.getBundle(ResourceBundles.java:37) > at > org.apache.hadoop.mapreduce.util.ResourceBundles.getValue(ResourceBundles.java:56) > - locked <0x000197cc6218> (a java.lang.Class for > org.apache.hadoop.mapreduce.util.ResourceBundles) > at > org.apache.hadoop.mapreduce.util.ResourceBundles.getCounterName(ResourceBundles.java:89) > at > org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.localizeCounterName(FrameworkCounterGroup.java:135) > at > org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.access$000(FrameworkCounterGroup.java:47) > at > org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup$FrameworkCounter.getDisplayName(FrameworkCounterGroup.java:75) > at > org.apache.hadoop.mapred.Counters$Counter.getDisplayName(Counters.java:130) > at > org.apache.hadoop.mapred.Counters.incrAllCounters(Counters.java:534) > - locked <0x0007ed1024b8> (a org.apache.hadoop.mapred.Counters) > at >
[jira] [Commented] (MAPREDUCE-6530) Jobtracker is slow when more JT UI requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14990324#comment-14990324 ] Karthik Kambatla commented on MAPREDUCE-6530: - Hadoop 2.0 have only MR2 running on Yarn. > Jobtracker is slow when more JT UI requests > --- > > Key: MAPREDUCE-6530 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6530 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: Prabhu Joseph > > JobTracker is slow when there are huge number of Jobs running and 30 > connections were established to info port to view Job status and counters. > hadoop job -list took 4m22.412s > We took Jstack traces and found most of the server threads waiting on > JobTracker object and the thread which has the lock on JobTracker waits for > ResourceBundle object. > "retireJobs" prio=10 tid=0x7f2345200800 nid=0x11c1 waiting for > monitor entry [0x7f22e3499000] >java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.hadoop.mapreduce.util.ResourceBundles.getValue(ResourceBundles.java:56) > - waiting to lock <0x000197cc6218> (a java.lang.Class for > org.apache.hadoop.mapreduce.util.ResourceBundles) > at > org.apache.hadoop.mapreduce.util.ResourceBundles.getCounterName(ResourceBundles.java:89) > at > org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.localizeCounterName(FrameworkCounterGroup.java:135) > at > org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.access$000(FrameworkCounterGroup.java:47) > at > org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup$FrameworkCounter.getDisplayName(FrameworkCounterGroup.java:75) > at > org.apache.hadoop.mapred.Counters$Counter.getDisplayName(Counters.java:130) > at > org.apache.hadoop.mapred.Counters.incrAllCounters(Counters.java:534) > - locked <0x0007f8411608> (a org.apache.hadoop.mapred.Counters) > at > org.apache.hadoop.mapred.JobInProgress.incrementTaskCounters(JobInProgress.java:1728) > at > org.apache.hadoop.mapred.JobInProgress.getMapCounters(JobInProgress.java:1669) > at > org.apache.hadoop.mapred.JobTracker$RetireJobs.addToCache(JobTracker.java:657) > - locked <0x9644ae08> (a > org.apache.hadoop.mapred.JobTracker$RetireJobs) > at > org.apache.hadoop.mapred.JobTracker$RetireJobs.run(JobTracker.java:769) > - locked <0x964c5550> (a > org.apache.hadoop.mapred.FairScheduler) > - locked <0x9644a9d0> (a > java.util.Collections$SynchronizedMap) > - locked <0x962ac660> (a org.apache.hadoop.mapred.JobTracker) > at java.lang.Thread.run(Thread.java:745) > The ResourceBundle object is locked most of the time by JT GUI jobtracker_jsp > and does getMapCounters(). > "926410165@qtp-1732070199-56" daemon prio=10 tid=0x7f232c4df000 nid=0x27c0 > runnable [0x7f22db7bf000] >java.lang.Thread.State: RUNNABLE > at java.lang.Throwable.fillInStackTrace(Native Method) > at java.lang.Throwable.fillInStackTrace(Throwable.java:783) > - locked <0x00061a49ede0> (a java.util.MissingResourceException) > at java.lang.Throwable.(Throwable.java:287) > at java.lang.Exception.(Exception.java:84) > at java.lang.RuntimeException.(RuntimeException.java:80) > at > java.util.MissingResourceException.(MissingResourceException.java:85) > at > java.util.ResourceBundle.throwMissingResourceException(ResourceBundle.java:1499) > at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1322) > at java.util.ResourceBundle.getBundle(ResourceBundle.java:1028) > at > org.apache.hadoop.mapreduce.util.ResourceBundles.getBundle(ResourceBundles.java:37) > at > org.apache.hadoop.mapreduce.util.ResourceBundles.getValue(ResourceBundles.java:56) > - locked <0x000197cc6218> (a java.lang.Class for > org.apache.hadoop.mapreduce.util.ResourceBundles) > at > org.apache.hadoop.mapreduce.util.ResourceBundles.getCounterName(ResourceBundles.java:89) > at > org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.localizeCounterName(FrameworkCounterGroup.java:135) > at > org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.access$000(FrameworkCounterGroup.java:47) > at > org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup$FrameworkCounter.getDisplayName(FrameworkCounterGroup.java:75) > at > org.apache.hadoop.mapred.Counters$Counter.getDisplayName(Counters.java:130) > at > org.apache.hadoop.mapred.Counters.incrAllCounters(Counters.java:534) > - locked <0x0007ed1024b8> (a org.apache.hadoop.mapred.Counters) > at >
[jira] [Updated] (MAPREDUCE-6531) CLONE - Mumak: Map-Reduce Simulator
[ https://issues.apache.org/jira/browse/MAPREDUCE-6531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-6531: Hadoop Flags: (was: Reviewed) > CLONE - Mumak: Map-Reduce Simulator > --- > > Key: MAPREDUCE-6531 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6531 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 0.21.0 >Reporter: GD >Assignee: Hong Tang > Fix For: 0.21.0 > > Attachments: 19-jobs.topology.json.gz, 19-jobs.trace.json.gz, > mapreduce-728-20090917-3.patch, mapreduce-728-20090917-4.patch, > mapreduce-728-20090917.patch, mapreduce-728-20090918-2.patch, > mapreduce-728-20090918-3.patch, mapreduce-728-20090918-5.patch, > mapreduce-728-20090918-6.patch, mapreduce-728-20090918.patch, mumak.png > > > h3. Vision: > We want to build a Simulator to simulate large-scale Hadoop clusters, > applications and workloads. This would be invaluable in furthering Hadoop by > providing a tool for researchers and developers to prototype features (e.g. > pluggable block-placement for HDFS, Map-Reduce schedulers etc.) and predict > their behaviour and performance with reasonable amount of confidence, > there-by aiding rapid innovation. > > h3. First Cut: Simulator for the Map-Reduce Scheduler > The Map-Reduce Scheduler is a fertile area of interest with at least four > schedulers, each with their own set of features, currently in existence: > Default Scheduler, Capacity Scheduler, Fairshare Scheduler & Priority > Scheduler. > Each scheduler's scheduling decisions are driven by many factors, such as > fairness, capacity guarantee, resource availability, data-locality etc. > Given that, it is non-trivial to accurately choose a single scheduler or even > a set of desired features to predict the right scheduler (or features) for a > given workload. Hence a simulator which can predict how well a particular > scheduler works for some specific workload by quickly iterating over > schedulers and/or scheduler features would be quite useful. > So, the first cut is to implement a simulator for the Map-Reduce scheduler > which take as input a job trace derived from production workload and a > cluster definition, and simulates the execution of the jobs in as defined in > the trace in this virtual cluster. As output, the detailed job execution > trace (recorded in relation to virtual simulated time) could then be analyzed > to understand various traits of individual schedulers (individual jobs turn > around time, throughput, faireness, capacity guarantee, etc). To support > this, we would need a simulator which could accurately model the conditions > of the actual system which would affect a schedulers decisions. These include > very large-scale clusters (thousands of nodes), the detailed characteristics > of the workload thrown at the clusters, job or task failures, data locality, > and cluster hardware (cpu, memory, disk i/o, network i/o, network topology) > etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6533) testDetermineCacheVisibilities of TestClientDistributedCacheManager is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-6533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chang Li updated MAPREDUCE-6533: Attachment: MAPREDUCE-6533.2.patch Thanks [~jlowe] for review! have updated .2 patch according to your suggestion to create them as Path object > testDetermineCacheVisibilities of TestClientDistributedCacheManager is broken > - > > Key: MAPREDUCE-6533 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6533 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Chang Li >Assignee: Chang Li > Attachments: MAPREDUCE-6533.2.patch, MAPREDUCE-6533.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5485) Allow repeating job commit by extending OutputCommitter API
[ https://issues.apache.org/jira/browse/MAPREDUCE-5485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-5485: -- Attachment: MAPREDUCE-5485-v1.patch Attach a new patch to address Bikas comments above, include: 1. Make retry logic go to committer.commitJob() rather than MRAppMaster 2. It will fail AM instead of Job when exception happens during jobCommit if commitJob() is repeatable. 3. Add related unit tests. Verify this feature works well on a small scale cluster that kill AM during job committing stage, and the job can continue and succeed after AM restarted. > Allow repeating job commit by extending OutputCommitter API > --- > > Key: MAPREDUCE-5485 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5485 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 2.1.0-beta >Reporter: Nemon Lou >Assignee: Junping Du > Attachments: MAPREDUCE-5485-demo-2.patch, MAPREDUCE-5485-demo.patch, > MAPREDUCE-5485-v1.patch > > > There are chances MRAppMaster crush during job committing,or NodeManager > restart cause the committing AM exit due to container expire.In these cases > ,the job will fail. > However,some jobs can redo commit so failing the job becomes unnecessary. > Let clients tell AM to allow redo commit or not is a better choice. > This idea comes from Jason Lowe's comments in MAPREDUCE-4819 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6533) testDetermineCacheVisibilities of TestClientDistributedCacheManager is broken
[
https://issues.apache.org/jira/browse/MAPREDUCE-6533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14990003#comment-14990003
]
Jason Lowe commented on MAPREDUCE-6533:
---
The problem is not the File constructor but rather the result when it gets
turned into a URI. The File constructor call is building the path
{{file:/Users/changli/hadoop/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/target/test-dir/TestCacheVisibility}}
correctly.
Since TEST_ROOT_DIR and TEST_VISIBILITY_DIR always want to be treated as
{{Path}} objects then a better fix would be to simply create them as such. I
don't see a need for the extraneous string replacement and URI conversions.
Then it's not hardcoded on a certain path separator and the code is cleaner.
> testDetermineCacheVisibilities of TestClientDistributedCacheManager is broken
> -
>
> Key: MAPREDUCE-6533
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6533
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Chang Li
>Assignee: Chang Li
> Attachments: MAPREDUCE-6533.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14989960#comment-14989960 ] Jason Lowe commented on MAPREDUCE-5870: --- TestUberAM failure appears to be related, and I'm not sure about the others. > Support for passing Job priority through Application Submission Context in > Mapreduce Side > - > > Key: MAPREDUCE-5870 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch, > 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch, > 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch, > 0007-MAPREDUCE-5870.patch, 0008-MAPREDUCE-5870.patch, Yarn-2002.1.patch > > > Job Prioirty can be set from client side as below [Configuration and api]. > a. JobConf.getJobPriority() and > Job.setPriority(JobPriority priority) > b. We can also use configuration > "mapreduce.job.priority". > Now this Job priority can be passed in Application Submission > context from Client side. > Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5485) Allow repeating job commit by extending OutputCommitter API
[ https://issues.apache.org/jira/browse/MAPREDUCE-5485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-5485: -- Priority: Critical (was: Major) > Allow repeating job commit by extending OutputCommitter API > --- > > Key: MAPREDUCE-5485 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5485 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 2.1.0-beta >Reporter: Nemon Lou >Assignee: Junping Du >Priority: Critical > Attachments: MAPREDUCE-5485-demo-2.patch, MAPREDUCE-5485-demo.patch, > MAPREDUCE-5485-v1.patch > > > There are chances MRAppMaster crush during job committing,or NodeManager > restart cause the committing AM exit due to container expire.In these cases > ,the job will fail. > However,some jobs can redo commit so failing the job becomes unnecessary. > Let clients tell AM to allow redo commit or not is a better choice. > This idea comes from Jason Lowe's comments in MAPREDUCE-4819 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5485) Allow repeating job commit by extending OutputCommitter API
[ https://issues.apache.org/jira/browse/MAPREDUCE-5485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-5485: -- Target Version/s: 2.6.3, 2.7.3 > Allow repeating job commit by extending OutputCommitter API > --- > > Key: MAPREDUCE-5485 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5485 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 2.1.0-beta >Reporter: Nemon Lou >Assignee: Junping Du > Attachments: MAPREDUCE-5485-demo-2.patch, MAPREDUCE-5485-demo.patch, > MAPREDUCE-5485-v1.patch > > > There are chances MRAppMaster crush during job committing,or NodeManager > restart cause the committing AM exit due to container expire.In these cases > ,the job will fail. > However,some jobs can redo commit so failing the job becomes unnecessary. > Let clients tell AM to allow redo commit or not is a better choice. > This idea comes from Jason Lowe's comments in MAPREDUCE-4819 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil G updated MAPREDUCE-5870:
---
Attachment: 0009-MAPREDUCE-5870.patch
Thank You [~jlowe]
Yes. {{TestUberAM}} failures were related. Other test cases are failing even
w/o the patch in my local. I will see one more jenkins result with this patch,
and if fails, will raise a different test failure ticket to track the same.
> Support for passing Job priority through Application Submission Context in
> Mapreduce Side
> -
>
> Key: MAPREDUCE-5870
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch,
> 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch,
> 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch,
> 0007-MAPREDUCE-5870.patch, 0008-MAPREDUCE-5870.patch,
> 0009-MAPREDUCE-5870.patch, Yarn-2002.1.patch
>
>
> Job Prioirty can be set from client side as below [Configuration and api].
> a. JobConf.getJobPriority() and
> Job.setPriority(JobPriority priority)
> b. We can also use configuration
> "mapreduce.job.priority".
> Now this Job priority can be passed in Application Submission
> context from Client side.
> Here we can reuse the MRJobConfig.PRIORITY configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
