[jira] [Commented] (MAPREDUCE-6298) Job#toString throws an exception when not in state RUNNING
[
https://issues.apache.org/jira/browse/MAPREDUCE-6298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14537635#comment-14537635
]
Lars Francke commented on MAPREDUCE-6298:
-
[~ozawa] Thanks for taking a look!
Job#toString throws an exception when not in state RUNNING
--
Key: MAPREDUCE-6298
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6298
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Lars Francke
Assignee: Lars Francke
Priority: Minor
Labels: BB2015-05-TBR
Attachments: MAPREDUCE-6298.1.patch
Job#toString calls {{ensureState(JobState.RUNNING);}} as the very first
thing. That method causes an Exception to be thrown which is not nice.
One thing this breaks is usage of Job on the Scala (e.g. Spark) REPL as that
calls toString after every invocation and that fails every time.
I'll attach a patch that checks state and if it's RUNNING prints the original
message and if not prints something else.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14537761#comment-14537761 ] sam liu commented on MAPREDUCE-6204: These ut should not fail, as they could pass in my env. I studied the outputs of ut execution and seems the failures were caused by env issue, like 'java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/exceptions/ApplicationIdNotProvidedException'. Could please help rerun the ut? Thanks! TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS --- Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Minor Labels: BB2015-05-RFC Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204-2.patch, MAPREDUCE-6204.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6360) TestMapreduceConfigFields is placed in wrong dir, introducing compile error
[
https://issues.apache.org/jira/browse/MAPREDUCE-6360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinayakumar B updated MAPREDUCE-6360:
-
Description:
MAPREDUCE-6192 has introduced a Test file {{TestMapreduceConfigFields }}
which was committed in package {{org.apache.hadoop.mapred}}
But the package declaration inside file is {{org.apache.hadoop.mapreduce}}.
By surprise, this is not giving any compile errors in maven build. But eclipse
catches it.
So move {{TestMapreduceConfigFields }} to correct package
{{org.apache.hadoop.mapreduce}}
was:
MAPREDUCE-6192 has introduced a Test file {{TestHdfsConfigFields}}
which was committed in package {{org.apache.hadoop.tools}}
But the package declaration inside file is {{org.apache.hadoop.hdfs.tools}}.
By surprise, this is not giving any compile errors in maven build. But eclipse
catches it.
So move {{TestHdfsConfigFields}} to correct package
{{org.apache.hadoop.hdfs.tools}}
TestMapreduceConfigFields is placed in wrong dir, introducing compile error
---
Key: MAPREDUCE-6360
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6360
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Vinayakumar B
Assignee: Vinayakumar B
MAPREDUCE-6192 has introduced a Test file {{TestMapreduceConfigFields }}
which was committed in package {{org.apache.hadoop.mapred}}
But the package declaration inside file is {{org.apache.hadoop.mapreduce}}.
By surprise, this is not giving any compile errors in maven build. But
eclipse catches it.
So move {{TestMapreduceConfigFields }} to correct package
{{org.apache.hadoop.mapreduce}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6360) TestHdfsConfigFields is placed in wrong dir, introducing compile error
Vinayakumar B created MAPREDUCE-6360:
Summary: TestHdfsConfigFields is placed in wrong dir, introducing
compile error
Key: MAPREDUCE-6360
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6360
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Vinayakumar B
HDFS-7559 has introduced a Test file {{TestHdfsConfigFields}}
which was committed in package {{org.apache.hadoop.tools}}
But the package declaration inside file is {{org.apache.hadoop.hdfs.tools}}.
By surprise, this is not giving any compile errors in maven build. But eclipse
catches it.
So move {{TestHdfsConfigFields}} to correct package
{{org.apache.hadoop.hdfs.tools}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (MAPREDUCE-6360) TestHdfsConfigFields is placed in wrong dir, introducing compile error
[
https://issues.apache.org/jira/browse/MAPREDUCE-6360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinayakumar B reassigned MAPREDUCE-6360:
Assignee: Vinayakumar B
TestHdfsConfigFields is placed in wrong dir, introducing compile error
--
Key: MAPREDUCE-6360
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6360
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Vinayakumar B
Assignee: Vinayakumar B
HDFS-7559 has introduced a Test file {{TestHdfsConfigFields}}
which was committed in package {{org.apache.hadoop.tools}}
But the package declaration inside file is {{org.apache.hadoop.hdfs.tools}}.
By surprise, this is not giving any compile errors in maven build. But
eclipse catches it.
So move {{TestHdfsConfigFields}} to correct package
{{org.apache.hadoop.hdfs.tools}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6360) TestMapreduceConfigFields is placed in wrong dir, introducing compile error
[
https://issues.apache.org/jira/browse/MAPREDUCE-6360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinayakumar B updated MAPREDUCE-6360:
-
Summary: TestMapreduceConfigFields is placed in wrong dir, introducing
compile error (was: TestHdfsConfigFields is placed in wrong dir, introducing
compile error)
TestMapreduceConfigFields is placed in wrong dir, introducing compile error
---
Key: MAPREDUCE-6360
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6360
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Vinayakumar B
Assignee: Vinayakumar B
HDFS-7559 has introduced a Test file {{TestHdfsConfigFields}}
which was committed in package {{org.apache.hadoop.tools}}
But the package declaration inside file is {{org.apache.hadoop.hdfs.tools}}.
By surprise, this is not giving any compile errors in maven build. But
eclipse catches it.
So move {{TestHdfsConfigFields}} to correct package
{{org.apache.hadoop.hdfs.tools}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6360) TestMapreduceConfigFields is placed in wrong dir, introducing compile error
[
https://issues.apache.org/jira/browse/MAPREDUCE-6360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinayakumar B updated MAPREDUCE-6360:
-
Description:
MAPREDUCE-6192 has introduced a Test file {{TestHdfsConfigFields}}
which was committed in package {{org.apache.hadoop.tools}}
But the package declaration inside file is {{org.apache.hadoop.hdfs.tools}}.
By surprise, this is not giving any compile errors in maven build. But eclipse
catches it.
So move {{TestHdfsConfigFields}} to correct package
{{org.apache.hadoop.hdfs.tools}}
was:
HDFS-7559 has introduced a Test file {{TestHdfsConfigFields}}
which was committed in package {{org.apache.hadoop.tools}}
But the package declaration inside file is {{org.apache.hadoop.hdfs.tools}}.
By surprise, this is not giving any compile errors in maven build. But eclipse
catches it.
So move {{TestHdfsConfigFields}} to correct package
{{org.apache.hadoop.hdfs.tools}}
TestMapreduceConfigFields is placed in wrong dir, introducing compile error
---
Key: MAPREDUCE-6360
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6360
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Vinayakumar B
Assignee: Vinayakumar B
MAPREDUCE-6192 has introduced a Test file {{TestHdfsConfigFields}}
which was committed in package {{org.apache.hadoop.tools}}
But the package declaration inside file is {{org.apache.hadoop.hdfs.tools}}.
By surprise, this is not giving any compile errors in maven build. But
eclipse catches it.
So move {{TestHdfsConfigFields}} to correct package
{{org.apache.hadoop.hdfs.tools}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3881) building fail under Windows
[
https://issues.apache.org/jira/browse/MAPREDUCE-3881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-3881:
-
Labels: (was: BB2015-05-RFC)
building fail under Windows
---
Key: MAPREDUCE-3881
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3881
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: build
Environment: D:\os\hadoopcommonmvn --version
Apache Maven 3.0.4 (r1232337; 2012-01-17 16:44:56+0800)
Maven home: C:\portable\maven\bin\..
Java version: 1.7.0_02, vendor: Oracle Corporation
Java home: C:\Program Files (x86)\Java\jdk1.7.0_02\jre
Default locale: zh_CN, platform encoding: GBK
OS name: windows 7, version: 6.1, arch: x86, family: windows
Reporter: Changming Sun
Priority: Minor
Attachments: pom.xml.patch
Original Estimate: 1h
Remaining Estimate: 1h
hadoop-mapreduce-project\hadoop-yarn\hadoop-yarn-common\pom.xml is not
portable.
execution
idgenerate-version/id
phasegenerate-sources/phase
configuration
executablescripts/saveVersion.sh/executable
arguments
argument${project.version}/argument
argument${project.build.directory}/argument
/arguments
/configuration
goals
goalexec/goal
/goals
/execution
when I built it under windows , I got a such error:
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2:exec
(gen
erate-version) on project hadoop-yarn-common: Command execution failed.
Cannot r
un program scripts\saveVersion.sh (in directory
D:\os\hadoopcommon\hadoop-map
reduce-project\hadoop-yarn\hadoop-yarn-common): CreateProcess error=2,
? - [Help 1]
we should modify it like this: (copied from
hadoop-common-project\hadoop-common\pom.xml)
configuration
target
mkdir
dir=${project.build.directory}/generated-sources/java/
exec executable=sh
arg
line=${basedir}/dev-support/saveVersion.sh
${project.version} ${project.build.directory}/generated-sources/java/
/exec
/target
/configuration
/execution
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3881) building fail under Windows
[
https://issues.apache.org/jira/browse/MAPREDUCE-3881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-3881:
-
Resolution: Not A Problem
Status: Resolved (was: Patch Available)
Closing it as 'Not A Problem' since it works fine in the latest.
building fail under Windows
---
Key: MAPREDUCE-3881
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3881
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: build
Environment: D:\os\hadoopcommonmvn --version
Apache Maven 3.0.4 (r1232337; 2012-01-17 16:44:56+0800)
Maven home: C:\portable\maven\bin\..
Java version: 1.7.0_02, vendor: Oracle Corporation
Java home: C:\Program Files (x86)\Java\jdk1.7.0_02\jre
Default locale: zh_CN, platform encoding: GBK
OS name: windows 7, version: 6.1, arch: x86, family: windows
Reporter: Changming Sun
Priority: Minor
Labels: BB2015-05-RFC
Attachments: pom.xml.patch
Original Estimate: 1h
Remaining Estimate: 1h
hadoop-mapreduce-project\hadoop-yarn\hadoop-yarn-common\pom.xml is not
portable.
execution
idgenerate-version/id
phasegenerate-sources/phase
configuration
executablescripts/saveVersion.sh/executable
arguments
argument${project.version}/argument
argument${project.build.directory}/argument
/arguments
/configuration
goals
goalexec/goal
/goals
/execution
when I built it under windows , I got a such error:
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2:exec
(gen
erate-version) on project hadoop-yarn-common: Command execution failed.
Cannot r
un program scripts\saveVersion.sh (in directory
D:\os\hadoopcommon\hadoop-map
reduce-project\hadoop-yarn\hadoop-yarn-common): CreateProcess error=2,
? - [Help 1]
we should modify it like this: (copied from
hadoop-common-project\hadoop-common\pom.xml)
configuration
target
mkdir
dir=${project.build.directory}/generated-sources/java/
exec executable=sh
arg
line=${basedir}/dev-support/saveVersion.sh
${project.version} ${project.build.directory}/generated-sources/java/
/exec
/target
/configuration
/execution
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6360) TestMapreduceConfigFields is placed in wrong dir, introducing compile error
[
https://issues.apache.org/jira/browse/MAPREDUCE-6360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14537783#comment-14537783
]
Brahma Reddy Battula commented on MAPREDUCE-6360:
-
Resloved as duplicate of HDFS-8362, since it's raised before.. Please let me
know your opinion..
TestMapreduceConfigFields is placed in wrong dir, introducing compile error
---
Key: MAPREDUCE-6360
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6360
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Vinayakumar B
Assignee: Vinayakumar B
MAPREDUCE-6192 has introduced a Test file {{TestMapreduceConfigFields }}
which was committed in package {{org.apache.hadoop.mapred}}
But the package declaration inside file is {{org.apache.hadoop.mapreduce}}.
By surprise, this is not giving any compile errors in maven build. But
eclipse catches it.
So move {{TestMapreduceConfigFields }} to correct package
{{org.apache.hadoop.mapreduce}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-6360) TestMapreduceConfigFields is placed in wrong dir, introducing compile error
[
https://issues.apache.org/jira/browse/MAPREDUCE-6360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula resolved MAPREDUCE-6360.
-
Resolution: Duplicate
TestMapreduceConfigFields is placed in wrong dir, introducing compile error
---
Key: MAPREDUCE-6360
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6360
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Vinayakumar B
Assignee: Vinayakumar B
MAPREDUCE-6192 has introduced a Test file {{TestMapreduceConfigFields }}
which was committed in package {{org.apache.hadoop.mapred}}
But the package declaration inside file is {{org.apache.hadoop.mapreduce}}.
By surprise, this is not giving any compile errors in maven build. But
eclipse catches it.
So move {{TestMapreduceConfigFields }} to correct package
{{org.apache.hadoop.mapreduce}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-3936) Clients should not enforce counter limits
[ https://issues.apache.org/jira/browse/MAPREDUCE-3936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White resolved MAPREDUCE-3936. -- Resolution: Won't Fix Clients should not enforce counter limits -- Key: MAPREDUCE-3936 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3936 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1 Reporter: Tom White Assignee: Tom White Labels: BB2015-05-TBR Attachments: MAPREDUCE-3936.patch, MAPREDUCE-3936.patch The code for enforcing counter limits (from MAPREDUCE-1943) creates a static JobConf instance to load the limits, which may throw an exception if the client limit is set to be lower than the limit on the cluster (perhaps because the cluster limit was raised from the default). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6360) TestMapreduceConfigFields is placed in wrong dir, introducing compile error
[
https://issues.apache.org/jira/browse/MAPREDUCE-6360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14537782#comment-14537782
]
Brahma Reddy Battula commented on MAPREDUCE-6360:
-
Duple of HDFS-8362..?
TestMapreduceConfigFields is placed in wrong dir, introducing compile error
---
Key: MAPREDUCE-6360
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6360
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Vinayakumar B
Assignee: Vinayakumar B
MAPREDUCE-6192 has introduced a Test file {{TestMapreduceConfigFields }}
which was committed in package {{org.apache.hadoop.mapred}}
But the package declaration inside file is {{org.apache.hadoop.mapreduce}}.
By surprise, this is not giving any compile errors in maven build. But
eclipse catches it.
So move {{TestMapreduceConfigFields }} to correct package
{{org.apache.hadoop.mapreduce}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6360) TestMapreduceConfigFields is placed in wrong dir, introducing compile error
[
https://issues.apache.org/jira/browse/MAPREDUCE-6360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arshad Mohammad updated MAPREDUCE-6360:
---
Attachment: MAPREDUCE-6360-1.patch
TestMapreduceConfigFields is placed in wrong dir, introducing compile error
---
Key: MAPREDUCE-6360
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6360
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Vinayakumar B
Assignee: Arshad Mohammad
Attachments: MAPREDUCE-6360-1.patch
MAPREDUCE-6192 has introduced a Test file {{TestMapreduceConfigFields }}
which was committed in package {{org.apache.hadoop.mapred}}
But the package declaration inside file is {{org.apache.hadoop.mapreduce}}.
By surprise, this is not giving any compile errors in maven build. But
eclipse catches it.
So move {{TestMapreduceConfigFields }} to correct package
{{org.apache.hadoop.mapreduce}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6360) TestMapreduceConfigFields is placed in wrong dir, introducing compile error
[
https://issues.apache.org/jira/browse/MAPREDUCE-6360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arshad Mohammad updated MAPREDUCE-6360:
---
Target Version/s: 2.8.0
Status: Patch Available (was: Reopened)
TestMapreduceConfigFields is placed in wrong dir, introducing compile error
---
Key: MAPREDUCE-6360
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6360
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Vinayakumar B
Assignee: Arshad Mohammad
Attachments: MAPREDUCE-6360-1.patch
MAPREDUCE-6192 has introduced a Test file {{TestMapreduceConfigFields }}
which was committed in package {{org.apache.hadoop.mapred}}
But the package declaration inside file is {{org.apache.hadoop.mapreduce}}.
By surprise, this is not giving any compile errors in maven build. But
eclipse catches it.
So move {{TestMapreduceConfigFields }} to correct package
{{org.apache.hadoop.mapreduce}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-4461) Resourcemanager UI does not show the queue details in IE
[ https://issues.apache.org/jira/browse/MAPREDUCE-4461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith resolved MAPREDUCE-4461. --- Resolution: Cannot Reproduce IE browser rendering problem should had fixed by YARN-1868. I can not reproduce the issue. Closing as Can not reproduce. Feel free to reopen if problem still exist. Resourcemanager UI does not show the queue details in IE Key: MAPREDUCE-4461 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4461 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Nishan Shetty Attachments: ASF.LICENSE.NOT.GRANTED--screenshot-1.jpg Resourcemanager UI does not show the queue details in IE -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-4378) hadoop-validate-setup.sh fails to execute kinit command in secure mode
[ https://issues.apache.org/jira/browse/MAPREDUCE-4378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith resolved MAPREDUCE-4378. --- Resolution: Cannot Reproduce In trunk or branch-2, I dont see hadoop-validate-setup.sh script file anymore. Closing as 'Cannot reproduce'. Reopen the issue if any script files fails while execution kinit. hadoop-validate-setup.sh fails to execute kinit command in secure mode -- Key: MAPREDUCE-4378 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4378 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.0.0-alpha, 3.0.0 Environment: SUSE Linux Enterprise Server 11 (x86_64) VERSION = 11 PATCHLEVEL = 1 Reporter: Nishan Shetty hadoop-validate-setup.sh is refering to the invalid kinit location. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6304) Specifying node labels when submitting MR jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-6304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated MAPREDUCE-6304: - Attachment: MAPREDUCE-6304.20150511-1.patch Attaching patch with applicable white-space and check style issues fixed Specifying node labels when submitting MR jobs -- Key: MAPREDUCE-6304 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6304 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Jian Fang Assignee: Naganarasimha G R Labels: mapreduce Attachments: MAPREDUCE-6304.20150410-1.patch, MAPREDUCE-6304.20150411-1.patch, MAPREDUCE-6304.20150501-1.patch, MAPREDUCE-6304.20150510-1.patch, MAPREDUCE-6304.20150511-1.patch Per the discussion on YARN-796, we need a mechanism in MAPREDUCE to specify node labels when submitting MR jobs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (MAPREDUCE-6360) TestMapreduceConfigFields is placed in wrong dir, introducing compile error
[
https://issues.apache.org/jira/browse/MAPREDUCE-6360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arshad Mohammad reopened MAPREDUCE-6360:
Assignee: Arshad Mohammad (was: Vinayakumar B)
TestMapreduceConfigFields is placed in wrong dir, introducing compile error
---
Key: MAPREDUCE-6360
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6360
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Vinayakumar B
Assignee: Arshad Mohammad
MAPREDUCE-6192 has introduced a Test file {{TestMapreduceConfigFields }}
which was committed in package {{org.apache.hadoop.mapred}}
But the package declaration inside file is {{org.apache.hadoop.mapreduce}}.
By surprise, this is not giving any compile errors in maven build. But
eclipse catches it.
So move {{TestMapreduceConfigFields }} to correct package
{{org.apache.hadoop.mapreduce}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6360) TestMapreduceConfigFields is placed in wrong dir, introducing compile error
[
https://issues.apache.org/jira/browse/MAPREDUCE-6360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14537921#comment-14537921
]
Arshad Mohammad commented on MAPREDUCE-6360:
In HDFS-8362 only HDFS portion of modified code is accepted. So now this issue
is no more a duplicate of HDFS-8362.
TestMapreduceConfigFields is placed in wrong dir, introducing compile error
---
Key: MAPREDUCE-6360
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6360
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Vinayakumar B
Assignee: Arshad Mohammad
Attachments: MAPREDUCE-6360-1.patch
MAPREDUCE-6192 has introduced a Test file {{TestMapreduceConfigFields }}
which was committed in package {{org.apache.hadoop.mapred}}
But the package declaration inside file is {{org.apache.hadoop.mapreduce}}.
By surprise, this is not giving any compile errors in maven build. But
eclipse catches it.
So move {{TestMapreduceConfigFields }} to correct package
{{org.apache.hadoop.mapreduce}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4754) Job is marked as FAILED and also throwing the TransitonException instead of KILLED when issues a KILL command
[
https://issues.apache.org/jira/browse/MAPREDUCE-4754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14537834#comment-14537834
]
Rohith commented on MAPREDUCE-4754:
---
I see current JobImpl also does not handle JOB_TASK_COMPLETED at KILLED, is it
potential issue still exist in trunk or branch-2?
Job is marked as FAILED and also throwing the TransitonException instead of
KILLED when issues a KILL command
-
Key: MAPREDUCE-4754
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4754
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.1-alpha, 2.0.2-alpha
Reporter: Nishan Shetty
{code}
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
JOB_TASK_COMPLETED at KILLED
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:695)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:119)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:893)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:889)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
at java.lang.Thread.run(Thread.java:662)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4754) Job is marked as FAILED and also throwing the TransitonException instead of KILLED when issues a KILL command
[
https://issues.apache.org/jira/browse/MAPREDUCE-4754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14537835#comment-14537835
]
Rohith commented on MAPREDUCE-4754:
---
Any folks has seen this issue recently?
Job is marked as FAILED and also throwing the TransitonException instead of
KILLED when issues a KILL command
-
Key: MAPREDUCE-4754
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4754
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.1-alpha, 2.0.2-alpha
Reporter: Nishan Shetty
{code}
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
JOB_TASK_COMPLETED at KILLED
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:695)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:119)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:893)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:889)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
at java.lang.Thread.run(Thread.java:662)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-4453) Jobs should be executed as same user in hadoop-validate-setup.sh
[ https://issues.apache.org/jira/browse/MAPREDUCE-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith resolved MAPREDUCE-4453. --- Resolution: Cannot Reproduce In trunk or branch-2 , *su -c* dont exist any more in the script files. Closing as cannot reporduce' Jobs should be executed as same user in hadoop-validate-setup.sh Key: MAPREDUCE-4453 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4453 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.0.1-alpha Reporter: Nishan Shetty 'su -c' command should be removed in hadoop-validate-setup.sh as TeraGen, Terasort and teravalidate jobs should be executed as same user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6174) Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14537932#comment-14537932 ] Eric Payne commented on MAPREDUCE-6174: --- Thanks, [~jira.shegalov], for the detailed review. Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput. --- Key: MAPREDUCE-6174 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6174 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 3.0.0, 2.6.0 Reporter: Eric Payne Assignee: Eric Payne Labels: BB2015-05-RFC Attachments: MAPREDUCE-6174.002.patch, MAPREDUCE-6174.003.patch, MAPREDUCE-6174.004.patch, MAPREDUCE-6174.v1.txt Per MAPREDUCE-6166, both InMemoryMapOutput and OnDiskMapOutput will be doing similar things with regards to IFile streams. In order to make it explicit that InMemoryMapOutput and OnDiskMapOutput are different from 3rd-party implementations, this JIRA will make them subclass a common class (see https://issues.apache.org/jira/browse/MAPREDUCE-6166?focusedCommentId=14223368page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14223368) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6174) Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14537933#comment-14537933 ] Eric Payne commented on MAPREDUCE-6174: --- I will work on it later today or tomorrow. Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput. --- Key: MAPREDUCE-6174 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6174 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 3.0.0, 2.6.0 Reporter: Eric Payne Assignee: Eric Payne Labels: BB2015-05-RFC Attachments: MAPREDUCE-6174.002.patch, MAPREDUCE-6174.003.patch, MAPREDUCE-6174.004.patch, MAPREDUCE-6174.v1.txt Per MAPREDUCE-6166, both InMemoryMapOutput and OnDiskMapOutput will be doing similar things with regards to IFile streams. In order to make it explicit that InMemoryMapOutput and OnDiskMapOutput are different from 3rd-party implementations, this JIRA will make them subclass a common class (see https://issues.apache.org/jira/browse/MAPREDUCE-6166?focusedCommentId=14223368page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14223368) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6360) TestMapreduceConfigFields is placed in wrong dir, introducing compile error
[
https://issues.apache.org/jira/browse/MAPREDUCE-6360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14537960#comment-14537960
]
Hadoop QA commented on MAPREDUCE-6360:
--
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 5m 7s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 26s | There were no new javac warning
messages. |
| {color:green}+1{color} | release audit | 0m 20s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 32s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 35s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 55s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 9m 15s | Tests passed in
hadoop-mapreduce-client-app. |
| | | 25m 46s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12731924/MAPREDUCE-6360-1.patch
|
| Optional Tests | javac unit findbugs checkstyle |
| git revision | trunk / 7e543c2 |
| hadoop-mapreduce-client-app test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5701/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5701/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5701/console |
This message was automatically generated.
TestMapreduceConfigFields is placed in wrong dir, introducing compile error
---
Key: MAPREDUCE-6360
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6360
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Vinayakumar B
Assignee: Arshad Mohammad
Attachments: MAPREDUCE-6360-1.patch
MAPREDUCE-6192 has introduced a Test file {{TestMapreduceConfigFields }}
which was committed in package {{org.apache.hadoop.mapred}}
But the package declaration inside file is {{org.apache.hadoop.mapreduce}}.
By surprise, this is not giving any compile errors in maven build. But
eclipse catches it.
So move {{TestMapreduceConfigFields }} to correct package
{{org.apache.hadoop.mapreduce}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6200) Streaming job -reducer option for Map only job
[ https://issues.apache.org/jira/browse/MAPREDUCE-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kengo Seki updated MAPREDUCE-6200: -- Labels: newbie streaming (was: ) Streaming job -reducer option for Map only job --- Key: MAPREDUCE-6200 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6200 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: DeepakVohra Assignee: Kengo Seki Labels: newbie, streaming -reducer is listed as a Required option, but is a -reducer still required if the Streaming job is Map only? http://hadoop.apache.org/docs/r1.2.1/streaming.html#Specifying+Map-Only+Jobs -reducer should not be listed as a Required option. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6200) Streaming job -reducer option for Map only job
[ https://issues.apache.org/jira/browse/MAPREDUCE-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kengo Seki updated MAPREDUCE-6200: -- Component/s: documentation Streaming job -reducer option for Map only job --- Key: MAPREDUCE-6200 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6200 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation Reporter: DeepakVohra Assignee: Kengo Seki Labels: newbie, streaming -reducer is listed as a Required option, but is a -reducer still required if the Streaming job is Map only? http://hadoop.apache.org/docs/r1.2.1/streaming.html#Specifying+Map-Only+Jobs -reducer should not be listed as a Required option. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6304) Specifying node labels when submitting MR jobs
[
https://issues.apache.org/jira/browse/MAPREDUCE-6304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14537972#comment-14537972
]
Hadoop QA commented on MAPREDUCE-6304:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 38s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 30s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 39s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 30s | The applied patch generated 8
new checkstyle issues (total was 497, now 505). |
| {color:green}+1{color} | whitespace | 0m 2s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 38s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 2m 53s | The patch appears to introduce 1
new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 8m 58s | Tests passed in
hadoop-mapreduce-client-app. |
| {color:green}+1{color} | mapreduce tests | 1m 36s | Tests passed in
hadoop-mapreduce-client-core. |
| {color:green}+1{color} | mapreduce tests | 106m 15s | Tests passed in
hadoop-mapreduce-client-jobclient. |
| | | 155m 44s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-mapreduce-client-app |
| | Inconsistent synchronization of
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.reduceNodeLabelExpression;
locked 66% of time Unsynchronized access at RMContainerAllocator.java:66% of
time Unsynchronized access at RMContainerAllocator.java:[line 218] |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12731906/MAPREDUCE-6304.20150511-1.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 3fa2efc |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5700/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5700/artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
|
| hadoop-mapreduce-client-app test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5700/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5700/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| hadoop-mapreduce-client-jobclient test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5700/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5700/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5700/console |
This message was automatically generated.
Specifying node labels when submitting MR jobs
--
Key: MAPREDUCE-6304
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6304
Project: Hadoop Map/Reduce
Issue Type: New Feature
Reporter: Jian Fang
Assignee: Naganarasimha G R
Labels: mapreduce
Attachments: MAPREDUCE-6304.20150410-1.patch,
MAPREDUCE-6304.20150411-1.patch, MAPREDUCE-6304.20150501-1.patch,
MAPREDUCE-6304.20150510-1.patch, MAPREDUCE-6304.20150511-1.patch
Per the discussion on YARN-796, we need a mechanism in MAPREDUCE to specify
node labels when submitting MR jobs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (MAPREDUCE-6200) Streaming job -reducer option for Map only job
[ https://issues.apache.org/jira/browse/MAPREDUCE-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kengo Seki reassigned MAPREDUCE-6200: - Assignee: Kengo Seki I think -mapper is also optional. I'd like to fix the document. Streaming job -reducer option for Map only job --- Key: MAPREDUCE-6200 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6200 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: DeepakVohra Assignee: Kengo Seki -reducer is listed as a Required option, but is a -reducer still required if the Streaming job is Map only? http://hadoop.apache.org/docs/r1.2.1/streaming.html#Specifying+Map-Only+Jobs -reducer should not be listed as a Required option. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6361) NPE issue in shuffle caused by concurrent issue between copySucceeded() in one thread and copyFailed() in another thread on the same host
Junping Du created MAPREDUCE-6361: - Summary: NPE issue in shuffle caused by concurrent issue between copySucceeded() in one thread and copyFailed() in another thread on the same host Key: MAPREDUCE-6361 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6361 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Junping Du Assignee: Junping Du The failure in log: 2015-05-08 21:00:00,513 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#25 at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.NullPointerException at org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.copyFailed(ShuffleSchedulerImpl.java:267) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:308) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6361) NPE issue in shuffle caused by concurrent issue between copySucceeded() in one thread and copyFailed() in another thread on the same host
[
https://issues.apache.org/jira/browse/MAPREDUCE-6361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14538071#comment-14538071
]
Junping Du commented on MAPREDUCE-6361:
---
NPE get throw in copyFailed() in ShuffleSchedulerImpl.java:267:
{code}
boolean hostFail = hostFailures.get(hostname).get() getMaxHostFailures() ?
true : false;
{code}
It means hostFailures doesn't include hostname that just failed, which is not
expected because we call hostFailed() to put host into hostFailures before
anytime to call copyFailed():
{code}
scheduler.hostFailed(host.getHostName());
for(TaskAttemptID left: failedTasks) {
scheduler.copyFailed(left, host, true, false);
}
{code}
Although hostFailed() and copyFailed() are both synchronized method (so as
copySucceeded()), it is still possible (like the only reason) to cause this NPE
for the other thread calls copySucceeded() on the same host (for other map
output) between we call hostFailed() and copyFailed() in this thread when
taking care of one map output failure.
We need to fix this concurrent issue to get rid of NPE issue which failed map
output copy directly without any retry.
NPE issue in shuffle caused by concurrent issue between copySucceeded() in
one thread and copyFailed() in another thread on the same host
-
Key: MAPREDUCE-6361
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6361
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Junping Du
Assignee: Junping Du
The failure in log:
2015-05-08 21:00:00,513 WARN [main] org.apache.hadoop.mapred.YarnChild:
Exception running child :
org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
shuffle in fetcher#25
at
org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.copyFailed(ShuffleSchedulerImpl.java:267)
at
org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:308)
at
org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6361) NPE issue in shuffle caused by concurrent issue between copySucceeded() in one thread and copyFailed() in another thread on the same host
[ https://issues.apache.org/jira/browse/MAPREDUCE-6361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-6361: -- Priority: Critical (was: Major) NPE issue in shuffle caused by concurrent issue between copySucceeded() in one thread and copyFailed() in another thread on the same host - Key: MAPREDUCE-6361 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6361 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Junping Du Assignee: Junping Du Priority: Critical The failure in log: 2015-05-08 21:00:00,513 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#25 at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.NullPointerException at org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.copyFailed(ShuffleSchedulerImpl.java:267) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:308) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6362) History Plugin should be updated
[
https://issues.apache.org/jira/browse/MAPREDUCE-6362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14538499#comment-14538499
]
Hadoop QA commented on MAPREDUCE-6362:
--
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 34s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 30s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 33s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 27s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 39s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 26s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 0m 17s | Tests passed in
hadoop-mapreduce-client-hs-plugins. |
| {color:green}+1{color} | yarn tests | 1m 56s | Tests passed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 0m 22s | Tests passed in
hadoop-yarn-server-web-proxy. |
| | | 40m 41s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12731975/MAPREDUCE-6362.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 2d671e5 |
| hadoop-mapreduce-client-hs-plugins test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5703/artifact/patchprocess/testrun_hadoop-mapreduce-client-hs-plugins.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5703/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-web-proxy test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5703/artifact/patchprocess/testrun_hadoop-yarn-server-web-proxy.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5703/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5703/console |
This message was automatically generated.
History Plugin should be updated
Key: MAPREDUCE-6362
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6362
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.6.1
Reporter: Mit Desai
Assignee: Mit Desai
Attachments: MAPREDUCE-6362.patch
As applications complete, the RM tracks their IDs in a completed list. This
list is routinely truncated to limit the total number of application
remembered by the RM.
When a user clicks the History for a job, either the browser is redirected to
the application's tracking link obtained from the stored application
instance. But when the application has been purged from the RM, an error is
displayed.
In very busy clusters the rate at which applications complete can cause
applications to be purged from the RM's internal list within hours, which
breaks the proxy URLs users have saved for their jobs.
We would like the RM to provide valid tracking links persist so that users
are not frustrated by broken links.
With the current plugin in place, redirections for the Mapreduce jobs works
but we need the add functionality for tez jobs
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6360) TestMapreduceConfigFields is placed in wrong dir, introducing compile error
[
https://issues.apache.org/jira/browse/MAPREDUCE-6360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14538361#comment-14538361
]
Ray Chiang commented on MAPREDUCE-6360:
---
As with HDFS-8362, it might be better to move into the
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce
directory (where the majority of the tests are). The mapred directory is
probably for tests inherited from MR1.
TestMapreduceConfigFields is placed in wrong dir, introducing compile error
---
Key: MAPREDUCE-6360
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6360
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Vinayakumar B
Assignee: Arshad Mohammad
Attachments: MAPREDUCE-6360-1.patch
MAPREDUCE-6192 has introduced a Test file {{TestMapreduceConfigFields }}
which was committed in package {{org.apache.hadoop.mapred}}
But the package declaration inside file is {{org.apache.hadoop.mapreduce}}.
By surprise, this is not giving any compile errors in maven build. But
eclipse catches it.
So move {{TestMapreduceConfigFields }} to correct package
{{org.apache.hadoop.mapreduce}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6362) History Plugin should be updated
[ https://issues.apache.org/jira/browse/MAPREDUCE-6362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated MAPREDUCE-6362: - Attachment: MAPREDUCE-6362.patch Attached the patch History Plugin should be updated Key: MAPREDUCE-6362 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6362 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.6.1 Reporter: Mit Desai Assignee: Mit Desai Attachments: MAPREDUCE-6362.patch As applications complete, the RM tracks their IDs in a completed list. This list is routinely truncated to limit the total number of application remembered by the RM. When a user clicks the History for a job, either the browser is redirected to the application's tracking link obtained from the stored application instance. But when the application has been purged from the RM, an error is displayed. In very busy clusters the rate at which applications complete can cause applications to be purged from the RM's internal list within hours, which breaks the proxy URLs users have saved for their jobs. We would like the RM to provide valid tracking links persist so that users are not frustrated by broken links. With the current plugin in place, redirections for the Mapreduce jobs works but we need the add functionality for tez jobs -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6298) Job#toString throws an exception when not in state RUNNING
[
https://issues.apache.org/jira/browse/MAPREDUCE-6298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14538729#comment-14538729
]
Lars Francke commented on MAPREDUCE-6298:
-
Thanks for the update [~jira.shegalov]. I'll look into the changes you
mentioned but it'll take me a bit before I can get to this. I'll ping you guys
when I'm ready
Job#toString throws an exception when not in state RUNNING
--
Key: MAPREDUCE-6298
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6298
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Lars Francke
Assignee: Lars Francke
Priority: Minor
Labels: BB2015-05-TBR
Attachments: MAPREDUCE-6298.1.patch
Job#toString calls {{ensureState(JobState.RUNNING);}} as the very first
thing. That method causes an Exception to be thrown which is not nice.
One thing this breaks is usage of Job on the Scala (e.g. Spark) REPL as that
calls toString after every invocation and that fails every time.
I'll attach a patch that checks state and if it's RUNNING prints the original
message and if not prints something else.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6362) History Plugin should be updated
Mit Desai created MAPREDUCE-6362: Summary: History Plugin should be updated Key: MAPREDUCE-6362 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6362 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.6.1 Reporter: Mit Desai As applications complete, the RM tracks their IDs in a completed list. This list is routinely truncated to limit the total number of application remembered by the RM. When a user clicks the History for a job, either the browser is redirected to the application's tracking link obtained from the stored application instance. But when the application has been purged from the RM, an error is displayed. In very busy clusters the rate at which applications complete can cause applications to be purged from the RM's internal list within hours, which breaks the proxy URLs users have saved for their jobs. We would like the RM to provide valid tracking links persist so that users are not frustrated by broken links. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5762) Port MAPREDUCE-3223 (Remove MRv1 config from mapred-default.xml) to branch-2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14538608#comment-14538608 ] Ming Ma commented on MAPREDUCE-5762: [~ajisakaa] it looks like mapreduce.cluster.local.dir was removed as part of this and caused some branch-2 unit tests to fail. Port MAPREDUCE-3223 (Remove MRv1 config from mapred-default.xml) to branch-2 Key: MAPREDUCE-5762 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5762 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Affects Versions: 2.3.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Priority: Minor Fix For: 2.8.0 Attachments: MAPREDUCE-5762-branch-2-002.patch, MAPREDUCE-5762-branch-2.patch MRv1 configs are removed in trunk, but they are not removed in branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5465) Container killed before hprof dumps profile.out
[
https://issues.apache.org/jira/browse/MAPREDUCE-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14538604#comment-14538604
]
Hadoop QA commented on MAPREDUCE-5465:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12732001/MAPREDUCE-5465-branch-2.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / ea11590 |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5704/console |
This message was automatically generated.
Container killed before hprof dumps profile.out
---
Key: MAPREDUCE-5465
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5465
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mr-am, mrv2
Reporter: Radim Kolar
Assignee: Ming Ma
Attachments: MAPREDUCE-5465-2.patch, MAPREDUCE-5465-3.patch,
MAPREDUCE-5465-4.patch, MAPREDUCE-5465-5.patch, MAPREDUCE-5465-6.patch,
MAPREDUCE-5465-7.patch, MAPREDUCE-5465-8.patch, MAPREDUCE-5465-9.patch,
MAPREDUCE-5465-branch-2.patch, MAPREDUCE-5465.patch
If there is profiling enabled for mapper or reducer then hprof dumps
profile.out at process exit. It is dumped after task signaled to AM that work
is finished.
AM kills container with finished work without waiting for hprof to finish
dumps. If hprof is dumping larger outputs (such as with depth=4 while depth=3
works) , it could not finish dump in time before being killed making entire
dump unusable because cpu and heap stats are missing.
There needs to be better delay before container is killed if profiling is
enabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5465) Container killed before hprof dumps profile.out
[ https://issues.apache.org/jira/browse/MAPREDUCE-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated MAPREDUCE-5465: --- Attachment: MAPREDUCE-5465-branch-2.patch Thanks [~jlowe] for spending time on this. For the handling of TA_KILL event when TA is in SUCCESS_CONTAINER_CLEANUP, do you mean https://issues.apache.org/jira/browse/MAPREDUCE-5776? If so, we can address the issue in that jira. Here is the patch for branch-2. Container killed before hprof dumps profile.out --- Key: MAPREDUCE-5465 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5465 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mr-am, mrv2 Reporter: Radim Kolar Assignee: Ming Ma Attachments: MAPREDUCE-5465-2.patch, MAPREDUCE-5465-3.patch, MAPREDUCE-5465-4.patch, MAPREDUCE-5465-5.patch, MAPREDUCE-5465-6.patch, MAPREDUCE-5465-7.patch, MAPREDUCE-5465-8.patch, MAPREDUCE-5465-9.patch, MAPREDUCE-5465-branch-2.patch, MAPREDUCE-5465.patch If there is profiling enabled for mapper or reducer then hprof dumps profile.out at process exit. It is dumped after task signaled to AM that work is finished. AM kills container with finished work without waiting for hprof to finish dumps. If hprof is dumping larger outputs (such as with depth=4 while depth=3 works) , it could not finish dump in time before being killed making entire dump unusable because cpu and heap stats are missing. There needs to be better delay before container is killed if profiling is enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6304) Specifying node labels when submitting MR jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-6304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14539024#comment-14539024 ] Wangda Tan commented on MAPREDUCE-6304: --- Thanks [~Naganarasimha] for working on this and testing, mostly LGTM, could you add the overwriting behavior in mapred-default.xml? (For example, by default is using queue's default-node-label-expression, AM.expression can overwrite job.expression, etc. Specifying node labels when submitting MR jobs -- Key: MAPREDUCE-6304 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6304 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Jian Fang Assignee: Naganarasimha G R Labels: mapreduce Attachments: MAPREDUCE-6304.20150410-1.patch, MAPREDUCE-6304.20150411-1.patch, MAPREDUCE-6304.20150501-1.patch, MAPREDUCE-6304.20150510-1.patch, MAPREDUCE-6304.20150511-1.patch Per the discussion on YARN-796, we need a mechanism in MAPREDUCE to specify node labels when submitting MR jobs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6337) add a mode to replay MR job history files to the timeline service
[ https://issues.apache.org/jira/browse/MAPREDUCE-6337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-6337: --- Labels: (was: BB2015-05-TBR) add a mode to replay MR job history files to the timeline service - Key: MAPREDUCE-6337 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6337 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: MAPREDUCE-6337-YARN-2928.001.patch The subtask covers the work on top of YARN-3437 to add a mode to replay MR job history files to the timeline service storage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6337) add a mode to replay MR job history files to the timeline service
[ https://issues.apache.org/jira/browse/MAPREDUCE-6337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-6337: --- Attachment: MAPREDUCE-6337-YARN-2928.002.patch Patch v.2 posted. Changes: - added the replay mode (1: write all entities at once, 2: write one entity at a time) - used the timeline collector manager to provide the writer to the timeline collectors - refactored the entity writers to provide the base functionality With regards to writing one per event, since this is based on the data structures generated by the job history file parser, there is no easy access to the job history events. Trying to reverse-engineer the events from the info would be considerable effort. Hopefully, writing one entity at a time generates enough chattiness for the write performance. Let me know what you think. Also, on the point of creating timeline entities out of job history, yes, I do agree that there may be benefits in a shared tool for creating timeline entities. But as you point out, the job history code bases it on events whereas the job history parser provides *info classes that are rather different. Since this is test code, I think it is OK to be close enough. If I'm missing some important data, I can add them to the entities. add a mode to replay MR job history files to the timeline service - Key: MAPREDUCE-6337 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6337 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: MAPREDUCE-6337-YARN-2928.001.patch, MAPREDUCE-6337-YARN-2928.002.patch The subtask covers the work on top of YARN-3437 to add a mode to replay MR job history files to the timeline service storage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6363) [NNBench mark] Lease mismatch error when running with multiple mappers
[
https://issues.apache.org/jira/browse/MAPREDUCE-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14539213#comment-14539213
]
Hadoop QA commented on MAPREDUCE-6363:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 5m 21s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 31s | There were no new javac warning
messages. |
| {color:green}+1{color} | release audit | 0m 20s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 35s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 38s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 47s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:red}-1{color} | mapreduce tests | 102m 35s | Tests failed in
hadoop-mapreduce-client-jobclient. |
| | | 119m 22s | |
\\
\\
|| Reason || Tests ||
| Timed out tests |
org.apache.hadoop.mapreduce.v2.TestSpeculativeExecutionWithMRApp |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12693611/HDFS4929.patch |
| Optional Tests | javac unit findbugs checkstyle |
| git revision | trunk / 3d28611 |
| hadoop-mapreduce-client-jobclient test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5705/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5705/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5705/console |
This message was automatically generated.
[NNBench mark] Lease mismatch error when running with multiple mappers
--
Key: MAPREDUCE-6363
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6363
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: benchmarks
Reporter: Brahma Reddy Battula
Assignee: Brahma Reddy Battula
Priority: Critical
Labels: BB2015-05-TBR
Attachments: HDFS4929.patch
Command :
./yarn jar
../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.0.1-tests.jar
nnbench -operation create_write -numberOfFiles 1000 -blockSize 268435456
-bytesToWrite 102400 -baseDir /benchmarks/NNBench`hostname -s`
-replicationFactorPerFile 3 -maps 100 -reduces 10
Trace :
013-06-21 10:44:53,763 INFO org.apache.hadoop.ipc.Server: IPC Server handler
7 on 9005, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from
192.168.105.214:36320: error:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: Lease mismatch
on /benchmarks/NNBenchlinux-185/data/file_linux-214__0 owned by
DFSClient_attempt_1371782327901_0001_m_48_0_1383437860_1 but is accessed
by DFSClient_attempt_1371782327901_0001_m_84_0_1880545303_1
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: Lease mismatch
on /benchmarks/NNBenchlinux-185/data/file_linux-214__0 owned by
DFSClient_attempt_1371782327901_0001_m_48_0_1383437860_1 but is accessed
by DFSClient_attempt_1371782327901_0001_m_84_0_1880545303_1
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2351)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2098)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2019)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:501)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:213)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:52012)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:435)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:925)
at
[jira] [Commented] (MAPREDUCE-6359) RM HA setup, Cluster tab links populated with AM hostname instead of RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-6359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14539066#comment-14539066 ] zhaoyunjiong commented on MAPREDUCE-6359: - Thanks for you time to review the patch. RM HA setup, Cluster tab links populated with AM hostname instead of RM -- Key: MAPREDUCE-6359 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6359 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.4.0 Environment: centOS-6.x Reporter: Aroop Maliakkal Assignee: zhaoyunjiong Priority: Minor Fix For: 2.8.0 Attachments: YARN-3423.patch In RM HA setup ( e.g. http://rm-1.vip.abc.com:50030/proxy/application_1427789305393_0002/ ), go to the job details and click on the Cluster tab on left top side. Click on any of the links , About, Applications , Scheduler. You can see that the hyperlink is pointing to http://am-1.vip.abc.com:port/cluster ). The port details for secure and unsecure cluster is given below :- 8088 ( DEFAULT_RM_WEBAPP_PORT = 8088 ) 8090 ( DEFAULT_RM_WEBAPP_HTTPS_PORT = 8090 ) Ideally, it should have pointed to resourcemanager hostname instead of AM hostname. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6298) Job#toString throws an exception when not in state RUNNING
[
https://issues.apache.org/jira/browse/MAPREDUCE-6298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14538309#comment-14538309
]
Gera Shegalov commented on MAPREDUCE-6298:
--
Thanks for the pointer, [~ozawa]! Thanks for the filing the JIRA and the patch,
[~lars_francke]!
Having toString rely on RUNNING does not make sense, indeed. I believe the
intent is to avoid doing RPC's for jobs that are finished, or the {{cluster}}
is reset. Note the conditions under which ensureState throws exceptions. I
suggest we do this in a more graceful manner. if the job is known not to be
running, simply skip the block with {{updateStatus}} for the conditions above.
However, if {{Job#status}} is already set we should just proceed. Otherwise,
return something to the tune of the patch.
Job#toString throws an exception when not in state RUNNING
--
Key: MAPREDUCE-6298
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6298
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Lars Francke
Assignee: Lars Francke
Priority: Minor
Labels: BB2015-05-TBR
Attachments: MAPREDUCE-6298.1.patch
Job#toString calls {{ensureState(JobState.RUNNING);}} as the very first
thing. That method causes an Exception to be thrown which is not nice.
One thing this breaks is usage of Job on the Scala (e.g. Spark) REPL as that
calls toString after every invocation and that fails every time.
I'll attach a patch that checks state and if it's RUNNING prints the original
message and if not prints something else.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (MAPREDUCE-6362) History Plugin should be updated
[ https://issues.apache.org/jira/browse/MAPREDUCE-6362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai reassigned MAPREDUCE-6362: Assignee: Mit Desai History Plugin should be updated Key: MAPREDUCE-6362 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6362 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.6.1 Reporter: Mit Desai Assignee: Mit Desai As applications complete, the RM tracks their IDs in a completed list. This list is routinely truncated to limit the total number of application remembered by the RM. When a user clicks the History for a job, either the browser is redirected to the application's tracking link obtained from the stored application instance. But when the application has been purged from the RM, an error is displayed. In very busy clusters the rate at which applications complete can cause applications to be purged from the RM's internal list within hours, which breaks the proxy URLs users have saved for their jobs. We would like the RM to provide valid tracking links persist so that users are not frustrated by broken links. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6362) History Plugin should be updated
[ https://issues.apache.org/jira/browse/MAPREDUCE-6362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated MAPREDUCE-6362: - Description: As applications complete, the RM tracks their IDs in a completed list. This list is routinely truncated to limit the total number of application remembered by the RM. When a user clicks the History for a job, either the browser is redirected to the application's tracking link obtained from the stored application instance. But when the application has been purged from the RM, an error is displayed. In very busy clusters the rate at which applications complete can cause applications to be purged from the RM's internal list within hours, which breaks the proxy URLs users have saved for their jobs. We would like the RM to provide valid tracking links persist so that users are not frustrated by broken links. With the current plugin in place, redirections for the Mapreduce jobs works but we need the add functionality for tez jobs was: As applications complete, the RM tracks their IDs in a completed list. This list is routinely truncated to limit the total number of application remembered by the RM. When a user clicks the History for a job, either the browser is redirected to the application's tracking link obtained from the stored application instance. But when the application has been purged from the RM, an error is displayed. In very busy clusters the rate at which applications complete can cause applications to be purged from the RM's internal list within hours, which breaks the proxy URLs users have saved for their jobs. We would like the RM to provide valid tracking links persist so that users are not frustrated by broken links. History Plugin should be updated Key: MAPREDUCE-6362 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6362 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.6.1 Reporter: Mit Desai Assignee: Mit Desai As applications complete, the RM tracks their IDs in a completed list. This list is routinely truncated to limit the total number of application remembered by the RM. When a user clicks the History for a job, either the browser is redirected to the application's tracking link obtained from the stored application instance. But when the application has been purged from the RM, an error is displayed. In very busy clusters the rate at which applications complete can cause applications to be purged from the RM's internal list within hours, which breaks the proxy URLs users have saved for their jobs. We would like the RM to provide valid tracking links persist so that users are not frustrated by broken links. With the current plugin in place, redirections for the Mapreduce jobs works but we need the add functionality for tez jobs -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5465) Tasks are often killed before they exit on their own
[ https://issues.apache.org/jira/browse/MAPREDUCE-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14538802#comment-14538802 ] Hudson commented on MAPREDUCE-5465: --- FAILURE: Integrated in Hadoop-trunk-Commit #7799 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7799/]) MAPREDUCE-5465. Tasks are often killed before they exit on their own. Contributed by Ming Ma (jlowe: rev 444836b3dcd3ee28238af7b5e753d644e8095788) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/TaskAttemptStateInternal.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestKill.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/AppContext.java * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestTaskAttemptFinishingMonitor.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/TaskAttemptFinishingMonitor.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/event/TaskAttemptEventType.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFail.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncher.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecutionWithMRApp.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockAppContext.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java Tasks are often killed before they exit on their own Key: MAPREDUCE-5465 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5465 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mr-am, mrv2 Reporter: Radim Kolar Assignee: Ming Ma Fix For: 2.8.0 Attachments: MAPREDUCE-5465-2.patch, MAPREDUCE-5465-3.patch, MAPREDUCE-5465-4.patch, MAPREDUCE-5465-5.patch, MAPREDUCE-5465-6.patch, MAPREDUCE-5465-7.patch, MAPREDUCE-5465-8.patch, MAPREDUCE-5465-9.patch, MAPREDUCE-5465-branch-2.patch, MAPREDUCE-5465.patch If there is profiling enabled for mapper or reducer then hprof dumps profile.out at process exit. It is dumped after task signaled to AM that work is finished. AM kills container with finished work without waiting for hprof to finish dumps. If hprof is dumping larger outputs (such as with depth=4 while depth=3 works) , it could not finish dump in time before being killed making entire dump unusable because cpu and heap stats are missing. There needs to be better delay before container is killed if profiling is enabled. -- This message
[jira] [Updated] (MAPREDUCE-5465) Tasks are often killed before they exit on their own
[ https://issues.apache.org/jira/browse/MAPREDUCE-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-5465: -- Resolution: Fixed Fix Version/s: 2.8.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks to Ming for the contribution and to Ray for additional review! I committed this to trunk and branch-2. Tasks are often killed before they exit on their own Key: MAPREDUCE-5465 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5465 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mr-am, mrv2 Reporter: Radim Kolar Assignee: Ming Ma Fix For: 2.8.0 Attachments: MAPREDUCE-5465-2.patch, MAPREDUCE-5465-3.patch, MAPREDUCE-5465-4.patch, MAPREDUCE-5465-5.patch, MAPREDUCE-5465-6.patch, MAPREDUCE-5465-7.patch, MAPREDUCE-5465-8.patch, MAPREDUCE-5465-9.patch, MAPREDUCE-5465-branch-2.patch, MAPREDUCE-5465.patch If there is profiling enabled for mapper or reducer then hprof dumps profile.out at process exit. It is dumped after task signaled to AM that work is finished. AM kills container with finished work without waiting for hprof to finish dumps. If hprof is dumping larger outputs (such as with depth=4 while depth=3 works) , it could not finish dump in time before being killed making entire dump unusable because cpu and heap stats are missing. There needs to be better delay before container is killed if profiling is enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5465) Tasks are often killed before they exit on their own
[ https://issues.apache.org/jira/browse/MAPREDUCE-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14538833#comment-14538833 ] Ming Ma commented on MAPREDUCE-5465: Thanks Jason and Ray. Tasks are often killed before they exit on their own Key: MAPREDUCE-5465 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5465 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mr-am, mrv2 Reporter: Radim Kolar Assignee: Ming Ma Fix For: 2.8.0 Attachments: MAPREDUCE-5465-2.patch, MAPREDUCE-5465-3.patch, MAPREDUCE-5465-4.patch, MAPREDUCE-5465-5.patch, MAPREDUCE-5465-6.patch, MAPREDUCE-5465-7.patch, MAPREDUCE-5465-8.patch, MAPREDUCE-5465-9.patch, MAPREDUCE-5465-branch-2.patch, MAPREDUCE-5465.patch If there is profiling enabled for mapper or reducer then hprof dumps profile.out at process exit. It is dumped after task signaled to AM that work is finished. AM kills container with finished work without waiting for hprof to finish dumps. If hprof is dumping larger outputs (such as with depth=4 while depth=3 works) , it could not finish dump in time before being killed making entire dump unusable because cpu and heap stats are missing. There needs to be better delay before container is killed if profiling is enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5465) Tasks are often killed before they exit on their own
[ https://issues.apache.org/jira/browse/MAPREDUCE-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-5465: -- Summary: Tasks are often killed before they exit on their own (was: Container killed before hprof dumps profile.out) Thanks for the branch-2 patch, Ming! +1 lgtm. Committing this. Tasks are often killed before they exit on their own Key: MAPREDUCE-5465 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5465 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mr-am, mrv2 Reporter: Radim Kolar Assignee: Ming Ma Attachments: MAPREDUCE-5465-2.patch, MAPREDUCE-5465-3.patch, MAPREDUCE-5465-4.patch, MAPREDUCE-5465-5.patch, MAPREDUCE-5465-6.patch, MAPREDUCE-5465-7.patch, MAPREDUCE-5465-8.patch, MAPREDUCE-5465-9.patch, MAPREDUCE-5465-branch-2.patch, MAPREDUCE-5465.patch If there is profiling enabled for mapper or reducer then hprof dumps profile.out at process exit. It is dumped after task signaled to AM that work is finished. AM kills container with finished work without waiting for hprof to finish dumps. If hprof is dumping larger outputs (such as with depth=4 while depth=3 works) , it could not finish dump in time before being killed making entire dump unusable because cpu and heap stats are missing. There needs to be better delay before container is killed if profiling is enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6360) TestMapreduceConfigFields is placed in wrong dir, introducing compile error
[
https://issues.apache.org/jira/browse/MAPREDUCE-6360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14539263#comment-14539263
]
Vinayakumar B commented on MAPREDUCE-6360:
--
bq. it might be better to move into the
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce
directory
Yes, thats sounds reasonable to be as well.
[~arshad.mohammad], can you post one more patch which moves the file itself
instead of importing other things to current file. ?
TestMapreduceConfigFields is placed in wrong dir, introducing compile error
---
Key: MAPREDUCE-6360
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6360
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Vinayakumar B
Assignee: Arshad Mohammad
Attachments: MAPREDUCE-6360-1.patch
MAPREDUCE-6192 has introduced a Test file {{TestMapreduceConfigFields }}
which was committed in package {{org.apache.hadoop.mapred}}
But the package declaration inside file is {{org.apache.hadoop.mapreduce}}.
By surprise, this is not giving any compile errors in maven build. But
eclipse catches it.
So move {{TestMapreduceConfigFields }} to correct package
{{org.apache.hadoop.mapreduce}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6360) TestMapreduceConfigFields is placed in wrong dir, introducing compile error
[
https://issues.apache.org/jira/browse/MAPREDUCE-6360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arshad Mohammad updated MAPREDUCE-6360:
---
Attachment: MAPREDUCE-6360-2.patch
moved TestMapreduceConfigFields to org.apache.hadoop.mapreduce package
TestMapreduceConfigFields is placed in wrong dir, introducing compile error
---
Key: MAPREDUCE-6360
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6360
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Vinayakumar B
Assignee: Arshad Mohammad
Attachments: MAPREDUCE-6360-1.patch, MAPREDUCE-6360-2.patch
MAPREDUCE-6192 has introduced a Test file {{TestMapreduceConfigFields }}
which was committed in package {{org.apache.hadoop.mapred}}
But the package declaration inside file is {{org.apache.hadoop.mapreduce}}.
By surprise, this is not giving any compile errors in maven build. But
eclipse catches it.
So move {{TestMapreduceConfigFields }} to correct package
{{org.apache.hadoop.mapreduce}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6304) Specifying node labels when submitting MR jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-6304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated MAPREDUCE-6304: - Attachment: MAPREDUCE-6304.20150512-1.patch Hi [~Wangda], have modified the documentation in Mapred-default.xml. Please check. Specifying node labels when submitting MR jobs -- Key: MAPREDUCE-6304 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6304 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Jian Fang Assignee: Naganarasimha G R Labels: mapreduce Attachments: MAPREDUCE-6304.20150410-1.patch, MAPREDUCE-6304.20150411-1.patch, MAPREDUCE-6304.20150501-1.patch, MAPREDUCE-6304.20150510-1.patch, MAPREDUCE-6304.20150511-1.patch, MAPREDUCE-6304.20150512-1.patch Per the discussion on YARN-796, we need a mechanism in MAPREDUCE to specify node labels when submitting MR jobs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6360) TestMapreduceConfigFields is placed in wrong dir, introducing compile error
[
https://issues.apache.org/jira/browse/MAPREDUCE-6360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14539302#comment-14539302
]
Vinayakumar B commented on MAPREDUCE-6360:
--
Patch didnt include the deleted file. This will be considered as new file. not
moved file.
Plz check again.
TestMapreduceConfigFields is placed in wrong dir, introducing compile error
---
Key: MAPREDUCE-6360
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6360
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Vinayakumar B
Assignee: Arshad Mohammad
Attachments: MAPREDUCE-6360-1.patch, MAPREDUCE-6360-2.patch
MAPREDUCE-6192 has introduced a Test file {{TestMapreduceConfigFields }}
which was committed in package {{org.apache.hadoop.mapred}}
But the package declaration inside file is {{org.apache.hadoop.mapreduce}}.
By surprise, this is not giving any compile errors in maven build. But
eclipse catches it.
So move {{TestMapreduceConfigFields }} to correct package
{{org.apache.hadoop.mapreduce}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Moved] (MAPREDUCE-6363) [NNBench mark] Lease mismatch error when running with multiple mappers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula moved HDFS-4929 to MAPREDUCE-6363: --- Component/s: (was: benchmarks) benchmarks Key: MAPREDUCE-6363 (was: HDFS-4929) Project: Hadoop Map/Reduce (was: Hadoop HDFS) [NNBench mark] Lease mismatch error when running with multiple mappers -- Key: MAPREDUCE-6363 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6363 Project: Hadoop Map/Reduce Issue Type: Bug Components: benchmarks Reporter: Brahma Reddy Battula Assignee: Brahma Reddy Battula Priority: Critical Labels: BB2015-05-TBR Attachments: HDFS4929.patch Command : ./yarn jar ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.0.1-tests.jar nnbench -operation create_write -numberOfFiles 1000 -blockSize 268435456 -bytesToWrite 102400 -baseDir /benchmarks/NNBench`hostname -s` -replicationFactorPerFile 3 -maps 100 -reduces 10 Trace : 013-06-21 10:44:53,763 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 9005, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from 192.168.105.214:36320: error: org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: Lease mismatch on /benchmarks/NNBenchlinux-185/data/file_linux-214__0 owned by DFSClient_attempt_1371782327901_0001_m_48_0_1383437860_1 but is accessed by DFSClient_attempt_1371782327901_0001_m_84_0_1880545303_1 org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: Lease mismatch on /benchmarks/NNBenchlinux-185/data/file_linux-214__0 owned by DFSClient_attempt_1371782327901_0001_m_48_0_1383437860_1 but is accessed by DFSClient_attempt_1371782327901_0001_m_84_0_1880545303_1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2351) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2098) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2019) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:501) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:213) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:52012) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:435) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:925) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1710) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1706) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-1439) Learning Scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-1439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14539113#comment-14539113 ] CichaoLee commented on MAPREDUCE-1439: -- I find it very intresting and useful, but I can not open the source code link, can you share me? my email: cichaole...@163.com Thank you very much Learning Scheduler -- Key: MAPREDUCE-1439 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1439 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobtracker Reporter: Jaideep Attachments: learning-scheduler-description.pdf I would like to contribute the scheduler I have written to the MapReduce project. Presently the scheduler source code is available on http://code.google.com/p/learnsched/. It has been tested to work with Hadoop 0.20, although the code available at the URL had been modified to build with trunk and needs testing. Currently the scheduler is in experimental stages, and any feedback for improvement will be extremely useful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4901) JobHistoryEventHandler errors should be fatal
[ https://issues.apache.org/jira/browse/MAPREDUCE-4901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14538213#comment-14538213 ] Robert Joseph Evans commented on MAPREDUCE-4901: [~mitdesai], I really don't have a lot of time to devote to this right now. If you are willing to wait I can try to take a look at upmerging, but it would probably be faster for someone else to take it if it is important to them. JobHistoryEventHandler errors should be fatal - Key: MAPREDUCE-4901 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4901 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 2.0.0-alpha Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Labels: BB2015-05-TBR Attachments: MR-4901-trunk.txt To be able to truly fix issues like MAPREDUCE-4819 and MAPREDUCE-4832, we need a 2 phase commit where a subsequent AM can be sure that at a specific point in time it knows exactly if any tasks/jobs are committing. The job history log is already used for similar functionality so we would like to reuse this, but we need to be sure that errors while writing out to the job history log are now fatal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4901) JobHistoryEventHandler errors should be fatal
[
https://issues.apache.org/jira/browse/MAPREDUCE-4901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14538233#comment-14538233
]
Hadoop QA commented on MAPREDUCE-4901:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | patch | 0m 1s | The patch file was not named
according to hadoop's naming conventions. Please see
https://wiki.apache.org/hadoop/HowToContribute for instructions. |
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12562403/MR-4901-trunk.txt |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / b9cebfc |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5702/console |
This message was automatically generated.
JobHistoryEventHandler errors should be fatal
-
Key: MAPREDUCE-4901
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4901
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0, 2.0.0-alpha
Reporter: Robert Joseph Evans
Assignee: Robert Joseph Evans
Labels: BB2015-05-TBR
Attachments: MR-4901-trunk.txt
To be able to truly fix issues like MAPREDUCE-4819 and MAPREDUCE-4832, we
need a 2 phase commit where a subsequent AM can be sure that at a specific
point in time it knows exactly if any tasks/jobs are committing. The job
history log is already used for similar functionality so we would like to
reuse this, but we need to be sure that errors while writing out to the job
history log are now fatal.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6164) mapreduce.reduce.shuffle.fetch.retry.timeout-ms should be set to 3 minutes instead of 30 seconds by default to be consistent with other retry timeout
[ https://issues.apache.org/jira/browse/MAPREDUCE-6164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-6164: -- Target Version/s: 2.8.0 (was: 2.6.1) mapreduce.reduce.shuffle.fetch.retry.timeout-ms should be set to 3 minutes instead of 30 seconds by default to be consistent with other retry timeout Key: MAPREDUCE-6164 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6164 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Junping Du Assignee: Junping Du Labels: BB2015-05-TBR Attachments: MAPREDUCE-6164.patch In MAPREDUCE-5891, we are adding retry logic to MAPREDUCE shuffle stage for fetcher can be survival during NM downtime (with shuffle service down as well). In many places, we are setting the default timeout to be 3 minutes (connection timeout, etc.) to tolerant possible more time for NM down, but we are making mapreduce.reduce.shuffle.fetch.retry.timeout-ms to be 30 seconds which is not consistent here. We should change this to 180 seconds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
