[jira] [Created] (MAPREDUCE-5292) ShuffleConsumerPlugin.Context should support org.apache.hadoop.mapreduce.Reducer
Tsuyoshi OZAWA created MAPREDUCE-5292: - Summary: ShuffleConsumerPlugin.Context should support org.apache.hadoop.mapreduce.Reducer Key: MAPREDUCE-5292 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5292 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: trunk Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Priority: Blocker ShuffleConsumerPlugin.Context only supports org.apache.hadoop.mapred.Reducer currently. Because of this, Reduce side Combiner is not used when using the new API, and just ignored. Please see MAPREDUCE-5221 for more detail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5293) Shuffle#MergeManager should support org.apache.hadoop.mapreduce.Reducer
Tsuyoshi OZAWA created MAPREDUCE-5293: - Summary: Shuffle#MergeManager should support org.apache.hadoop.mapreduce.Reducer Key: MAPREDUCE-5293 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5293 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: trunk Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Priority: Blocker Shuffle#MergeManager only accepts org.apache.hadoop.mapred.Reducer currently. Because of this, Reduce-side Combiner is not used when using the new API, and just ignored. By supporting it and using the feature from ReduceTask, Reduce-side combiner can be enabled with new API. Please see MAPREDUCE-5221 for more detail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5295) ShuffleConsumerPlugin.Context should support org.apache.hadoop.mapreduce.Reducer
Tsuyoshi OZAWA created MAPREDUCE-5295: - Summary: ShuffleConsumerPlugin.Context should support org.apache.hadoop.mapreduce.Reducer Key: MAPREDUCE-5295 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5295 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: trunk, 2.1.0-beta, 2.0.5-alpha Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA ShuffleConsumerPlugin.Context only supports org.apache.hadoop.mapred.Reducer currently. Because of this, Reduce side Combiner is not used when using the new API, and just ignored. Please see MAPREDUCE-5221 for more detail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5294) Shuffle#MergeManager should support org.apache.hadoop.mapreduce.Reducer
Tsuyoshi OZAWA created MAPREDUCE-5294: - Summary: Shuffle#MergeManager should support org.apache.hadoop.mapreduce.Reducer Key: MAPREDUCE-5294 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5294 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: trunk, 2.1.0-beta, 2.0.5-alpha Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Shuffle#MergeManager only accepts org.apache.hadoop.mapred.Reducer currently. Because of this, Reduce-side Combiner is not used when using the new API, and just ignored. By supporting it and using the feature from ReduceTask, Reduce-side combiner can be enabled with new API. Please see MAPREDUCE-5221 for more detail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5292) ShuffleConsumerPlugin.Context should support org.apache.hadoop.mapreduce.Reducer
[ https://issues.apache.org/jira/browse/MAPREDUCE-5292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA resolved MAPREDUCE-5292. --- Resolution: Duplicate Moved to MAPREDUCE-5294. ShuffleConsumerPlugin.Context should support org.apache.hadoop.mapreduce.Reducer Key: MAPREDUCE-5292 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5292 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: trunk Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Priority: Blocker ShuffleConsumerPlugin.Context only supports org.apache.hadoop.mapred.Reducer currently. Because of this, Reduce side Combiner is not used when using the new API, and just ignored. Please see MAPREDUCE-5221 for more detail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5294) Shuffle#MergeManager should support org.apache.hadoop.mapreduce.Reducer
[ https://issues.apache.org/jira/browse/MAPREDUCE-5294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA resolved MAPREDUCE-5294. --- Resolution: Duplicate Moved to MAPREDUCE-5294, subtask of MAPREDUCE-5221. Shuffle#MergeManager should support org.apache.hadoop.mapreduce.Reducer --- Key: MAPREDUCE-5294 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5294 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: trunk, 2.1.0-beta, 2.0.5-alpha Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Shuffle#MergeManager only accepts org.apache.hadoop.mapred.Reducer currently. Because of this, Reduce-side Combiner is not used when using the new API, and just ignored. By supporting it and using the feature from ReduceTask, Reduce-side combiner can be enabled with new API. Please see MAPREDUCE-5221 for more detail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5293) Shuffle#MergeManager should support org.apache.hadoop.mapreduce.Reducer
[ https://issues.apache.org/jira/browse/MAPREDUCE-5293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA resolved MAPREDUCE-5293. --- Resolution: Duplicate Moved to MAPREDUCE-5294, subtask of MAPREDUCE-5221. Shuffle#MergeManager should support org.apache.hadoop.mapreduce.Reducer --- Key: MAPREDUCE-5293 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5293 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: trunk Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Priority: Blocker Shuffle#MergeManager only accepts org.apache.hadoop.mapred.Reducer currently. Because of this, Reduce-side Combiner is not used when using the new API, and just ignored. By supporting it and using the feature from ReduceTask, Reduce-side combiner can be enabled with new API. Please see MAPREDUCE-5221 for more detail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-5294) Shuffle#MergeManager should support org.apache.hadoop.mapreduce.Reducer
[ https://issues.apache.org/jira/browse/MAPREDUCE-5294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA reopened MAPREDUCE-5294: --- Sorry, I resolved this jira wrongly, so reopen it. Shuffle#MergeManager should support org.apache.hadoop.mapreduce.Reducer --- Key: MAPREDUCE-5294 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5294 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: trunk, 2.1.0-beta, 2.0.5-alpha Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Shuffle#MergeManager only accepts org.apache.hadoop.mapred.Reducer currently. Because of this, Reduce-side Combiner is not used when using the new API, and just ignored. By supporting it and using the feature from ReduceTask, Reduce-side combiner can be enabled with new API. Please see MAPREDUCE-5221 for more detail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5288) ResourceEstimator#getEstimatedTotalMapOutputSize suffers from divide by zero issues
[
https://issues.apache.org/jira/browse/MAPREDUCE-5288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13672980#comment-13672980
]
Fengdong Yu commented on MAPREDUCE-5288:
This was fixed in hadoop-1.1.2 stable release.
if we determine completedMapsInputSize is zero, then job's map tasks MUST be
zero, so the estimated output size is zero.
below is the code:
{code}
long getEstimatedMapOutputSize() {
long estimate = 0L;
if (job.desiredMaps() 0) {
estimate = getEstimatedTotalMapOutputSize() / job.desiredMaps();
}
return estimate;
}
{code}
ResourceEstimator#getEstimatedTotalMapOutputSize suffers from divide by zero
issues
---
Key: MAPREDUCE-5288
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5288
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv1
Affects Versions: 1.2.0
Reporter: Harsh J
Assignee: Karthik Kambatla
The computation in the above mentioned class-method is below:
{code}
long estimate = Math.round(((double)inputSize *
completedMapsOutputSize * 2.0)/completedMapsInputSize);
{code}
Given
http://docs.oracle.com/javase/6/docs/api/java/lang/Math.html#round(double),
its possible that the returned estimate could be Long.MAX_VALUE if
completedMapsInputSize is determined to be zero.
This can be proven with a simple code snippet:
{code}
class Foo {
public static void main(String... args) {
long inputSize = 600L + 2;
long estimate = Math.round(((double)inputSize *
1L * 2.0)/0L);
System.out.println(estimate);
}
}
{code}
The above conveniently prints out: {{9223372036854775807}}, which is
Long.MAX_VALUE (or 8 Exbibytes per MapReduce).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5291) Change MR App to use update property names in container-log4j.properties
[ https://issues.apache.org/jira/browse/MAPREDUCE-5291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13672998#comment-13672998 ] Hudson commented on MAPREDUCE-5291: --- Integrated in Hadoop-Yarn-trunk #229 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/229/]) YARN-720 and MAPREDUCE-5291. container-log4j.properties should not refer to mapreduce properties. Update MRApp to use YARN properties for log setup. Contributed by Zhijie Shen. (Revision 1488829) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1488829 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestMapReduceChildJVM.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestTaskLog.java * /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/streaming/TestStreamingTaskLog.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/container-log4j.properties Change MR App to use update property names in container-log4j.properties Key: MAPREDUCE-5291 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5291 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Siddharth Seth Assignee: Zhijie Shen Fix For: 2.1.0-beta Attachments: MAPREDUCE-5291.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5) Shuffle's getMapOutput() fails with EofException, followed by IllegalStateException
[ https://issues.apache.org/jira/browse/MAPREDUCE-5?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673021#comment-13673021 ] Tan Cheng Wee commented on MAPREDUCE-5: --- Will cut down the number of parallel copies help to address this issue? Shuffle's getMapOutput() fails with EofException, followed by IllegalStateException --- Key: MAPREDUCE-5 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.2, 1.1.1 Environment: Sun Java 1.6.0_13, OpenSolaris, running on a SunFire 4150 (x64) 10 node cluster Reporter: George Porter Attachments: temp.rar During the shuffle phase, I'm seeing a large sequence of the following actions: 1) WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_200905181452_0002_m_10_0,0) failed : org.mortbay.jetty.EofException 2) WARN org.mortbay.log: Committed before 410 getMapOutput(attempt_200905181452_0002_m_10_0,0) failed : org.mortbay.jetty.EofException 3) ERROR org.mortbay.log: /mapOutput java.lang.IllegalStateException: Committed The map phase completes with 100%, and then the reduce phase crawls along with the above errors in each of the TaskTracker logs. None of the tasktrackers get lost. When I run non-data jobs like the 'pi' test from the example jar, everything works fine. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4071) NPE while executing MRAppMaster shutdown hook
[
https://issues.apache.org/jira/browse/MAPREDUCE-4071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673022#comment-13673022
]
nemon lou commented on MAPREDUCE-4071:
--
When split metadata size exceeds 1000,MRAppMaster will throw an IOException
during start and the NPE shows up.
Bhallamudi,are you still working on this?
NPE while executing MRAppMaster shutdown hook
-
Key: MAPREDUCE-4071
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4071
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am, mrv2
Affects Versions: 0.23.3, 2.0.0-alpha, trunk
Reporter: Bhallamudi Venkata Siva Kamesh
Attachments: MAPREDUCE-4071-1.patch, MAPREDUCE-4071-2.patch,
MAPREDUCE-4071-2.patch, MAPREDUCE-4071.patch
While running the shutdown hook of MRAppMaster, hit NPE
{noformat}
Exception in thread Thread-1 java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.setSignalled(MRAppMaster.java:668)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1004)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5291) Change MR App to use update property names in container-log4j.properties
[ https://issues.apache.org/jira/browse/MAPREDUCE-5291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673088#comment-13673088 ] Hudson commented on MAPREDUCE-5291: --- Integrated in Hadoop-Hdfs-trunk #1419 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1419/]) YARN-720 and MAPREDUCE-5291. container-log4j.properties should not refer to mapreduce properties. Update MRApp to use YARN properties for log setup. Contributed by Zhijie Shen. (Revision 1488829) Result = FAILURE sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1488829 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestMapReduceChildJVM.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestTaskLog.java * /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/streaming/TestStreamingTaskLog.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/container-log4j.properties Change MR App to use update property names in container-log4j.properties Key: MAPREDUCE-5291 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5291 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Siddharth Seth Assignee: Zhijie Shen Fix For: 2.1.0-beta Attachments: MAPREDUCE-5291.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4548) M/R jobs can not access S3 if Kerberos is enabled
[
https://issues.apache.org/jira/browse/MAPREDUCE-4548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohamed Abdul Rasheed updated MAPREDUCE-4548:
-
Environment:
hadoop-1.0.0;MIT kerberos;java 1.6.0_26
CDH4.3.0(hadoop 2.0.0-alpha);MIT kerberos;java 1.6.0_26
was:hadoop-1.0.0;MIT kerberos;java 1.6.0_26
M/R jobs can not access S3 if Kerberos is enabled
-
Key: MAPREDUCE-4548
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4548
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.0.0-alpha
Environment: hadoop-1.0.0;MIT kerberos;java 1.6.0_26
CDH4.3.0(hadoop 2.0.0-alpha);MIT kerberos;java 1.6.0_26
Reporter: Manuel DE FERRAN
With Kerberos enabled, any job that is taking as input or output s3 files
fails.
It can be easily reproduced with wordcount shipped in hadoop-examples.jar and
a public S3 file:
{code}
/opt/hadoop/bin/hadoop --config /opt/hadoop/conf/ jar
/opt/hadoop/hadoop-examples-1.0.0.jar wordcount s3n://ubikodpublic/test out01
{code}
returns:
{code}
12/08/10 12:40:19 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
192 for hadoop on 10.85.151.233:9000
12/08/10 12:40:19 INFO security.TokenCache: Got dt for
hdfs://aws04.machine.com:9000/mapred/staging/hadoop/.staging/job_201208101229_0004;uri=10.85.151.233:9000;t.service=10.85.151.233:9000
12/08/10 12:40:19 INFO mapred.JobClient: Cleaning up the staging area
hdfs://aws04.machine.com:9000/mapred/staging/hadoop/.staging/job_201208101229_0004
java.lang.IllegalArgumentException: java.net.UnknownHostException:
ubikodpublic
at
org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:293)
at
org.apache.hadoop.security.SecurityUtil.buildDTServiceName(SecurityUtil.java:317)
at
org.apache.hadoop.fs.FileSystem.getCanonicalServiceName(FileSystem.java:189)
at
org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:92)
at
org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:79)
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:197)
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:252)
SNIP
{code}
This patch seems to fix it.
{code}
Index: core/org/apache/hadoop/security/SecurityUtil.java
===
--- core/org/apache/hadoop/security/SecurityUtil.java (révision 1305278)
+++ core/org/apache/hadoop/security/SecurityUtil.java (copie de travail)
@@ -313,6 +313,9 @@
if (authority == null || authority.isEmpty()) {
return null;
}
+if (uri.getScheme().equals(s3n) || uri.getScheme().equals(s3)) {
+ return null;
+}
InetSocketAddress addr = NetUtils.createSocketAddr(authority, defPort);
return buildTokenService(addr).toString();
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4548) M/R jobs can not access S3 if Kerberos is enabled
[
https://issues.apache.org/jira/browse/MAPREDUCE-4548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohamed Abdul Rasheed updated MAPREDUCE-4548:
-
Affects Version/s: 2.0.0-alpha
M/R jobs can not access S3 if Kerberos is enabled
-
Key: MAPREDUCE-4548
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4548
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.0.0-alpha
Environment: hadoop-1.0.0;MIT kerberos;java 1.6.0_26
Reporter: Manuel DE FERRAN
With Kerberos enabled, any job that is taking as input or output s3 files
fails.
It can be easily reproduced with wordcount shipped in hadoop-examples.jar and
a public S3 file:
{code}
/opt/hadoop/bin/hadoop --config /opt/hadoop/conf/ jar
/opt/hadoop/hadoop-examples-1.0.0.jar wordcount s3n://ubikodpublic/test out01
{code}
returns:
{code}
12/08/10 12:40:19 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
192 for hadoop on 10.85.151.233:9000
12/08/10 12:40:19 INFO security.TokenCache: Got dt for
hdfs://aws04.machine.com:9000/mapred/staging/hadoop/.staging/job_201208101229_0004;uri=10.85.151.233:9000;t.service=10.85.151.233:9000
12/08/10 12:40:19 INFO mapred.JobClient: Cleaning up the staging area
hdfs://aws04.machine.com:9000/mapred/staging/hadoop/.staging/job_201208101229_0004
java.lang.IllegalArgumentException: java.net.UnknownHostException:
ubikodpublic
at
org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:293)
at
org.apache.hadoop.security.SecurityUtil.buildDTServiceName(SecurityUtil.java:317)
at
org.apache.hadoop.fs.FileSystem.getCanonicalServiceName(FileSystem.java:189)
at
org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:92)
at
org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:79)
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:197)
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:252)
SNIP
{code}
This patch seems to fix it.
{code}
Index: core/org/apache/hadoop/security/SecurityUtil.java
===
--- core/org/apache/hadoop/security/SecurityUtil.java (révision 1305278)
+++ core/org/apache/hadoop/security/SecurityUtil.java (copie de travail)
@@ -313,6 +313,9 @@
if (authority == null || authority.isEmpty()) {
return null;
}
+if (uri.getScheme().equals(s3n) || uri.getScheme().equals(s3)) {
+ return null;
+}
InetSocketAddress addr = NetUtils.createSocketAddr(authority, defPort);
return buildTokenService(addr).toString();
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4548) M/R jobs can not access S3 if Kerberos is enabled
[
https://issues.apache.org/jira/browse/MAPREDUCE-4548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673103#comment-13673103
]
Mohamed Abdul Rasheed commented on MAPREDUCE-4548:
--
I encountered this bug while using DistCp to S3. The patch fixed the problem.
Thanks!!!
M/R jobs can not access S3 if Kerberos is enabled
-
Key: MAPREDUCE-4548
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4548
Project: Hadoop Map/Reduce
Issue Type: Bug
Environment: hadoop-1.0.0;MIT kerberos;java 1.6.0_26
Reporter: Manuel DE FERRAN
With Kerberos enabled, any job that is taking as input or output s3 files
fails.
It can be easily reproduced with wordcount shipped in hadoop-examples.jar and
a public S3 file:
{code}
/opt/hadoop/bin/hadoop --config /opt/hadoop/conf/ jar
/opt/hadoop/hadoop-examples-1.0.0.jar wordcount s3n://ubikodpublic/test out01
{code}
returns:
{code}
12/08/10 12:40:19 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
192 for hadoop on 10.85.151.233:9000
12/08/10 12:40:19 INFO security.TokenCache: Got dt for
hdfs://aws04.machine.com:9000/mapred/staging/hadoop/.staging/job_201208101229_0004;uri=10.85.151.233:9000;t.service=10.85.151.233:9000
12/08/10 12:40:19 INFO mapred.JobClient: Cleaning up the staging area
hdfs://aws04.machine.com:9000/mapred/staging/hadoop/.staging/job_201208101229_0004
java.lang.IllegalArgumentException: java.net.UnknownHostException:
ubikodpublic
at
org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:293)
at
org.apache.hadoop.security.SecurityUtil.buildDTServiceName(SecurityUtil.java:317)
at
org.apache.hadoop.fs.FileSystem.getCanonicalServiceName(FileSystem.java:189)
at
org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:92)
at
org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:79)
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:197)
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:252)
SNIP
{code}
This patch seems to fix it.
{code}
Index: core/org/apache/hadoop/security/SecurityUtil.java
===
--- core/org/apache/hadoop/security/SecurityUtil.java (révision 1305278)
+++ core/org/apache/hadoop/security/SecurityUtil.java (copie de travail)
@@ -313,6 +313,9 @@
if (authority == null || authority.isEmpty()) {
return null;
}
+if (uri.getScheme().equals(s3n) || uri.getScheme().equals(s3)) {
+ return null;
+}
InetSocketAddress addr = NetUtils.createSocketAddr(authority, defPort);
return buildTokenService(addr).toString();
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5291) Change MR App to use update property names in container-log4j.properties
[ https://issues.apache.org/jira/browse/MAPREDUCE-5291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673137#comment-13673137 ] Hudson commented on MAPREDUCE-5291: --- Integrated in Hadoop-Mapreduce-trunk #1445 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1445/]) YARN-720 and MAPREDUCE-5291. container-log4j.properties should not refer to mapreduce properties. Update MRApp to use YARN properties for log setup. Contributed by Zhijie Shen. (Revision 1488829) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1488829 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestMapReduceChildJVM.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestTaskLog.java * /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/streaming/TestStreamingTaskLog.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/container-log4j.properties Change MR App to use update property names in container-log4j.properties Key: MAPREDUCE-5291 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5291 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Siddharth Seth Assignee: Zhijie Shen Fix For: 2.1.0-beta Attachments: MAPREDUCE-5291.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5268) Improve history server startup performance
[ https://issues.apache.org/jira/browse/MAPREDUCE-5268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-5268: -- Resolution: Fixed Fix Version/s: 0.23.9 2.1.0-beta Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks, Karthik! I committed this to trunk, branch-2, and branch-0.23. Improve history server startup performance -- Key: MAPREDUCE-5268 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5268 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobhistoryserver Affects Versions: 0.23.7, 2.0.4-alpha Reporter: Jason Lowe Assignee: Karthik Kambatla Fix For: 2.1.0-beta, 0.23.9 Attachments: mr-5268.patch, mr-5268.patch, mr-5268.patch, mr-5268.patch, mr-5268.patch, mr-5268-prelim.patch The history server can easily take many minutes to startup when there are a significant number of jobs to scan in the done directory. However the scanning of files is not the bottleneck, rather it's the heavy use of ConcurrentSkipListMap.size in HistoryFileManager. ConcurrentSkipListMap.size is a very expensive operation, especially on maps with many entries, as it has to scan every entry to compute the size. We should avoid calling this method or at least minimize its use. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5268) Improve history server startup performance
[ https://issues.apache.org/jira/browse/MAPREDUCE-5268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673181#comment-13673181 ] Hudson commented on MAPREDUCE-5268: --- Integrated in Hadoop-trunk-Commit #3837 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3837/]) MAPREDUCE-5268. Improve history server startup performance. Contributed by Karthik Kambatla (Revision 1489012) Result = SUCCESS jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1489012 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobIdHistoryFileInfoMap.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobListCache.java Improve history server startup performance -- Key: MAPREDUCE-5268 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5268 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobhistoryserver Affects Versions: 0.23.7, 2.0.4-alpha Reporter: Jason Lowe Assignee: Karthik Kambatla Fix For: 2.1.0-beta, 0.23.9 Attachments: mr-5268.patch, mr-5268.patch, mr-5268.patch, mr-5268.patch, mr-5268.patch, mr-5268-prelim.patch The history server can easily take many minutes to startup when there are a significant number of jobs to scan in the done directory. However the scanning of files is not the bottleneck, rather it's the heavy use of ConcurrentSkipListMap.size in HistoryFileManager. ConcurrentSkipListMap.size is a very expensive operation, especially on maps with many entries, as it has to scan every entry to compute the size. We should avoid calling this method or at least minimize its use. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5283) Over 10 different tests have near identical implementations of AppContext
[ https://issues.apache.org/jira/browse/MAPREDUCE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-5283: -- Status: Open (was: Patch Available) Argh, the patch has gone stale since YARN-735 went in. Sandy, could you update it? Over 10 different tests have near identical implementations of AppContext - Key: MAPREDUCE-5283 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5283 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, test Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-5283-1.patch, MAPREDUCE-5283.patch I'm trying to add a method to AppContext for MAPREDUCE-5171, and I have to go into nearly every test file for MR web services to make sure their TestAppContext implements it. I propose having a common implementation of AppContext that all these tests can use. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5267) History server should be more robust when cleaning old jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-5267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673195#comment-13673195 ] Jason Lowe commented on MAPREDUCE-5267: --- The proposed listStatus change would help but not be sufficient to prevent problems from cropping up. For example, someone accidentally creating a directory under /mapred/done where the history server can read the directory but not delete any files underneath it would still cause problems. Bottom line is that HistoryFileManager.clean needs to protect itself from IOExceptions that can occur when interacting with the filesystem and try to make cleanup progress despite those exceptions. The listStatus inconsistency is best handled by a separate JIRA. History server should be more robust when cleaning old jobs --- Key: MAPREDUCE-5267 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5267 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobhistoryserver Affects Versions: 0.23.7, 2.0.4-alpha Reporter: Jason Lowe Ran across a situation where an admin user had accidentally created a directory in one of the date directories under /mapred/history/done/ that was not readable by the historyserver user. That effectively prevented the history server from cleaning any jobs from that date forward, as it hit an IOException trying to scan the directory and that aborted the entire clean process. The history server should localize IOException handling to the directory/file being processed and move on to the next entry in the list rather than aborting the entire cleaning process. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5199) AppTokens file can/should be removed
[ https://issues.apache.org/jira/browse/MAPREDUCE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated MAPREDUCE-5199: --- Attachment: MAPREDUCE-5199.patch This should work now. There were problems with the creds in the jobConf. Everything is addressed except removal of the extra UGI. AppTokens file can/should be removed Key: MAPREDUCE-5199 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5199 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: security Affects Versions: 3.0.0, 2.1.0-beta Reporter: Vinod Kumar Vavilapalli Assignee: Daryn Sharp Priority: Blocker Attachments: MAPREDUCE-5199.patch, MAPREDUCE-5199.patch, MAPREDUCE-5199.patch, MAPREDUCE-5199.patch All the required tokens are propagated to AMs and containers via startContainer(), no need for explicitly creating the app-token file that we have today.. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5199) AppTokens file can/should be removed
[
https://issues.apache.org/jira/browse/MAPREDUCE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673256#comment-13673256
]
Hadoop QA commented on MAPREDUCE-5199:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12585880/MAPREDUCE-5199.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3715//console
This message is automatically generated.
AppTokens file can/should be removed
Key: MAPREDUCE-5199
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5199
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: security
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Vinod Kumar Vavilapalli
Assignee: Daryn Sharp
Priority: Blocker
Attachments: MAPREDUCE-5199.patch, MAPREDUCE-5199.patch,
MAPREDUCE-5199.patch, MAPREDUCE-5199.patch
All the required tokens are propagated to AMs and containers via
startContainer(), no need for explicitly creating the app-token file that we
have today..
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5036) Default shuffle handler port should not be 8080
[ https://issues.apache.org/jira/browse/MAPREDUCE-5036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-5036: - Resolution: Fixed Status: Resolved (was: Patch Available) I just committed the fix. Thanks Sandy. Default shuffle handler port should not be 8080 --- Key: MAPREDUCE-5036 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5036 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.1.0-beta Attachments: MAPREDUCE-5036-13562.patch, MAPREDUCE-5036.patch The shuffle handler port (mapreduce.shuffle.port) defaults to 8080. This is a pretty common port for web services, and is likely to cause unnecessary port conflicts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5036) Default shuffle handler port should not be 8080
[ https://issues.apache.org/jira/browse/MAPREDUCE-5036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673320#comment-13673320 ] Hudson commented on MAPREDUCE-5036: --- Integrated in Hadoop-trunk-Commit #3839 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3839/]) MAPREDUCE-5036. Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. (Revision 1489060) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1489060 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java Default shuffle handler port should not be 8080 --- Key: MAPREDUCE-5036 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5036 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.1.0-beta Attachments: MAPREDUCE-5036-13562.patch, MAPREDUCE-5036.patch The shuffle handler port (mapreduce.shuffle.port) defaults to 8080. This is a pretty common port for web services, and is likely to cause unnecessary port conflicts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5296) Mapred API: Function signature change in JobControl
Zhijie Shen created MAPREDUCE-5296: -- Summary: Mapred API: Function signature change in JobControl Key: MAPREDUCE-5296 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5296 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen String addJob(Job) - String addJob(ControlledJob) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5296) Mapred API: Function signature change in JobControl
[ https://issues.apache.org/jira/browse/MAPREDUCE-5296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5296: --- Attachment: MAPREDUCE-5296.1.patch Add String addJob(Job). Mapred API: Function signature change in JobControl --- Key: MAPREDUCE-5296 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5296 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Attachments: MAPREDUCE-5296.1.patch String addJob(Job) - String addJob(ControlledJob) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5296) Mapred API: Function signature change in JobControl
[ https://issues.apache.org/jira/browse/MAPREDUCE-5296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5296: --- Status: Patch Available (was: Open) Mapred API: Function signature change in JobControl --- Key: MAPREDUCE-5296 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5296 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Attachments: MAPREDUCE-5296.1.patch String addJob(Job) - String addJob(ControlledJob) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5250) Searching for ';' in JobTracker History throws ArrayOutOfBoundException
[
https://issues.apache.org/jira/browse/MAPREDUCE-5250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673365#comment-13673365
]
Alejandro Abdelnur commented on MAPREDUCE-5250:
---
+1
Searching for ';' in JobTracker History throws ArrayOutOfBoundException
Key: MAPREDUCE-5250
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5250
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 1.2.0
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Minor
Labels: history
Attachments: mr-5250.patch
Searching for ';' in JobTracker History throws ArrayOutOfBoundException
{noformat}
Problem accessing /jobhistoryhome.jsp. Reason:
0
Caused by:
java.lang.ArrayIndexOutOfBoundsException: 0
at
org.apache.hadoop.mapred.jobhistoryhome_jsp._jspService(jobhistoryhome_jsp.java:221)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:914)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5250) Searching for ';' in JobTracker History throws ArrayOutOfBoundException
[
https://issues.apache.org/jira/browse/MAPREDUCE-5250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated MAPREDUCE-5250:
--
Resolution: Fixed
Fix Version/s: 1.3.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks Karthik. Committed to branch-1.
Searching for ';' in JobTracker History throws ArrayOutOfBoundException
Key: MAPREDUCE-5250
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5250
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 1.2.0
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Minor
Labels: history
Fix For: 1.3.0
Attachments: mr-5250.patch
Searching for ';' in JobTracker History throws ArrayOutOfBoundException
{noformat}
Problem accessing /jobhistoryhome.jsp. Reason:
0
Caused by:
java.lang.ArrayIndexOutOfBoundsException: 0
at
org.apache.hadoop.mapred.jobhistoryhome_jsp._jspService(jobhistoryhome_jsp.java:221)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:914)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5275) Mapreduce API: TokenCache incompatibility issues with MR1
[
https://issues.apache.org/jira/browse/MAPREDUCE-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673373#comment-13673373
]
Daryn Sharp commented on MAPREDUCE-5275:
Retro -1.
Why are we resurrecting APIs marked @InterfaceAudience.Private in 1.x? And
now making them public in 2.x?
{{TokenCache#getDelegationToken}} was intentionally removed since it is
fundamentally incompatible with multi-token filesystems like viewfs. It's also
not legitimate for external callers to be guessing/assuming the token key for a
fs. In fact, it's about to change soon, so this method cannot be retained.
Mapreduce API: TokenCache incompatibility issues with MR1
-
Key: MAPREDUCE-5275
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5275
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Mayank Bansal
Fix For: 2.1.0-beta
Attachments: MAPREDUCE-5275-trunk-1.patch,
MAPREDUCE-5275-trunk-2.patch, MAPREDUCE-5275-trunk-3.patch,
MAPREDUCE-5275-trunk-4.patch
There're following incompatibility issues:
* TokenDelegationTokenIdentifier getDelegationToken(Credentials, String) is
removed
* Credentials loadTokens(String, Configuration) changes to Credentials
loadTokens(String, JobConf)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-5275) Mapreduce API: TokenCache incompatibility issues with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp reopened MAPREDUCE-5275: Mapreduce API: TokenCache incompatibility issues with MR1 - Key: MAPREDUCE-5275 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5275 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Mayank Bansal Fix For: 2.1.0-beta Attachments: MAPREDUCE-5275-trunk-1.patch, MAPREDUCE-5275-trunk-2.patch, MAPREDUCE-5275-trunk-3.patch, MAPREDUCE-5275-trunk-4.patch There're following incompatibility issues: * TokenDelegationTokenIdentifier getDelegationToken(Credentials, String) is removed * Credentials loadTokens(String, Configuration) changes to Credentials loadTokens(String, JobConf) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5296) Mapred API: Function signature change in JobControl
[ https://issues.apache.org/jira/browse/MAPREDUCE-5296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673379#comment-13673379 ] Mayank Bansal commented on MAPREDUCE-5296: -- +1 looks good Thanks, Mayank Mapred API: Function signature change in JobControl --- Key: MAPREDUCE-5296 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5296 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Attachments: MAPREDUCE-5296.1.patch String addJob(Job) - String addJob(ControlledJob) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5275) Mapreduce API: TokenCache incompatibility issues with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673391#comment-13673391 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-5275: I just verified again and it IS public in 1.x. If and when 'it changes soon', you can remove it and mark it as incompatible at that point of time. Even so, the API is taking in a 'namenode', so it can be supported even for viewfs. Mapreduce API: TokenCache incompatibility issues with MR1 - Key: MAPREDUCE-5275 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5275 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Mayank Bansal Fix For: 2.1.0-beta Attachments: MAPREDUCE-5275-trunk-1.patch, MAPREDUCE-5275-trunk-2.patch, MAPREDUCE-5275-trunk-3.patch, MAPREDUCE-5275-trunk-4.patch There're following incompatibility issues: * TokenDelegationTokenIdentifier getDelegationToken(Credentials, String) is removed * Credentials loadTokens(String, Configuration) changes to Credentials loadTokens(String, JobConf) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-5263) filecache.DistributedCache incompatiblity issues with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen reopened MAPREDUCE-5263: Two more incompatibility issues: * long[] getArchiveTimestamps(Configuration) - String[] getArchiveTimestamps(Configuration) *long[] getFileTimestamps(Configuration) - String[] getFileTimestamps(Configuration) Changes may break 0.23 filecache.DistributedCache incompatiblity issues with MR1 - Key: MAPREDUCE-5263 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5263 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Fix For: 2.1.0-beta Attachments: MAPREDUCE-5263.1.patch, MAPREDUCE-5263.2.patch, MAPREDUCE-5263.3.patch A couple of methods and variables have been removed: void addLocalArchives(Configuration, String) void addLocalFiles(Configuration, String) void createAllSymlink(Configuration, File, File) FileStatus getFileStatus(Configuration, URI) long getTimestamp(Configuration, URI) void setArchiveTimestamps(Configuration, String) void setFileTimestamps(Configuration, String) void setLocalArchives(Configuration, String) void setLocalFiles(Configuration, String) String CACHE_ARCHIVES String CACHE_ARCHIVES_SIZES String CACHE_ARCHIVES_TIMESTAMPS String CACHE_FILES String CACHE_FILES_SIZES String CACHE_FILES_TIMESTAMPS String CACHE_LOCALARCHIVES String CACHE_LOCALFILES String CACHE_SYMLINK -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5283) Over 10 different tests have near identical implementations of AppContext
[ https://issues.apache.org/jira/browse/MAPREDUCE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5283: -- Status: Patch Available (was: Open) Over 10 different tests have near identical implementations of AppContext - Key: MAPREDUCE-5283 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5283 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, test Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-5283-1.patch, MAPREDUCE-5283-2.patch, MAPREDUCE-5283.patch I'm trying to add a method to AppContext for MAPREDUCE-5171, and I have to go into nearly every test file for MR web services to make sure their TestAppContext implements it. I propose having a common implementation of AppContext that all these tests can use. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5283) Over 10 different tests have near identical implementations of AppContext
[ https://issues.apache.org/jira/browse/MAPREDUCE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5283: -- Attachment: MAPREDUCE-5283-2.patch Over 10 different tests have near identical implementations of AppContext - Key: MAPREDUCE-5283 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5283 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, test Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-5283-1.patch, MAPREDUCE-5283-2.patch, MAPREDUCE-5283.patch I'm trying to add a method to AppContext for MAPREDUCE-5171, and I have to go into nearly every test file for MR web services to make sure their TestAppContext implements it. I propose having a common implementation of AppContext that all these tests can use. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5283) Over 10 different tests have near identical implementations of AppContext
[ https://issues.apache.org/jira/browse/MAPREDUCE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673399#comment-13673399 ] Sandy Ryza commented on MAPREDUCE-5283: --- Sure thing - attached a rebased patch Over 10 different tests have near identical implementations of AppContext - Key: MAPREDUCE-5283 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5283 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, test Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-5283-1.patch, MAPREDUCE-5283-2.patch, MAPREDUCE-5283.patch I'm trying to add a method to AppContext for MAPREDUCE-5171, and I have to go into nearly every test file for MR web services to make sure their TestAppContext implements it. I propose having a common implementation of AppContext that all these tests can use. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5296) Mapred API: Function signature change in JobControl
[
https://issues.apache.org/jira/browse/MAPREDUCE-5296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673417#comment-13673417
]
Hadoop QA commented on MAPREDUCE-5296:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12585898/MAPREDUCE-5296.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3716//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3716//console
This message is automatically generated.
Mapred API: Function signature change in JobControl
---
Key: MAPREDUCE-5296
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5296
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Attachments: MAPREDUCE-5296.1.patch
String addJob(Job) - String addJob(ControlledJob)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5297) Update MR App after YARN-748
Jian He created MAPREDUCE-5297: -- Summary: Update MR App after YARN-748 Key: MAPREDUCE-5297 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5297 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jian He -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5297) Update MR App since BuilderUtils is moved to yarn-server-common after YARN-748
[ https://issues.apache.org/jira/browse/MAPREDUCE-5297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated MAPREDUCE-5297: --- Summary: Update MR App since BuilderUtils is moved to yarn-server-common after YARN-748 (was: Update MR App after YARN-748) Update MR App since BuilderUtils is moved to yarn-server-common after YARN-748 --- Key: MAPREDUCE-5297 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5297 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jian He -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5263) filecache.DistributedCache incompatiblity issues with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5263: --- Status: Patch Available (was: Reopened) filecache.DistributedCache incompatiblity issues with MR1 - Key: MAPREDUCE-5263 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5263 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Fix For: 2.1.0-beta Attachments: MAPREDUCE-5263.1.patch, MAPREDUCE-5263.2.patch, MAPREDUCE-5263.3.patch, MAPREDUCE-5263.4.patch A couple of methods and variables have been removed: void addLocalArchives(Configuration, String) void addLocalFiles(Configuration, String) void createAllSymlink(Configuration, File, File) FileStatus getFileStatus(Configuration, URI) long getTimestamp(Configuration, URI) void setArchiveTimestamps(Configuration, String) void setFileTimestamps(Configuration, String) void setLocalArchives(Configuration, String) void setLocalFiles(Configuration, String) String CACHE_ARCHIVES String CACHE_ARCHIVES_SIZES String CACHE_ARCHIVES_TIMESTAMPS String CACHE_FILES String CACHE_FILES_SIZES String CACHE_FILES_TIMESTAMPS String CACHE_LOCALARCHIVES String CACHE_LOCALFILES String CACHE_SYMLINK -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5263) filecache.DistributedCache incompatiblity issues with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5263: --- Attachment: MAPREDUCE-5263.4.patch Change the return types of the two functions. They will break 0.23, but not significantly. filecache.DistributedCache incompatiblity issues with MR1 - Key: MAPREDUCE-5263 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5263 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Fix For: 2.1.0-beta Attachments: MAPREDUCE-5263.1.patch, MAPREDUCE-5263.2.patch, MAPREDUCE-5263.3.patch, MAPREDUCE-5263.4.patch A couple of methods and variables have been removed: void addLocalArchives(Configuration, String) void addLocalFiles(Configuration, String) void createAllSymlink(Configuration, File, File) FileStatus getFileStatus(Configuration, URI) long getTimestamp(Configuration, URI) void setArchiveTimestamps(Configuration, String) void setFileTimestamps(Configuration, String) void setLocalArchives(Configuration, String) void setLocalFiles(Configuration, String) String CACHE_ARCHIVES String CACHE_ARCHIVES_SIZES String CACHE_ARCHIVES_TIMESTAMPS String CACHE_FILES String CACHE_FILES_SIZES String CACHE_FILES_TIMESTAMPS String CACHE_LOCALARCHIVES String CACHE_LOCALFILES String CACHE_SYMLINK -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-5297) Update MR App since BuilderUtils is moved to yarn-server-common after YARN-748
[ https://issues.apache.org/jira/browse/MAPREDUCE-5297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He reassigned MAPREDUCE-5297: -- Assignee: Jian He Update MR App since BuilderUtils is moved to yarn-server-common after YARN-748 --- Key: MAPREDUCE-5297 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5297 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jian He Assignee: Jian He -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-5220) Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen reopened MAPREDUCE-5220: void setTaskID(TaskAttemptID) is missing in TaskCompletionEvent Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2 Key: MAPREDUCE-5220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5220 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: client Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.1.0-beta Attachments: MAPREDUCE-5220-1.patch, MAPREDUCE-5220.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5263) filecache.DistributedCache incompatiblity issues with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673462#comment-13673462 ] Jason Lowe commented on MAPREDUCE-5263: --- bq. They will break 0.23, but not significantly. Could you elaborate more on not significantly? Do you know that code in 0.23 doesn't call this method, for example? We were under the impression that the mapred APIs were paramount to 1.x binary compatibility, but since this is not going to be the case for the mapreduce APIs that the 0.23 mapreduce APIs would be preserved. Is that no longer the case? filecache.DistributedCache incompatiblity issues with MR1 - Key: MAPREDUCE-5263 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5263 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Fix For: 2.1.0-beta Attachments: MAPREDUCE-5263.1.patch, MAPREDUCE-5263.2.patch, MAPREDUCE-5263.3.patch, MAPREDUCE-5263.4.patch A couple of methods and variables have been removed: void addLocalArchives(Configuration, String) void addLocalFiles(Configuration, String) void createAllSymlink(Configuration, File, File) FileStatus getFileStatus(Configuration, URI) long getTimestamp(Configuration, URI) void setArchiveTimestamps(Configuration, String) void setFileTimestamps(Configuration, String) void setLocalArchives(Configuration, String) void setLocalFiles(Configuration, String) String CACHE_ARCHIVES String CACHE_ARCHIVES_SIZES String CACHE_ARCHIVES_TIMESTAMPS String CACHE_FILES String CACHE_FILES_SIZES String CACHE_FILES_TIMESTAMPS String CACHE_LOCALARCHIVES String CACHE_LOCALFILES String CACHE_SYMLINK -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5263) filecache.DistributedCache incompatiblity issues with MR1
[
https://issues.apache.org/jira/browse/MAPREDUCE-5263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673463#comment-13673463
]
Hadoop QA commented on MAPREDUCE-5263:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12585918/MAPREDUCE-5263.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3718//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3718//console
This message is automatically generated.
filecache.DistributedCache incompatiblity issues with MR1
-
Key: MAPREDUCE-5263
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5263
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Fix For: 2.1.0-beta
Attachments: MAPREDUCE-5263.1.patch, MAPREDUCE-5263.2.patch,
MAPREDUCE-5263.3.patch, MAPREDUCE-5263.4.patch
A couple of methods and variables have been removed:
void addLocalArchives(Configuration, String)
void addLocalFiles(Configuration, String)
void createAllSymlink(Configuration, File, File)
FileStatus getFileStatus(Configuration, URI)
long getTimestamp(Configuration, URI)
void setArchiveTimestamps(Configuration, String)
void setFileTimestamps(Configuration, String)
void setLocalArchives(Configuration, String)
void setLocalFiles(Configuration, String)
String CACHE_ARCHIVES
String CACHE_ARCHIVES_SIZES
String CACHE_ARCHIVES_TIMESTAMPS
String CACHE_FILES
String CACHE_FILES_SIZES
String CACHE_FILES_TIMESTAMPS
String CACHE_LOCALARCHIVES
String CACHE_LOCALFILES
String CACHE_SYMLINK
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5283) Over 10 different tests have near identical implementations of AppContext
[
https://issues.apache.org/jira/browse/MAPREDUCE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673466#comment-13673466
]
Hadoop QA commented on MAPREDUCE-5283:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12585907/MAPREDUCE-5283-2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3717//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3717//console
This message is automatically generated.
Over 10 different tests have near identical implementations of AppContext
-
Key: MAPREDUCE-5283
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5283
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: applicationmaster, test
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: MAPREDUCE-5283-1.patch, MAPREDUCE-5283-2.patch,
MAPREDUCE-5283.patch
I'm trying to add a method to AppContext for MAPREDUCE-5171, and I have to go
into nearly every test file for MR web services to make sure their
TestAppContext implements it. I propose having a common implementation of
AppContext that all these tests can use.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5275) Mapreduce API: TokenCache incompatibility issues with MR1
[
https://issues.apache.org/jira/browse/MAPREDUCE-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673470#comment-13673470
]
Daryn Sharp commented on MAPREDUCE-5275:
bq. I just verified again and it IS public in 1.x.
Yes, in 1.x it has the {{public}} keyword, but it's annotated
{{@InterfaceAudience.Private}} and the class is annotated
{{@InterfaceStability.Evolving}}. A method assuming a 1-to-1 fs to token
mapping cannot work with viewfs and 1-to-many.
The real question is why are we re-adding evolving/private methods removed 9
months ago? What broke? Because it's already broken for viewfs, based on
fragile assumptions if not broken in other cases, and soon the whole assumption
changes. It won't be a problem if a retired private api stays retired.
Mapreduce API: TokenCache incompatibility issues with MR1
-
Key: MAPREDUCE-5275
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5275
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Mayank Bansal
Fix For: 2.1.0-beta
Attachments: MAPREDUCE-5275-trunk-1.patch,
MAPREDUCE-5275-trunk-2.patch, MAPREDUCE-5275-trunk-3.patch,
MAPREDUCE-5275-trunk-4.patch
There're following incompatibility issues:
* TokenDelegationTokenIdentifier getDelegationToken(Credentials, String) is
removed
* Credentials loadTokens(String, Configuration) changes to Credentials
loadTokens(String, JobConf)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5275) Mapreduce API: TokenCache incompatibility issues with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673491#comment-13673491 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-5275: bq. Yes, in 1.x it has the public keyword, but it's annotated @InterfaceAudience.Private and the class is annotated @InterfaceStability.Evolving. A method assuming a 1-to-1 fs to token mapping cannot work with viewfs and 1-to-many. I didn't mean java public. I don't see the annotation in branch-1. It is commented in code - that doesn't mean it is private. Which branch are you seeing? bq. The real question is why are we re-adding evolving/private methods removed 9 months ago? We are just comparing 1.x with 2.x and fixing all changes *if possible*. Mapreduce API: TokenCache incompatibility issues with MR1 - Key: MAPREDUCE-5275 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5275 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Mayank Bansal Fix For: 2.1.0-beta Attachments: MAPREDUCE-5275-trunk-1.patch, MAPREDUCE-5275-trunk-2.patch, MAPREDUCE-5275-trunk-3.patch, MAPREDUCE-5275-trunk-4.patch There're following incompatibility issues: * TokenDelegationTokenIdentifier getDelegationToken(Credentials, String) is removed * Credentials loadTokens(String, Configuration) changes to Credentials loadTokens(String, JobConf) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5220) Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673513#comment-13673513 ] Zhijie Shen commented on MAPREDUCE-5220: [~sandyr], would you like to work on this issue, or should I can take it over? Thanks! Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2 Key: MAPREDUCE-5220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5220 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: client Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.1.0-beta Attachments: MAPREDUCE-5220-1.patch, MAPREDUCE-5220.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5220) Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673515#comment-13673515 ] Sandy Ryza commented on MAPREDUCE-5220: --- [~zjshen], feel free to take it over. Thanks! Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2 Key: MAPREDUCE-5220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5220 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: client Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.1.0-beta Attachments: MAPREDUCE-5220-1.patch, MAPREDUCE-5220.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5263) filecache.DistributedCache incompatiblity issues with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673530#comment-13673530 ] Zhijie Shen commented on MAPREDUCE-5263: bq. Could you elaborate more on not significantly? Do you know that code in 0.23 doesn't call this method, for example? It seems that the common way in MR2 is to call JobContext#getArchiveTimestamps and JobContext#getFileTimestamps, which are not changed. JobContextImpl makes use of DistributedCache#getArchiveTimestamps and JobContext#getFileTimestamps, and converts longs to strings. bq. We were under the impression that the mapred APIs were paramount to 1.x binary compatibility, but since this is not going to be the case for the mapreduce APIs that the 0.23 mapreduce APIs would be preserved. Is that no longer the case? It's still the case. It needs to be clarified here that the code changes are ultimately for org.apache.hadoop.filecache.DistributedCache, though they are within org.apache.hadoop.mapreduce.filecache.DistributedCache. The former class extends the latter one in MR2, and was used in MR1. filecache.DistributedCache incompatiblity issues with MR1 - Key: MAPREDUCE-5263 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5263 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Fix For: 2.1.0-beta Attachments: MAPREDUCE-5263.1.patch, MAPREDUCE-5263.2.patch, MAPREDUCE-5263.3.patch, MAPREDUCE-5263.4.patch A couple of methods and variables have been removed: void addLocalArchives(Configuration, String) void addLocalFiles(Configuration, String) void createAllSymlink(Configuration, File, File) FileStatus getFileStatus(Configuration, URI) long getTimestamp(Configuration, URI) void setArchiveTimestamps(Configuration, String) void setFileTimestamps(Configuration, String) void setLocalArchives(Configuration, String) void setLocalFiles(Configuration, String) String CACHE_ARCHIVES String CACHE_ARCHIVES_SIZES String CACHE_ARCHIVES_TIMESTAMPS String CACHE_FILES String CACHE_FILES_SIZES String CACHE_FILES_TIMESTAMPS String CACHE_LOCALARCHIVES String CACHE_LOCALFILES String CACHE_SYMLINK -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-5220) Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen reassigned MAPREDUCE-5220: -- Assignee: Zhijie Shen (was: Sandy Ryza) Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2 Key: MAPREDUCE-5220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5220 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: client Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Zhijie Shen Fix For: 2.1.0-beta Attachments: MAPREDUCE-5220-1.patch, MAPREDUCE-5220.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5298) Move MapReduce services to YARN-117 stricter lifecycle
Steve Loughran created MAPREDUCE-5298: - Summary: Move MapReduce services to YARN-117 stricter lifecycle Key: MAPREDUCE-5298 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5298 Project: Hadoop Map/Reduce Issue Type: New Feature Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Steve Loughran Assignee: Steve Loughran The MR services need to be in sync with the YARN-117 lifecycle enhancements -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5220) Mapred API: TaskCompletionEvent incompatibility issues with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5220: --- Description: 1. Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2. 2. void setTaskID(TaskAttemptID) is missing. Summary: Mapred API: TaskCompletionEvent incompatibility issues with MR1 (was: Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2) Mapred API: TaskCompletionEvent incompatibility issues with MR1 --- Key: MAPREDUCE-5220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5220 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: client Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Zhijie Shen Fix For: 2.1.0-beta Attachments: MAPREDUCE-5220-1.patch, MAPREDUCE-5220.patch 1. Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2. 2. void setTaskID(TaskAttemptID) is missing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5275) Mapreduce API: TokenCache incompatibility issues with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673560#comment-13673560 ] Daryn Sharp commented on MAPREDUCE-5275: It's been annotated, although commented out, as far back as branch-0.20.203. The annotations are not commented out in branch-21 and later. A number of methods in the TC were added in Jan 2010 to branch-21 via MAPREDUCE-143. The methods were annotated private. The backport of the patch to branch-20 commented out the annotations, as seemed somewhat common at the time - probably dependency issues. If you agree there's not a compelling reason to restore broken methods never intended to be public, could you please revert the patch? Mapreduce API: TokenCache incompatibility issues with MR1 - Key: MAPREDUCE-5275 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5275 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Mayank Bansal Fix For: 2.1.0-beta Attachments: MAPREDUCE-5275-trunk-1.patch, MAPREDUCE-5275-trunk-2.patch, MAPREDUCE-5275-trunk-3.patch, MAPREDUCE-5275-trunk-4.patch There're following incompatibility issues: * TokenDelegationTokenIdentifier getDelegationToken(Credentials, String) is removed * Credentials loadTokens(String, Configuration) changes to Credentials loadTokens(String, JobConf) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5220) Mapred API: TaskCompletionEvent incompatibility issues with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5220: --- Status: Patch Available (was: Reopened) Mapred API: TaskCompletionEvent incompatibility issues with MR1 --- Key: MAPREDUCE-5220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5220 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: client Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Zhijie Shen Fix For: 2.1.0-beta Attachments: MAPREDUCE-5220-1.patch, MAPREDUCE-5220.2.patch, MAPREDUCE-5220.patch 1. Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2. 2. void setTaskID(TaskAttemptID) is missing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5220) Mapred API: TaskCompletionEvent incompatibility issues with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5220: --- Attachment: MAPREDUCE-5220.2.patch Thanks! Take it over. Straightforward patch. Mapred API: TaskCompletionEvent incompatibility issues with MR1 --- Key: MAPREDUCE-5220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5220 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: client Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Zhijie Shen Fix For: 2.1.0-beta Attachments: MAPREDUCE-5220-1.patch, MAPREDUCE-5220.2.patch, MAPREDUCE-5220.patch 1. Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2. 2. void setTaskID(TaskAttemptID) is missing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5220) Mapred API: TaskCompletionEvent incompatibility issues with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5220: - Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. Thanks Zhijie! Mapred API: TaskCompletionEvent incompatibility issues with MR1 --- Key: MAPREDUCE-5220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5220 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: client Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Zhijie Shen Fix For: 2.1.0-beta Attachments: MAPREDUCE-5220-1.patch, MAPREDUCE-5220.2.patch, MAPREDUCE-5220.patch 1. Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2. 2. void setTaskID(TaskAttemptID) is missing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5220) Mapred API: TaskCompletionEvent incompatibility issues with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673626#comment-13673626 ] Hudson commented on MAPREDUCE-5220: --- Integrated in Hadoop-trunk-Commit #3846 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3846/]) MAPREDUCE-5220. Fix backward incompatibility for TaskCompletionEvent by adding back setTaskID. Contributed by Zhijie Shen. (Revision 1489187) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1489187 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskCompletionEvent.java Mapred API: TaskCompletionEvent incompatibility issues with MR1 --- Key: MAPREDUCE-5220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5220 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: client Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Zhijie Shen Fix For: 2.1.0-beta Attachments: MAPREDUCE-5220-1.patch, MAPREDUCE-5220.2.patch, MAPREDUCE-5220.patch 1. Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2. 2. void setTaskID(TaskAttemptID) is missing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4942) mapreduce.Job has a bunch of methods that throw InterruptedException so its incompatible with MR1
[
https://issues.apache.org/jira/browse/MAPREDUCE-4942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673667#comment-13673667
]
Zhijie Shen commented on MAPREDUCE-4942:
Just found three more methods that have to break either MR1 or 0.23 binary
compatibility.
* boolean failTask(TaskAttemptID): Change in return type from void to boolean.
* boolean killTask(TaskAttemptID): Change in return type from void to boolean.
* TaskCompletionEvent[] getTaskCompletionEvents(int): Change in return type
from org.apache.hadoop.mapred.TaskCompletionEvent[] to
org.apache.hadoop.mapreduce.TaskCompletionEvent[].
mapreduce.Job has a bunch of methods that throw InterruptedException so its
incompatible with MR1
-
Key: MAPREDUCE-4942
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4942
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv2
Affects Versions: 2.0.2-alpha
Reporter: Robert Kanter
Assignee: Robert Kanter
Fix For: 2.1.0-beta
Attachments: MAPREDUCE-4942.patch
The following methods in MR2's {{org.apache.hadoop.mapreduce.Job}} throw an
{{InterruptedException}} but don't in MR1, which makes them incompatible.
(Their Javadoc comments are also missing that they throw an
{{InterruptedException}} anyway)
I propose that we wrap the {{InterruptedException}} in a
{{RuntimeException}}.
{code}
public float setupProgress() throws IOException, InterruptedException
public float mapProgress() throws IOException, InterruptedException
public float reduceProgress() throws IOException, InterruptedException
public boolean isComplete() throws IOException, InterruptedException
public boolean isSuccessful() throws IOException, InterruptedException
public void killJob() throws IOException, InterruptedException
public void killTask(org.apache.hadoop.mapreduce.TaskAttemptID) throws
IOException, InterruptedException
public void failTask(org.apache.hadoop.mapreduce.TaskAttemptID) throws
IOException, InterruptedException
public Counters getCounters() throws IOException, InterruptedException
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5298) Move MapReduce services to YARN-117 stricter lifecycle
[ https://issues.apache.org/jira/browse/MAPREDUCE-5298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-5298: -- Attachment: MAPREDUCE-5298-016.patch Patch in sync w/ YARN-530-016.patch Move MapReduce services to YARN-117 stricter lifecycle -- Key: MAPREDUCE-5298 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5298 Project: Hadoop Map/Reduce Issue Type: New Feature Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Steve Loughran Assignee: Steve Loughran Attachments: MAPREDUCE-5298-016.patch The MR services need to be in sync with the YARN-117 lifecycle enhancements -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5298) Move MapReduce services to YARN-117 stricter lifecycle
[ https://issues.apache.org/jira/browse/MAPREDUCE-5298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-5298: -- Target Version/s: 2.1.0-beta Status: Patch Available (was: Open) This patch is in sync with the YARN-530 lifecycle changes # moves them all to the new serviceStop/serviceStart methods # incorporates MAPREDUCE-3502: Review all Service.stop() operations and make sure that they work before a service is started # adds more service stopping in test runs JobHistoryServer is the most troublesome here -though I can't see any obvious cause Move MapReduce services to YARN-117 stricter lifecycle -- Key: MAPREDUCE-5298 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5298 Project: Hadoop Map/Reduce Issue Type: New Feature Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Steve Loughran Assignee: Steve Loughran Attachments: MAPREDUCE-5298-016.patch The MR services need to be in sync with the YARN-117 lifecycle enhancements -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5284) Mapreduce API: CounterGroup changes from non-abstract class to interface
[ https://issues.apache.org/jira/browse/MAPREDUCE-5284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen resolved MAPREDUCE-5284. Resolution: Won't Fix Mapreduce API: CounterGroup changes from non-abstract class to interface Key: MAPREDUCE-5284 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5284 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Therefore, constructors and implemented methods are removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5281) Mapreduce API: Counter changes from non-abstract class to interface
[ https://issues.apache.org/jira/browse/MAPREDUCE-5281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen resolved MAPREDUCE-5281. Resolution: Won't Fix Mapreduce API: Counter changes from non-abstract class to interface --- Key: MAPREDUCE-5281 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5281 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Therefore, significant changes in Counter: 1. Two Constructors are removed; 2. Following methods are removed: * boolean equals(Object) * int hashCode() * void readFields(DataInput) * void write(DataOutput) Fix of this issue may break 0.23. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5298) Move MapReduce services to YARN-117 stricter lifecycle
[
https://issues.apache.org/jira/browse/MAPREDUCE-5298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673732#comment-13673732
]
Hadoop QA commented on MAPREDUCE-5298:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12585975/MAPREDUCE-5298-016.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3720//console
This message is automatically generated.
Move MapReduce services to YARN-117 stricter lifecycle
--
Key: MAPREDUCE-5298
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5298
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: applicationmaster
Affects Versions: 2.0.4-alpha
Reporter: Steve Loughran
Assignee: Steve Loughran
Attachments: MAPREDUCE-5298-016.patch
The MR services need to be in sync with the YARN-117 lifecycle enhancements
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5296) Mapred API: Function signature change in JobControl
[ https://issues.apache.org/jira/browse/MAPREDUCE-5296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5296: - Resolution: Fixed Fix Version/s: 2.1.0-beta Status: Resolved (was: Patch Available) I just committed this. Thanks [~zjshen]! Mapred API: Function signature change in JobControl --- Key: MAPREDUCE-5296 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5296 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Fix For: 2.1.0-beta Attachments: MAPREDUCE-5296.1.patch String addJob(Job) - String addJob(ControlledJob) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5299) Mapred API: void setTaskID(TaskAttemptID) is missing in TaskCompletionEvent
Zhijie Shen created MAPREDUCE-5299: -- Summary: Mapred API: void setTaskID(TaskAttemptID) is missing in TaskCompletionEvent Key: MAPREDUCE-5299 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5299 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Move the add-on patch of MAPREDUCE-5220 here. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5299) Mapred API: void setTaskID(TaskAttemptID) is missing in TaskCompletionEvent
[ https://issues.apache.org/jira/browse/MAPREDUCE-5299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5299: --- Attachment: MAPREDUCE-5299.1.patch Move to the add-on patch. Mapred API: void setTaskID(TaskAttemptID) is missing in TaskCompletionEvent Key: MAPREDUCE-5299 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5299 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Attachments: MAPREDUCE-5299.1.patch Move the add-on patch of MAPREDUCE-5220 here. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5299) Mapred API: void setTaskID(TaskAttemptID) is missing in TaskCompletionEvent
[ https://issues.apache.org/jira/browse/MAPREDUCE-5299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5299: --- Status: Patch Available (was: Open) Mapred API: void setTaskID(TaskAttemptID) is missing in TaskCompletionEvent Key: MAPREDUCE-5299 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5299 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Attachments: MAPREDUCE-5299.1.patch Move the add-on patch of MAPREDUCE-5220 here. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5270) Migrate from using BuilderUtil factory methods to individual record factory method on MapReduce side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated MAPREDUCE-5270: --- Attachment: MAPREDUCE-5270.4.patch Upload a new patch to cover all MR side not use BuilderUtil factory methods Migrate from using BuilderUtil factory methods to individual record factory method on MapReduce side Key: MAPREDUCE-5270 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5270 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jian He Assignee: Jian He Fix For: 2.1.0-beta Attachments: MAPREDUCE-5270.1.patch, MAPREDUCE-5270.2.patch, MAPREDUCE-5270.3.patch, MAPREDUCE-5270.4.patch Migrate the factory method on map reduce side. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5263) filecache.DistributedCache incompatiblity issues with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5263: - Resolution: Fixed Status: Resolved (was: Patch Available) Zhijie, since this has already been committed. I'm closing this. Please reopen a new one. Tx. filecache.DistributedCache incompatiblity issues with MR1 - Key: MAPREDUCE-5263 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5263 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Fix For: 2.1.0-beta Attachments: MAPREDUCE-5263.1.patch, MAPREDUCE-5263.2.patch, MAPREDUCE-5263.3.patch, MAPREDUCE-5263.4.patch A couple of methods and variables have been removed: void addLocalArchives(Configuration, String) void addLocalFiles(Configuration, String) void createAllSymlink(Configuration, File, File) FileStatus getFileStatus(Configuration, URI) long getTimestamp(Configuration, URI) void setArchiveTimestamps(Configuration, String) void setFileTimestamps(Configuration, String) void setLocalArchives(Configuration, String) void setLocalFiles(Configuration, String) String CACHE_ARCHIVES String CACHE_ARCHIVES_SIZES String CACHE_ARCHIVES_TIMESTAMPS String CACHE_FILES String CACHE_FILES_SIZES String CACHE_FILES_TIMESTAMPS String CACHE_LOCALARCHIVES String CACHE_LOCALFILES String CACHE_SYMLINK -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5300) Two function signature changes in filecache.DistributedCache
Zhijie Shen created MAPREDUCE-5300: -- Summary: Two function signature changes in filecache.DistributedCache Key: MAPREDUCE-5300 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5300 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Two more incompatibility issues: * long[] getArchiveTimestamps(Configuration) - String[] getArchiveTimestamps(Configuration) * long[] getFileTimestamps(Configuration) - String[] getFileTimestamps(Configuration) Changes will break 0.23 *Move the add-on patch of MAPREDUCE-5263 here.* -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5299) Mapred API: void setTaskID(TaskAttemptID) is missing in TaskCompletionEvent
[
https://issues.apache.org/jira/browse/MAPREDUCE-5299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673755#comment-13673755
]
Hadoop QA commented on MAPREDUCE-5299:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12585981/MAPREDUCE-5299.1.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3721//console
This message is automatically generated.
Mapred API: void setTaskID(TaskAttemptID) is missing in TaskCompletionEvent
Key: MAPREDUCE-5299
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5299
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Attachments: MAPREDUCE-5299.1.patch
Move the add-on patch of MAPREDUCE-5220 here.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5300) Two function signature changes in filecache.DistributedCache
[ https://issues.apache.org/jira/browse/MAPREDUCE-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5300: --- Status: Patch Available (was: Open) Two function signature changes in filecache.DistributedCache Key: MAPREDUCE-5300 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5300 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Attachments: MAPREDUCE-5300.1.patch Two more incompatibility issues: * long[] getArchiveTimestamps(Configuration) - String[] getArchiveTimestamps(Configuration) * long[] getFileTimestamps(Configuration) - String[] getFileTimestamps(Configuration) Changes will break 0.23 *Move the add-on patch of MAPREDUCE-5263 here.* -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5300) Two function signature changes in filecache.DistributedCache
[ https://issues.apache.org/jira/browse/MAPREDUCE-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5300: --- Attachment: MAPREDUCE-5300.1.patch Move the add-on patch here Two function signature changes in filecache.DistributedCache Key: MAPREDUCE-5300 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5300 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Attachments: MAPREDUCE-5300.1.patch Two more incompatibility issues: * long[] getArchiveTimestamps(Configuration) - String[] getArchiveTimestamps(Configuration) * long[] getFileTimestamps(Configuration) - String[] getFileTimestamps(Configuration) Changes will break 0.23 *Move the add-on patch of MAPREDUCE-5263 here.* -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5297) Update MR App since BuilderUtils is moved to yarn-server-common after YARN-748
[ https://issues.apache.org/jira/browse/MAPREDUCE-5297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated MAPREDUCE-5297: --- Attachment: MAPREDUCE-5297.1.patch Changed MR not to use BuilderUtils at all Update MR App since BuilderUtils is moved to yarn-server-common after YARN-748 --- Key: MAPREDUCE-5297 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5297 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jian He Assignee: Jian He Attachments: MAPREDUCE-5297.1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5300) Two function signature changes in filecache.DistributedCache
[
https://issues.apache.org/jira/browse/MAPREDUCE-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673787#comment-13673787
]
Hadoop QA commented on MAPREDUCE-5300:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12585985/MAPREDUCE-5300.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3722//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3722//console
This message is automatically generated.
Two function signature changes in filecache.DistributedCache
Key: MAPREDUCE-5300
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5300
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Attachments: MAPREDUCE-5300.1.patch
Two more incompatibility issues:
* long[] getArchiveTimestamps(Configuration) - String[]
getArchiveTimestamps(Configuration)
* long[] getFileTimestamps(Configuration) - String[]
getFileTimestamps(Configuration)
Changes will break 0.23
*Move the add-on patch of MAPREDUCE-5263 here.*
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5300) Two function signature changes in filecache.DistributedCache
[ https://issues.apache.org/jira/browse/MAPREDUCE-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5300: - Resolution: Fixed Fix Version/s: 2.1.0-beta Status: Resolved (was: Patch Available) I just committed this. Thanks [~zjshen]! Two function signature changes in filecache.DistributedCache Key: MAPREDUCE-5300 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5300 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Fix For: 2.1.0-beta Attachments: MAPREDUCE-5300.1.patch Two more incompatibility issues: * long[] getArchiveTimestamps(Configuration) - String[] getArchiveTimestamps(Configuration) * long[] getFileTimestamps(Configuration) - String[] getFileTimestamps(Configuration) Changes will break 0.23 *Move the add-on patch of MAPREDUCE-5263 here.* -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5299) Mapred API: void setTaskID(TaskAttemptID) is missing in TaskCompletionEvent
[ https://issues.apache.org/jira/browse/MAPREDUCE-5299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5299: - Resolution: Fixed Fix Version/s: 2.1.0-beta Status: Resolved (was: Patch Available) I already committed this via MAPREDUCE-5220, changed jira number in CHANGES.txt now. Mapred API: void setTaskID(TaskAttemptID) is missing in TaskCompletionEvent Key: MAPREDUCE-5299 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5299 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Fix For: 2.1.0-beta Attachments: MAPREDUCE-5299.1.patch Move the add-on patch of MAPREDUCE-5220 here. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5270) Migrate from using BuilderUtil factory methods to individual record factory method on MapReduce side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673800#comment-13673800 ] Jian He commented on MAPREDUCE-5270: Put the new patch on wrong place, remove it here. it should be on MAPREDUCE-5297 Migrate from using BuilderUtil factory methods to individual record factory method on MapReduce side Key: MAPREDUCE-5270 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5270 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jian He Assignee: Jian He Fix For: 2.1.0-beta Attachments: MAPREDUCE-5270.1.patch, MAPREDUCE-5270.2.patch, MAPREDUCE-5270.3.patch Migrate the factory method on map reduce side. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-5300) Two function signature changes in filecache.DistributedCache
[ https://issues.apache.org/jira/browse/MAPREDUCE-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reopened MAPREDUCE-5300: -- Two function signature changes in filecache.DistributedCache Key: MAPREDUCE-5300 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5300 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Fix For: 2.1.0-beta Attachments: MAPREDUCE-5300.1.patch Two more incompatibility issues: * long[] getArchiveTimestamps(Configuration) - String[] getArchiveTimestamps(Configuration) * long[] getFileTimestamps(Configuration) - String[] getFileTimestamps(Configuration) Changes will break 0.23 *Move the add-on patch of MAPREDUCE-5263 here.* -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5270) Migrate from using BuilderUtil factory methods to individual record factory method on MapReduce side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated MAPREDUCE-5270: --- Attachment: (was: MAPREDUCE-5270.4.patch) Migrate from using BuilderUtil factory methods to individual record factory method on MapReduce side Key: MAPREDUCE-5270 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5270 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jian He Assignee: Jian He Fix For: 2.1.0-beta Attachments: MAPREDUCE-5270.1.patch, MAPREDUCE-5270.2.patch, MAPREDUCE-5270.3.patch Migrate the factory method on map reduce side. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5300) Two function signature changes in filecache.DistributedCache
[ https://issues.apache.org/jira/browse/MAPREDUCE-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-5300. -- Resolution: Fixed Hadoop Flags: Incompatible change Re-opened to close with 'Incompatible Change' flag. Two function signature changes in filecache.DistributedCache Key: MAPREDUCE-5300 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5300 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Fix For: 2.1.0-beta Attachments: MAPREDUCE-5300.1.patch Two more incompatibility issues: * long[] getArchiveTimestamps(Configuration) - String[] getArchiveTimestamps(Configuration) * long[] getFileTimestamps(Configuration) - String[] getFileTimestamps(Configuration) Changes will break 0.23 *Move the add-on patch of MAPREDUCE-5263 here.* -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5299) Mapred API: void setTaskID(TaskAttemptID) is missing in TaskCompletionEvent
[ https://issues.apache.org/jira/browse/MAPREDUCE-5299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673819#comment-13673819 ] Hudson commented on MAPREDUCE-5299: --- Integrated in Hadoop-trunk-Commit #3848 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3848/]) MAPREDUCE-5299. Changed jira number for the commit in CHANGES.txt. (Revision 1489229) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1489229 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt Mapred API: void setTaskID(TaskAttemptID) is missing in TaskCompletionEvent Key: MAPREDUCE-5299 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5299 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Fix For: 2.1.0-beta Attachments: MAPREDUCE-5299.1.patch Move the add-on patch of MAPREDUCE-5220 here. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5300) Two function signature changes in filecache.DistributedCache
[ https://issues.apache.org/jira/browse/MAPREDUCE-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673820#comment-13673820 ] Hudson commented on MAPREDUCE-5300: --- Integrated in Hadoop-trunk-Commit #3848 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3848/]) MAPREDUCE-5300. Moved to INCOMPATIBLE CHANGES section in CHANGES.txt. (Revision 1489232) MAPREDUCE-5300. Fix backward incompatibility for o.a.h.mapreduce.filecache.DistributedCache. Contributed by Zhijie Shen. (Revision 1489227) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1489232 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1489227 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapred/TestMRWithDistributedCache.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/filecache/DistributedCache.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/JobContextImpl.java Two function signature changes in filecache.DistributedCache Key: MAPREDUCE-5300 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5300 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Fix For: 2.1.0-beta Attachments: MAPREDUCE-5300.1.patch Two more incompatibility issues: * long[] getArchiveTimestamps(Configuration) - String[] getArchiveTimestamps(Configuration) * long[] getFileTimestamps(Configuration) - String[] getFileTimestamps(Configuration) Changes will break 0.23 *Move the add-on patch of MAPREDUCE-5263 here.* -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5245) A number of public static variables are removed from JobConf
[ https://issues.apache.org/jira/browse/MAPREDUCE-5245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5245: - Resolution: Fixed Fix Version/s: 2.1.0-beta Status: Resolved (was: Patch Available) +1. I just committed this. Thanks [~zjshen]! A number of public static variables are removed from JobConf Key: MAPREDUCE-5245 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5245 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Fix For: 2.1.0-beta Attachments: MAPREDUCE-5245.1.patch, MAPREDUCE-5245.2.patch A number of public static variables are removed from JobConf: boolean DEFAULT_MAPREDUCE_RECOVER_JOB String MAPREDUCE_RECOVER_JOB String WORKFLOW_ADJACENCY_PREFIX_PATTERN String WORKFLOW_ADJACENCY_PREFIX_STRING String WORKFLOW_ID String WORKFLOW_NAME String WORKFLOW_NODE_NAME String WORKFLOW_TAGS The workflow related variables are moved to MRJobConfig. The follwing public static variables becomes default: String MAPRED_JOB_MAP_MEMORY_MB_PROPERTY String MAPRED_JOB_REDUCE_MEMORY_MB_PROPERTY The variables there are no longer referred internally in 2.x, but they might be used by users as they were public. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5245) A number of public static variables are removed from JobConf
[ https://issues.apache.org/jira/browse/MAPREDUCE-5245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673846#comment-13673846 ] Hudson commented on MAPREDUCE-5245: --- Integrated in Hadoop-trunk-Commit #3849 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3849/]) MAPREDUCE-5245. Added back constants to JobConf to fix incompatibilities. Contributed by Zhijie Shen. (Revision 1489236) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1489236 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestJobConf.java A number of public static variables are removed from JobConf Key: MAPREDUCE-5245 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5245 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Fix For: 2.1.0-beta Attachments: MAPREDUCE-5245.1.patch, MAPREDUCE-5245.2.patch A number of public static variables are removed from JobConf: boolean DEFAULT_MAPREDUCE_RECOVER_JOB String MAPREDUCE_RECOVER_JOB String WORKFLOW_ADJACENCY_PREFIX_PATTERN String WORKFLOW_ADJACENCY_PREFIX_STRING String WORKFLOW_ID String WORKFLOW_NAME String WORKFLOW_NODE_NAME String WORKFLOW_TAGS The workflow related variables are moved to MRJobConfig. The follwing public static variables becomes default: String MAPRED_JOB_MAP_MEMORY_MB_PROPERTY String MAPRED_JOB_REDUCE_MEMORY_MB_PROPERTY The variables there are no longer referred internally in 2.x, but they might be used by users as they were public. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5221) Reduce side Combiner is not used when using the new API
[ https://issues.apache.org/jira/browse/MAPREDUCE-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673862#comment-13673862 ] Alejandro Abdelnur commented on MAPREDUCE-5221: --- I've pinged [~masokan] to verify this was no a regression because of the plugglable shuffle and pluggable sort changes (MAPREDUCE-4049 MAPREDUCE-4808). It is not. After digging a bit it seems Hadoop2 ReduceTask is not wired to handle mapreduce combiner on the reducer side. This may have been overlooked when inverting mapred/mapreduce implementations in Hadoop 2. Reduce side Combiner is not used when using the new API --- Key: MAPREDUCE-5221 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5221 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.0.4-alpha Reporter: Siddharth Seth Assignee: Tsuyoshi OZAWA Attachments: MAPREDUCE-5221.1.patch, MAPREDUCE-5221.2.patch, MAPREDUCE-5221.3.patch If a combiner is specified using o.a.h.mapreduce.Job.setCombinerClass - this will silently ignored on the reduce side since the reduce side usage is only aware of the old api combiner. This doesn't fail the job - since the new combiner key does not deprecate the old key. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5297) Update MR App since BuilderUtils is moved to yarn-server-common after YARN-748
[ https://issues.apache.org/jira/browse/MAPREDUCE-5297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673881#comment-13673881 ] Hudson commented on MAPREDUCE-5297: --- Integrated in Hadoop-trunk-Commit #3850 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3850/]) YARN-748. Moved BuilderUtils from yarn-common to yarn-server-common for eventual retirement. Contributed by Jian He. MAPREDUCE-5297. Updated MR App since BuilderUtils is no longer public after YARN-748. Contributed by Jian He. (Revision 1489257) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1489257 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/local/LocalContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRecovery.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestTaskHeartbeatHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttemptContainerRequest.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncherImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/local/TestLocalContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/security/MRDelegationTokenRenewer.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRBuilderUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/api/records/TestIds.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestMapReduceTrackingUriPlugin.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryClientService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryEvents.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHSWebApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/NotRunningJob.java *
[jira] [Resolved] (MAPREDUCE-5297) Update MR App since BuilderUtils is moved to yarn-server-common after YARN-748
[ https://issues.apache.org/jira/browse/MAPREDUCE-5297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli resolved MAPREDUCE-5297. Resolution: Fixed Fix Version/s: 2.1.0-beta Hadoop Flags: Reviewed Reviewed and committed this together with YARN-748 to trunk and branch-2. Closing this. Update MR App since BuilderUtils is moved to yarn-server-common after YARN-748 --- Key: MAPREDUCE-5297 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5297 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jian He Assignee: Jian He Fix For: 2.1.0-beta Attachments: MAPREDUCE-5297.1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5283) Over 10 different tests have near identical implementations of AppContext
[ https://issues.apache.org/jira/browse/MAPREDUCE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-5283: -- Status: Open (was: Patch Available) Thanks for the update, Sandy. Unfortunately I wasn't fast enough, the patch is already out of date again. This time looks like YARN-748 swooped in. Sandy, could you refresh it again? Over 10 different tests have near identical implementations of AppContext - Key: MAPREDUCE-5283 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5283 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, test Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-5283-1.patch, MAPREDUCE-5283-2.patch, MAPREDUCE-5283.patch I'm trying to add a method to AppContext for MAPREDUCE-5171, and I have to go into nearly every test file for MR web services to make sure their TestAppContext implements it. I propose having a common implementation of AppContext that all these tests can use. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5283) Over 10 different tests have near identical implementations of AppContext
[ https://issues.apache.org/jira/browse/MAPREDUCE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5283: -- Status: Patch Available (was: Open) Over 10 different tests have near identical implementations of AppContext - Key: MAPREDUCE-5283 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5283 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, test Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-5283-1.patch, MAPREDUCE-5283-2.patch, MAPREDUCE-5283-3.patch, MAPREDUCE-5283.patch I'm trying to add a method to AppContext for MAPREDUCE-5171, and I have to go into nearly every test file for MR web services to make sure their TestAppContext implements it. I propose having a common implementation of AppContext that all these tests can use. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5283) Over 10 different tests have near identical implementations of AppContext
[ https://issues.apache.org/jira/browse/MAPREDUCE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5283: -- Attachment: MAPREDUCE-5283-3.patch Over 10 different tests have near identical implementations of AppContext - Key: MAPREDUCE-5283 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5283 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, test Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-5283-1.patch, MAPREDUCE-5283-2.patch, MAPREDUCE-5283-3.patch, MAPREDUCE-5283.patch I'm trying to add a method to AppContext for MAPREDUCE-5171, and I have to go into nearly every test file for MR web services to make sure their TestAppContext implements it. I propose having a common implementation of AppContext that all these tests can use. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5283) Over 10 different tests have near identical implementations of AppContext
[ https://issues.apache.org/jira/browse/MAPREDUCE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673900#comment-13673900 ] Sandy Ryza commented on MAPREDUCE-5283: --- Attached another refresh Over 10 different tests have near identical implementations of AppContext - Key: MAPREDUCE-5283 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5283 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, test Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-5283-1.patch, MAPREDUCE-5283-2.patch, MAPREDUCE-5283-3.patch, MAPREDUCE-5283.patch I'm trying to add a method to AppContext for MAPREDUCE-5171, and I have to go into nearly every test file for MR web services to make sure their TestAppContext implements it. I propose having a common implementation of AppContext that all these tests can use. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5283) Over 10 different tests have near identical implementations of AppContext
[
https://issues.apache.org/jira/browse/MAPREDUCE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673923#comment-13673923
]
Hadoop QA commented on MAPREDUCE-5283:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12586011/MAPREDUCE-5283-3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3723//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3723//console
This message is automatically generated.
Over 10 different tests have near identical implementations of AppContext
-
Key: MAPREDUCE-5283
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5283
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: applicationmaster, test
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: MAPREDUCE-5283-1.patch, MAPREDUCE-5283-2.patch,
MAPREDUCE-5283-3.patch, MAPREDUCE-5283.patch
I'm trying to add a method to AppContext for MAPREDUCE-5171, and I have to go
into nearly every test file for MR web services to make sure their
TestAppContext implements it. I propose having a common implementation of
AppContext that all these tests can use.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5283) Over 10 different tests have near identical implementations of AppContext
[ https://issues.apache.org/jira/browse/MAPREDUCE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673926#comment-13673926 ] Jason Lowe commented on MAPREDUCE-5283: --- +1, lgtm. Will commit shortly. Over 10 different tests have near identical implementations of AppContext - Key: MAPREDUCE-5283 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5283 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, test Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-5283-1.patch, MAPREDUCE-5283-2.patch, MAPREDUCE-5283-3.patch, MAPREDUCE-5283.patch I'm trying to add a method to AppContext for MAPREDUCE-5171, and I have to go into nearly every test file for MR web services to make sure their TestAppContext implements it. I propose having a common implementation of AppContext that all these tests can use. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5283) Over 10 different tests have near identical implementations of AppContext
[ https://issues.apache.org/jira/browse/MAPREDUCE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-5283: -- Resolution: Fixed Fix Version/s: 2.1.0-beta Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks, Sandy! I committed this to trunk and branch-2. Over 10 different tests have near identical implementations of AppContext - Key: MAPREDUCE-5283 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5283 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, test Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.1.0-beta Attachments: MAPREDUCE-5283-1.patch, MAPREDUCE-5283-2.patch, MAPREDUCE-5283-3.patch, MAPREDUCE-5283.patch I'm trying to add a method to AppContext for MAPREDUCE-5171, and I have to go into nearly every test file for MR web services to make sure their TestAppContext implements it. I propose having a common implementation of AppContext that all these tests can use. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5283) Over 10 different tests have near identical implementations of AppContext
[ https://issues.apache.org/jira/browse/MAPREDUCE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673940#comment-13673940 ] Sandy Ryza commented on MAPREDUCE-5283: --- Thanks Jason! Over 10 different tests have near identical implementations of AppContext - Key: MAPREDUCE-5283 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5283 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, test Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.1.0-beta Attachments: MAPREDUCE-5283-1.patch, MAPREDUCE-5283-2.patch, MAPREDUCE-5283-3.patch, MAPREDUCE-5283.patch I'm trying to add a method to AppContext for MAPREDUCE-5171, and I have to go into nearly every test file for MR web services to make sure their TestAppContext implements it. I propose having a common implementation of AppContext that all these tests can use. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
