[jira] [Commented] (MAPREDUCE-5222) Add missing methods to JobClient
[
https://issues.apache.org/jira/browse/MAPREDUCE-5222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13655793#comment-13655793
]
Zhijie Shen commented on MAPREDUCE-5222:
{quote}
Also, is this everything in JobClient?
{quote}
submitJobInternal(). Thought it should be called internally, however, it's
accessible to the public.
And 4 public static variables, which are no longer used in 2.x. The problem
again is that they were accessible to the public:
boolean MAPREDUCE_CLIENT_RETRY_POLICY_ENABLED_DEFAULT
String MAPREDUCE_CLIENT_RETRY_POLICY_ENABLED_KEY
String MAPREDUCE_CLIENT_RETRY_POLICY_SPEC_DEFAULT
String MAPREDUCE_CLIENT_RETRY_POLICY_SPEC_KEY
Add missing methods to JobClient
-
Key: MAPREDUCE-5222
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5222
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Affects Versions: 2.0.4-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Fix For: 2.0.5-beta
Attachments: mr-5222.patch, mr-5222.patch, mr-5222-rte.patch
JobClient is missing the following two public methods we need to add for
binary compatibility:
# static isJobDirValid(Path, FileSystem)
# Path getStagingAreaDir()
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5233) Functions are changed or removed from Job in jobcontrol
[
https://issues.apache.org/jira/browse/MAPREDUCE-5233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13655825#comment-13655825
]
Zhijie Shen commented on MAPREDUCE-5233:
1. In setState, it is good to add default block to handle the case that the int
param is not within the value domain of the State enum.
2. getMapredJobID is changed to return String to be compatible with 1.x. Is it
good to keep a similar function (maybe called getMapredJobId) to still return
the JobID object? In case the function will be potentially used in 2.x.
3. I'm not clear why getJobID will be called twice with the patch. Perhaps some
inline comments here to explain the two usages?
{code}
-verify(mockjob).getJobID();
+verify(mockjob,times(2)).getJobID();
{code}
Functions are changed or removed from Job in jobcontrol
---
Key: MAPREDUCE-5233
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5233
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Mayank Bansal
Fix For: 2.0.5-beta
Attachments: MAPREDUCE-5233-trunk-1.patch,
MAPREDUCE-5233-trunk-2.patch, MAPREDUCE-5233-trunk-3.patch
The functions are removed from Job in jobcontrol:
1. setMapredJobID(String)
2. setState(int)
The function signatures are changed:
1. addDependingJob(ControlledJob)
2. getMapredJobID()
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-4942) mapreduce.Job has a bunch of methods that throw InterruptedException so its incompatible with MR1
[
https://issues.apache.org/jira/browse/MAPREDUCE-4942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy reopened MAPREDUCE-4942:
--
Sigh, I missed this.
There are multiple issues with this patch:
# It's incompatible with hadoop-0.23.x
# Has anyone checked Pig and other projects which have already done necessary
to work with hadoop-0.23?
# This badly hoses all users at Yahoo who were brave enough to move to
hadoop-0.23 to test YARN.
The above reasons, plus given the fact that compatibility b/w hadoop-1 for
o.a.h.mapreduce is a non-starter I propose we revert this.
Thoughts?
In future, I'd appreciate a ping if/when major changes to user-facing MR apis
occur - I can provide a lot of context. Thanks.
mapreduce.Job has a bunch of methods that throw InterruptedException so its
incompatible with MR1
-
Key: MAPREDUCE-4942
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4942
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.2-alpha
Reporter: Robert Kanter
Assignee: Robert Kanter
Fix For: 2.0.5-beta
Attachments: MAPREDUCE-4942.patch
The following methods in MR2's {{org.apache.hadoop.mapreduce.Job}} throw an
{{InterruptedException}} but don't in MR1, which makes them incompatible.
(Their Javadoc comments are also missing that they throw an
{{InterruptedException}} anyway)
I propose that we wrap the {{InterruptedException}} in a
{{RuntimeException}}.
{code}
public float setupProgress() throws IOException, InterruptedException
public float mapProgress() throws IOException, InterruptedException
public float reduceProgress() throws IOException, InterruptedException
public boolean isComplete() throws IOException, InterruptedException
public boolean isSuccessful() throws IOException, InterruptedException
public void killJob() throws IOException, InterruptedException
public void killTask(org.apache.hadoop.mapreduce.TaskAttemptID) throws
IOException, InterruptedException
public void failTask(org.apache.hadoop.mapreduce.TaskAttemptID) throws
IOException, InterruptedException
public Counters getCounters() throws IOException, InterruptedException
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5208) SpillRecord and ShuffleHandler should use SecureIOUtils for reading index file and map output
[ https://issues.apache.org/jira/browse/MAPREDUCE-5208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13655899#comment-13655899 ] Hudson commented on MAPREDUCE-5208: --- Integrated in Hadoop-Yarn-trunk #208 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/208/]) MAPREDUCE-5208. Modified ShuffleHandler to use SecureIOUtils for reading local files. Contributed by Omkar Vinit Joshi. (Revision 1481657) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1481657 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/SpillRecord.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/resources/krb5.conf SpillRecord and ShuffleHandler should use SecureIOUtils for reading index file and map output - Key: MAPREDUCE-5208 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5208 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Fix For: 2.0.5-beta Attachments: MAPREDUCE-5208-20130506.patch, MAPREDUCE-5208-20130507.patch, MAPREDUCE-5208.20130510.patch ShuffleHandler (map output file) and SpillRecord (index file) are reading file using unsecured input stream. There exists a possibility for symlink attack. related to YARN-578 . Creating this issue to track map reduce changes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5208) SpillRecord and ShuffleHandler should use SecureIOUtils for reading index file and map output
[ https://issues.apache.org/jira/browse/MAPREDUCE-5208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13655903#comment-13655903 ] Dennis Y commented on MAPREDUCE-5208: - this patch breaks compilation for branch-2: 2013-05-13 14:59:09.643 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile (default-testCompile) on project hadoop-mapreduce-client-shuffle: Compilation failure: Compilation failure: 2013-05-13 14:59:09.645 [ERROR] /home/jenkins/jenkins/workspace/Hadoop-common-FB-Integration-fb-YAHOO-259-branch-2-gd/yhadoop-common/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java:[341,29] cannot find symbol 2013-05-13 14:59:09.646 [ERROR] symbol : variable POSIX 2013-05-13 14:59:09.648 [ERROR] location: class org.apache.hadoop.io.nativeio.NativeIO 2013-05-13 14:59:09.649 [ERROR] /home/jenkins/jenkins/workspace/Hadoop-common-FB-Integration-fb-YAHOO-259-branch-2-gd/yhadoop-common/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java:[357,19] join(java.lang.CharSequence,java.lang.Iterable?) in org.apache.hadoop.util.StringUtils cannot be applied to (java.lang.String,java.lang.String[]) SpillRecord and ShuffleHandler should use SecureIOUtils for reading index file and map output - Key: MAPREDUCE-5208 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5208 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Fix For: 2.0.5-beta Attachments: MAPREDUCE-5208-20130506.patch, MAPREDUCE-5208-20130507.patch, MAPREDUCE-5208.20130510.patch ShuffleHandler (map output file) and SpillRecord (index file) are reading file using unsecured input stream. There exists a possibility for symlink attack. related to YARN-578 . Creating this issue to track map reduce changes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5143) TestLineRecordReader was no test case for compressed files
[ https://issues.apache.org/jira/browse/MAPREDUCE-5143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated MAPREDUCE-5143: -- Attachment: MAPREDUCE-5143.1.patch Add a test case for compressed input file. TestLineRecordReader was no test case for compressed files -- Key: MAPREDUCE-5143 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5143 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Sonu Prathap Priority: Minor Attachments: MAPREDUCE-5143.1.patch TestLineRecordReader was no test case for compressed files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5143) TestLineRecordReader was no test case for compressed files
[ https://issues.apache.org/jira/browse/MAPREDUCE-5143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated MAPREDUCE-5143: -- Assignee: Tsuyoshi OZAWA TestLineRecordReader was no test case for compressed files -- Key: MAPREDUCE-5143 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5143 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Sonu Prathap Assignee: Tsuyoshi OZAWA Priority: Minor Attachments: MAPREDUCE-5143.1.patch TestLineRecordReader was no test case for compressed files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5143) TestLineRecordReader was no test case for compressed files
[ https://issues.apache.org/jira/browse/MAPREDUCE-5143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated MAPREDUCE-5143: -- Affects Version/s: 2.0.5-beta trunk 3.0.0 Status: Patch Available (was: Open) TestLineRecordReader was no test case for compressed files -- Key: MAPREDUCE-5143 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5143 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0, trunk, 2.0.5-beta Reporter: Sonu Prathap Assignee: Tsuyoshi OZAWA Priority: Minor Attachments: MAPREDUCE-5143.1.patch TestLineRecordReader was no test case for compressed files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5143) TestLineRecordReader was no test case for compressed files
[
https://issues.apache.org/jira/browse/MAPREDUCE-5143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13655939#comment-13655939
]
Hadoop QA commented on MAPREDUCE-5143:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582919/MAPREDUCE-5143.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3619//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3619//console
This message is automatically generated.
TestLineRecordReader was no test case for compressed files
--
Key: MAPREDUCE-5143
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5143
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 3.0.0, trunk, 2.0.5-beta
Reporter: Sonu Prathap
Assignee: Tsuyoshi OZAWA
Priority: Minor
Attachments: MAPREDUCE-5143.1.patch
TestLineRecordReader was no test case for compressed files
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4067) Replace YarnRemoteException with IOException in MRv2 APIs
[ https://issues.apache.org/jira/browse/MAPREDUCE-4067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13655946#comment-13655946 ] Hudson commented on MAPREDUCE-4067: --- Integrated in Hadoop-Hdfs-trunk #1397 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1397/]) MAPREDUCE-4067. Changed MRClientProtocol api to throw IOException only (Xuan Gong via vinodkv) (Revision 1481695) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1481695 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRAppBenchmark.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncherImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/MRClientProtocol.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/impl/pb/client/MRClientProtocolPBClientImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/impl/pb/service/MRClientProtocolPBServiceImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/security/MRDelegationTokenRenewer.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/TestRPCFactories.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryClientService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/NotRunningJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientRedirect.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestResourceMgrDelegate.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/TestJHSSecurity.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobsWithHistoryService.java Replace YarnRemoteException with IOException in MRv2 APIs - Key: MAPREDUCE-4067 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4067 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Xuan Gong Priority: Critical Fix For: 2.0.5-beta Attachments: MAPREDUCE-4067.1.patch, MAPREDUCE-4067.2.patch, MAPREDUCE-4067.3.patch, MAPREDUCE-4067.4.patch, MR4067.txt YarnRemoteException is defined as a generic wrapper for all the exceptions in yarn. I think we should instead throw IOExceptions in the API, which can later be extended for more
[jira] [Commented] (MAPREDUCE-5208) SpillRecord and ShuffleHandler should use SecureIOUtils for reading index file and map output
[ https://issues.apache.org/jira/browse/MAPREDUCE-5208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13655947#comment-13655947 ] Hudson commented on MAPREDUCE-5208: --- Integrated in Hadoop-Hdfs-trunk #1397 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1397/]) MAPREDUCE-5208. Modified ShuffleHandler to use SecureIOUtils for reading local files. Contributed by Omkar Vinit Joshi. (Revision 1481657) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1481657 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/SpillRecord.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/resources/krb5.conf SpillRecord and ShuffleHandler should use SecureIOUtils for reading index file and map output - Key: MAPREDUCE-5208 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5208 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Fix For: 2.0.5-beta Attachments: MAPREDUCE-5208-20130506.patch, MAPREDUCE-5208-20130507.patch, MAPREDUCE-5208.20130510.patch ShuffleHandler (map output file) and SpillRecord (index file) are reading file using unsecured input stream. There exists a possibility for symlink attack. related to YARN-578 . Creating this issue to track map reduce changes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4067) Replace YarnRemoteException with IOException in MRv2 APIs
[ https://issues.apache.org/jira/browse/MAPREDUCE-4067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13655984#comment-13655984 ] Hudson commented on MAPREDUCE-4067: --- Integrated in Hadoop-Mapreduce-trunk #1424 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1424/]) MAPREDUCE-4067. Changed MRClientProtocol api to throw IOException only (Xuan Gong via vinodkv) (Revision 1481695) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1481695 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRAppBenchmark.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncherImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/MRClientProtocol.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/impl/pb/client/MRClientProtocolPBClientImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/impl/pb/service/MRClientProtocolPBServiceImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/security/MRDelegationTokenRenewer.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/TestRPCFactories.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryClientService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/NotRunningJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientRedirect.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestResourceMgrDelegate.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/TestJHSSecurity.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobsWithHistoryService.java Replace YarnRemoteException with IOException in MRv2 APIs - Key: MAPREDUCE-4067 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4067 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Xuan Gong Priority: Critical Fix For: 2.0.5-beta Attachments: MAPREDUCE-4067.1.patch, MAPREDUCE-4067.2.patch, MAPREDUCE-4067.3.patch, MAPREDUCE-4067.4.patch, MR4067.txt YarnRemoteException is defined as a generic wrapper for all the exceptions in yarn. I think we should instead throw IOExceptions in the API, which can later be extended for more
[jira] [Commented] (MAPREDUCE-5208) SpillRecord and ShuffleHandler should use SecureIOUtils for reading index file and map output
[ https://issues.apache.org/jira/browse/MAPREDUCE-5208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13655985#comment-13655985 ] Hudson commented on MAPREDUCE-5208: --- Integrated in Hadoop-Mapreduce-trunk #1424 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1424/]) MAPREDUCE-5208. Modified ShuffleHandler to use SecureIOUtils for reading local files. Contributed by Omkar Vinit Joshi. (Revision 1481657) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1481657 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/SpillRecord.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/resources/krb5.conf SpillRecord and ShuffleHandler should use SecureIOUtils for reading index file and map output - Key: MAPREDUCE-5208 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5208 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Fix For: 2.0.5-beta Attachments: MAPREDUCE-5208-20130506.patch, MAPREDUCE-5208-20130507.patch, MAPREDUCE-5208.20130510.patch ShuffleHandler (map output file) and SpillRecord (index file) are reading file using unsecured input stream. There exists a possibility for symlink attack. related to YARN-578 . Creating this issue to track map reduce changes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4942) mapreduce.Job has a bunch of methods that throw InterruptedException so its incompatible with MR1
[
https://issues.apache.org/jira/browse/MAPREDUCE-4942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656106#comment-13656106
]
Alejandro Abdelnur commented on MAPREDUCE-4942:
---
Arun, this is then a catch22. If we don't fix it, we are breaking Hadoop 1
users moving to Hadoop 2. I guess we'll have have to decide what is more
important, to keep compatibility with Hadoop 1 or Hadoop 0.23. Y! folks may not
be happy with the following, but given the adoption of Hadoop 1 and Hadoop
0.23, we should incline to Hadoop 1. Thoughts?
mapreduce.Job has a bunch of methods that throw InterruptedException so its
incompatible with MR1
-
Key: MAPREDUCE-4942
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4942
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.2-alpha
Reporter: Robert Kanter
Assignee: Robert Kanter
Fix For: 2.0.5-beta
Attachments: MAPREDUCE-4942.patch
The following methods in MR2's {{org.apache.hadoop.mapreduce.Job}} throw an
{{InterruptedException}} but don't in MR1, which makes them incompatible.
(Their Javadoc comments are also missing that they throw an
{{InterruptedException}} anyway)
I propose that we wrap the {{InterruptedException}} in a
{{RuntimeException}}.
{code}
public float setupProgress() throws IOException, InterruptedException
public float mapProgress() throws IOException, InterruptedException
public float reduceProgress() throws IOException, InterruptedException
public boolean isComplete() throws IOException, InterruptedException
public boolean isSuccessful() throws IOException, InterruptedException
public void killJob() throws IOException, InterruptedException
public void killTask(org.apache.hadoop.mapreduce.TaskAttemptID) throws
IOException, InterruptedException
public void failTask(org.apache.hadoop.mapreduce.TaskAttemptID) throws
IOException, InterruptedException
public Counters getCounters() throws IOException, InterruptedException
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5215) mapreduce.Job is missing getJobClient() so its incompatible with MR1
[
https://issues.apache.org/jira/browse/MAPREDUCE-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated MAPREDUCE-5215:
-
Status: Open (was: Patch Available)
mapreduce.Job is missing getJobClient() so its incompatible with MR1
Key: MAPREDUCE-5215
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5215
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.2-alpha
Reporter: Robert Kanter
Assignee: Robert Kanter
Attachments: MAPREDUCE-5215.patch
The method {{org.apache.hadoop.mapred.JobClient getJobClient()}} is in MR1's
{{mapreduce.Job}} but doesn't exist in MR2's, which makes them incompatible.
MR2's implementation of {{Job}} doesn't use a JobClient object, but we can
create one and return it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5130) Add missing job config options to mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656120#comment-13656120 ] Sandy Ryza commented on MAPREDUCE-5130: --- Sorry, linked to the wrong ticket. MAPREDUCE-5238 is the one I filed for the test failure Add missing job config options to mapred-default.xml Key: MAPREDUCE-5130 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5130 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-5130-1.patch, MAPREDUCE-5130-1.patch, MAPREDUCE-5130-2.patch, MAPREDUCE-5130.patch I came across that mapreduce.map.child.java.opts and mapreduce.reduce.child.java.opts were missing in mapred-default.xml. I'll do a fuller sweep to see what else is missing before posting a patch. List so far: mapreduce.map/reduce.child.java.opts mapreduce.map/reduce.memory.mb mapreduce.job.jvm.numtasks mapreduce.input.lineinputformat.linespermap mapreduce.task.combine.progress.records -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5236) references to JobConf.DISABLE_MEMORY_LIMIT don't make sense in the context of MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656130#comment-13656130 ] Sandy Ryza commented on MAPREDUCE-5236: --- I didn't mean to suggest removing the constant, just references to it within the code. Sure, I can do this in MAPREDUCE-5130. references to JobConf.DISABLE_MEMORY_LIMIT don't make sense in the context of MR2 - Key: MAPREDUCE-5236 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5236 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza In MR1, a special value of -1 could be given for mapreduce.job.map|reduce.memory.mb when memory limits were disabled. In MR2, this makes no sense, as with slots gone, this value is used for requesting resources and scheduling. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5232) log classpath and other key properties on child JVM start
[ https://issues.apache.org/jira/browse/MAPREDUCE-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656149#comment-13656149 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-5232: bq. My reason for adding those lines was to make it look similar to what other nodes were doing: StringUtils.toStartupShutdownString(). Also, these marker lines are not printed after each line, but only once when the JVM starts. Did I miss something? No, I thought you were printing it after every property. Just checked the patch, that isn't the case. So it's fine as it is. log classpath and other key properties on child JVM start - Key: MAPREDUCE-5232 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5232 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, mrv2 Affects Versions: 2.0.4-alpha Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: MAPREDUCE-5232-branch-2.patch, MAPREDUCE-5232-branch-2.patch, MAPREDUCE-5232.patch, MAPREDUCE-5232.patch It would be great if we log vital information such as classpath, etc. upon a mapreduce child JVM start. This would help a great deal in terms of troubleshooting classpath issues, etc. Today it is pretty difficult to debug this unless you preserve the container script. Maybe it can log things like classpath, os name/version, java version, etc. at the beginning of the child JVM start. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5208) SpillRecord and ShuffleHandler should use SecureIOUtils for reading index file and map output
[ https://issues.apache.org/jira/browse/MAPREDUCE-5208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656162#comment-13656162 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-5208: [~dennisyv], checking it now. I forgot to compile and run tests on branch-2. Did that only on trunk. SpillRecord and ShuffleHandler should use SecureIOUtils for reading index file and map output - Key: MAPREDUCE-5208 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5208 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Fix For: 2.0.5-beta Attachments: MAPREDUCE-5208-20130506.patch, MAPREDUCE-5208-20130507.patch, MAPREDUCE-5208.20130510.patch ShuffleHandler (map output file) and SpillRecord (index file) are reading file using unsecured input stream. There exists a possibility for symlink attack. related to YARN-578 . Creating this issue to track map reduce changes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5232) log classpath and other key properties on child JVM start
[ https://issues.apache.org/jira/browse/MAPREDUCE-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-5232: --- Attachment: MAPREDUCE-5232.patch log classpath and other key properties on child JVM start - Key: MAPREDUCE-5232 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5232 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, mrv2 Affects Versions: 2.0.4-alpha Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: MAPREDUCE-5232-branch-2.patch, MAPREDUCE-5232-branch-2.patch, MAPREDUCE-5232.patch, MAPREDUCE-5232.patch, MAPREDUCE-5232.patch It would be great if we log vital information such as classpath, etc. upon a mapreduce child JVM start. This would help a great deal in terms of troubleshooting classpath issues, etc. Today it is pretty difficult to debug this unless you preserve the container script. Maybe it can log things like classpath, os name/version, java version, etc. at the beginning of the child JVM start. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5232) log classpath and other key properties on child JVM start
[ https://issues.apache.org/jira/browse/MAPREDUCE-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-5232: --- Attachment: MAPREDUCE-5232-branch-2.patch log classpath and other key properties on child JVM start - Key: MAPREDUCE-5232 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5232 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, mrv2 Affects Versions: 2.0.4-alpha Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: MAPREDUCE-5232-branch-2.patch, MAPREDUCE-5232-branch-2.patch, MAPREDUCE-5232-branch-2.patch, MAPREDUCE-5232.patch, MAPREDUCE-5232.patch, MAPREDUCE-5232.patch It would be great if we log vital information such as classpath, etc. upon a mapreduce child JVM start. This would help a great deal in terms of troubleshooting classpath issues, etc. Today it is pretty difficult to debug this unless you preserve the container script. Maybe it can log things like classpath, os name/version, java version, etc. at the beginning of the child JVM start. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5232) log classpath and other key properties on child JVM start
[ https://issues.apache.org/jira/browse/MAPREDUCE-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-5232: --- Status: Patch Available (was: Open) Updated the patches. By the way, the build seems broken on branch-2 currently (before this patch). [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile (default-testCompile) on project hadoop-mapreduce-client-shuffle: Compilation failure: Compilation failure: [ERROR] /Users/sjlee/git/hadoop-common/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java:[341,29] cannot find symbol [ERROR] symbol : variable POSIX [ERROR] location: class org.apache.hadoop.io.nativeio.NativeIO [ERROR] /Users/sjlee/git/hadoop-common/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java:[357,19] join(java.lang.CharSequence,java.lang.Iterable?) in org.apache.hadoop.util.StringUtils cannot be applied to (java.lang.String,java.lang.String[]) log classpath and other key properties on child JVM start - Key: MAPREDUCE-5232 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5232 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, mrv2 Affects Versions: 2.0.4-alpha Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: MAPREDUCE-5232-branch-2.patch, MAPREDUCE-5232-branch-2.patch, MAPREDUCE-5232-branch-2.patch, MAPREDUCE-5232.patch, MAPREDUCE-5232.patch, MAPREDUCE-5232.patch It would be great if we log vital information such as classpath, etc. upon a mapreduce child JVM start. This would help a great deal in terms of troubleshooting classpath issues, etc. Today it is pretty difficult to debug this unless you preserve the container script. Maybe it can log things like classpath, os name/version, java version, etc. at the beginning of the child JVM start. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4942) mapreduce.Job has a bunch of methods that throw InterruptedException so its incompatible with MR1
[
https://issues.apache.org/jira/browse/MAPREDUCE-4942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated MAPREDUCE-4942:
---
Issue Type: Sub-task (was: Bug)
Parent: MAPREDUCE-5108
mapreduce.Job has a bunch of methods that throw InterruptedException so its
incompatible with MR1
-
Key: MAPREDUCE-4942
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4942
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv2
Affects Versions: 2.0.2-alpha
Reporter: Robert Kanter
Assignee: Robert Kanter
Fix For: 2.0.5-beta
Attachments: MAPREDUCE-4942.patch
The following methods in MR2's {{org.apache.hadoop.mapreduce.Job}} throw an
{{InterruptedException}} but don't in MR1, which makes them incompatible.
(Their Javadoc comments are also missing that they throw an
{{InterruptedException}} anyway)
I propose that we wrap the {{InterruptedException}} in a
{{RuntimeException}}.
{code}
public float setupProgress() throws IOException, InterruptedException
public float mapProgress() throws IOException, InterruptedException
public float reduceProgress() throws IOException, InterruptedException
public boolean isComplete() throws IOException, InterruptedException
public boolean isSuccessful() throws IOException, InterruptedException
public void killJob() throws IOException, InterruptedException
public void killTask(org.apache.hadoop.mapreduce.TaskAttemptID) throws
IOException, InterruptedException
public void failTask(org.apache.hadoop.mapreduce.TaskAttemptID) throws
IOException, InterruptedException
public Counters getCounters() throws IOException, InterruptedException
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5215) mapreduce.Job is missing getJobClient() so its incompatible with MR1
[
https://issues.apache.org/jira/browse/MAPREDUCE-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated MAPREDUCE-5215:
---
Issue Type: Sub-task (was: Bug)
Parent: MAPREDUCE-5108
mapreduce.Job is missing getJobClient() so its incompatible with MR1
Key: MAPREDUCE-5215
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5215
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv2
Affects Versions: 2.0.2-alpha
Reporter: Robert Kanter
Assignee: Robert Kanter
Attachments: MAPREDUCE-5215.patch
The method {{org.apache.hadoop.mapred.JobClient getJobClient()}} is in MR1's
{{mapreduce.Job}} but doesn't exist in MR2's, which makes them incompatible.
MR2's implementation of {{Job}} doesn't use a JobClient object, but we can
create one and return it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4251) API Incompatibility - Sampler
[ https://issues.apache.org/jira/browse/MAPREDUCE-4251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-4251: --- Issue Type: Sub-task (was: Bug) Parent: MAPREDUCE-5108 API Incompatibility - Sampler - Key: MAPREDUCE-4251 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4251 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv1 Affects Versions: 0.22.0 Reporter: Benoy Antony Assignee: Benoy Antony Priority: Minor Attachments: MAPREDUCE-4251.patch org.apache.hadoop.mapred.lib.InputSampler#Sampler in Hadoop 0.20 has been moved to org.apache.hadoop.mapreduce.lib.partition.InputSampler#Sampler in Hadoop 0.22 The arguments of the getSample method in the Sampler class have also been changed, 0.22 use the new InputFormat, and 0.20 use the deprecated InputFormat, 0.22 use org.apache.hadoop.mapreduce.Job and 0.20use org.apache.hadoop.mapred.JobConf. So the programs compiled with old api has to be changed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4251) API Incompatibility - Sampler
[ https://issues.apache.org/jira/browse/MAPREDUCE-4251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen resolved MAPREDUCE-4251. Resolution: Duplicate It duplicates MAPREDUCE-5157. Reopen it if it is needed for 22 also. API Incompatibility - Sampler - Key: MAPREDUCE-4251 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4251 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv1 Affects Versions: 0.22.0 Reporter: Benoy Antony Assignee: Benoy Antony Priority: Minor Attachments: MAPREDUCE-4251.patch org.apache.hadoop.mapred.lib.InputSampler#Sampler in Hadoop 0.20 has been moved to org.apache.hadoop.mapreduce.lib.partition.InputSampler#Sampler in Hadoop 0.22 The arguments of the getSample method in the Sampler class have also been changed, 0.22 use the new InputFormat, and 0.20 use the deprecated InputFormat, 0.22 use org.apache.hadoop.mapreduce.Job and 0.20use org.apache.hadoop.mapred.JobConf. So the programs compiled with old api has to be changed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5208) SpillRecord and ShuffleHandler should use SecureIOUtils for reading index file and map output
[ https://issues.apache.org/jira/browse/MAPREDUCE-5208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656220#comment-13656220 ] Omkar Vinit Joshi commented on MAPREDUCE-5208: -- [~vinodkv] , [~dennisyv] ... branch-2 patch needs to updated... There is nothing like 'NativeIO.POSIX' in branch-2.. updating the patch SpillRecord and ShuffleHandler should use SecureIOUtils for reading index file and map output - Key: MAPREDUCE-5208 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5208 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Fix For: 2.0.5-beta Attachments: MAPREDUCE-5208-20130506.patch, MAPREDUCE-5208-20130507.patch, MAPREDUCE-5208.20130510.patch ShuffleHandler (map output file) and SpillRecord (index file) are reading file using unsecured input stream. There exists a possibility for symlink attack. related to YARN-578 . Creating this issue to track map reduce changes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5220) Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5220: -- Attachment: MAPREDUCE-5220.patch Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2 Key: MAPREDUCE-5220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5220 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: client Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.0.5-beta Attachments: MAPREDUCE-5220.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5220) Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5220: -- Status: Patch Available (was: Open) Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2 Key: MAPREDUCE-5220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5220 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: client Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.0.5-beta Attachments: MAPREDUCE-5220.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5220) Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656260#comment-13656260 ] Sandy Ryza commented on MAPREDUCE-5220: --- Uploaded a patch that changes the methods to public and marks them as @Private. Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2 Key: MAPREDUCE-5220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5220 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: client Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.0.5-beta Attachments: MAPREDUCE-5220.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5220) Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656263#comment-13656263 ] Alejandro Abdelnur commented on MAPREDUCE-5220: --- +1. Differently from MAPREDUCE-4942 this will not break 0.23 as the visibility is widening, [~acmurthy]? Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2 Key: MAPREDUCE-5220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5220 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: client Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.0.5-beta Attachments: MAPREDUCE-5220.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5168) Reducer can OOM during shuffle because on-disk output stream not released
[ https://issues.apache.org/jira/browse/MAPREDUCE-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656267#comment-13656267 ] Jason Lowe commented on MAPREDUCE-5168: --- [~vinodkv], do you have any reservations about this going into trunk and branch-2? I can commit this just to branch-0.23 if desired. Reducer can OOM during shuffle because on-disk output stream not released - Key: MAPREDUCE-5168 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5168 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.7, 2.0.5-beta Reporter: Jason Lowe Assignee: Jason Lowe Priority: Critical Attachments: MAPREDUCE-5168-branch-0.23.patch, MAPREDUCE-5168.patch If a reducer needs to shuffle a map output to disk, it opens an output stream and writes the data to disk. However it does not release the reference to the output stream within the MapOutput, and the output stream can have a 128K buffer attached to it. If enough of these on-disk outputs are queued up waiting to be merged, it can cause the reducer to OOM during the shuffle phase. In one case I saw there were 1200 on-disk outputs queued up to be merged, leading to an extra 150MB of pressure on the heap due to the output stream buffers that were no longer necessary. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4366) mapred metrics shows negative count of waiting maps and reduces
[ https://issues.apache.org/jira/browse/MAPREDUCE-4366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656268#comment-13656268 ] Alejandro Abdelnur commented on MAPREDUCE-4366: --- +1, LGTM. [~acmurthy]? mapred metrics shows negative count of waiting maps and reduces --- Key: MAPREDUCE-4366 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4366 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 1.0.2 Reporter: Thomas Graves Assignee: Sandy Ryza Attachments: MAPREDUCE-4366-branch-1-1.patch, MAPREDUCE-4366-branch-1.patch Negative waiting_maps and waiting_reduces count is observed in the mapred metrics. MAPREDUCE-1238 partially fixed this but it appears there is still issues as we are seeing it, but not as bad. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5208) SpillRecord and ShuffleHandler should use SecureIOUtils for reading index file and map output
[ https://issues.apache.org/jira/browse/MAPREDUCE-5208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated MAPREDUCE-5208: - Attachment: MAPREDUCE-5208.20130513.branch2.update.patch Patch update for branch-2 SpillRecord and ShuffleHandler should use SecureIOUtils for reading index file and map output - Key: MAPREDUCE-5208 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5208 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Fix For: 2.0.5-beta Attachments: MAPREDUCE-5208-20130506.patch, MAPREDUCE-5208-20130507.patch, MAPREDUCE-5208.20130510.patch, MAPREDUCE-5208.20130513.branch2.update.patch ShuffleHandler (map output file) and SpillRecord (index file) are reading file using unsecured input stream. There exists a possibility for symlink attack. related to YARN-578 . Creating this issue to track map reduce changes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4942) mapreduce.Job has a bunch of methods that throw InterruptedException so its incompatible with MR1
[
https://issues.apache.org/jira/browse/MAPREDUCE-4942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656278#comment-13656278
]
Robert Joseph Evans commented on MAPREDUCE-4942:
This is one of the odd places where the code is binary compatible but not
source compatible. 0.23 binaries should run against the 2.0 API, but they are
likely to not compile against the changed API unless they are catching
Exception :).
I can see your reasoning about wanting to maintain compatibility with 1.0, we
got so much flack from asking everyone to recompile that we are not going to be
able to do it again for a long time. Because of that I personally am against
putting in any changes that are incompatible with 0.23. If we do it will
either delay Yahoo adopting 2.0 significantly or it will force us to fork until
we can build up enough good will that we can risk asking everyone to recompile
again. I am not going to -1 this because I don't think that is good for the
community, but it will mean that Yahoo! is not going to be able to run on stock
2.0 for a while.
mapreduce.Job has a bunch of methods that throw InterruptedException so its
incompatible with MR1
-
Key: MAPREDUCE-4942
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4942
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv2
Affects Versions: 2.0.2-alpha
Reporter: Robert Kanter
Assignee: Robert Kanter
Fix For: 2.0.5-beta
Attachments: MAPREDUCE-4942.patch
The following methods in MR2's {{org.apache.hadoop.mapreduce.Job}} throw an
{{InterruptedException}} but don't in MR1, which makes them incompatible.
(Their Javadoc comments are also missing that they throw an
{{InterruptedException}} anyway)
I propose that we wrap the {{InterruptedException}} in a
{{RuntimeException}}.
{code}
public float setupProgress() throws IOException, InterruptedException
public float mapProgress() throws IOException, InterruptedException
public float reduceProgress() throws IOException, InterruptedException
public boolean isComplete() throws IOException, InterruptedException
public boolean isSuccessful() throws IOException, InterruptedException
public void killJob() throws IOException, InterruptedException
public void killTask(org.apache.hadoop.mapreduce.TaskAttemptID) throws
IOException, InterruptedException
public void failTask(org.apache.hadoop.mapreduce.TaskAttemptID) throws
IOException, InterruptedException
public Counters getCounters() throws IOException, InterruptedException
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5240) inside of FileOutputCommitter the initialized Credentials cache appears to be empty
[
https://issues.apache.org/jira/browse/MAPREDUCE-5240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656281#comment-13656281
]
Mark Grover commented on MAPREDUCE-5240:
Thanks, Vinod, for the patch.
I tested Sqoop with Hadoop built with this patch and was able to run an import
and export job successfully.
For those interested, I installed patched Hadoop from
http://bigtop01.cloudera.org:8080/job/Hadoop-2.0.4/label=centos6/lastSuccessfulBuild/artifact/output/hadoop/
and ran it in pseudo distributed mode. I also downloaded and installed sqoop
(v. 1.99.2) from Bigtop. I created a MySQL table with a primary key column and
a bunch of other columns and imported data from there to HDFS and then exported
to the same data back to MySQL to a different empty table with same schema as
the other table.
You have my +1 (non-committer).
inside of FileOutputCommitter the initialized Credentials cache appears to be
empty
---
Key: MAPREDUCE-5240
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5240
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.4-alpha
Reporter: Roman Shaposhnik
Assignee: Vinod Kumar Vavilapalli
Priority: Blocker
Fix For: 2.0.5-beta
Attachments: LostCreds.java, MAPREDUCE-5240-20130512.txt
I am attaching a modified wordcount job that clearly demonstrates the problem
we've encountered in running Sqoop2 on YARN (BIGTOP-949).
Here's what running it produces:
{noformat}
$ hadoop fs -mkdir in
$ hadoop fs -put /etc/passwd in
$ hadoop jar ./bug.jar org.myorg.LostCreds
13/05/12 03:13:46 WARN mapred.JobConf: The variable mapred.child.ulimit is no
longer used.
numberOfSecretKeys: 1
numberOfTokens: 0
..
..
..
13/05/12 03:05:35 INFO mapreduce.Job: Job job_1368318686284_0013 failed with
state FAILED due to: Job commit failed: java.io.IOException:
numberOfSecretKeys: 0
numberOfTokens: 0
at
org.myorg.LostCreds$DestroyerFileOutputCommitter.commitJob(LostCreds.java:43)
at
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:249)
at
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:212)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
{noformat}
As you can see, even though we've clearly initialized the creds via:
{noformat}
job.getCredentials().addSecretKey(new Text(mykey), mysecret.getBytes());
{noformat}
It doesn't seem to appear later in the job.
This is a pretty critical issue for Sqoop 2 since it appears to be DOA for
YARN in Hadoop 2.0.4-alpha
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5208) SpillRecord and ShuffleHandler should use SecureIOUtils for reading index file and map output
[ https://issues.apache.org/jira/browse/MAPREDUCE-5208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656288#comment-13656288 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-5208: Tx for the response, Omkar. Can you just post an update patch as I already committed the original one to branch-2? Also, please run all YARN tests on branch-2. Tx. SpillRecord and ShuffleHandler should use SecureIOUtils for reading index file and map output - Key: MAPREDUCE-5208 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5208 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Fix For: 2.0.5-beta Attachments: MAPREDUCE-5208-20130506.patch, MAPREDUCE-5208-20130507.patch, MAPREDUCE-5208.20130510.patch, MAPREDUCE-5208.20130513.branch2.update.patch ShuffleHandler (map output file) and SpillRecord (index file) are reading file using unsecured input stream. There exists a possibility for symlink attack. related to YARN-578 . Creating this issue to track map reduce changes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5240) inside of FileOutputCommitter the initialized Credentials cache appears to be empty
[
https://issues.apache.org/jira/browse/MAPREDUCE-5240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656290#comment-13656290
]
Jarek Jarcec Cecho commented on MAPREDUCE-5240:
---
[~mgrover] thank you Mark for testing this, greatly appreciated!
inside of FileOutputCommitter the initialized Credentials cache appears to be
empty
---
Key: MAPREDUCE-5240
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5240
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.4-alpha
Reporter: Roman Shaposhnik
Assignee: Vinod Kumar Vavilapalli
Priority: Blocker
Fix For: 2.0.5-beta
Attachments: LostCreds.java, MAPREDUCE-5240-20130512.txt
I am attaching a modified wordcount job that clearly demonstrates the problem
we've encountered in running Sqoop2 on YARN (BIGTOP-949).
Here's what running it produces:
{noformat}
$ hadoop fs -mkdir in
$ hadoop fs -put /etc/passwd in
$ hadoop jar ./bug.jar org.myorg.LostCreds
13/05/12 03:13:46 WARN mapred.JobConf: The variable mapred.child.ulimit is no
longer used.
numberOfSecretKeys: 1
numberOfTokens: 0
..
..
..
13/05/12 03:05:35 INFO mapreduce.Job: Job job_1368318686284_0013 failed with
state FAILED due to: Job commit failed: java.io.IOException:
numberOfSecretKeys: 0
numberOfTokens: 0
at
org.myorg.LostCreds$DestroyerFileOutputCommitter.commitJob(LostCreds.java:43)
at
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:249)
at
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:212)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
{noformat}
As you can see, even though we've clearly initialized the creds via:
{noformat}
job.getCredentials().addSecretKey(new Text(mykey), mysecret.getBytes());
{noformat}
It doesn't seem to appear later in the job.
This is a pretty critical issue for Sqoop 2 since it appears to be DOA for
YARN in Hadoop 2.0.4-alpha
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5208) SpillRecord and ShuffleHandler should use SecureIOUtils for reading index file and map output
[ https://issues.apache.org/jira/browse/MAPREDUCE-5208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated MAPREDUCE-5208: - Attachment: MAPREDUCE-5208.20130513.branch2.update.1.patch Changes for branch-2. SpillRecord and ShuffleHandler should use SecureIOUtils for reading index file and map output - Key: MAPREDUCE-5208 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5208 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Fix For: 2.0.5-beta Attachments: MAPREDUCE-5208-20130506.patch, MAPREDUCE-5208-20130507.patch, MAPREDUCE-5208.20130510.patch, MAPREDUCE-5208.20130513.branch2.update.1.patch, MAPREDUCE-5208.20130513.branch2.update.patch ShuffleHandler (map output file) and SpillRecord (index file) are reading file using unsecured input stream. There exists a possibility for symlink attack. related to YARN-578 . Creating this issue to track map reduce changes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5222) Add missing methods to JobClient
[ https://issues.apache.org/jira/browse/MAPREDUCE-5222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656300#comment-13656300 ] Karthik Kambatla commented on MAPREDUCE-5222: - Thanks Alejandro, Vinod and Zhijie for your inputs. Was not sure how we wanted proceed with the APIs in question given they refer to old nomenclature of Job. Regarding submitJobInternal(), the javadoc for the method specifically mentions it is for internal use. Should we still add it? Will address rest of the comments. Thanks again. Add missing methods to JobClient - Key: MAPREDUCE-5222 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5222 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: 2.0.4-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Fix For: 2.0.5-beta Attachments: mr-5222.patch, mr-5222.patch, mr-5222-rte.patch JobClient is missing the following two public methods we need to add for binary compatibility: # static isJobDirValid(Path, FileSystem) # Path getStagingAreaDir() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5220) Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5220: --- Status: Open (was: Patch Available) bq. Differently from MAPREDUCE-4942 this will not break 0.23 as the visibility is widening Not clear what you are saying. This will not break 0.23, right? Also, I don't see setTaskAttemptId at all in 1.x. That change should be reverted. Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2 Key: MAPREDUCE-5220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5220 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: client Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.0.5-beta Attachments: MAPREDUCE-5220.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5240) inside of FileOutputCommitter the initialized Credentials cache appears to be empty
[
https://issues.apache.org/jira/browse/MAPREDUCE-5240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656321#comment-13656321
]
Vinod Kumar Vavilapalli commented on MAPREDUCE-5240:
Thanks Mark, I'll ping another committer to quickly review this.
inside of FileOutputCommitter the initialized Credentials cache appears to be
empty
---
Key: MAPREDUCE-5240
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5240
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.4-alpha
Reporter: Roman Shaposhnik
Assignee: Vinod Kumar Vavilapalli
Priority: Blocker
Fix For: 2.0.5-beta
Attachments: LostCreds.java, MAPREDUCE-5240-20130512.txt
I am attaching a modified wordcount job that clearly demonstrates the problem
we've encountered in running Sqoop2 on YARN (BIGTOP-949).
Here's what running it produces:
{noformat}
$ hadoop fs -mkdir in
$ hadoop fs -put /etc/passwd in
$ hadoop jar ./bug.jar org.myorg.LostCreds
13/05/12 03:13:46 WARN mapred.JobConf: The variable mapred.child.ulimit is no
longer used.
numberOfSecretKeys: 1
numberOfTokens: 0
..
..
..
13/05/12 03:05:35 INFO mapreduce.Job: Job job_1368318686284_0013 failed with
state FAILED due to: Job commit failed: java.io.IOException:
numberOfSecretKeys: 0
numberOfTokens: 0
at
org.myorg.LostCreds$DestroyerFileOutputCommitter.commitJob(LostCreds.java:43)
at
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:249)
at
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:212)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
{noformat}
As you can see, even though we've clearly initialized the creds via:
{noformat}
job.getCredentials().addSecretKey(new Text(mykey), mysecret.getBytes());
{noformat}
It doesn't seem to appear later in the job.
This is a pretty critical issue for Sqoop 2 since it appears to be DOA for
YARN in Hadoop 2.0.4-alpha
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5208) SpillRecord and ShuffleHandler should use SecureIOUtils for reading index file and map output
[ https://issues.apache.org/jira/browse/MAPREDUCE-5208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656333#comment-13656333 ] Omkar Vinit Joshi commented on MAPREDUCE-5208: -- I compiled and tested this on branch-2 .. for below 2 machines. unsecured (and native io disabled) single node mac machine and secured (native io enabled) ubuntu.12.04 machine SpillRecord and ShuffleHandler should use SecureIOUtils for reading index file and map output - Key: MAPREDUCE-5208 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5208 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Fix For: 2.0.5-beta Attachments: MAPREDUCE-5208-20130506.patch, MAPREDUCE-5208-20130507.patch, MAPREDUCE-5208.20130510.patch, MAPREDUCE-5208.20130513.branch2.update.1.patch, MAPREDUCE-5208.20130513.branch2.update.patch ShuffleHandler (map output file) and SpillRecord (index file) are reading file using unsecured input stream. There exists a possibility for symlink attack. related to YARN-578 . Creating this issue to track map reduce changes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4942) mapreduce.Job has a bunch of methods that throw InterruptedException so its incompatible with MR1
[
https://issues.apache.org/jira/browse/MAPREDUCE-4942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656340#comment-13656340
]
Arun C Murthy commented on MAPREDUCE-4942:
--
[~tucu00] Unfortunately, as we have talked about, it's well beyond a point
where we can support compatibility for o.a.h.mapreduce vis-a-vis MR1. Too much
water down the pipe with MAPREDUCE-777 etc.
Given that, I'd rather make it easy for Yahoo and other early adopters of YARN.
Makes sense?
mapreduce.Job has a bunch of methods that throw InterruptedException so its
incompatible with MR1
-
Key: MAPREDUCE-4942
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4942
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv2
Affects Versions: 2.0.2-alpha
Reporter: Robert Kanter
Assignee: Robert Kanter
Fix For: 2.0.5-beta
Attachments: MAPREDUCE-4942.patch
The following methods in MR2's {{org.apache.hadoop.mapreduce.Job}} throw an
{{InterruptedException}} but don't in MR1, which makes them incompatible.
(Their Javadoc comments are also missing that they throw an
{{InterruptedException}} anyway)
I propose that we wrap the {{InterruptedException}} in a
{{RuntimeException}}.
{code}
public float setupProgress() throws IOException, InterruptedException
public float mapProgress() throws IOException, InterruptedException
public float reduceProgress() throws IOException, InterruptedException
public boolean isComplete() throws IOException, InterruptedException
public boolean isSuccessful() throws IOException, InterruptedException
public void killJob() throws IOException, InterruptedException
public void killTask(org.apache.hadoop.mapreduce.TaskAttemptID) throws
IOException, InterruptedException
public void failTask(org.apache.hadoop.mapreduce.TaskAttemptID) throws
IOException, InterruptedException
public Counters getCounters() throws IOException, InterruptedException
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5222) Add missing methods to JobClient
[ https://issues.apache.org/jira/browse/MAPREDUCE-5222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656342#comment-13656342 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-5222: Tx for reporting the remaining issues in JobClient, Zhijie! Like I said, let's get that all fixed here instead of doing it piece-meal. bq. Regarding submitJobInternal().. Anything that is public should be supported, whether it is *supposed to* be internal or not. Unless we cannot support it all, of course. Add missing methods to JobClient - Key: MAPREDUCE-5222 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5222 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: 2.0.4-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Fix For: 2.0.5-beta Attachments: mr-5222.patch, mr-5222.patch, mr-5222-rte.patch JobClient is missing the following two public methods we need to add for binary compatibility: # static isJobDirValid(Path, FileSystem) # Path getStagingAreaDir() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5220) Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656346#comment-13656346 ] Sandy Ryza commented on MAPREDUCE-5220: --- This will not break 0.23. Uploading a new patch with the reverted setTaskAttemptId Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2 Key: MAPREDUCE-5220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5220 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: client Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.0.5-beta Attachments: MAPREDUCE-5220.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5220) Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5220: -- Attachment: MAPREDUCE-5220-1.patch Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2 Key: MAPREDUCE-5220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5220 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: client Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.0.5-beta Attachments: MAPREDUCE-5220-1.patch, MAPREDUCE-5220.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5233) Functions are changed or removed from Job in jobcontrol
[ https://issues.apache.org/jira/browse/MAPREDUCE-5233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656347#comment-13656347 ] Mayank Bansal commented on MAPREDUCE-5233: -- Thanks Zhijie for the review. 1. In setState, it is good to add default block to handle the case that the int param is not within the value domain of the State enum. Looks like its not needed as by default i m returning the default value. 2. getMapredJobID is changed to return String to be compatible with 1.x. Is it good to keep a similar function (maybe called getMapredJobId) to still return the JobID object? In case the function will be potentially used in 2.x. Good Idea , Fixed it. 3. I'm not clear why getJobID will be called twice with the patch. Perhaps some inline comments here to explain the two usages? -verify(mockjob).getJobID(); +verify(mockjob,times(2)).getJobID(); Fixed it in latest patch Thanks, Mayank Functions are changed or removed from Job in jobcontrol --- Key: MAPREDUCE-5233 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5233 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Mayank Bansal Fix For: 2.0.5-beta Attachments: MAPREDUCE-5233-trunk-1.patch, MAPREDUCE-5233-trunk-2.patch, MAPREDUCE-5233-trunk-3.patch The functions are removed from Job in jobcontrol: 1. setMapredJobID(String) 2. setState(int) The function signatures are changed: 1. addDependingJob(ControlledJob) 2. getMapredJobID() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5233) Functions are changed or removed from Job in jobcontrol
[ https://issues.apache.org/jira/browse/MAPREDUCE-5233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated MAPREDUCE-5233: - Status: Open (was: Patch Available) Functions are changed or removed from Job in jobcontrol --- Key: MAPREDUCE-5233 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5233 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Mayank Bansal Fix For: 2.0.5-beta Attachments: MAPREDUCE-5233-trunk-1.patch, MAPREDUCE-5233-trunk-2.patch, MAPREDUCE-5233-trunk-3.patch, MAPREDUCE-5233-trunk-4.patch The functions are removed from Job in jobcontrol: 1. setMapredJobID(String) 2. setState(int) The function signatures are changed: 1. addDependingJob(ControlledJob) 2. getMapredJobID() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5233) Functions are changed or removed from Job in jobcontrol
[ https://issues.apache.org/jira/browse/MAPREDUCE-5233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated MAPREDUCE-5233: - Attachment: MAPREDUCE-5233-trunk-4.patch Functions are changed or removed from Job in jobcontrol --- Key: MAPREDUCE-5233 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5233 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Mayank Bansal Fix For: 2.0.5-beta Attachments: MAPREDUCE-5233-trunk-1.patch, MAPREDUCE-5233-trunk-2.patch, MAPREDUCE-5233-trunk-3.patch, MAPREDUCE-5233-trunk-4.patch The functions are removed from Job in jobcontrol: 1. setMapredJobID(String) 2. setState(int) The function signatures are changed: 1. addDependingJob(ControlledJob) 2. getMapredJobID() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5233) Functions are changed or removed from Job in jobcontrol
[ https://issues.apache.org/jira/browse/MAPREDUCE-5233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated MAPREDUCE-5233: - Status: Patch Available (was: Open) Functions are changed or removed from Job in jobcontrol --- Key: MAPREDUCE-5233 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5233 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Mayank Bansal Fix For: 2.0.5-beta Attachments: MAPREDUCE-5233-trunk-1.patch, MAPREDUCE-5233-trunk-2.patch, MAPREDUCE-5233-trunk-3.patch, MAPREDUCE-5233-trunk-4.patch The functions are removed from Job in jobcontrol: 1. setMapredJobID(String) 2. setState(int) The function signatures are changed: 1. addDependingJob(ControlledJob) 2. getMapredJobID() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5208) SpillRecord and ShuffleHandler should use SecureIOUtils for reading index file and map output
[ https://issues.apache.org/jira/browse/MAPREDUCE-5208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656351#comment-13656351 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-5208: Thanks, Omkar, I committed the bug-fix to branch-2. SpillRecord and ShuffleHandler should use SecureIOUtils for reading index file and map output - Key: MAPREDUCE-5208 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5208 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi Fix For: 2.0.5-beta Attachments: MAPREDUCE-5208-20130506.patch, MAPREDUCE-5208-20130507.patch, MAPREDUCE-5208.20130510.patch, MAPREDUCE-5208.20130513.branch2.update.1.patch, MAPREDUCE-5208.20130513.branch2.update.patch ShuffleHandler (map output file) and SpillRecord (index file) are reading file using unsecured input stream. There exists a possibility for symlink attack. related to YARN-578 . Creating this issue to track map reduce changes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5242) Remove mapred-default.xml
Siddharth Seth created MAPREDUCE-5242: - Summary: Remove mapred-default.xml Key: MAPREDUCE-5242 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5242 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.0.4-alpha Reporter: Siddharth Seth Similar to YARN-673. (details there) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5220) Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5220: -- Status: Patch Available (was: Open) Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2 Key: MAPREDUCE-5220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5220 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: client Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.0.5-beta Attachments: MAPREDUCE-5220-1.patch, MAPREDUCE-5220.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5130) Add missing job config options to mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5130: -- Attachment: MAPREDUCE-5130-3.patch Add missing job config options to mapred-default.xml Key: MAPREDUCE-5130 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5130 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-5130-1.patch, MAPREDUCE-5130-1.patch, MAPREDUCE-5130-2.patch, MAPREDUCE-5130-3.patch, MAPREDUCE-5130.patch I came across that mapreduce.map.child.java.opts and mapreduce.reduce.child.java.opts were missing in mapred-default.xml. I'll do a fuller sweep to see what else is missing before posting a patch. List so far: mapreduce.map/reduce.child.java.opts mapreduce.map/reduce.memory.mb mapreduce.job.jvm.numtasks mapreduce.input.lineinputformat.linespermap mapreduce.task.combine.progress.records -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5130) Add missing job config options to mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5130: -- Status: Patch Available (was: Open) Uploaded a new patch that includes the changes to JobConf Add missing job config options to mapred-default.xml Key: MAPREDUCE-5130 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5130 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-5130-1.patch, MAPREDUCE-5130-1.patch, MAPREDUCE-5130-2.patch, MAPREDUCE-5130-3.patch, MAPREDUCE-5130.patch I came across that mapreduce.map.child.java.opts and mapreduce.reduce.child.java.opts were missing in mapred-default.xml. I'll do a fuller sweep to see what else is missing before posting a patch. List so far: mapreduce.map/reduce.child.java.opts mapreduce.map/reduce.memory.mb mapreduce.job.jvm.numtasks mapreduce.input.lineinputformat.linespermap mapreduce.task.combine.progress.records -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5220) Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656366#comment-13656366
]
Hadoop QA commented on MAPREDUCE-5220:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582987/MAPREDUCE-5220-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3623//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3623//console
This message is automatically generated.
Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2
Key: MAPREDUCE-5220
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5220
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: client
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Fix For: 2.0.5-beta
Attachments: MAPREDUCE-5220-1.patch, MAPREDUCE-5220.patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5211) Reducer intermediate files can collide during merge
[ https://issues.apache.org/jira/browse/MAPREDUCE-5211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-5211: -- Resolution: Fixed Fix Version/s: 0.23.8 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for the reviews. I committed this to branch-0.23. I do not believe the collision in branch-2/trunk is possible since it concatenates an absolute path of a source file to a working directory specific to the reduce attempt to build a destination path. The pathnames can become very long, but I don't think they will collide. If we want to change that behavior, we can handle it in another JIRA. Reducer intermediate files can collide during merge --- Key: MAPREDUCE-5211 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5211 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.7 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Blocker Fix For: 0.23.8 Attachments: MAPREDUCE-5211.branch-0.23.patch The OnDiskMerger.merge method constructs an output path that is not unique to a reduce attempt, and as a result can result in a file collision with other reducers from the same app that are running on the same node. In addition the name of the output file is based on MapOutput.toString which may not be unique in light of multi-pass merges on disk since the mapId will be null and the basename ends up as MapOutput(null, DISK) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5240) inside of FileOutputCommitter the initialized Credentials cache appears to be empty
[
https://issues.apache.org/jira/browse/MAPREDUCE-5240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656394#comment-13656394
]
Siddharth Seth commented on MAPREDUCE-5240:
---
Vinod, instead of replacing YarnConfiguration with JobConf in the app master,
can this change be localized. i.e. use a JobConf object only when initializing
the OutputCommitter.
We shouldn't need to load yarn-default.xml within the app master itself, since
it would've been loaded in the client - but I'd prefer being certain of this.
Alternately we can create the JobConf with a new YarnConfiguration as the
parameter.
inside of FileOutputCommitter the initialized Credentials cache appears to be
empty
---
Key: MAPREDUCE-5240
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5240
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.4-alpha
Reporter: Roman Shaposhnik
Assignee: Vinod Kumar Vavilapalli
Priority: Blocker
Fix For: 2.0.5-beta
Attachments: LostCreds.java, MAPREDUCE-5240-20130512.txt
I am attaching a modified wordcount job that clearly demonstrates the problem
we've encountered in running Sqoop2 on YARN (BIGTOP-949).
Here's what running it produces:
{noformat}
$ hadoop fs -mkdir in
$ hadoop fs -put /etc/passwd in
$ hadoop jar ./bug.jar org.myorg.LostCreds
13/05/12 03:13:46 WARN mapred.JobConf: The variable mapred.child.ulimit is no
longer used.
numberOfSecretKeys: 1
numberOfTokens: 0
..
..
..
13/05/12 03:05:35 INFO mapreduce.Job: Job job_1368318686284_0013 failed with
state FAILED due to: Job commit failed: java.io.IOException:
numberOfSecretKeys: 0
numberOfTokens: 0
at
org.myorg.LostCreds$DestroyerFileOutputCommitter.commitJob(LostCreds.java:43)
at
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:249)
at
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:212)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
{noformat}
As you can see, even though we've clearly initialized the creds via:
{noformat}
job.getCredentials().addSecretKey(new Text(mykey), mysecret.getBytes());
{noformat}
It doesn't seem to appear later in the job.
This is a pretty critical issue for Sqoop 2 since it appears to be DOA for
YARN in Hadoop 2.0.4-alpha
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5232) log classpath and other key properties on child JVM start
[ https://issues.apache.org/jira/browse/MAPREDUCE-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656405#comment-13656405 ] Sangjin Lee commented on MAPREDUCE-5232: Looks like the pre-commit build failed for an unrelated reason: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3620/console Is it possible to re-kick the pre-commit build? log classpath and other key properties on child JVM start - Key: MAPREDUCE-5232 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5232 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, mrv2 Affects Versions: 2.0.4-alpha Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: MAPREDUCE-5232-branch-2.patch, MAPREDUCE-5232-branch-2.patch, MAPREDUCE-5232-branch-2.patch, MAPREDUCE-5232.patch, MAPREDUCE-5232.patch, MAPREDUCE-5232.patch It would be great if we log vital information such as classpath, etc. upon a mapreduce child JVM start. This would help a great deal in terms of troubleshooting classpath issues, etc. Today it is pretty difficult to debug this unless you preserve the container script. Maybe it can log things like classpath, os name/version, java version, etc. at the beginning of the child JVM start. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5243) MRAdmin is removed from M/R while RMAdmin is added to Yarn
Zhijie Shen created MAPREDUCE-5243: -- Summary: MRAdmin is removed from M/R while RMAdmin is added to Yarn Key: MAPREDUCE-5243 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5243 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Though in the 2.x mapred script, MRAdmin will not be called, MRAdmin class is better to be there in case users call it programmatically. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5243) MRAdmin is removed from M/R while RMAdmin is added to Yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-5243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla resolved MAPREDUCE-5243. - Resolution: Duplicate Marking this as a duplicate of MAPREDUCE-5214 MRAdmin is removed from M/R while RMAdmin is added to Yarn -- Key: MAPREDUCE-5243 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5243 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Fix For: 2.0.5-beta Though in the 2.x mapred script, MRAdmin will not be called, MRAdmin class is better to be there in case users call it programmatically. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5214) Compatibility: Add a deprecated MRAdmin that wraps around RMAdmin
[ https://issues.apache.org/jira/browse/MAPREDUCE-5214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656422#comment-13656422 ] Karthik Kambatla commented on MAPREDUCE-5214: - [~zjshen] created MAPREDUCE-5243 to address the same. May be we should still get this in. Comments? Compatibility: Add a deprecated MRAdmin that wraps around RMAdmin - Key: MAPREDUCE-5214 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5214 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.0.4-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: mr-5214-v0.patch MRAdmin doesn't apply to MR2. However, to maintain compatibility against 1.x releases, it might be a good idea to add a deprecated version of MRAdmin that wraps around RMAdmin, prints out a deprecated message and calls the relevant RMAdmin methods. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5214) Compatibility: Add a deprecated MRAdmin that wraps around RMAdmin
[ https://issues.apache.org/jira/browse/MAPREDUCE-5214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-5214: Issue Type: Sub-task (was: Bug) Parent: MAPREDUCE-5108 Compatibility: Add a deprecated MRAdmin that wraps around RMAdmin - Key: MAPREDUCE-5214 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5214 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: 2.0.4-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: mr-5214-v0.patch MRAdmin doesn't apply to MR2. However, to maintain compatibility against 1.x releases, it might be a good idea to add a deprecated version of MRAdmin that wraps around RMAdmin, prints out a deprecated message and calls the relevant RMAdmin methods. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5028) Maps fail when io.sort.mb is set to high value
[
https://issues.apache.org/jira/browse/MAPREDUCE-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656425#comment-13656425
]
Karthik Kambatla commented on MAPREDUCE-5028:
-
Hey [~acmurthy]: just checking if you had a chance to take a look at this.
Maps fail when io.sort.mb is set to high value
--
Key: MAPREDUCE-5028
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5028
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 1.1.1, 2.0.3-alpha, 0.23.5
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Critical
Fix For: 1.2.0, 2.0.5-beta
Attachments: mr-5028-branch1.patch, mr-5028-branch1.patch,
mr-5028-branch1.patch, mr-5028-trunk.patch, mr-5028-trunk.patch,
mr-5028-trunk.patch, repro-mr-5028.patch
Verified the problem exists on branch-1 with the following configuration:
Pseudo-dist mode: 2 maps/ 1 reduce, mapred.child.java.opts=-Xmx2048m,
io.sort.mb=1280, dfs.block.size=2147483648
Run teragen to generate 4 GB data
Maps fail when you run wordcount on this configuration with the following
error:
{noformat}
java.io.IOException: Spill failed
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1031)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:692)
at
org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at
org.apache.hadoop.examples.WordCount$TokenizerMapper.map(WordCount.java:45)
at
org.apache.hadoop.examples.WordCount$TokenizerMapper.map(WordCount.java:34)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at org.apache.hadoop.io.IntWritable.readFields(IntWritable.java:38)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
at
org.apache.hadoop.mapreduce.ReduceContext.nextKeyValue(ReduceContext.java:116)
at
org.apache.hadoop.mapreduce.ReduceContext.nextKey(ReduceContext.java:92)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:175)
at
org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1505)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1438)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:855)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1346)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5244) Two functions change their visibility in JobStatus
Zhijie Shen created MAPREDUCE-5244: -- Summary: Two functions change their visibility in JobStatus Key: MAPREDUCE-5244 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5244 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Two functions change their visibility in JobStatus from public to protected: void setRunState(int) void setSchedulingInfo(String) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5240) inside of FileOutputCommitter the initialized Credentials cache appears to be empty
[
https://issues.apache.org/jira/browse/MAPREDUCE-5240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656443#comment-13656443
]
Vinod Kumar Vavilapalli commented on MAPREDUCE-5240:
bq, Vinod, instead of replacing YarnConfiguration with JobConf in the app
master, can this change be localized. i.e. use a JobConf object only when
initializing the OutputCommitter.
JobConf is heavy, I didn't want to clone another one unnecessarily.
bq. We shouldn't need to load yarn-default.xml within the app master itself,
since it would've been loaded in the client - but I'd prefer being certain of
this. Alternately we can create the JobConf with a new YarnConfiguration as the
parameter.
Good point. Don't know if something in MRAppMaster uses stuff in
YarnConfiguration, but doing what you are suggesting to be safe.
inside of FileOutputCommitter the initialized Credentials cache appears to be
empty
---
Key: MAPREDUCE-5240
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5240
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.4-alpha
Reporter: Roman Shaposhnik
Assignee: Vinod Kumar Vavilapalli
Priority: Blocker
Fix For: 2.0.5-beta
Attachments: LostCreds.java, MAPREDUCE-5240-20130512.txt,
MAPREDUCE-5240-20130513.txt
I am attaching a modified wordcount job that clearly demonstrates the problem
we've encountered in running Sqoop2 on YARN (BIGTOP-949).
Here's what running it produces:
{noformat}
$ hadoop fs -mkdir in
$ hadoop fs -put /etc/passwd in
$ hadoop jar ./bug.jar org.myorg.LostCreds
13/05/12 03:13:46 WARN mapred.JobConf: The variable mapred.child.ulimit is no
longer used.
numberOfSecretKeys: 1
numberOfTokens: 0
..
..
..
13/05/12 03:05:35 INFO mapreduce.Job: Job job_1368318686284_0013 failed with
state FAILED due to: Job commit failed: java.io.IOException:
numberOfSecretKeys: 0
numberOfTokens: 0
at
org.myorg.LostCreds$DestroyerFileOutputCommitter.commitJob(LostCreds.java:43)
at
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:249)
at
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:212)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
{noformat}
As you can see, even though we've clearly initialized the creds via:
{noformat}
job.getCredentials().addSecretKey(new Text(mykey), mysecret.getBytes());
{noformat}
It doesn't seem to appear later in the job.
This is a pretty critical issue for Sqoop 2 since it appears to be DOA for
YARN in Hadoop 2.0.4-alpha
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5240) inside of FileOutputCommitter the initialized Credentials cache appears to be empty
[
https://issues.apache.org/jira/browse/MAPREDUCE-5240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated MAPREDUCE-5240:
---
Status: Open (was: Patch Available)
inside of FileOutputCommitter the initialized Credentials cache appears to be
empty
---
Key: MAPREDUCE-5240
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5240
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.4-alpha
Reporter: Roman Shaposhnik
Assignee: Vinod Kumar Vavilapalli
Priority: Blocker
Fix For: 2.0.5-beta
Attachments: LostCreds.java, MAPREDUCE-5240-20130512.txt,
MAPREDUCE-5240-20130513.txt
I am attaching a modified wordcount job that clearly demonstrates the problem
we've encountered in running Sqoop2 on YARN (BIGTOP-949).
Here's what running it produces:
{noformat}
$ hadoop fs -mkdir in
$ hadoop fs -put /etc/passwd in
$ hadoop jar ./bug.jar org.myorg.LostCreds
13/05/12 03:13:46 WARN mapred.JobConf: The variable mapred.child.ulimit is no
longer used.
numberOfSecretKeys: 1
numberOfTokens: 0
..
..
..
13/05/12 03:05:35 INFO mapreduce.Job: Job job_1368318686284_0013 failed with
state FAILED due to: Job commit failed: java.io.IOException:
numberOfSecretKeys: 0
numberOfTokens: 0
at
org.myorg.LostCreds$DestroyerFileOutputCommitter.commitJob(LostCreds.java:43)
at
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:249)
at
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:212)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
{noformat}
As you can see, even though we've clearly initialized the creds via:
{noformat}
job.getCredentials().addSecretKey(new Text(mykey), mysecret.getBytes());
{noformat}
It doesn't seem to appear later in the job.
This is a pretty critical issue for Sqoop 2 since it appears to be DOA for
YARN in Hadoop 2.0.4-alpha
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5240) inside of FileOutputCommitter the initialized Credentials cache appears to be empty
[
https://issues.apache.org/jira/browse/MAPREDUCE-5240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated MAPREDUCE-5240:
---
Attachment: MAPREDUCE-5240-20130513.txt
Updated patch to address Sid's comment.
inside of FileOutputCommitter the initialized Credentials cache appears to be
empty
---
Key: MAPREDUCE-5240
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5240
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.4-alpha
Reporter: Roman Shaposhnik
Assignee: Vinod Kumar Vavilapalli
Priority: Blocker
Fix For: 2.0.5-beta
Attachments: LostCreds.java, MAPREDUCE-5240-20130512.txt,
MAPREDUCE-5240-20130513.txt
I am attaching a modified wordcount job that clearly demonstrates the problem
we've encountered in running Sqoop2 on YARN (BIGTOP-949).
Here's what running it produces:
{noformat}
$ hadoop fs -mkdir in
$ hadoop fs -put /etc/passwd in
$ hadoop jar ./bug.jar org.myorg.LostCreds
13/05/12 03:13:46 WARN mapred.JobConf: The variable mapred.child.ulimit is no
longer used.
numberOfSecretKeys: 1
numberOfTokens: 0
..
..
..
13/05/12 03:05:35 INFO mapreduce.Job: Job job_1368318686284_0013 failed with
state FAILED due to: Job commit failed: java.io.IOException:
numberOfSecretKeys: 0
numberOfTokens: 0
at
org.myorg.LostCreds$DestroyerFileOutputCommitter.commitJob(LostCreds.java:43)
at
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:249)
at
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:212)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
{noformat}
As you can see, even though we've clearly initialized the creds via:
{noformat}
job.getCredentials().addSecretKey(new Text(mykey), mysecret.getBytes());
{noformat}
It doesn't seem to appear later in the job.
This is a pretty critical issue for Sqoop 2 since it appears to be DOA for
YARN in Hadoop 2.0.4-alpha
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5240) inside of FileOutputCommitter the initialized Credentials cache appears to be empty
[
https://issues.apache.org/jira/browse/MAPREDUCE-5240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated MAPREDUCE-5240:
---
Status: Patch Available (was: Open)
inside of FileOutputCommitter the initialized Credentials cache appears to be
empty
---
Key: MAPREDUCE-5240
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5240
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.4-alpha
Reporter: Roman Shaposhnik
Assignee: Vinod Kumar Vavilapalli
Priority: Blocker
Fix For: 2.0.5-beta
Attachments: LostCreds.java, MAPREDUCE-5240-20130512.txt,
MAPREDUCE-5240-20130513.txt
I am attaching a modified wordcount job that clearly demonstrates the problem
we've encountered in running Sqoop2 on YARN (BIGTOP-949).
Here's what running it produces:
{noformat}
$ hadoop fs -mkdir in
$ hadoop fs -put /etc/passwd in
$ hadoop jar ./bug.jar org.myorg.LostCreds
13/05/12 03:13:46 WARN mapred.JobConf: The variable mapred.child.ulimit is no
longer used.
numberOfSecretKeys: 1
numberOfTokens: 0
..
..
..
13/05/12 03:05:35 INFO mapreduce.Job: Job job_1368318686284_0013 failed with
state FAILED due to: Job commit failed: java.io.IOException:
numberOfSecretKeys: 0
numberOfTokens: 0
at
org.myorg.LostCreds$DestroyerFileOutputCommitter.commitJob(LostCreds.java:43)
at
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:249)
at
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:212)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
{noformat}
As you can see, even though we've clearly initialized the creds via:
{noformat}
job.getCredentials().addSecretKey(new Text(mykey), mysecret.getBytes());
{noformat}
It doesn't seem to appear later in the job.
This is a pretty critical issue for Sqoop 2 since it appears to be DOA for
YARN in Hadoop 2.0.4-alpha
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5245) A number of public static variables are removed from JobConf
Zhijie Shen created MAPREDUCE-5245: -- Summary: A number of public static variables are removed from JobConf Key: MAPREDUCE-5245 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5245 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen A number of public static variables are removed from JobConf: boolean DEFAULT_MAPREDUCE_RECOVER_JOB String MAPREDUCE_RECOVER_JOB String WORKFLOW_ADJACENCY_PREFIX_PATTERN String WORKFLOW_ADJACENCY_PREFIX_STRING String WORKFLOW_ID String WORKFLOW_NAME String WORKFLOW_NODE_NAME String WORKFLOW_TAGS The workflow related variables are moved to MRJobConfig. The follwing public static variables becomes default: String MAPRED_JOB_MAP_MEMORY_MB_PROPERTY String MAPRED_JOB_REDUCE_MEMORY_MB_PROPERTY The variables there are no longer referred internally in 2.x, but they might be used by users as they were public. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5240) inside of FileOutputCommitter the initialized Credentials cache appears to be empty
[
https://issues.apache.org/jira/browse/MAPREDUCE-5240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656450#comment-13656450
]
Siddharth Seth commented on MAPREDUCE-5240:
---
+1, subject to nothing being reported by jenkins.
inside of FileOutputCommitter the initialized Credentials cache appears to be
empty
---
Key: MAPREDUCE-5240
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5240
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.4-alpha
Reporter: Roman Shaposhnik
Assignee: Vinod Kumar Vavilapalli
Priority: Blocker
Fix For: 2.0.5-beta
Attachments: LostCreds.java, MAPREDUCE-5240-20130512.txt,
MAPREDUCE-5240-20130513.txt
I am attaching a modified wordcount job that clearly demonstrates the problem
we've encountered in running Sqoop2 on YARN (BIGTOP-949).
Here's what running it produces:
{noformat}
$ hadoop fs -mkdir in
$ hadoop fs -put /etc/passwd in
$ hadoop jar ./bug.jar org.myorg.LostCreds
13/05/12 03:13:46 WARN mapred.JobConf: The variable mapred.child.ulimit is no
longer used.
numberOfSecretKeys: 1
numberOfTokens: 0
..
..
..
13/05/12 03:05:35 INFO mapreduce.Job: Job job_1368318686284_0013 failed with
state FAILED due to: Job commit failed: java.io.IOException:
numberOfSecretKeys: 0
numberOfTokens: 0
at
org.myorg.LostCreds$DestroyerFileOutputCommitter.commitJob(LostCreds.java:43)
at
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:249)
at
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:212)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
{noformat}
As you can see, even though we've clearly initialized the creds via:
{noformat}
job.getCredentials().addSecretKey(new Text(mykey), mysecret.getBytes());
{noformat}
It doesn't seem to appear later in the job.
This is a pretty critical issue for Sqoop 2 since it appears to be DOA for
YARN in Hadoop 2.0.4-alpha
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5222) Add missing methods to JobClient
[ https://issues.apache.org/jira/browse/MAPREDUCE-5222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-5222: Attachment: mr-5222-2.patch Uploading a patch that addresses all comments. Added @Private annotation to methods that are public, but are *supposed* to be for internal use; this ensures compatibility but also captures the intended users. Add missing methods to JobClient - Key: MAPREDUCE-5222 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5222 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: 2.0.4-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Fix For: 2.0.5-beta Attachments: mr-5222-2.patch, mr-5222.patch, mr-5222.patch, mr-5222-rte.patch JobClient is missing the following two public methods we need to add for binary compatibility: # static isJobDirValid(Path, FileSystem) # Path getStagingAreaDir() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5222) Add missing methods to JobClient
[ https://issues.apache.org/jira/browse/MAPREDUCE-5222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-5222: Status: Patch Available (was: Open) Add missing methods to JobClient - Key: MAPREDUCE-5222 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5222 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: 2.0.4-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Fix For: 2.0.5-beta Attachments: mr-5222-2.patch, mr-5222.patch, mr-5222.patch, mr-5222-rte.patch JobClient is missing the following two public methods we need to add for binary compatibility: # static isJobDirValid(Path, FileSystem) # Path getStagingAreaDir() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5220) Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656456#comment-13656456 ] Arun C Murthy commented on MAPREDUCE-5220: -- +1, not sure this is a big deal. Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2 Key: MAPREDUCE-5220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5220 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: client Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.0.5-beta Attachments: MAPREDUCE-5220-1.patch, MAPREDUCE-5220.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5233) Functions are changed or removed from Job in jobcontrol
[ https://issues.apache.org/jira/browse/MAPREDUCE-5233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated MAPREDUCE-5233: - Status: Open (was: Patch Available) Functions are changed or removed from Job in jobcontrol --- Key: MAPREDUCE-5233 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5233 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Mayank Bansal Fix For: 2.0.5-beta Attachments: MAPREDUCE-5233-trunk-1.patch, MAPREDUCE-5233-trunk-2.patch, MAPREDUCE-5233-trunk-3.patch, MAPREDUCE-5233-trunk-4.patch The functions are removed from Job in jobcontrol: 1. setMapredJobID(String) 2. setState(int) The function signatures are changed: 1. addDependingJob(ControlledJob) 2. getMapredJobID() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5233) Functions are changed or removed from Job in jobcontrol
[ https://issues.apache.org/jira/browse/MAPREDUCE-5233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated MAPREDUCE-5233: - Status: Patch Available (was: Open) Functions are changed or removed from Job in jobcontrol --- Key: MAPREDUCE-5233 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5233 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Mayank Bansal Fix For: 2.0.5-beta Attachments: MAPREDUCE-5233-trunk-1.patch, MAPREDUCE-5233-trunk-2.patch, MAPREDUCE-5233-trunk-3.patch, MAPREDUCE-5233-trunk-4.patch The functions are removed from Job in jobcontrol: 1. setMapredJobID(String) 2. setState(int) The function signatures are changed: 1. addDependingJob(ControlledJob) 2. getMapredJobID() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5214) Compatibility: Add a deprecated MRAdmin that wraps around RMAdmin
[ https://issues.apache.org/jira/browse/MAPREDUCE-5214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656460#comment-13656460 ] Arun C Murthy commented on MAPREDUCE-5214: -- Agree with [~tucu00]. We don't need to support this for YARN. Compatibility: Add a deprecated MRAdmin that wraps around RMAdmin - Key: MAPREDUCE-5214 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5214 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: 2.0.4-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: mr-5214-v0.patch MRAdmin doesn't apply to MR2. However, to maintain compatibility against 1.x releases, it might be a good idea to add a deprecated version of MRAdmin that wraps around RMAdmin, prints out a deprecated message and calls the relevant RMAdmin methods. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5214) Compatibility: Add a deprecated MRAdmin that wraps around RMAdmin
[ https://issues.apache.org/jira/browse/MAPREDUCE-5214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-5214. -- Resolution: Won't Fix Compatibility: Add a deprecated MRAdmin that wraps around RMAdmin - Key: MAPREDUCE-5214 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5214 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: 2.0.4-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: mr-5214-v0.patch MRAdmin doesn't apply to MR2. However, to maintain compatibility against 1.x releases, it might be a good idea to add a deprecated version of MRAdmin that wraps around RMAdmin, prints out a deprecated message and calls the relevant RMAdmin methods. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5214) Compatibility: Add a deprecated MRAdmin that wraps around RMAdmin
[ https://issues.apache.org/jira/browse/MAPREDUCE-5214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656463#comment-13656463 ] Karthik Kambatla commented on MAPREDUCE-5214: - FWIW, MAPREDUCE-5218 annotates MRAdmin as private to make sure the intent is communicated. Compatibility: Add a deprecated MRAdmin that wraps around RMAdmin - Key: MAPREDUCE-5214 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5214 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: 2.0.4-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: mr-5214-v0.patch MRAdmin doesn't apply to MR2. However, to maintain compatibility against 1.x releases, it might be a good idea to add a deprecated version of MRAdmin that wraps around RMAdmin, prints out a deprecated message and calls the relevant RMAdmin methods. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5244) Two functions change their visibility in JobStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-5244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5244: --- Target Version/s: 2.0.5-beta Fix Version/s: (was: 2.0.5-beta) Two functions change their visibility in JobStatus -- Key: MAPREDUCE-5244 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5244 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Two functions change their visibility in JobStatus from public to protected: void setRunState(int) void setSchedulingInfo(String) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5214) Compatibility: Add a deprecated MRAdmin that wraps around RMAdmin
[ https://issues.apache.org/jira/browse/MAPREDUCE-5214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656482#comment-13656482 ] Zhijie Shen commented on MAPREDUCE-5214: So it is good to document that the scope of binary compatibility is the APIs that MR 1.x applications may use. Compatibility: Add a deprecated MRAdmin that wraps around RMAdmin - Key: MAPREDUCE-5214 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5214 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: 2.0.4-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: mr-5214-v0.patch MRAdmin doesn't apply to MR2. However, to maintain compatibility against 1.x releases, it might be a good idea to add a deprecated version of MRAdmin that wraps around RMAdmin, prints out a deprecated message and calls the relevant RMAdmin methods. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5244) Two functions change their visibility in JobStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-5244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656483#comment-13656483 ] Sandy Ryza commented on MAPREDUCE-5244: --- [~zjshen], we're wrapping up a release and would like to include this. If you're not planning on working on it today or tomorrow, do you mind if I take it on? Two functions change their visibility in JobStatus -- Key: MAPREDUCE-5244 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5244 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Two functions change their visibility in JobStatus from public to protected: void setRunState(int) void setSchedulingInfo(String) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5244) Two functions change their visibility in JobStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-5244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656486#comment-13656486 ] Zhijie Shen commented on MAPREDUCE-5244: I'll fix it shortly. Two functions change their visibility in JobStatus -- Key: MAPREDUCE-5244 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5244 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Two functions change their visibility in JobStatus from public to protected: void setRunState(int) void setSchedulingInfo(String) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5222) Fix JobClient incompatibilities with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5222: --- Summary: Fix JobClient incompatibilities with MR1 (was: Add missing methods to JobClient ) Editing title as we are doing a little more than adding some missing methods. Fix JobClient incompatibilities with MR1 Key: MAPREDUCE-5222 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5222 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: 2.0.4-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Fix For: 2.0.5-beta Attachments: mr-5222-2.patch, mr-5222.patch, mr-5222.patch, mr-5222-rte.patch JobClient is missing the following two public methods we need to add for binary compatibility: # static isJobDirValid(Path, FileSystem) # Path getStagingAreaDir() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5222) Fix JobClient incompatibilities with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5222: --- Fix Version/s: (was: 2.0.5-beta) Also, Karthik, a general suggestion: Fix version is set when a committer checks in the code - use TargetVersion instead indicating your intention. Tx. Fix JobClient incompatibilities with MR1 Key: MAPREDUCE-5222 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5222 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: 2.0.4-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: mr-5222-2.patch, mr-5222.patch, mr-5222.patch, mr-5222-rte.patch JobClient is missing the following two public methods we need to add for binary compatibility: # static isJobDirValid(Path, FileSystem) # Path getStagingAreaDir() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5244) Two functions change their visibility in JobStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-5244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656499#comment-13656499 ] Sandy Ryza commented on MAPREDUCE-5244: --- Thanks a ton. Two functions change their visibility in JobStatus -- Key: MAPREDUCE-5244 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5244 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Two functions change their visibility in JobStatus from public to protected: void setRunState(int) void setSchedulingInfo(String) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5228) Enum Counter is removed from FileInputFormat and FileOutputFormat of both mapred and mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5228: --- Target Version/s: 2.0.5-beta Enum Counter is removed from FileInputFormat and FileOutputFormat of both mapred and mapreduce -- Key: MAPREDUCE-5228 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5228 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen The enum was used by findCounter(Enum key) to find a specific counter object. Now it seems to be replaced by FileInputFormatCounter and FileOutputFormatCounter. Now the enum seems to be only used internally, but not sure whether it will be used externally when users extend FileFormat. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5229) TEMP_DIR_NAME is removed from of FileOutputCommitter of mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5229: --- Target Version/s: 2.0.5-beta TEMP_DIR_NAME is removed from of FileOutputCommitter of mapreduce - Key: MAPREDUCE-5229 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5229 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen TEMP_DIR_NAME is removed from of FileOutputCommitter of mapreduce. As FileOutputFormat and FileOutputCommitter may be extend by users, and the extended user classes can configured to use. Therefore, this missing public static variable may cause compatibility problem when the extended classes refer it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5245) A number of public static variables are removed from JobConf
[ https://issues.apache.org/jira/browse/MAPREDUCE-5245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5245: --- Target Version/s: 2.0.5-beta Fix Version/s: (was: 2.0.5-beta) A number of public static variables are removed from JobConf Key: MAPREDUCE-5245 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5245 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen A number of public static variables are removed from JobConf: boolean DEFAULT_MAPREDUCE_RECOVER_JOB String MAPREDUCE_RECOVER_JOB String WORKFLOW_ADJACENCY_PREFIX_PATTERN String WORKFLOW_ADJACENCY_PREFIX_STRING String WORKFLOW_ID String WORKFLOW_NAME String WORKFLOW_NODE_NAME String WORKFLOW_TAGS The workflow related variables are moved to MRJobConfig. The follwing public static variables becomes default: String MAPRED_JOB_MAP_MEMORY_MB_PROPERTY String MAPRED_JOB_REDUCE_MEMORY_MB_PROPERTY The variables there are no longer referred internally in 2.x, but they might be used by users as they were public. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5237) Three functions are removed in ClusterStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-5237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5237: --- Target Version/s: 2.0.5-beta Fix Version/s: (was: 2.0.5-beta) Three functions are removed in ClusterStatus - Key: MAPREDUCE-5237 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5237 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen The three functions are: CollectionString getGraylistedTrackerNames() int getGraylistedTrackers() State getJobTrackerState() The're tracker related, such that they are no longer used in MR2. Maybe we should add them and throw UnsupportedOperationException? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5230) createFileSplit is removed from NLineInputFormat of mapred
[ https://issues.apache.org/jira/browse/MAPREDUCE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5230: --- Target Version/s: 2.0.5-beta Fix Version/s: (was: 2.0.5-beta) createFileSplit is removed from NLineInputFormat of mapred -- Key: MAPREDUCE-5230 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5230 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen createFileSplit is removed from NLineInputFormat of mapred, because it's no longer used in the new getSplit implementation. However, since function is protected before, there is still the potential risk that the user defined format class which extends old NLineInputFormat uses the protected function. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5231) Constructor of DBInputFormat.DBRecordReader in mapred is changed
[ https://issues.apache.org/jira/browse/MAPREDUCE-5231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5231: --- Target Version/s: 2.0.5-beta Fix Version/s: (was: 2.0.5-beta) Constructor of DBInputFormat.DBRecordReader in mapred is changed Key: MAPREDUCE-5231 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5231 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen The constructor of DBInputFormat.DBRecordReader in mapred is changed from MR1 to RM2. Though MAPREDUCE-716 tried to deal with the API difference. However, if DBInputFormat.DBRecordReader is extended, the incompatibility around the constructor is still there. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5220) Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-5220: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks Sandy. Committed to trunk and branch-2. Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2 Key: MAPREDUCE-5220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5220 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: client Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.0.5-beta Attachments: MAPREDUCE-5220-1.patch, MAPREDUCE-5220.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5234) Signature changes for getTaskId of TaskReport in mapred
[ https://issues.apache.org/jira/browse/MAPREDUCE-5234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5234: --- Target Version/s: 2.0.5-beta Fix Version/s: (was: 2.0.5-beta) Signature changes for getTaskId of TaskReport in mapred --- Key: MAPREDUCE-5234 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5234 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen TaskReport in mapred of MR2 extends TaskReport in mapreduce, and inherits getTaskId, which return TaskID object. in MR1, this function returns String. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5233) Functions are changed or removed from Job in jobcontrol
[ https://issues.apache.org/jira/browse/MAPREDUCE-5233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5233: --- Target Version/s: 2.0.5-beta Fix Version/s: (was: 2.0.5-beta) Functions are changed or removed from Job in jobcontrol --- Key: MAPREDUCE-5233 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5233 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Mayank Bansal Attachments: MAPREDUCE-5233-trunk-1.patch, MAPREDUCE-5233-trunk-2.patch, MAPREDUCE-5233-trunk-3.patch, MAPREDUCE-5233-trunk-4.patch The functions are removed from Job in jobcontrol: 1. setMapredJobID(String) 2. setState(int) The function signatures are changed: 1. addDependingJob(ControlledJob) 2. getMapredJobID() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (MAPREDUCE-5220) Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656505#comment-13656505 ] Alejandro Abdelnur edited comment on MAPREDUCE-5220 at 5/13/13 11:15 PM: - Thanks Sandy. Committed to trunk and branch-2. And thanks Arun for looking at it. was (Author: tucu00): Thanks Sandy. Committed to trunk and branch-2. Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2 Key: MAPREDUCE-5220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5220 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: client Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.0.5-beta Attachments: MAPREDUCE-5220-1.patch, MAPREDUCE-5220.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5222) Fix JobClient incompatibilities with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656507#comment-13656507 ] Karthik Kambatla commented on MAPREDUCE-5222: - Thanks Vinod. My bad - the fix version was unintentional. Fix JobClient incompatibilities with MR1 Key: MAPREDUCE-5222 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5222 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: 2.0.4-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: mr-5222-2.patch, mr-5222.patch, mr-5222.patch, mr-5222-rte.patch JobClient is missing the following two public methods we need to add for binary compatibility: # static isJobDirValid(Path, FileSystem) # Path getStagingAreaDir() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5235) MAX_GROUP_LIMIT is removed from Counters in mapred
[ https://issues.apache.org/jira/browse/MAPREDUCE-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5235: --- Target Version/s: 2.0.5-beta Fix Version/s: (was: 2.0.5-beta) MAX_GROUP_LIMIT is removed from Counters in mapred -- Key: MAPREDUCE-5235 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5235 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen MAX_GROUP_LIMIT is removed from Counters in mapred in MR2. Though it seems not to be the variable that will be referred by the user code. It was actually configurable value MR1. We should investigate why the upper bound doesn't need to be checked in MR2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5038) old API CombineFileInputFormat missing fixes that are in new API
[ https://issues.apache.org/jira/browse/MAPREDUCE-5038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656509#comment-13656509 ] Sandy Ryza commented on MAPREDUCE-5038: --- [~hagleitn], did you get a chance to look at this? old API CombineFileInputFormat missing fixes that are in new API - Key: MAPREDUCE-5038 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5038 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1.1.1 Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 1.3.0 Attachments: MAPREDUCE-5038-1.patch, MAPREDUCE-5038.patch, MAPREDUCE-5038-revised-1.patch, MAPREDUCE-5038-revised-1.patch, MAPREDUCE-5038-revised.patch The following changes patched the CombineFileInputFormat in mapreduce, but neglected the one in mapred MAPREDUCE-1597 enabled the CombineFileInputFormat to work on splittable files MAPREDUCE-2021 solved returning duplicate hostnames in split locations MAPREDUCE-1806 CombineFileInputFormat does not work with paths not on default FS In trunk this is not an issue as the one in mapred extends the one in mapreduce. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
