[jira] [Commented] (MAPREDUCE-4052) Windows eclipse cannot submit job from Windows client to Linux/Unix Hadoop cluster.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13855737#comment-13855737 ] Attila Pados commented on MAPREDUCE-4052: - Hi, as i found, the patch attached to this jira ticket, is not working for 2.2.0, Hadoop2 GA. i made another jira issue: https://issues.apache.org/jira/browse/MAPREDUCE-5655 and also added a patch that fixes it. Please keep in mind, that this is generaly a workaround. Attila Windows eclipse cannot submit job from Windows client to Linux/Unix Hadoop cluster. --- Key: MAPREDUCE-4052 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4052 Project: Hadoop Map/Reduce Issue Type: Bug Components: job submission Affects Versions: 0.23.1, 2.2.0 Environment: client on the Windows, the the cluster on the suse Reporter: xieguiming Assignee: xieguiming Attachments: MAPREDUCE-4052-0.patch, MAPREDUCE-4052.patch when I use the eclipse on the windows to submit the job. and the applicationmaster throw the exception: Exception in thread main java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/v2/app/MRAppMaster Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.v2.app.MRAppMaster at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) Could not find the main class: org.apache.hadoop.mapreduce.v2.app.MRAppMaster. Program will exit. The reasion is : class Apps addToEnvironment function, use the private static final String SYSTEM_PATH_SEPARATOR = System.getProperty(path.separator); and will result the MRApplicationMaster classpath use the ; separator. I suggest that nodemanger do the replace. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (MAPREDUCE-5694) MR AM container syslog is empty
[ https://issues.apache.org/jira/browse/MAPREDUCE-5694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Kamrul Islam updated MAPREDUCE-5694: - Attachment: MAPREDUCE-5694.1.patch patch included. MR AM container syslog is empty - Key: MAPREDUCE-5694 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5694 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.2.0 Reporter: Mohammad Kamrul Islam Assignee: Mohammad Kamrul Islam Attachments: MAPREDUCE-5694.1.patch When the property mapreduce.task.userlog.limit.kb is set non-zero in mapped-site.xml, AM container syslog remains empty. Without the log, it is hard to identify the cause of any MR AM failure. However, if mapreduce.task.userlog.limit.kb is not set (or defaulted to 0), syslog contents are correct. Bug details: For non zero mapreduce.task.userlog.limit.kb, the code caches the log contents into memory, waited to be written to file when close() method is called at the end(Ref: ContainerLogAppender.java). Explicit call of LogManager.shutdown() from the main() (REF:MRAppMaster.java), ultimately call the *Appender.close(). Root Cause: LogManager.shutdown() is not being called from MRAppMaster.java. Similar steps were taken correctly in YarnChild.java. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (MAPREDUCE-5694) MR AM container syslog is empty
[ https://issues.apache.org/jira/browse/MAPREDUCE-5694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Kamrul Islam updated MAPREDUCE-5694: - Status: Patch Available (was: Open) MR AM container syslog is empty - Key: MAPREDUCE-5694 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5694 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.2.0 Reporter: Mohammad Kamrul Islam Assignee: Mohammad Kamrul Islam Attachments: MAPREDUCE-5694.1.patch When the property mapreduce.task.userlog.limit.kb is set non-zero in mapped-site.xml, AM container syslog remains empty. Without the log, it is hard to identify the cause of any MR AM failure. However, if mapreduce.task.userlog.limit.kb is not set (or defaulted to 0), syslog contents are correct. Bug details: For non zero mapreduce.task.userlog.limit.kb, the code caches the log contents into memory, waited to be written to file when close() method is called at the end(Ref: ContainerLogAppender.java). Explicit call of LogManager.shutdown() from the main() (REF:MRAppMaster.java), ultimately call the *Appender.close(). Root Cause: LogManager.shutdown() is not being called from MRAppMaster.java. Similar steps were taken correctly in YarnChild.java. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (MAPREDUCE-5694) MR AM container syslog is empty
[
https://issues.apache.org/jira/browse/MAPREDUCE-5694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13855914#comment-13855914
]
Hadoop QA commented on MAPREDUCE-5694:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12620250/MAPREDUCE-5694.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4280//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4280//console
This message is automatically generated.
MR AM container syslog is empty
-
Key: MAPREDUCE-5694
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5694
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Mohammad Kamrul Islam
Assignee: Mohammad Kamrul Islam
Attachments: MAPREDUCE-5694.1.patch
When the property mapreduce.task.userlog.limit.kb is set non-zero in
mapped-site.xml, AM container syslog remains empty. Without the log, it is
hard to identify the cause of any MR AM failure.
However, if mapreduce.task.userlog.limit.kb is not set (or defaulted to
0), syslog contents are correct.
Bug details:
For non zero mapreduce.task.userlog.limit.kb, the code caches the log
contents into memory, waited to be written to file when close() method is
called at the end(Ref: ContainerLogAppender.java).
Explicit call of LogManager.shutdown() from the main()
(REF:MRAppMaster.java), ultimately call the *Appender.close().
Root Cause:
LogManager.shutdown() is not being called from MRAppMaster.java.
Similar steps were taken correctly in YarnChild.java.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (MAPREDUCE-3310) Custom grouping comparator cannot be set for Combiners
[
https://issues.apache.org/jira/browse/MAPREDUCE-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated MAPREDUCE-3310:
--
Attachment: MAPREDUCE-3310-branch-1.patch
Custom grouping comparator cannot be set for Combiners
--
Key: MAPREDUCE-3310
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3310
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client
Affects Versions: 0.20.1
Environment: All
Reporter: Mathias Herberts
Assignee: Alejandro Abdelnur
Attachments: MAPREDUCE-3310-branch-1.patch,
MAPREDUCE-3310-branch-1.patch, MAPREDUCE-3310-trunk.patch,
MAPREDUCE-3310-trunk.patch, MAPREDUCE-3310-trunk.patch,
MAPREDUCE-3310-trunk.patch
Combiners are often described as 'Reducers running on the Map side'.
As Reducers, Combiners are fed K,{V}, where {V} is built by grouping values
associated with the 'same' key.
For Reducers, the comparator used for grouping values can be set
independently of that used to sort the keys (using
Job.setGroupingComparatorClass).
Such a configuration is not possible for Combiners, meaning some things done
in Reducers cannot be done in Combiners (such as secondary sort).
It would be handy to have a Job.setCombinerGroupingComparatorClass method
that would allow the setting of the grouping comparator used when applying a
Combiner.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (MAPREDUCE-3310) Custom grouping comparator cannot be set for Combiners
[
https://issues.apache.org/jira/browse/MAPREDUCE-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated MAPREDUCE-3310:
--
Attachment: MAPREDUCE-3310-trunk.patch
new patches with the suggested method names changes.
Custom grouping comparator cannot be set for Combiners
--
Key: MAPREDUCE-3310
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3310
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client
Affects Versions: 0.20.1
Environment: All
Reporter: Mathias Herberts
Assignee: Alejandro Abdelnur
Attachments: MAPREDUCE-3310-branch-1.patch,
MAPREDUCE-3310-branch-1.patch, MAPREDUCE-3310-trunk.patch,
MAPREDUCE-3310-trunk.patch, MAPREDUCE-3310-trunk.patch,
MAPREDUCE-3310-trunk.patch
Combiners are often described as 'Reducers running on the Map side'.
As Reducers, Combiners are fed K,{V}, where {V} is built by grouping values
associated with the 'same' key.
For Reducers, the comparator used for grouping values can be set
independently of that used to sort the keys (using
Job.setGroupingComparatorClass).
Such a configuration is not possible for Combiners, meaning some things done
in Reducers cannot be done in Combiners (such as secondary sort).
It would be handy to have a Job.setCombinerGroupingComparatorClass method
that would allow the setting of the grouping comparator used when applying a
Combiner.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (MAPREDUCE-5695) Jobclient API getCleanupTaskReports() is returning error Unrecognized task type: JOB_CLEANUP
[ https://issues.apache.org/jira/browse/MAPREDUCE-5695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5695: --- Priority: Blocker (was: Major) Jobclient API getCleanupTaskReports() is returning error Unrecognized task type: JOB_CLEANUP --- Key: MAPREDUCE-5695 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5695 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 2.3.0 Reporter: Ashutosh Jindal Priority: Blocker Exception in thread main org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unrecognized task type: JOB_CLEANUP at org.apache.hadoop.mapreduce.TypeConverter.toYarn(TypeConverter.java:113) at org.apache.hadoop.mapred.ClientServiceDelegate.getTaskReports(ClientServiceDelegate.java:413) at org.apache.hadoop.mapred.YARNRunner.getTaskReports(YARNRunner.java:549) at org.apache.hadoop.mapreduce.Job$3.run(Job.java:533) at org.apache.hadoop.mapreduce.Job$3.run(Job.java:531) -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (MAPREDUCE-3310) Custom grouping comparator cannot be set for Combiners
[
https://issues.apache.org/jira/browse/MAPREDUCE-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13856000#comment-13856000
]
Hadoop QA commented on MAPREDUCE-3310:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12620265/MAPREDUCE-3310-trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
14 warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.mapreduce.security.TestJHSSecurity
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4281//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4281//console
This message is automatically generated.
Custom grouping comparator cannot be set for Combiners
--
Key: MAPREDUCE-3310
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3310
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client
Affects Versions: 0.20.1
Environment: All
Reporter: Mathias Herberts
Assignee: Alejandro Abdelnur
Attachments: MAPREDUCE-3310-branch-1.patch,
MAPREDUCE-3310-branch-1.patch, MAPREDUCE-3310-trunk.patch,
MAPREDUCE-3310-trunk.patch, MAPREDUCE-3310-trunk.patch,
MAPREDUCE-3310-trunk.patch
Combiners are often described as 'Reducers running on the Map side'.
As Reducers, Combiners are fed K,{V}, where {V} is built by grouping values
associated with the 'same' key.
For Reducers, the comparator used for grouping values can be set
independently of that used to sort the keys (using
Job.setGroupingComparatorClass).
Such a configuration is not possible for Combiners, meaning some things done
in Reducers cannot be done in Combiners (such as secondary sort).
It would be handy to have a Job.setCombinerGroupingComparatorClass method
that would allow the setting of the grouping comparator used when applying a
Combiner.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (MAPREDUCE-3310) Custom grouping comparator cannot be set for Combiners
[
https://issues.apache.org/jira/browse/MAPREDUCE-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13856006#comment-13856006
]
Alejandro Abdelnur commented on MAPREDUCE-3310:
---
test failure seems unrelated.
Custom grouping comparator cannot be set for Combiners
--
Key: MAPREDUCE-3310
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3310
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client
Affects Versions: 0.20.1
Environment: All
Reporter: Mathias Herberts
Assignee: Alejandro Abdelnur
Attachments: MAPREDUCE-3310-branch-1.patch,
MAPREDUCE-3310-branch-1.patch, MAPREDUCE-3310-trunk.patch,
MAPREDUCE-3310-trunk.patch, MAPREDUCE-3310-trunk.patch,
MAPREDUCE-3310-trunk.patch
Combiners are often described as 'Reducers running on the Map side'.
As Reducers, Combiners are fed K,{V}, where {V} is built by grouping values
associated with the 'same' key.
For Reducers, the comparator used for grouping values can be set
independently of that used to sort the keys (using
Job.setGroupingComparatorClass).
Such a configuration is not possible for Combiners, meaning some things done
in Reducers cannot be done in Combiners (such as secondary sort).
It would be handy to have a Job.setCombinerGroupingComparatorClass method
that would allow the setting of the grouping comparator used when applying a
Combiner.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Created] (MAPREDUCE-5696) Add Localization counters to MR
Gera Shegalov created MAPREDUCE-5696: Summary: Add Localization counters to MR Key: MAPREDUCE-5696 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5696 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Gera Shegalov -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Assigned] (MAPREDUCE-5696) Add Localization counters to MR
[ https://issues.apache.org/jira/browse/MAPREDUCE-5696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gera Shegalov reassigned MAPREDUCE-5696: Assignee: Gera Shegalov Add Localization counters to MR --- Key: MAPREDUCE-5696 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5696 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Gera Shegalov Assignee: Gera Shegalov -- This message was sent by Atlassian JIRA (v6.1.5#6160)
