[jira] [Commented] (MAPREDUCE-5799) add default value of MR_AM_ADMIN_USER_ENV
[
https://issues.apache.org/jira/browse/MAPREDUCE-5799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14342044#comment-14342044
]
Allen Wittenauer commented on MAPREDUCE-5799:
-
Let's get this re-based and then we can commit it...
add default value of MR_AM_ADMIN_USER_ENV
-
Key: MAPREDUCE-5799
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5799
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 2.3.0
Reporter: Liyin Liang
Assignee: Liyin Liang
Attachments: MAPREDUCE-5799.diff
Submit a 1 map + 1 reduce sleep job with the following config:
{code}
property
namemapreduce.map.output.compress/name
valuetrue/value
/property
property
namemapreduce.map.output.compress.codec/name
valueorg.apache.hadoop.io.compress.SnappyCodec/value
/property
property
namemapreduce.job.ubertask.enable/name
valuetrue/value
/property
{code}
And the LinuxContainerExecutor is enable on NodeManager.
This job will fail with the following error:
{code}
2014-03-18 21:28:20,153 FATAL [uber-SubtaskRunner]
org.apache.hadoop.mapred.LocalContainerLauncher: Error running local
(uberized) 'child' : java.lang.UnsatisfiedLinkError:
org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native
Method)
at
org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63)
at
org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:132)
at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
at org.apache.hadoop.mapred.IFile$Writer.init(IFile.java:115)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1583)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1462)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:700)
at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:1990)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:774)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at
org.apache.hadoop.mapred.LocalContainerLauncher$SubtaskRunner.runSubtask(LocalContainerLauncher.java:317)
at
org.apache.hadoop.mapred.LocalContainerLauncher$SubtaskRunner.run(LocalContainerLauncher.java:232)
at java.lang.Thread.run(Thread.java:662)
{code}
When create a ContainerLaunchContext for task in
TaskAttemptImpl.createCommonContainerLaunchContext(), the
DEFAULT_MAPRED_ADMIN_USER_ENV which is
LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native is added to the environment.
Where when create a ContainerLaunchContext for mrappmaster in
YARNRunner.createApplicationSubmissionContext(), there is no default
environment. So the ubermode job fails to find native lib.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5653) DistCp does not honour config-overrides for mapreduce.[map,reduce].memory.mb
[ https://issues.apache.org/jira/browse/MAPREDUCE-5653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14342146#comment-14342146 ] Hudson commented on MAPREDUCE-5653: --- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #119 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/119/]) MAPREDUCE-5653. DistCp does not honour config-overrides for mapreduce.[map,reduce].memory.mb (Ratandeep Ratti via aw) (aw: rev 039366e3b430ff7d9a7ff30405a0431292069a8a) * hadoop-tools/hadoop-distcp/src/main/resources/distcp-default.xml * hadoop-mapreduce-project/CHANGES.txt DistCp does not honour config-overrides for mapreduce.[map,reduce].memory.mb Key: MAPREDUCE-5653 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5653 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp Affects Versions: trunk, 0.23.9, 2.2.0 Reporter: Mithun Radhakrishnan Assignee: Ratandeep Ratti Fix For: 3.0.0 Attachments: MAPREDUCE-5653.branch-0.23.patch, MAPREDUCE-5653.branch-2.patch, MAPREDUCE-5653.trunk.2.patch, MAPREDUCE-5653.trunk.patch When a DistCp job is run through Oozie (through a Java action that launches DistCp), one sees that mapred.child.java.opts as set from the caller is honoured by DistCp. But, DistCp doesn't seem to honour any overrides for configs mapreduce.[map,reduce].memory.mb. Problem has been identified. I'll post a patch shortly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5965) Hadoop streaming throws error if list of input files is high. Error is: error=7, Argument list too long at if number of input file is high
[
https://issues.apache.org/jira/browse/MAPREDUCE-5965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated MAPREDUCE-5965:
Status: Open (was: Patch Available)
Cancelling the patch... again.
It needs to be rebased for the current source tree.
Hadoop streaming throws error if list of input files is high. Error is:
error=7, Argument list too long at if number of input file is high
Key: MAPREDUCE-5965
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5965
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Arup Malakar
Assignee: Arup Malakar
Attachments: MAPREDUCE-5965.1.patch, MAPREDUCE-5965.patch
Hadoop streaming exposes all the key values in job conf as environment
variables when it forks a process for streaming code to run. Unfortunately
the variable mapreduce_input_fileinputformat_inputdir contains the list of
input files, and Linux has a limit on size of environment variables +
arguments.
Based on how long the list of files and their full path is this could be
pretty huge. And given all of these variables are not even used it stops user
from running hadoop job with large number of files, even though it could be
run.
Linux throws E2BIG if the size is greater than certain size which is error
code 7. And java translates that to error=7, Argument list too long. More:
http://man7.org/linux/man-pages/man2/execve.2.html I suggest skipping
variables if it is greater than certain length. That way if user code
requires the environment variable it would fail. It should also introduce a
config variable to skip long variables, and set it to false by default. That
way user has to specifically set it to true to invoke this feature.
Here is the exception:
{code}
Error: java.lang.RuntimeException: Error in configuring object at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:426) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:415) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by:
java.lang.reflect.InvocationTargetException at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
... 9 more Caused by: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38) ... 14
more Caused by: java.lang.reflect.InvocationTargetException at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
... 17 more Caused by: java.lang.RuntimeException: configuration exception at
org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:222) at
org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66) ... 22
more Caused by: java.io.IOException: Cannot run program
/data/hadoop/hadoop-yarn/cache/yarn/nm-local-dir/usercache/oo-analytics/appcache/application_1403599726264_13177/container_1403599726264_13177_01_06/./rbenv_runner.sh:
error=7, Argument list too long at
java.lang.ProcessBuilder.start(ProcessBuilder.java:1041) at
org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:209) ... 23
more Caused by: java.io.IOException: error=7, Argument list too long at
java.lang.UNIXProcess.forkAndExec(Native Method) at
java.lang.UNIXProcess.init(UNIXProcess.java:135) at
java.lang.ProcessImpl.start(ProcessImpl.java:130) at
[jira] [Updated] (MAPREDUCE-6136) MRAppMaster doesn't shutdown file systems
[ https://issues.apache.org/jira/browse/MAPREDUCE-6136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated MAPREDUCE-6136: -- Affects Version/s: 2.6.0 MRAppMaster doesn't shutdown file systems - Key: MAPREDUCE-6136 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6136 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.4.1, 2.6.0 Reporter: Noah Watkins Assignee: Brahma Reddy Battula When MRAppMaster exit it doesn't call close on its open file systems instances. MAPREDUCE-3614 sets conf.setBoolean(fs.automatic.close, false); in MRAppMaster::main and then called FileSystem.closeAll() in MRAppMasterShutdownHook. However, MAPREDUCE-4205 removed the call to FileSystem.closeAll() MRAppMasterShutdownHook but left `fs.automatic.close` set to false. Removing `conf.setBoolean(fs.automatic.close, false);` worked for me, but it wasn't clear if this had other implications. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5653) DistCp does not honour config-overrides for mapreduce.[map,reduce].memory.mb
[ https://issues.apache.org/jira/browse/MAPREDUCE-5653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14342169#comment-14342169 ] Hudson commented on MAPREDUCE-5653: --- SUCCESS: Integrated in Hadoop-Yarn-trunk #853 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/853/]) MAPREDUCE-5653. DistCp does not honour config-overrides for mapreduce.[map,reduce].memory.mb (Ratandeep Ratti via aw) (aw: rev 039366e3b430ff7d9a7ff30405a0431292069a8a) * hadoop-mapreduce-project/CHANGES.txt * hadoop-tools/hadoop-distcp/src/main/resources/distcp-default.xml DistCp does not honour config-overrides for mapreduce.[map,reduce].memory.mb Key: MAPREDUCE-5653 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5653 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp Affects Versions: trunk, 0.23.9, 2.2.0 Reporter: Mithun Radhakrishnan Assignee: Ratandeep Ratti Fix For: 3.0.0 Attachments: MAPREDUCE-5653.branch-0.23.patch, MAPREDUCE-5653.branch-2.patch, MAPREDUCE-5653.trunk.2.patch, MAPREDUCE-5653.trunk.patch When a DistCp job is run through Oozie (through a Java action that launches DistCp), one sees that mapred.child.java.opts as set from the caller is honoured by DistCp. But, DistCp doesn't seem to honour any overrides for configs mapreduce.[map,reduce].memory.mb. Problem has been identified. I'll post a patch shortly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6057) Remove obsolete entries from mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-6057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-6057: Status: Patch Available (was: Open) Remove obsolete entries from mapred-default.xml --- Key: MAPREDUCE-6057 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6057 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.5.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: newbie Attachments: MAPREDUCE-6057-01.patch, MAPREDUCE-6057-02.patch, MAPREDUCE-6057-03.patch, MAPREDUCE-6057.004.patch The following properties are defined in mapred-default.xml but no longer exist in MRJobConfig. map.sort.class mapred.child.env mapred.child.java.opts mapreduce.app-submission.cross-platform mapreduce.client.completion.pollinterval mapreduce.client.output.filter mapreduce.client.progressmonitor.pollinterval mapreduce.client.submit.file.replication mapreduce.cluster.acls.enabled mapreduce.cluster.local.dir mapreduce.framework.name mapreduce.ifile.readahead mapreduce.ifile.readahead.bytes mapreduce.input.fileinputformat.list-status.num-threads mapreduce.input.fileinputformat.split.minsize mapreduce.input.lineinputformat.linespermap mapreduce.job.counters.limit mapreduce.job.max.split.locations mapreduce.job.reduce.shuffle.consumer.plugin.class mapreduce.jobhistory.address mapreduce.jobhistory.admin.acl mapreduce.jobhistory.admin.address mapreduce.jobhistory.cleaner.enable mapreduce.jobhistory.cleaner.interval-ms mapreduce.jobhistory.client.thread-count mapreduce.jobhistory.datestring.cache.size mapreduce.jobhistory.done-dir mapreduce.jobhistory.http.policy mapreduce.jobhistory.intermediate-done-dir mapreduce.jobhistory.joblist.cache.size mapreduce.jobhistory.keytab mapreduce.jobhistory.loadedjobs.cache.size mapreduce.jobhistory.max-age-ms mapreduce.jobhistory.minicluster.fixed.ports mapreduce.jobhistory.move.interval-ms mapreduce.jobhistory.move.thread-count mapreduce.jobhistory.principal mapreduce.jobhistory.recovery.enable mapreduce.jobhistory.recovery.store.class mapreduce.jobhistory.recovery.store.fs.uri mapreduce.jobhistory.store.class mapreduce.jobhistory.webapp.address mapreduce.local.clientfactory.class.name mapreduce.map.skip.proc.count.autoincr mapreduce.output.fileoutputformat.compress mapreduce.output.fileoutputformat.compress.codec mapreduce.output.fileoutputformat.compress.type mapreduce.reduce.skip.proc.count.autoincr mapreduce.shuffle.connection-keep-alive.enable mapreduce.shuffle.connection-keep-alive.timeout mapreduce.shuffle.max.connections mapreduce.shuffle.max.threads mapreduce.shuffle.port mapreduce.shuffle.ssl.enabled mapreduce.shuffle.ssl.file.buffer.size mapreduce.shuffle.transfer.buffer.size mapreduce.shuffle.transferTo.allowed yarn.app.mapreduce.client-am.ipc.max-retries-on-timeouts Submitting bug for comment/feedback about which properties should be kept in mapred-default.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6057) Remove obsolete entries from mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-6057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-6057: Status: Open (was: Patch Available) Remove obsolete entries from mapred-default.xml --- Key: MAPREDUCE-6057 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6057 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.5.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: newbie Attachments: MAPREDUCE-6057-01.patch, MAPREDUCE-6057-02.patch, MAPREDUCE-6057-03.patch, MAPREDUCE-6057.004.patch The following properties are defined in mapred-default.xml but no longer exist in MRJobConfig. map.sort.class mapred.child.env mapred.child.java.opts mapreduce.app-submission.cross-platform mapreduce.client.completion.pollinterval mapreduce.client.output.filter mapreduce.client.progressmonitor.pollinterval mapreduce.client.submit.file.replication mapreduce.cluster.acls.enabled mapreduce.cluster.local.dir mapreduce.framework.name mapreduce.ifile.readahead mapreduce.ifile.readahead.bytes mapreduce.input.fileinputformat.list-status.num-threads mapreduce.input.fileinputformat.split.minsize mapreduce.input.lineinputformat.linespermap mapreduce.job.counters.limit mapreduce.job.max.split.locations mapreduce.job.reduce.shuffle.consumer.plugin.class mapreduce.jobhistory.address mapreduce.jobhistory.admin.acl mapreduce.jobhistory.admin.address mapreduce.jobhistory.cleaner.enable mapreduce.jobhistory.cleaner.interval-ms mapreduce.jobhistory.client.thread-count mapreduce.jobhistory.datestring.cache.size mapreduce.jobhistory.done-dir mapreduce.jobhistory.http.policy mapreduce.jobhistory.intermediate-done-dir mapreduce.jobhistory.joblist.cache.size mapreduce.jobhistory.keytab mapreduce.jobhistory.loadedjobs.cache.size mapreduce.jobhistory.max-age-ms mapreduce.jobhistory.minicluster.fixed.ports mapreduce.jobhistory.move.interval-ms mapreduce.jobhistory.move.thread-count mapreduce.jobhistory.principal mapreduce.jobhistory.recovery.enable mapreduce.jobhistory.recovery.store.class mapreduce.jobhistory.recovery.store.fs.uri mapreduce.jobhistory.store.class mapreduce.jobhistory.webapp.address mapreduce.local.clientfactory.class.name mapreduce.map.skip.proc.count.autoincr mapreduce.output.fileoutputformat.compress mapreduce.output.fileoutputformat.compress.codec mapreduce.output.fileoutputformat.compress.type mapreduce.reduce.skip.proc.count.autoincr mapreduce.shuffle.connection-keep-alive.enable mapreduce.shuffle.connection-keep-alive.timeout mapreduce.shuffle.max.connections mapreduce.shuffle.max.threads mapreduce.shuffle.port mapreduce.shuffle.ssl.enabled mapreduce.shuffle.ssl.file.buffer.size mapreduce.shuffle.transfer.buffer.size mapreduce.shuffle.transferTo.allowed yarn.app.mapreduce.client-am.ipc.max-retries-on-timeouts Submitting bug for comment/feedback about which properties should be kept in mapred-default.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6136) MRAppMaster doesn't shutdown file systems
[ https://issues.apache.org/jira/browse/MAPREDUCE-6136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated MAPREDUCE-6136: Attachment: MAPRED-6136.patch MRAppMaster doesn't shutdown file systems - Key: MAPREDUCE-6136 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6136 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.4.1, 2.6.0 Reporter: Noah Watkins Assignee: Brahma Reddy Battula Attachments: MAPRED-6136.patch When MRAppMaster exit it doesn't call close on its open file systems instances. MAPREDUCE-3614 sets conf.setBoolean(fs.automatic.close, false); in MRAppMaster::main and then called FileSystem.closeAll() in MRAppMasterShutdownHook. However, MAPREDUCE-4205 removed the call to FileSystem.closeAll() MRAppMasterShutdownHook but left `fs.automatic.close` set to false. Removing `conf.setBoolean(fs.automatic.close, false);` worked for me, but it wasn't clear if this had other implications. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (MAPREDUCE-6136) MRAppMaster doesn't shutdown file systems
[ https://issues.apache.org/jira/browse/MAPREDUCE-6136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on MAPREDUCE-6136 started by Brahma Reddy Battula. --- MRAppMaster doesn't shutdown file systems - Key: MAPREDUCE-6136 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6136 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.4.1, 2.6.0 Reporter: Noah Watkins Assignee: Brahma Reddy Battula Attachments: MAPRED-6136.patch When MRAppMaster exit it doesn't call close on its open file systems instances. MAPREDUCE-3614 sets conf.setBoolean(fs.automatic.close, false); in MRAppMaster::main and then called FileSystem.closeAll() in MRAppMasterShutdownHook. However, MAPREDUCE-4205 removed the call to FileSystem.closeAll() MRAppMasterShutdownHook but left `fs.automatic.close` set to false. Removing `conf.setBoolean(fs.automatic.close, false);` worked for me, but it wasn't clear if this had other implications. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5653) DistCp does not honour config-overrides for mapreduce.[map,reduce].memory.mb
[ https://issues.apache.org/jira/browse/MAPREDUCE-5653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14342243#comment-14342243 ] Hudson commented on MAPREDUCE-5653: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk #2051 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2051/]) MAPREDUCE-5653. DistCp does not honour config-overrides for mapreduce.[map,reduce].memory.mb (Ratandeep Ratti via aw) (aw: rev 039366e3b430ff7d9a7ff30405a0431292069a8a) * hadoop-mapreduce-project/CHANGES.txt * hadoop-tools/hadoop-distcp/src/main/resources/distcp-default.xml DistCp does not honour config-overrides for mapreduce.[map,reduce].memory.mb Key: MAPREDUCE-5653 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5653 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp Affects Versions: trunk, 0.23.9, 2.2.0 Reporter: Mithun Radhakrishnan Assignee: Ratandeep Ratti Fix For: 3.0.0 Attachments: MAPREDUCE-5653.branch-0.23.patch, MAPREDUCE-5653.branch-2.patch, MAPREDUCE-5653.trunk.2.patch, MAPREDUCE-5653.trunk.patch When a DistCp job is run through Oozie (through a Java action that launches DistCp), one sees that mapred.child.java.opts as set from the caller is honoured by DistCp. But, DistCp doesn't seem to honour any overrides for configs mapreduce.[map,reduce].memory.mb. Problem has been identified. I'll post a patch shortly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5653) DistCp does not honour config-overrides for mapreduce.[map,reduce].memory.mb
[ https://issues.apache.org/jira/browse/MAPREDUCE-5653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14342251#comment-14342251 ] Hudson commented on MAPREDUCE-5653: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #110 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/110/]) MAPREDUCE-5653. DistCp does not honour config-overrides for mapreduce.[map,reduce].memory.mb (Ratandeep Ratti via aw) (aw: rev 039366e3b430ff7d9a7ff30405a0431292069a8a) * hadoop-mapreduce-project/CHANGES.txt * hadoop-tools/hadoop-distcp/src/main/resources/distcp-default.xml DistCp does not honour config-overrides for mapreduce.[map,reduce].memory.mb Key: MAPREDUCE-5653 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5653 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp Affects Versions: trunk, 0.23.9, 2.2.0 Reporter: Mithun Radhakrishnan Assignee: Ratandeep Ratti Fix For: 3.0.0 Attachments: MAPREDUCE-5653.branch-0.23.patch, MAPREDUCE-5653.branch-2.patch, MAPREDUCE-5653.trunk.2.patch, MAPREDUCE-5653.trunk.patch When a DistCp job is run through Oozie (through a Java action that launches DistCp), one sees that mapred.child.java.opts as set from the caller is honoured by DistCp. But, DistCp doesn't seem to honour any overrides for configs mapreduce.[map,reduce].memory.mb. Problem has been identified. I'll post a patch shortly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5657) [JDK8] Fix Javadoc errors caused by incorrect or illegal tags in doc comments
[ https://issues.apache.org/jira/browse/MAPREDUCE-5657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated MAPREDUCE-5657: - Attachment: MAPREDUCE-5657-003.patch Rebased. [JDK8] Fix Javadoc errors caused by incorrect or illegal tags in doc comments - Key: MAPREDUCE-5657 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5657 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.7.0 Reporter: Andrew Purtell Assignee: Akira AJISAKA Priority: Minor Attachments: 5657-branch-2.patch, 5657-branch-2.patch, 5657-trunk.patch, 5657-trunk.patch, MAPREDUCE-5657-002.patch, MAPREDUCE-5657-003.patch, MAPREDUCE-5657-wip3.patch, build.log Javadoc is more strict by default in JDK8 and will error out on malformed or illegal tags found in doc comments. Although tagged as JDK8 all of the required changes are generic Javadoc cleanups. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5657) [JDK8] Fix Javadoc errors caused by incorrect or illegal tags in doc comments
[ https://issues.apache.org/jira/browse/MAPREDUCE-5657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14342802#comment-14342802 ] Akira AJISAKA commented on MAPREDUCE-5657: -- bq. Is this related? Looks unrelated since the patch is just to fix documents. [JDK8] Fix Javadoc errors caused by incorrect or illegal tags in doc comments - Key: MAPREDUCE-5657 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5657 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation Affects Versions: 2.7.0 Reporter: Andrew Purtell Assignee: Akira AJISAKA Priority: Minor Attachments: 5657-branch-2.patch, 5657-branch-2.patch, 5657-trunk.patch, 5657-trunk.patch, MAPREDUCE-5657-002.patch, MAPREDUCE-5657-003.patch, MAPREDUCE-5657-wip3.patch, build.log Javadoc is more strict by default in JDK8 and will error out on malformed or illegal tags found in doc comments. Although tagged as JDK8 all of the required changes are generic Javadoc cleanups. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5657) [JDK8] Fix Javadoc errors caused by incorrect or illegal tags in doc comments
[ https://issues.apache.org/jira/browse/MAPREDUCE-5657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated MAPREDUCE-5657: - Component/s: documentation [JDK8] Fix Javadoc errors caused by incorrect or illegal tags in doc comments - Key: MAPREDUCE-5657 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5657 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation Affects Versions: 2.7.0 Reporter: Andrew Purtell Assignee: Akira AJISAKA Priority: Minor Attachments: 5657-branch-2.patch, 5657-branch-2.patch, 5657-trunk.patch, 5657-trunk.patch, MAPREDUCE-5657-002.patch, MAPREDUCE-5657-003.patch, MAPREDUCE-5657-wip3.patch, build.log Javadoc is more strict by default in JDK8 and will error out on malformed or illegal tags found in doc comments. Although tagged as JDK8 all of the required changes are generic Javadoc cleanups. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6190) MR Job is stuck because of one mapper stuck in STARTING
[
https://issues.apache.org/jira/browse/MAPREDUCE-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14342806#comment-14342806
]
Rohith commented on MAPREDUCE-6190:
---
[~amalhotra159] [~tannin]
Would mind attaching full log of MRAppMaster. And did anyone observe scenario
mentioned in YARN-1680?
MR Job is stuck because of one mapper stuck in STARTING
---
Key: MAPREDUCE-6190
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6190
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Ankit Malhotra
Trying to figure out a weird issue we started seeing on our CDH5.1.0 cluster
with map reduce jobs on YARN.
We had a job stuck for hours because one of the mappers never started up
fully. Basically, the map task had 2 attempts, the first one failed and the
AM tried to schedule a second one and the second attempt was stuck on STATE:
STARTING, STATUS: NEW. A node never got assigned and the task along with the
job was stuck indefinitely.
The AM logs had this being logged again and again:
{code}
2014-12-09 19:25:12,347 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Ramping down 0
2014-12-09 19:25:13,352 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received
completed container container_1408745633994_450952_02_003807
2014-12-09 19:25:13,352 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce preemption
successful attempt_1408745633994_450952_r_48_1000
2014-12-09 19:25:13,352 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Ramping down all
scheduled reduces:0
2014-12-09 19:25:13,352 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to preempt 1
2014-12-09 19:25:13,353 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Preempting
attempt_1408745633994_450952_r_50_1000
2014-12-09 19:25:13,353 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating
schedule, headroom=0
2014-12-09 19:25:13,353 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
completedMapPercent 0.99968 totalMemLimit:1722880 finalMapMemLimit:2560
finalReduceMemLimit:1720320 netScheduledMapMem:2560
netScheduledReduceMem:1722880
2014-12-09 19:25:13,353 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Ramping down 0
2014-12-09 19:25:13,353 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling:
PendingReds:77 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:0
AssignedReds:673 CompletedMaps:3124 CompletedReds:0 ContAlloc:4789
ContRel:798 HostLocal:2944 RackLocal:155
2014-12-09 19:25:14,353 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before
Scheduling: PendingReds:78 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:0
AssignedReds:673 CompletedMaps:3124 CompletedReds:0 ContAlloc:4789
ContRel:798 HostLocal:2944 RackLocal:155
2014-12-09 19:25:14,359 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating
schedule, headroom=0
{code}
On killing the task manually, the AM started up the task again, scheduled and
ran it successfully completing the task and the job with it.
Some quick code grepping led us here:
http://grepcode.com/file/repo1.maven.org/maven2/org.apache.hadoop/hadoop-mapreduce-client-app/2.3.0/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java#397
But still dont quite understand why this would happen once in a while and why
the job would suddenly be ok once the stuck task is manually killed.
Note: Other jobs succeed on the cluster while this job is stuck.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6096) SummarizedJob class NPEs with some jhist files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangyubiao updated MAPREDUCE-6096: --- Attachment: MAPREDUCE-6096-v6.patch SummarizedJob class NPEs with some jhist files -- Key: MAPREDUCE-6096 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6096 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: trunk Reporter: zhangyubiao Labels: easyfix, patch Attachments: MAPREDUCE-6096-v2.patch, MAPREDUCE-6096-v3.patch, MAPREDUCE-6096-v4.patch, MAPREDUCE-6096-v5.patch, MAPREDUCE-6096-v6.patch, MAPREDUCE-6096.patch, job_1410427642147_0124-1411726671220-hadp-word+count-1411726696863-1-1-SUCCEEDED-default.jhist When I Parse the JobHistory in the HistoryFile,I use the Hadoop System's map-reduce-client-core project org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser class and HistoryViewer$SummarizedJob to Parse the JobHistoryFile(Just Like job_1408862281971_489761-1410883171851_XXX.jhist) and it throw an Exception Just Like Exception in thread pool-1-thread-1 java.lang.NullPointerException at org.apache.hadoop.mapreduce.jobhistory.HistoryViewer$SummarizedJob.init(HistoryViewer.java:626) at com.jd.hadoop.log.parse.ParseLogService.getJobDetail(ParseLogService.java:70) After I'm see the SummarizedJob class I find that attempt.getTaskStatus() is NULL ， So I change the order of attempt.getTaskStatus().equals (TaskStatus.State.FAILED.toString()) to TaskStatus.State.FAILED.toString().equals(attempt.getTaskStatus()) and it works well . So I wonder If we can change all attempt.getTaskStatus() after TaskStatus.State.XXX.toString() ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6096) SummarizedJob class NPEs with some jhist files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14342815#comment-14342815 ] zhangyubiao commented on MAPREDUCE-6096: MAPREDUCE-6096-v6.patch is commit SummarizedJob class NPEs with some jhist files -- Key: MAPREDUCE-6096 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6096 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: trunk Reporter: zhangyubiao Labels: easyfix, patch Attachments: MAPREDUCE-6096-v2.patch, MAPREDUCE-6096-v3.patch, MAPREDUCE-6096-v4.patch, MAPREDUCE-6096-v5.patch, MAPREDUCE-6096-v6.patch, MAPREDUCE-6096.patch, job_1410427642147_0124-1411726671220-hadp-word+count-1411726696863-1-1-SUCCEEDED-default.jhist When I Parse the JobHistory in the HistoryFile,I use the Hadoop System's map-reduce-client-core project org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser class and HistoryViewer$SummarizedJob to Parse the JobHistoryFile(Just Like job_1408862281971_489761-1410883171851_XXX.jhist) and it throw an Exception Just Like Exception in thread pool-1-thread-1 java.lang.NullPointerException at org.apache.hadoop.mapreduce.jobhistory.HistoryViewer$SummarizedJob.init(HistoryViewer.java:626) at com.jd.hadoop.log.parse.ParseLogService.getJobDetail(ParseLogService.java:70) After I'm see the SummarizedJob class I find that attempt.getTaskStatus() is NULL ， So I change the order of attempt.getTaskStatus().equals (TaskStatus.State.FAILED.toString()) to TaskStatus.State.FAILED.toString().equals(attempt.getTaskStatus()) and it works well . So I wonder If we can change all attempt.getTaskStatus() after TaskStatus.State.XXX.toString() ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6096) SummarizedJob class NPEs with some jhist files
[
https://issues.apache.org/jira/browse/MAPREDUCE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14342818#comment-14342818
]
Hadoop QA commented on MAPREDUCE-6096:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12701801/MAPREDUCE-6096-v6.patch
against trunk revision dd9cd07.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5234//console
This message is automatically generated.
SummarizedJob class NPEs with some jhist files
--
Key: MAPREDUCE-6096
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6096
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver
Affects Versions: trunk
Reporter: zhangyubiao
Labels: easyfix, patch
Attachments: MAPREDUCE-6096-v2.patch, MAPREDUCE-6096-v3.patch,
MAPREDUCE-6096-v4.patch, MAPREDUCE-6096-v5.patch, MAPREDUCE-6096-v6.patch,
MAPREDUCE-6096.patch,
job_1410427642147_0124-1411726671220-hadp-word+count-1411726696863-1-1-SUCCEEDED-default.jhist
When I Parse the JobHistory in the HistoryFile,I use the Hadoop System's
map-reduce-client-core project
org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser class and
HistoryViewer$SummarizedJob to Parse the JobHistoryFile(Just Like
job_1408862281971_489761-1410883171851_XXX.jhist)
and it throw an Exception Just Like
Exception in thread pool-1-thread-1 java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.jobhistory.HistoryViewer$SummarizedJob.init(HistoryViewer.java:626)
at
com.jd.hadoop.log.parse.ParseLogService.getJobDetail(ParseLogService.java:70)
After I'm see the SummarizedJob class I find that attempt.getTaskStatus() is
NULL ， So I change the order of
attempt.getTaskStatus().equals (TaskStatus.State.FAILED.toString()) to
TaskStatus.State.FAILED.toString().equals(attempt.getTaskStatus())
and it works well .
So I wonder If we can change all attempt.getTaskStatus() after
TaskStatus.State.XXX.toString() ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6096) SummarizedJob class NPEs with some jhist files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangyubiao updated MAPREDUCE-6096: --- Attachment: MAPREDUCE-6096-v7.patch SummarizedJob class NPEs with some jhist files -- Key: MAPREDUCE-6096 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6096 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: trunk Reporter: zhangyubiao Labels: easyfix, patch Attachments: MAPREDUCE-6096-v2.patch, MAPREDUCE-6096-v3.patch, MAPREDUCE-6096-v4.patch, MAPREDUCE-6096-v5.patch, MAPREDUCE-6096-v6.patch, MAPREDUCE-6096-v7.patch, MAPREDUCE-6096.patch, job_1410427642147_0124-1411726671220-hadp-word+count-1411726696863-1-1-SUCCEEDED-default.jhist When I Parse the JobHistory in the HistoryFile,I use the Hadoop System's map-reduce-client-core project org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser class and HistoryViewer$SummarizedJob to Parse the JobHistoryFile(Just Like job_1408862281971_489761-1410883171851_XXX.jhist) and it throw an Exception Just Like Exception in thread pool-1-thread-1 java.lang.NullPointerException at org.apache.hadoop.mapreduce.jobhistory.HistoryViewer$SummarizedJob.init(HistoryViewer.java:626) at com.jd.hadoop.log.parse.ParseLogService.getJobDetail(ParseLogService.java:70) After I'm see the SummarizedJob class I find that attempt.getTaskStatus() is NULL ， So I change the order of attempt.getTaskStatus().equals (TaskStatus.State.FAILED.toString()) to TaskStatus.State.FAILED.toString().equals(attempt.getTaskStatus()) and it works well . So I wonder If we can change all attempt.getTaskStatus() after TaskStatus.State.XXX.toString() ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6096) SummarizedJob class NPEs with some jhist files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14342826#comment-14342826 ] zhangyubiao commented on MAPREDUCE-6096: MAPREDUCE-6096-v7.patch is commit SummarizedJob class NPEs with some jhist files -- Key: MAPREDUCE-6096 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6096 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: trunk Reporter: zhangyubiao Labels: easyfix, patch Attachments: MAPREDUCE-6096-v2.patch, MAPREDUCE-6096-v3.patch, MAPREDUCE-6096-v4.patch, MAPREDUCE-6096-v5.patch, MAPREDUCE-6096-v6.patch, MAPREDUCE-6096-v7.patch, MAPREDUCE-6096.patch, job_1410427642147_0124-1411726671220-hadp-word+count-1411726696863-1-1-SUCCEEDED-default.jhist When I Parse the JobHistory in the HistoryFile,I use the Hadoop System's map-reduce-client-core project org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser class and HistoryViewer$SummarizedJob to Parse the JobHistoryFile(Just Like job_1408862281971_489761-1410883171851_XXX.jhist) and it throw an Exception Just Like Exception in thread pool-1-thread-1 java.lang.NullPointerException at org.apache.hadoop.mapreduce.jobhistory.HistoryViewer$SummarizedJob.init(HistoryViewer.java:626) at com.jd.hadoop.log.parse.ParseLogService.getJobDetail(ParseLogService.java:70) After I'm see the SummarizedJob class I find that attempt.getTaskStatus() is NULL ， So I change the order of attempt.getTaskStatus().equals (TaskStatus.State.FAILED.toString()) to TaskStatus.State.FAILED.toString().equals(attempt.getTaskStatus()) and it works well . So I wonder If we can change all attempt.getTaskStatus() after TaskStatus.State.XXX.toString() ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6223) TestJobConf#testNegativeValueForTaskVmem failures
[
https://issues.apache.org/jira/browse/MAPREDUCE-6223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14342827#comment-14342827
]
Varun Saxena commented on MAPREDUCE-6223:
-
Thanks [~kasha]/[~ozawa] for the review and commit
TestJobConf#testNegativeValueForTaskVmem failures
-
Key: MAPREDUCE-6223
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6223
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 3.0.0
Reporter: Gera Shegalov
Assignee: Varun Saxena
Fix For: 3.0.0
Attachments: MAPREDUCE-6223.001.patch, MAPREDUCE-6223.002.patch,
MAPREDUCE-6223.003.patch, MAPREDUCE-6223.004.patch, MAPREDUCE-6223.005.patch,
MAPREDUCE-6223.006.patch
{code}
Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 3.328 sec
FAILURE! - in org.apache.hadoop.conf.TestJobConf
testNegativeValueForTaskVmem(org.apache.hadoop.conf.TestJobConf) Time
elapsed: 0.089 sec FAILURE!
java.lang.AssertionError: expected:1024 but was:-1
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.conf.TestJobConf.testNegativeValueForTaskVmem(TestJobConf.java:111)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5657) [JDK8] Fix Javadoc errors caused by incorrect or illegal tags in doc comments
[
https://issues.apache.org/jira/browse/MAPREDUCE-5657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14342866#comment-14342866
]
Hadoop QA commented on MAPREDUCE-5657:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12701797/MAPREDUCE-5657-003.patch
against trunk revision 30e73eb.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-distcp
hadoop-tools/hadoop-extras hadoop-tools/hadoop-gridmix
hadoop-tools/hadoop-openstack hadoop-tools/hadoop-rumen.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5233//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5233//console
This message is automatically generated.
[JDK8] Fix Javadoc errors caused by incorrect or illegal tags in doc comments
-
Key: MAPREDUCE-5657
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5657
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: documentation
Affects Versions: 2.7.0
Reporter: Andrew Purtell
Assignee: Akira AJISAKA
Priority: Minor
Attachments: 5657-branch-2.patch, 5657-branch-2.patch,
5657-trunk.patch, 5657-trunk.patch, MAPREDUCE-5657-002.patch,
MAPREDUCE-5657-003.patch, MAPREDUCE-5657-wip3.patch, build.log
Javadoc is more strict by default in JDK8 and will error out on malformed or
illegal tags found in doc comments. Although tagged as JDK8 all of the
required changes are generic Javadoc cleanups.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5653) DistCp does not honour config-overrides for mapreduce.[map,reduce].memory.mb
[ https://issues.apache.org/jira/browse/MAPREDUCE-5653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14342283#comment-14342283 ] Hudson commented on MAPREDUCE-5653: --- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2069 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2069/]) MAPREDUCE-5653. DistCp does not honour config-overrides for mapreduce.[map,reduce].memory.mb (Ratandeep Ratti via aw) (aw: rev 039366e3b430ff7d9a7ff30405a0431292069a8a) * hadoop-tools/hadoop-distcp/src/main/resources/distcp-default.xml * hadoop-mapreduce-project/CHANGES.txt DistCp does not honour config-overrides for mapreduce.[map,reduce].memory.mb Key: MAPREDUCE-5653 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5653 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp Affects Versions: trunk, 0.23.9, 2.2.0 Reporter: Mithun Radhakrishnan Assignee: Ratandeep Ratti Fix For: 3.0.0 Attachments: MAPREDUCE-5653.branch-0.23.patch, MAPREDUCE-5653.branch-2.patch, MAPREDUCE-5653.trunk.2.patch, MAPREDUCE-5653.trunk.patch When a DistCp job is run through Oozie (through a Java action that launches DistCp), one sees that mapred.child.java.opts as set from the caller is honoured by DistCp. But, DistCp doesn't seem to honour any overrides for configs mapreduce.[map,reduce].memory.mb. Problem has been identified. I'll post a patch shortly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6136) MRAppMaster doesn't shutdown file systems
[ https://issues.apache.org/jira/browse/MAPREDUCE-6136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14342299#comment-14342299 ] Brahma Reddy Battula commented on MAPREDUCE-6136: - Thanks Tsuyoshing for taking a look into this issue...Removing `conf.setBoolean(fs.automatic.close, false), worked for me also...Let me anythoughts on this.. MRAppMaster doesn't shutdown file systems - Key: MAPREDUCE-6136 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6136 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.4.1, 2.6.0 Reporter: Noah Watkins Assignee: Brahma Reddy Battula Attachments: MAPRED-6136.patch When MRAppMaster exit it doesn't call close on its open file systems instances. MAPREDUCE-3614 sets conf.setBoolean(fs.automatic.close, false); in MRAppMaster::main and then called FileSystem.closeAll() in MRAppMasterShutdownHook. However, MAPREDUCE-4205 removed the call to FileSystem.closeAll() MRAppMasterShutdownHook but left `fs.automatic.close` set to false. Removing `conf.setBoolean(fs.automatic.close, false);` worked for me, but it wasn't clear if this had other implications. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6136) MRAppMaster doesn't shutdown file systems
[ https://issues.apache.org/jira/browse/MAPREDUCE-6136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14342282#comment-14342282 ] Tsuyoshi Ozawa commented on MAPREDUCE-6136: --- [~brahmareddy] please submit your patch. MRAppMaster doesn't shutdown file systems - Key: MAPREDUCE-6136 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6136 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.4.1, 2.6.0 Reporter: Noah Watkins Assignee: Brahma Reddy Battula Attachments: MAPRED-6136.patch When MRAppMaster exit it doesn't call close on its open file systems instances. MAPREDUCE-3614 sets conf.setBoolean(fs.automatic.close, false); in MRAppMaster::main and then called FileSystem.closeAll() in MRAppMasterShutdownHook. However, MAPREDUCE-4205 removed the call to FileSystem.closeAll() MRAppMasterShutdownHook but left `fs.automatic.close` set to false. Removing `conf.setBoolean(fs.automatic.close, false);` worked for me, but it wasn't clear if this had other implications. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6136) MRAppMaster doesn't shutdown file systems
[ https://issues.apache.org/jira/browse/MAPREDUCE-6136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated MAPREDUCE-6136: Status: Patch Available (was: In Progress) MRAppMaster doesn't shutdown file systems - Key: MAPREDUCE-6136 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6136 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.6.0, 2.4.1 Reporter: Noah Watkins Assignee: Brahma Reddy Battula Attachments: MAPRED-6136.patch When MRAppMaster exit it doesn't call close on its open file systems instances. MAPREDUCE-3614 sets conf.setBoolean(fs.automatic.close, false); in MRAppMaster::main and then called FileSystem.closeAll() in MRAppMasterShutdownHook. However, MAPREDUCE-4205 removed the call to FileSystem.closeAll() MRAppMasterShutdownHook but left `fs.automatic.close` set to false. Removing `conf.setBoolean(fs.automatic.close, false);` worked for me, but it wasn't clear if this had other implications. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5653) DistCp does not honour config-overrides for mapreduce.[map,reduce].memory.mb
[ https://issues.apache.org/jira/browse/MAPREDUCE-5653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14342270#comment-14342270 ] Hudson commented on MAPREDUCE-5653: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #119 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/119/]) MAPREDUCE-5653. DistCp does not honour config-overrides for mapreduce.[map,reduce].memory.mb (Ratandeep Ratti via aw) (aw: rev 039366e3b430ff7d9a7ff30405a0431292069a8a) * hadoop-tools/hadoop-distcp/src/main/resources/distcp-default.xml * hadoop-mapreduce-project/CHANGES.txt DistCp does not honour config-overrides for mapreduce.[map,reduce].memory.mb Key: MAPREDUCE-5653 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5653 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp Affects Versions: trunk, 0.23.9, 2.2.0 Reporter: Mithun Radhakrishnan Assignee: Ratandeep Ratti Fix For: 3.0.0 Attachments: MAPREDUCE-5653.branch-0.23.patch, MAPREDUCE-5653.branch-2.patch, MAPREDUCE-5653.trunk.2.patch, MAPREDUCE-5653.trunk.patch When a DistCp job is run through Oozie (through a Java action that launches DistCp), one sees that mapred.child.java.opts as set from the caller is honoured by DistCp. But, DistCp doesn't seem to honour any overrides for configs mapreduce.[map,reduce].memory.mb. Problem has been identified. I'll post a patch shortly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6136) MRAppMaster doesn't shutdown file systems
[ https://issues.apache.org/jira/browse/MAPREDUCE-6136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14342346#comment-14342346 ] Tsuyoshi Ozawa commented on MAPREDUCE-6136: --- LGTM. Waiting for committing this for 3 days. MRAppMaster doesn't shutdown file systems - Key: MAPREDUCE-6136 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6136 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.4.1, 2.6.0 Reporter: Noah Watkins Assignee: Brahma Reddy Battula Attachments: MAPRED-6136.patch When MRAppMaster exit it doesn't call close on its open file systems instances. MAPREDUCE-3614 sets conf.setBoolean(fs.automatic.close, false); in MRAppMaster::main and then called FileSystem.closeAll() in MRAppMasterShutdownHook. However, MAPREDUCE-4205 removed the call to FileSystem.closeAll() MRAppMasterShutdownHook but left `fs.automatic.close` set to false. Removing `conf.setBoolean(fs.automatic.close, false);` worked for me, but it wasn't clear if this had other implications. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5657) [JDK8] Fix Javadoc errors caused by incorrect or illegal tags in doc comments
[
https://issues.apache.org/jira/browse/MAPREDUCE-5657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14342348#comment-14342348
]
Tsuyoshi Ozawa commented on MAPREDUCE-5657:
---
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-enforcer-plugin:1.3.1:enforce (depcheck) on
project hadoop-rumen: Some Enforcer rules have failed. Look above for specific
messages explaining why the rule failed. - [Help 1]
{code}
Is this related?
[JDK8] Fix Javadoc errors caused by incorrect or illegal tags in doc comments
-
Key: MAPREDUCE-5657
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5657
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Andrew Purtell
Assignee: Akira AJISAKA
Priority: Minor
Attachments: 5657-branch-2.patch, 5657-branch-2.patch,
5657-trunk.patch, 5657-trunk.patch, MAPREDUCE-5657-002.patch,
MAPREDUCE-5657-wip3.patch, build.log
Javadoc is more strict by default in JDK8 and will error out on malformed or
illegal tags found in doc comments. Although tagged as JDK8 all of the
required changes are generic Javadoc cleanups.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
