[jira] [Commented] (MAPREDUCE-6791) remove unnecessary dependency from hadoop-mapreduce-client-jobclient to hadoop-mapreduce-client-shuffle
[
https://issues.apache.org/jira/browse/MAPREDUCE-6791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15584000#comment-15584000
]
Hadoop QA commented on MAPREDUCE-6791:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
2s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 33s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 109m 40s
{color} | {color:green} hadoop-mapreduce-client-jobclient in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 122m 29s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832712/mapreduce6791.001.patch
|
| JIRA Issue | MAPREDUCE-6791 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml |
| uname | Linux 46facad532cf 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / f5d9235 |
| Default Java | 1.8.0_101 |
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6763/testReport/ |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
|
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6763/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> remove unnecessary dependency from hadoop-mapreduce-client-jobclient to
> hadoop-mapreduce-client-shuffle
> ---
>
> Key: MAPREDUCE-6791
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6791
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 3.0.0-alpha1
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Minor
> Fix For: 3.0.0-alpha2
>
> Attachments: mapreduce6791.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To
[jira] [Commented] (MAPREDUCE-6536) hadoop-pipes doesn't use maven properties for openssl
[
https://issues.apache.org/jira/browse/MAPREDUCE-6536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583975#comment-15583975
]
Hadoop QA commented on MAPREDUCE-6536:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s {color}
| {color:red} MAPREDUCE-6536 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12781071/MAPREDUCE-6536.04.patch
|
| JIRA Issue | MAPREDUCE-6536 |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6764/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> hadoop-pipes doesn't use maven properties for openssl
> -
>
> Key: MAPREDUCE-6536
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6536
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: pipes
>Affects Versions: 3.0.0-alpha1
> Environment: OS X
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
>Priority: Blocker
> Attachments: HADOOP-12518.00.patch, HADOOP-12518.01.patch,
> HADOOP-12518.02.patch, HADOOP-12518.03.patch, MAPREDUCE-6536.04.patch
>
>
> hadoop-common has some maven properties that are used to define where OpenSSL
> lives. hadoop-pipes should also use them so we can enable automated testing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6536) hadoop-pipes doesn't use maven properties for openssl
[ https://issues.apache.org/jira/browse/MAPREDUCE-6536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583966#comment-15583966 ] Andrew Wang commented on MAPREDUCE-6536: Ping on this JIRA. Looks like it's pretty close, should we target for alpha2? > hadoop-pipes doesn't use maven properties for openssl > - > > Key: MAPREDUCE-6536 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6536 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Affects Versions: 3.0.0-alpha1 > Environment: OS X >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > Attachments: HADOOP-12518.00.patch, HADOOP-12518.01.patch, > HADOOP-12518.02.patch, HADOOP-12518.03.patch, MAPREDUCE-6536.04.patch > > > hadoop-common has some maven properties that are used to define where OpenSSL > lives. hadoop-pipes should also use them so we can enable automated testing. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-5506) Hadoop-1.1.1 occurs ArrayIndexOutOfBoundsException with MultithreadedMapRunner
[ https://issues.apache.org/jira/browse/MAPREDUCE-5506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved MAPREDUCE-5506. Resolution: Won't Fix Resolving as WONTFIX since mr1 has been removed. > Hadoop-1.1.1 occurs ArrayIndexOutOfBoundsException with MultithreadedMapRunner > -- > > Key: MAPREDUCE-5506 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5506 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1 >Affects Versions: 1.1.1 > Environment: RHEL 6.3 x86_64 >Reporter: sam liu >Priority: Blocker > > After I set: > - 'jobConf.setMapRunnerClass(MultithreadedMapRunner.class);' in MR app > - 'mapred.map.multithreadedrunner.threads = 2' in mapred-site.xml > A simple MR app failed as its Map task encountered > ArrayIndexOutOfBoundsException as below(please ignore the line numbers in the > exception as I added some log print codes): > java.lang.ArrayIndexOutOfBoundsException > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer$Buffer.write(MapTask.java:1331) > at java.io.DataOutputStream.write(DataOutputStream.java:101) > at org.apache.hadoop.io.Text.write(Text.java:282) > at > org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90) > at > org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1060) > at > org.apache.hadoop.mapred.MapTask$OldOutputCollector.collect(MapTask.java:591) > at study.hadoop.mapreduce.sample.WordCount$Map.map(WordCount.java:41) > at study.hadoop.mapreduce.sample.WordCount$Map.map(WordCount.java:1) > at > org.apache.hadoop.mapred.lib.MultithreadedMapRunner$MapperInvokeRunable.run(MultithreadedMapRunner.java:231) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919) > at java.lang.Thread.run(Thread.java:738) > And the exception happens on line 'System.arraycopy(b, off, kvbuffer, > bufindex, len)' in MapTask.java#MapOutputBuffer#Buffer#write(). When the > exception occurs, 'b.length=4' but 'len=9'. > Btw, if I set 'mapred.map.multithreadedrunner.threads = 1', no exception > happened. So it should be an issue caused by multiple threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6791) remove unnecessary dependency from hadoop-mapreduce-client-jobclient to hadoop-mapreduce-client-shuffle
[ https://issues.apache.org/jira/browse/MAPREDUCE-6791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-6791: - Status: Patch Available (was: Open) > remove unnecessary dependency from hadoop-mapreduce-client-jobclient to > hadoop-mapreduce-client-shuffle > --- > > Key: MAPREDUCE-6791 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6791 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 3.0.0-alpha1 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Minor > Fix For: 3.0.0-alpha2 > > Attachments: mapreduce6791.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-6375) Modify the JHS to be able to read the ConcatenatableAggregatedLogFormat
[
https://issues.apache.org/jira/browse/MAPREDUCE-6375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter resolved MAPREDUCE-6375.
--
Resolution: Won't Fix
Closing this as "Won't Fix" given we have MAPREDUCE-6415.
> Modify the JHS to be able to read the ConcatenatableAggregatedLogFormat
> ---
>
> Key: MAPREDUCE-6375
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6375
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>
> When serving logs, the JHS needs to be able to read the
> {{ConcatenatableAggregatedLogFormat}} or the {{AggregatedLogFormat}}
> transparently.
> (see YARN-2942)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6792) Allow user's full principal name as owner of MapReduce staging directory in JobSubmissionFiles#JobStagingDir()
[
https://issues.apache.org/jira/browse/MAPREDUCE-6792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated MAPREDUCE-6792:
---
Target Version/s: 2.9.0, 3.0.0-alpha2 (was: 2.9.0, 3.0.0-alpha1)
> Allow user's full principal name as owner of MapReduce staging directory in
> JobSubmissionFiles#JobStagingDir()
> --
>
> Key: MAPREDUCE-6792
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6792
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Attachments: MAPREDUCE-6792.1.patch
>
>
> Background -
> Currently, {{JobSubmissionFiles#JobStagingDir()}} assumes that file owner
> returned as part of {{FileSystem#getFileStatus()}} is always user's short
> principal name, which is true for HDFS. But, some file systems which are HDFS
> compatible like [Azure Data Lake Store (ADLS)
> |https://azure.microsoft.com/en-in/services/data-lake-store/] and work in
> multi tenant environment can have users with same names belonging to
> different domains. For example, {{us...@company1.com}} and
> {{us...@company2.com}}. It will be ambiguous, if
> {{FileSystem#getFileStatus()}} returns only the user's short principal name
> (without domain name) as the owner of the file/directory.
> The following code block allows only short user principal name as owner. It
> simply fails saying that ownership on the staging directory is not as
> expected, if owner returned by the {{FileStatus#getOwner()}} is not equal to
> short principal name of the current user.
> {code}
> String realUser;
> String currentUser;
> UserGroupInformation ugi = UserGroupInformation.getLoginUser();
> realUser = ugi.getShortUserName();
> currentUser = UserGroupInformation.getCurrentUser().getShortUserName();
> if (fs.exists(stagingArea)) {
> FileStatus fsStatus = fs.getFileStatus(stagingArea);
> String owner = fsStatus.getOwner();
> if (!(owner.equals(currentUser) || owner.equals(realUser))) {
> throw new IOException("The ownership on the staging directory " +
> stagingArea + " is not as expected. " +
> "It is owned by " + owner + ". The directory must " +
> "be owned by the submitter " + currentUser + " or " +
> "by " + realUser);
> }
> {code}
> The proposal is to remove the strict restriction on short principal name by
> allowing the user's full principal name as owner of staging area directory in
> {{JobSubmissionFiles#JobStagingDir()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6458) Figure out the way to pass build-in classpath (files in distributed cache, etc.) from parent to spawned shells
[ https://issues.apache.org/jira/browse/MAPREDUCE-6458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated MAPREDUCE-6458: --- Target Version/s: 3.0.0-alpha2 (was: 3.0.0-alpha1) > Figure out the way to pass build-in classpath (files in distributed cache, > etc.) from parent to spawned shells > -- > > Key: MAPREDUCE-6458 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6458 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Junping Du >Assignee: Dustin Cote > Attachments: MAPREDUCE-6458.00.patch > > > In MAPREDUCE-6454 (target for branch-2.x), we provide a way with constraints > to pass built-in classpath from parent to child shell, via HADOOP_CLASSPATH, > so jars in distributed cache can still work in child tasks. In trunk, we may > think some way different, like: involve additional env var to safely pass > build-in classpath. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6704) Container fail to launch for mapred application
[
https://issues.apache.org/jira/browse/MAPREDUCE-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583306#comment-15583306
]
Yongjun Zhang commented on MAPREDUCE-6704:
--
Thanks [~bibinchundatt], good catch, and sorry for missing that earlier. I
tried again after removing "+", it runs.
> Container fail to launch for mapred application
> ---
>
> Key: MAPREDUCE-6704
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6704
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Blocker
> Attachments: 0001-MAPREDUCE-6704.patch, 0001-YARN-5026.patch
>
>
> Container fail to launch for mapred application.
> As part for launch script {{HADOOP_MAPRED_HOME}} default value is not set
> .After
> https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c
>{{HADOOP_MAPRED_HOME}} is not able to get from {{builder.environment()}}
> since {{DefaultContainerExecutor#buildCommandExecutor}} sets inherit to false.
> {noformat}
> 16/05/02 09:16:05 INFO mapreduce.Job: Job job_1462155939310_0004 failed with
> state FAILED due to: Application application_1462155939310_0004 failed 2
> times due to AM Container for appattempt_1462155939310_0004_02 exited
> with exitCode: 1
> Failing this attempt.Diagnostics: Exception from container-launch.
> Container id: container_1462155939310_0004_02_01
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:946)
> at org.apache.hadoop.util.Shell.run(Shell.java:850)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:227)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:385)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:281)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:89)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> Error: Could not find or load main class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6749) MR AM should reuse containers for Map/Reduce Tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583132#comment-15583132 ] Devaraj K commented on MAPREDUCE-6749: -- I have created a branch MR-6749 for this. > MR AM should reuse containers for Map/Reduce Tasks > -- > > Key: MAPREDUCE-6749 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6749 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6749-Container Reuse-v0.pdf > > > It is with the continuation of MAPREDUCE-3902, MR AM should reuse containers > for Map/Reduce Tasks similar to the JVM Reuse feature we had in MRv1. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6704) Container fail to launch for mapred application
[
https://issues.apache.org/jira/browse/MAPREDUCE-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582894#comment-15582894
]
Bibin A Chundatt commented on MAPREDUCE-6704:
-
[~yzhangal]
Looks like you have made wrong configuration
{noformat}
=JAVA_HOME
{noformat}
Could please remove {{=}} and try again. I thought of documenting the white
list parameter configuration and closing the issue waiting for +1 which
solution to take.
> Container fail to launch for mapred application
> ---
>
> Key: MAPREDUCE-6704
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6704
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Blocker
> Attachments: 0001-MAPREDUCE-6704.patch, 0001-YARN-5026.patch
>
>
> Container fail to launch for mapred application.
> As part for launch script {{HADOOP_MAPRED_HOME}} default value is not set
> .After
> https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c
>{{HADOOP_MAPRED_HOME}} is not able to get from {{builder.environment()}}
> since {{DefaultContainerExecutor#buildCommandExecutor}} sets inherit to false.
> {noformat}
> 16/05/02 09:16:05 INFO mapreduce.Job: Job job_1462155939310_0004 failed with
> state FAILED due to: Application application_1462155939310_0004 failed 2
> times due to AM Container for appattempt_1462155939310_0004_02 exited
> with exitCode: 1
> Failing this attempt.Diagnostics: Exception from container-launch.
> Container id: container_1462155939310_0004_02_01
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:946)
> at org.apache.hadoop.util.Shell.run(Shell.java:850)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:227)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:385)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:281)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:89)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> Error: Could not find or load main class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6794) Remove unused properties from TTConfig.java
[ https://issues.apache.org/jira/browse/MAPREDUCE-6794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15581672#comment-15581672 ] Akira Ajisaka commented on MAPREDUCE-6794: -- Filed MAPREDUCE-6796 for removing unused properties from JTConfig.java. > Remove unused properties from TTConfig.java > --- > > Key: MAPREDUCE-6794 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6794 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka > Attachments: MAPREDUCE-6794.01.patch > > > There are many unused properties in TTConfig.java. Let's remove them. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6796) Remove unused properties from JTConfig.java
Akira Ajisaka created MAPREDUCE-6796: Summary: Remove unused properties from JTConfig.java Key: MAPREDUCE-6796 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6796 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Akira Ajisaka JobTracker-side of MAPREDUCE-6794. There are many unused properties in JTConfig.java. Let's remove them. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-5608) Replace and deprecate mapred.tasktracker.indexcache.mb
[
https://issues.apache.org/jira/browse/MAPREDUCE-5608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15581610#comment-15581610
]
Hadoop QA commented on MAPREDUCE-5608:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
36s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 9s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
50s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 29s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m 1s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 37s
{color} | {color:red} root: The patch generated 1 new + 1052 unchanged - 3
fixed = 1053 total (was 1055) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 13s
{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 34s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 25s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12833694/MAPREDUCE-5608.003.patch
|
| JIRA Issue | MAPREDUCE-5608 |
| Optional Tests | asflicense mvnsite compile javac javadoc mvninstall
unit findbugs checkstyle |
| uname | Linux 803e456bf9e3 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 1f304b0 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6762/artifact/patchprocess/diff-checkstyle-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6762/testReport/ |
| modules | C: hadoop-common-project/hadoop-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
U: . |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6762/console |
| Powered by | Apache Yetus 0.3.0
[jira] [Updated] (MAPREDUCE-5608) Replace and deprecate mapred.tasktracker.indexcache.mb
[ https://issues.apache.org/jira/browse/MAPREDUCE-5608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated MAPREDUCE-5608: - Attachment: MAPREDUCE-5608.003.patch Fixes checkstyle warnings. > Replace and deprecate mapred.tasktracker.indexcache.mb > -- > > Key: MAPREDUCE-5608 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5608 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 2.2.0 >Reporter: Sandy Ryza >Assignee: Akira Ajisaka > Labels: BB2015-05-TBR, configuration, newbie > Attachments: MAPREDUCE-5608-002.patch, MAPREDUCE-5608.003.patch, > MAPREDUCE-5608.patch > > > In MR2 mapred.tasktracker.indexcache.mb still works for configuring the size > of the shuffle service index cache. As the tasktracker no longer exists, we > should replace this with something like mapreduce.shuffle.indexcache.mb. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-5608) Replace and deprecate mapred.tasktracker.indexcache.mb
[ https://issues.apache.org/jira/browse/MAPREDUCE-5608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15581510#comment-15581510 ] Akira Ajisaka edited comment on MAPREDUCE-5608 at 10/17/16 7:46 AM: 03 patch: Fixes checkstyle warnings. was (Author: ajisakaa): Fixes checkstyle warnings. > Replace and deprecate mapred.tasktracker.indexcache.mb > -- > > Key: MAPREDUCE-5608 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5608 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 2.2.0 >Reporter: Sandy Ryza >Assignee: Akira Ajisaka > Labels: BB2015-05-TBR, configuration, newbie > Attachments: MAPREDUCE-5608-002.patch, MAPREDUCE-5608.003.patch, > MAPREDUCE-5608.patch > > > In MR2 mapred.tasktracker.indexcache.mb still works for configuring the size > of the shuffle service index cache. As the tasktracker no longer exists, we > should replace this with something like mapreduce.shuffle.indexcache.mb. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6794) Remove unused properties from TTConfig.java
[
https://issues.apache.org/jira/browse/MAPREDUCE-6794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15581493#comment-15581493
]
Akira Ajisaka commented on MAPREDUCE-6794:
--
bq.
./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java:52:
@SuppressWarnings("deprecation"):3: Method length is 435 lines (max allowed
is 150).
The checkstyle warning exists before applying this patch.
> Remove unused properties from TTConfig.java
> ---
>
> Key: MAPREDUCE-6794
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6794
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
> Attachments: MAPREDUCE-6794.01.patch
>
>
> There are many unused properties in TTConfig.java. Let's remove them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6794) Remove unused properties from TTConfig.java
[
https://issues.apache.org/jira/browse/MAPREDUCE-6794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15581387#comment-15581387
]
Hadoop QA commented on MAPREDUCE-6794:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
37s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 48s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
31s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 6s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
55s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
57s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 31s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 49s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 31s
{color} | {color:red} root: The patch generated 1 new + 405 unchanged - 100
fixed = 406 total (was 505) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 10s
{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 27s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 8s
{color} | {color:green} hadoop-mapreduce-client-hs in the patch passed. {color}
|
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 55s
{color} | {color:green} hadoop-sls in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 56m 9s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12833677/MAPREDUCE-6794.01.patch
|
| JIRA Issue | MAPREDUCE-6794 |
| Optional Tests | asflicense mvnsite compile javac javadoc mvninstall
unit findbugs checkstyle xml |
| uname | Linux b7e262a2ae6a 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 1f304b0 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
