[jira] [Commented] (MAPREDUCE-6579) Test failure : TestNetworkedJob
[
https://issues.apache.org/jira/browse/MAPREDUCE-6579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085455#comment-15085455
]
Akira AJISAKA commented on MAPREDUCE-6579:
--
Thanks [~Naganarasimha] for the comment.
bq. arguments for the assertEquals are swapped
I've noticed that but kept as-is because this issue is to fix the test failure.
bq. If you can accomodate in this patch then fine even if not its ok !
Okay, I'll update the patch.
> Test failure : TestNetworkedJob
> ---
>
> Key: MAPREDUCE-6579
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6579
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Rohith Sharma K S
>Assignee: Akira AJISAKA
> Attachments: MAPREDUCE-6579.01.patch
>
>
> From
> [https://builds.apache.org/job/PreCommit-YARN-Build/9976/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt]
> TestNetworkedJob are failed intermittently.
> {code}
> Running org.apache.hadoop.mapred.TestNetworkedJob
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 81.131 sec
> <<< FAILURE! - in org.apache.hadoop.mapred.TestNetworkedJob
> testNetworkedJob(org.apache.hadoop.mapred.TestNetworkedJob) Time elapsed:
> 30.55 sec <<< FAILURE!
> org.junit.ComparisonFailure: expected:<[[Tue Dec 15 14:02:45 + 2015]
> Application is Activated, waiting for resources to be assigned for AM.
> Details : AM Partition = ; Partition Resource =
> ; Queue's Absolute capacity = 100.0 % ; Queue's
> Absolute used capacity = 0.0 % ; Queue's Absolute max capacity = 100.0 % ; ]>
> but was:<[]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.mapred.TestNetworkedJob.testNetworkedJob(TestNetworkedJob.java:174)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6579) Test failure : TestNetworkedJob
[
https://issues.apache.org/jira/browse/MAPREDUCE-6579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085451#comment-15085451
]
Naganarasimha G R commented on MAPREDUCE-6579:
--
Hi [~ajisakaa],
Overall the fix is fine but only one issue (not related to your modifications),
arguments for the {{assertEquals}} are swapped (first is expected and seconds
arg is actual). If you can accomodate in this patch then fine even if not its
ok !
> Test failure : TestNetworkedJob
> ---
>
> Key: MAPREDUCE-6579
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6579
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Rohith Sharma K S
>Assignee: Akira AJISAKA
> Attachments: MAPREDUCE-6579.01.patch
>
>
> From
> [https://builds.apache.org/job/PreCommit-YARN-Build/9976/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt]
> TestNetworkedJob are failed intermittently.
> {code}
> Running org.apache.hadoop.mapred.TestNetworkedJob
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 81.131 sec
> <<< FAILURE! - in org.apache.hadoop.mapred.TestNetworkedJob
> testNetworkedJob(org.apache.hadoop.mapred.TestNetworkedJob) Time elapsed:
> 30.55 sec <<< FAILURE!
> org.junit.ComparisonFailure: expected:<[[Tue Dec 15 14:02:45 + 2015]
> Application is Activated, waiting for resources to be assigned for AM.
> Details : AM Partition = ; Partition Resource =
> ; Queue's Absolute capacity = 100.0 % ; Queue's
> Absolute used capacity = 0.0 % ; Queue's Absolute max capacity = 100.0 % ; ]>
> but was:<[]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.mapred.TestNetworkedJob.testNetworkedJob(TestNetworkedJob.java:174)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6579) Test failure : TestNetworkedJob
[
https://issues.apache.org/jira/browse/MAPREDUCE-6579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated MAPREDUCE-6579:
-
Attachment: MAPREDUCE-6579.02.patch
02 patch
* Corrected the order of the arguments for assertEquals method.
* Fixed the following useless assertions.
{code}
assertEquals(info, info);
assertEquals(info.toString(), info.toString());
{code}
* Implemented equals method in BlackListInfo class to pass the following
assertion.
{code}
assertEquals(info, info2);
{code}
> Test failure : TestNetworkedJob
> ---
>
> Key: MAPREDUCE-6579
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6579
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Rohith Sharma K S
>Assignee: Akira AJISAKA
> Attachments: MAPREDUCE-6579.01.patch, MAPREDUCE-6579.02.patch
>
>
> From
> [https://builds.apache.org/job/PreCommit-YARN-Build/9976/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt]
> TestNetworkedJob are failed intermittently.
> {code}
> Running org.apache.hadoop.mapred.TestNetworkedJob
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 81.131 sec
> <<< FAILURE! - in org.apache.hadoop.mapred.TestNetworkedJob
> testNetworkedJob(org.apache.hadoop.mapred.TestNetworkedJob) Time elapsed:
> 30.55 sec <<< FAILURE!
> org.junit.ComparisonFailure: expected:<[[Tue Dec 15 14:02:45 + 2015]
> Application is Activated, waiting for resources to be assigned for AM.
> Details : AM Partition = ; Partition Resource =
> ; Queue's Absolute capacity = 100.0 % ; Queue's
> Absolute used capacity = 0.0 % ; Queue's Absolute max capacity = 100.0 % ; ]>
> but was:<[]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.mapred.TestNetworkedJob.testNetworkedJob(TestNetworkedJob.java:174)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6068) Illegal progress value warnings in map tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-6068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085651#comment-15085651
]
Eduard Camaj commented on MAPREDUCE-6068:
-
What about these fails above? And who can decide if this is OK or not?
It is shame to have fix for such long and people are still facing this error in
their deployments with latest version.
> Illegal progress value warnings in map tasks
>
>
> Key: MAPREDUCE-6068
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6068
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, task
>Affects Versions: 3.0.0
>Reporter: Todd Lipcon
>Assignee: Binglin Chang
> Labels: BB2015-05-RFC
> Attachments: MAPREDUCE-6068.002.patch, MAPREDUCE-6068.v1.patch
>
>
> When running a terasort on latest trunk, I see the following in my task logs:
> {code}
> 2014-09-02 17:42:28,437 INFO [main] org.apache.hadoop.mapred.MapTask: Map
> output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
> 2014-09-02 17:42:42,238 WARN [main] org.apache.hadoop.util.Progress: Illegal
> progress value found, progress is larger than 1. Progress will be changed to 1
> 2014-09-02 17:42:42,238 WARN [main] org.apache.hadoop.util.Progress: Illegal
> progress value found, progress is larger than 1. Progress will be changed to 1
> 2014-09-02 17:42:42,241 INFO [main] org.apache.hadoop.mapred.MapTask:
> Starting flush of map output
> {code}
> We should eliminate these warnings.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[
https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085602#comment-15085602
]
Jason Lowe commented on MAPREDUCE-6577:
---
Thanks, Sangjin! Could you also commit this to branch-2.8? I find it odd that
the fix is important enough to backport to 2.6 and 2.7 but not important enough
for 2.8. Users updating from those older releases to 2.8 will accidentally
drop this fix.
> MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
>
>
> Key: MAPREDUCE-6577
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am
>Affects Versions: 2.6.0
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
> Fix For: 2.7.3, 2.9.0, 2.6.4
>
> Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch,
> MAPREDUCE-6577.03.patch, MAPREDUCE-6577.04.patch
>
>
> If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not
> configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library:
> {noformat}
> 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader:
> Unable to load native-hadoop library for your platform... using builtin-java
> classes where applicable
> {noformat}
> As a result, any code that needs the hadoop native library in the MR AM will
> fail. For example, an uber-AM with lz4 compression for the mapper task will
> fail:
> {noformat}
> 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner]
> org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local
> (uberized) 'child' : java.lang.RuntimeException: native lz4 library not
> available
> at
> org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
> at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114)
> at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6579) Test failure : TestNetworkedJob
[
https://issues.apache.org/jira/browse/MAPREDUCE-6579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085781#comment-15085781
]
Hadoop QA commented on MAPREDUCE-6579:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
44s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 56s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 5s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 1s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 14s
{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
in trunk has 2 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 12s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 30s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 30s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 22s
{color} | {color:red} Patch generated 1 new checkstyle issues in
hadoop-mapreduce-project/hadoop-mapreduce-client (total was 91, now 92).
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 24s
{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
introduced 1 new FindBugs issues. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m 27s
{color} | {color:red}
hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_66
with JDK v1.8.0_66 generated 2 new issues (was 100, now 100). {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 14s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 103m 37s
{color} | {color:red} hadoop-mapreduce-client-jobclient in the patch failed
with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 21s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 105m 15s
{color} | {color:green} hadoop-mapreduce-client-jobclient in the patch passed
with JDK v1.7.0_91. {color} |
|
[jira] [Updated] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[
https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee updated MAPREDUCE-6577:
---
Fix Version/s: 2.8.0
> MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
>
>
> Key: MAPREDUCE-6577
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am
>Affects Versions: 2.6.0
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
> Fix For: 2.8.0, 2.7.3, 2.9.0, 2.6.4
>
> Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch,
> MAPREDUCE-6577.03.patch, MAPREDUCE-6577.04.patch
>
>
> If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not
> configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library:
> {noformat}
> 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader:
> Unable to load native-hadoop library for your platform... using builtin-java
> classes where applicable
> {noformat}
> As a result, any code that needs the hadoop native library in the MR AM will
> fail. For example, an uber-AM with lz4 compression for the mapper task will
> fail:
> {noformat}
> 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner]
> org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local
> (uberized) 'child' : java.lang.RuntimeException: native lz4 library not
> available
> at
> org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
> at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114)
> at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[
https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085803#comment-15085803
]
Sangjin Lee commented on MAPREDUCE-6577:
Thanks for pointing that out. I somehow (mistakenly) thought that 2.8.0 was
already being cut. I just picked the fix to 2.8.0.
> MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
>
>
> Key: MAPREDUCE-6577
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am
>Affects Versions: 2.6.0
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
> Fix For: 2.8.0, 2.7.3, 2.9.0, 2.6.4
>
> Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch,
> MAPREDUCE-6577.03.patch, MAPREDUCE-6577.04.patch
>
>
> If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not
> configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library:
> {noformat}
> 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader:
> Unable to load native-hadoop library for your platform... using builtin-java
> classes where applicable
> {noformat}
> As a result, any code that needs the hadoop native library in the MR AM will
> fail. For example, an uber-AM with lz4 compression for the mapper task will
> fail:
> {noformat}
> 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner]
> org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local
> (uberized) 'child' : java.lang.RuntimeException: native lz4 library not
> available
> at
> org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
> at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114)
> at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6476) InvalidResourceException when Nodelabel don't have access to queue should be handled
[
https://issues.apache.org/jira/browse/MAPREDUCE-6476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085878#comment-15085878
]
Bibin A Chundatt commented on MAPREDUCE-6476:
-
Thank you Wangda for review
> InvalidResourceException when Nodelabel don't have access to queue should be
> handled
>
>
> Key: MAPREDUCE-6476
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6476
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 0001-MAPREDUCE-6476.patch, 0002-MAPREDUCE-6476.patch
>
>
> Steps to reproduce
> ===
> Submit mapreduce job
> # map to label x
> # reduce to label y
> Precondition
> # Queue b to which reduce is submitted not having access to label specified
> *Impact*
> # Jobs fail only of the RM-AM comunication timeout
> (About 10 mins i think)
> Should kill the job immediately when InvalidResourceException is received on
> {{RMContainerRequestor#makeRemoteRequest}}
> *Logs*
> {noformat}
> 2015-09-11 16:44:30,116 ERROR [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: ERROR IN CONTACTING RM.
> org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
> resource request, queue=b1 doesn't have permission to access all labels in
> resource request. labelExpression of resource request=1. Queue labels=3
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:304)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:234)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:250)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:106)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:457)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2230)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2226)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1667)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2224)
> at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown
> Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateYarnException(RPCUtil.java:75)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:116)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:79)
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:251)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> at com.sun.proxy.$Proxy37.allocate(Unknown Source)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor.makeRemoteRequest(RMContainerRequestor.java:203)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.getResources(RMContainerAllocator.java:694)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:263)
> at
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$AllocatorRunnable.run(RMCommunicator.java:281)
> at java.lang.Thread.run(Thread.java:745)
> Caused by:
>
[jira] [Commented] (MAPREDUCE-6579) Test failure : TestNetworkedJob
[
https://issues.apache.org/jira/browse/MAPREDUCE-6579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085807#comment-15085807
]
Naganarasimha G R commented on MAPREDUCE-6579:
--
[~ajisakaa], Thanks for considering and correcting the parameters other issues
mentioned. Just that the check style and findbugs are same and are technical
correction though it might not have any impact apart from it other corrections
are fine !
> Test failure : TestNetworkedJob
> ---
>
> Key: MAPREDUCE-6579
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6579
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Rohith Sharma K S
>Assignee: Akira AJISAKA
> Attachments: MAPREDUCE-6579.01.patch, MAPREDUCE-6579.02.patch
>
>
> From
> [https://builds.apache.org/job/PreCommit-YARN-Build/9976/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt]
> TestNetworkedJob are failed intermittently.
> {code}
> Running org.apache.hadoop.mapred.TestNetworkedJob
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 81.131 sec
> <<< FAILURE! - in org.apache.hadoop.mapred.TestNetworkedJob
> testNetworkedJob(org.apache.hadoop.mapred.TestNetworkedJob) Time elapsed:
> 30.55 sec <<< FAILURE!
> org.junit.ComparisonFailure: expected:<[[Tue Dec 15 14:02:45 + 2015]
> Application is Activated, waiting for resources to be assigned for AM.
> Details : AM Partition = ; Partition Resource =
> ; Queue's Absolute capacity = 100.0 % ; Queue's
> Absolute used capacity = 0.0 % ; Queue's Absolute max capacity = 100.0 % ; ]>
> but was:<[]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.mapred.TestNetworkedJob.testNetworkedJob(TestNetworkedJob.java:174)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6600) Split findbugs-exclude into submodules
[ https://issues.apache.org/jira/browse/MAPREDUCE-6600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15086817#comment-15086817 ] Akira AJISAKA commented on MAPREDUCE-6600: -- In Common and HDFS project, findbugs-exclude.xml has already been splitted. In YARN project, findbugs-exclude.xml has not been splitted, so I'll file a separate jira in YARN project. > Split findbugs-exclude into submodules > -- > > Key: MAPREDUCE-6600 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6600 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: test >Reporter: Akira AJISAKA > > Now findbugs-exclude.xml is in hadoop-mapreduce-project module but the java > source code is in other modules. The mismatch of the modules causes build > failure (see YETUS-271), so the findbugs-exclude.xml should be splitted into > submodules. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6600) Split findbugs-exclude into submodules
Akira AJISAKA created MAPREDUCE-6600: Summary: Split findbugs-exclude into submodules Key: MAPREDUCE-6600 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6600 Project: Hadoop Map/Reduce Issue Type: Improvement Components: test Reporter: Akira AJISAKA Now findbugs-exclude.xml is in hadoop-mapreduce-project module but the java source code is in other modules. The mismatch of the modules causes build failure (see YETUS-271), so the findbugs-exclude.xml should be splitted into submodules. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6579) Test failure : TestNetworkedJob
[
https://issues.apache.org/jira/browse/MAPREDUCE-6579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated MAPREDUCE-6579:
-
Attachment: MAPREDUCE-6579.03.patch
03 patch:
* Added hashCode method in BlackListInfo class.
* Added null check to equals method in BlackListInfo class.
> Test failure : TestNetworkedJob
> ---
>
> Key: MAPREDUCE-6579
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6579
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Rohith Sharma K S
>Assignee: Akira AJISAKA
> Attachments: MAPREDUCE-6579.01.patch, MAPREDUCE-6579.02.patch,
> MAPREDUCE-6579.03.patch
>
>
> From
> [https://builds.apache.org/job/PreCommit-YARN-Build/9976/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt]
> TestNetworkedJob are failed intermittently.
> {code}
> Running org.apache.hadoop.mapred.TestNetworkedJob
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 81.131 sec
> <<< FAILURE! - in org.apache.hadoop.mapred.TestNetworkedJob
> testNetworkedJob(org.apache.hadoop.mapred.TestNetworkedJob) Time elapsed:
> 30.55 sec <<< FAILURE!
> org.junit.ComparisonFailure: expected:<[[Tue Dec 15 14:02:45 + 2015]
> Application is Activated, waiting for resources to be assigned for AM.
> Details : AM Partition = ; Partition Resource =
> ; Queue's Absolute capacity = 100.0 % ; Queue's
> Absolute used capacity = 0.0 % ; Queue's Absolute max capacity = 100.0 % ; ]>
> but was:<[]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.mapred.TestNetworkedJob.testNetworkedJob(TestNetworkedJob.java:174)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6595) Fix findbugs warnings in OutputCommitter and FileOutputCommitter
[ https://issues.apache.org/jira/browse/MAPREDUCE-6595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15086810#comment-15086810 ] Akira AJISAKA commented on MAPREDUCE-6595: -- Rethinking this, even if YETUS-271 is fixed, findbugs will not run on hadoop-mapreduce-client-core module. To fix the mismatch of module, I propose to split findbugs-exclude.xml into submodules. I'll file a separate jira. > Fix findbugs warnings in OutputCommitter and FileOutputCommitter > > > Key: MAPREDUCE-6595 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6595 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA > Attachments: MAPREDUCE-6595.01.patch, MAPREDUCE-6595.testing.patch, > findbugsHtml.html > > > There are 2 findbugs warnings in hadoop-mapreduce-client-core module. > https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6237/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (MAPREDUCE-6600) Split findbugs-exclude into submodules
[ https://issues.apache.org/jira/browse/MAPREDUCE-6600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA reassigned MAPREDUCE-6600: Assignee: Akira AJISAKA > Split findbugs-exclude into submodules > -- > > Key: MAPREDUCE-6600 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6600 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: test >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA > > Now findbugs-exclude.xml is in hadoop-mapreduce-project module but the java > source code is in other modules. The mismatch of the modules causes build > failure (see YETUS-271), so the findbugs-exclude.xml should be splitted into > submodules. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[
https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated MAPREDUCE-6577:
--
Target Version/s: 2.6.4
> MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
>
>
> Key: MAPREDUCE-6577
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am
>Affects Versions: 2.6.0
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
> Fix For: 2.8.0, 2.7.3, 2.9.0, 2.6.4
>
> Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch,
> MAPREDUCE-6577.03.patch, MAPREDUCE-6577.04.patch
>
>
> If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not
> configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library:
> {noformat}
> 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader:
> Unable to load native-hadoop library for your platform... using builtin-java
> classes where applicable
> {noformat}
> As a result, any code that needs the hadoop native library in the MR AM will
> fail. For example, an uber-AM with lz4 compression for the mapper task will
> fail:
> {noformat}
> 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner]
> org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local
> (uberized) 'child' : java.lang.RuntimeException: native lz4 library not
> available
> at
> org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
> at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114)
> at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
