[jira] [Commented] (MAPREDUCE-7011) TestClientDistributedCacheManager::testDetermineCacheVisibilities assumes all parent dirs set other exec
[
https://issues.apache.org/jira/browse/MAPREDUCE-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261979#comment-16261979
]
Hudson commented on MAPREDUCE-7011:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13267 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13267/])
MAPREDUCE-7011. (cdouglas: rev 782ba3bf9da52699b27405a3f147464975d1df99)
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/filecache/TestClientDistributedCacheManager.java
> TestClientDistributedCacheManager::testDetermineCacheVisibilities assumes all
> parent dirs set other exec
>
>
> Key: MAPREDUCE-7011
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7011
> Project: Hadoop Map/Reduce
> Issue Type: Test
>Reporter: Chris Douglas
>Assignee: Chris Douglas
>Priority: Trivial
> Fix For: 3.0.1
>
> Attachments: MAPREDUCE-7011.000.patch
>
>
> {{TestClientDistributedCacheManager}} sets up some local directories to check
> the visibility set for dependencies, given their filesystem permissions.
> However, if it is run in an environment where the scratch directory is not
> itself PUBLIC ({{ClientDistributedCacheManager::isPublic}}), then it will
> fail.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7011) TestClientDistributedCacheManager::testDetermineCacheVisibilities assumes all parent dirs set other exec
[
https://issues.apache.org/jira/browse/MAPREDUCE-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated MAPREDUCE-7011:
-
Resolution: Fixed
Assignee: Chris Douglas
Hadoop Flags: Reviewed
Fix Version/s: 3.0.1
Status: Resolved (was: Patch Available)
I committed this. Thanks for the review, [~subru]
> TestClientDistributedCacheManager::testDetermineCacheVisibilities assumes all
> parent dirs set other exec
>
>
> Key: MAPREDUCE-7011
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7011
> Project: Hadoop Map/Reduce
> Issue Type: Test
>Reporter: Chris Douglas
>Assignee: Chris Douglas
>Priority: Trivial
> Fix For: 3.0.1
>
> Attachments: MAPREDUCE-7011.000.patch
>
>
> {{TestClientDistributedCacheManager}} sets up some local directories to check
> the visibility set for dependencies, given their filesystem permissions.
> However, if it is run in an environment where the scratch directory is not
> itself PUBLIC ({{ClientDistributedCacheManager::isPublic}}), then it will
> fail.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7011) TestClientDistributedCacheManager::testDetermineCacheVisibilities assumes all parent dirs set other exec
[
https://issues.apache.org/jira/browse/MAPREDUCE-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated MAPREDUCE-7011:
-
Priority: Trivial (was: Minor)
> TestClientDistributedCacheManager::testDetermineCacheVisibilities assumes all
> parent dirs set other exec
>
>
> Key: MAPREDUCE-7011
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7011
> Project: Hadoop Map/Reduce
> Issue Type: Test
>Reporter: Chris Douglas
>Priority: Trivial
> Attachments: MAPREDUCE-7011.000.patch
>
>
> {{TestClientDistributedCacheManager}} sets up some local directories to check
> the visibility set for dependencies, given their filesystem permissions.
> However, if it is run in an environment where the scratch directory is not
> itself PUBLIC ({{ClientDistributedCacheManager::isPublic}}), then it will
> fail.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7011) TestClientDistributedCacheManager::testDetermineCacheVisibilities assumes all parent dirs set other exec
[
https://issues.apache.org/jira/browse/MAPREDUCE-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261966#comment-16261966
]
Chris Douglas commented on MAPREDUCE-7011:
--
bq. Does it make sense to open another jira to refactor
TestClientDistributedCacheManager to not use the local FS?
It wouldn't hurt. I'll open something generic that includes this as an example.
> TestClientDistributedCacheManager::testDetermineCacheVisibilities assumes all
> parent dirs set other exec
>
>
> Key: MAPREDUCE-7011
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7011
> Project: Hadoop Map/Reduce
> Issue Type: Test
>Reporter: Chris Douglas
>Priority: Minor
> Attachments: MAPREDUCE-7011.000.patch
>
>
> {{TestClientDistributedCacheManager}} sets up some local directories to check
> the visibility set for dependencies, given their filesystem permissions.
> However, if it is run in an environment where the scratch directory is not
> itself PUBLIC ({{ClientDistributedCacheManager::isPublic}}), then it will
> fail.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7011) TestClientDistributedCacheManager::testDetermineCacheVisibilities assumes all parent dirs set other exec
[
https://issues.apache.org/jira/browse/MAPREDUCE-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261728#comment-16261728
]
Subru Krishnan commented on MAPREDUCE-7011:
---
+1 for the patch. Does it make sense to open another jira to refactor
{{TestClientDistributedCacheManager}} to not use the local FS?
> TestClientDistributedCacheManager::testDetermineCacheVisibilities assumes all
> parent dirs set other exec
>
>
> Key: MAPREDUCE-7011
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7011
> Project: Hadoop Map/Reduce
> Issue Type: Test
>Reporter: Chris Douglas
>Priority: Minor
> Attachments: MAPREDUCE-7011.000.patch
>
>
> {{TestClientDistributedCacheManager}} sets up some local directories to check
> the visibility set for dependencies, given their filesystem permissions.
> However, if it is run in an environment where the scratch directory is not
> itself PUBLIC ({{ClientDistributedCacheManager::isPublic}}), then it will
> fail.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7012) 3.0 deployment cannot work with old version MR tar ball which break rolling upgrade
[
https://issues.apache.org/jira/browse/MAPREDUCE-7012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261476#comment-16261476
]
Junping Du commented on MAPREDUCE-7012:
---
CC [~andrew.wang], [~vinodkv], [~jlowe].
> 3.0 deployment cannot work with old version MR tar ball which break rolling
> upgrade
> ---
>
> Key: MAPREDUCE-7012
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7012
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distributed-cache, mrv2
>Reporter: Junping Du
>Priority: Blocker
>
> I tried to deploy 3.0 cluster with 2.9 MR tar ball. The MR job is failed
> because following error:
> {noformat}
> 2017-11-21 12:42:50,911 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for
> application appattempt_1511295641738_0003_01
> 2017-11-21 12:42:51,070 WARN [main] org.apache.hadoop.util.NativeCodeLoader:
> Unable to load native-hadoop library for your platform... using builtin-java
> classes where applicable
> 2017-11-21 12:42:51,118 FATAL [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.lang.RuntimeException: Unable to determine current user
> at
> org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:254)
> at
> org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:220)
> at
> org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:212)
> at
> org.apache.hadoop.conf.Configuration.addResource(Configuration.java:888)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1638)
> Caused by: java.io.IOException: Exception reading
> /tmp/nm-local-dir/usercache/jdu/appcache/application_1511295641738_0003/container_e03_1511295641738_0003_01_01/container_tokens
> at
> org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:208)
> at
> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:907)
> at
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:820)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:689)
> at
> org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:252)
> ... 4 more
> Caused by: java.io.IOException: Unknown version 1 in token storage.
> at
> org.apache.hadoop.security.Credentials.readTokenStorageStream(Credentials.java:226)
> at
> org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:205)
> ... 8 more
> 2017-11-21 12:42:51,122 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
> with status 1: java.lang.RuntimeException: Unable to determine current user
> {noformat}
> I think it is due to token incompatiblity change between 2.9 and 3.0. As we
> claim "rolling upgrade" is supported in Hadoop 3, we should fix this before
> we ship 3.0 otherwise all MR running applications will get stuck during/after
> upgrade.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7012) 3.0 deployment cannot work with old version MR tar ball which break rolling upgrade
Junping Du created MAPREDUCE-7012:
-
Summary: 3.0 deployment cannot work with old version MR tar ball
which break rolling upgrade
Key: MAPREDUCE-7012
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7012
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distributed-cache, mrv2
Reporter: Junping Du
Priority: Blocker
I tried to deploy 3.0 cluster with 2.9 MR tar ball. The MR job is failed
because following error:
{noformat}
2017-11-21 12:42:50,911 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for
application appattempt_1511295641738_0003_01
2017-11-21 12:42:51,070 WARN [main] org.apache.hadoop.util.NativeCodeLoader:
Unable to load native-hadoop library for your platform... using builtin-java
classes where applicable
2017-11-21 12:42:51,118 FATAL [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.lang.RuntimeException: Unable to determine current user
at
org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:254)
at
org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:220)
at
org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:212)
at
org.apache.hadoop.conf.Configuration.addResource(Configuration.java:888)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1638)
Caused by: java.io.IOException: Exception reading
/tmp/nm-local-dir/usercache/jdu/appcache/application_1511295641738_0003/container_e03_1511295641738_0003_01_01/container_tokens
at
org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:208)
at
org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:907)
at
org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:820)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:689)
at
org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:252)
... 4 more
Caused by: java.io.IOException: Unknown version 1 in token storage.
at
org.apache.hadoop.security.Credentials.readTokenStorageStream(Credentials.java:226)
at
org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:205)
... 8 more
2017-11-21 12:42:51,122 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
with status 1: java.lang.RuntimeException: Unable to determine current user
{noformat}
I think it is due to token incompatiblity change between 2.9 and 3.0. As we
claim "rolling upgrade" is supported in Hadoop 3, we should fix this before we
ship 3.0 otherwise all MR running applications will get stuck during/after
upgrade.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7011) TestClientDistributedCacheManager::testDetermineCacheVisibilities assumes all parent dirs set other exec
[
https://issues.apache.org/jira/browse/MAPREDUCE-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261434#comment-16261434
]
Hadoop QA commented on MAPREDUCE-7011:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m
5s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 39s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
The patch generated 0 new + 2 unchanged - 1 fixed = 2 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 59s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
5s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 62m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | MAPREDUCE-7011 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12898711/MAPREDUCE-7011.000.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 0028a4c49265 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 03c311e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7232/testReport/ |
| Max. process+thread count | 354 (vs. ulimit of 5000) |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7232/console |
| Powered by | Apache Yetus
[jira] [Commented] (MAPREDUCE-5124) AM lacks flow control for task events
[
https://issues.apache.org/jira/browse/MAPREDUCE-5124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261427#comment-16261427
]
Hadoop QA commented on MAPREDUCE-5124:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 16s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 26s{color} | {color:orange}
hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 16 new +
258 unchanged - 5 fixed = 274 total (was 263) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 19s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
53s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}135m 16s{color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
25s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}189m 30s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.mapreduce.security.ssl.TestEncryptedShuffle |
| | hadoop.mapreduce.v2.TestUberAM |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | MAPREDUCE-5124 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12897988/MAPREDUCE-5124-001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux ae225d3068d4 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk /
[jira] [Updated] (MAPREDUCE-7011) TestClientDistributedCacheManager::testDetermineCacheVisibilities assumes all parent dirs set other exec
[
https://issues.apache.org/jira/browse/MAPREDUCE-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated MAPREDUCE-7011:
-
Status: Patch Available (was: Open)
> TestClientDistributedCacheManager::testDetermineCacheVisibilities assumes all
> parent dirs set other exec
>
>
> Key: MAPREDUCE-7011
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7011
> Project: Hadoop Map/Reduce
> Issue Type: Test
>Reporter: Chris Douglas
>Priority: Minor
> Attachments: MAPREDUCE-7011.000.patch
>
>
> {{TestClientDistributedCacheManager}} sets up some local directories to check
> the visibility set for dependencies, given their filesystem permissions.
> However, if it is run in an environment where the scratch directory is not
> itself PUBLIC ({{ClientDistributedCacheManager::isPublic}}), then it will
> fail.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7011) TestClientDistributedCacheManager::testDetermineCacheVisibilities assumes all parent dirs set other exec
[
https://issues.apache.org/jira/browse/MAPREDUCE-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated MAPREDUCE-7011:
-
Attachment: MAPREDUCE-7011.000.patch
While it would clearly be better if the test validated the visibility logic
without requiring the local filesystem, that would likely require some
refactoring. v000 simply skips the test.
> TestClientDistributedCacheManager::testDetermineCacheVisibilities assumes all
> parent dirs set other exec
>
>
> Key: MAPREDUCE-7011
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7011
> Project: Hadoop Map/Reduce
> Issue Type: Test
>Reporter: Chris Douglas
>Priority: Minor
> Attachments: MAPREDUCE-7011.000.patch
>
>
> {{TestClientDistributedCacheManager}} sets up some local directories to check
> the visibility set for dependencies, given their filesystem permissions.
> However, if it is run in an environment where the scratch directory is not
> itself PUBLIC ({{ClientDistributedCacheManager::isPublic}}), then it will
> fail.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Moved] (MAPREDUCE-7011) TestClientDistributedCacheManager::testDetermineCacheVisibilities assumes all parent dirs set other exec
[
https://issues.apache.org/jira/browse/MAPREDUCE-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas moved HDFS-12844 to MAPREDUCE-7011:
-
Key: MAPREDUCE-7011 (was: HDFS-12844)
Project: Hadoop Map/Reduce (was: Hadoop HDFS)
> TestClientDistributedCacheManager::testDetermineCacheVisibilities assumes all
> parent dirs set other exec
>
>
> Key: MAPREDUCE-7011
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7011
> Project: Hadoop Map/Reduce
> Issue Type: Test
>Reporter: Chris Douglas
>Priority: Minor
>
> {{TestClientDistributedCacheManager}} sets up some local directories to check
> the visibility set for dependencies, given their filesystem permissions.
> However, if it is run in an environment where the scratch directory is not
> itself PUBLIC ({{ClientDistributedCacheManager::isPublic}}), then it will
> fail.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-5124) AM lacks flow control for task events
[ https://issues.apache.org/jira/browse/MAPREDUCE-5124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261063#comment-16261063 ] Jason Lowe commented on MAPREDUCE-5124: --- I also kicked Jenkins which mysteriously did not comment on this patch. It looks like there will be some checkstyle issues that should be addressed with exceeding 80 columns and other nitty things, but it'd also be good to see a unit test run. > AM lacks flow control for task events > - > > Key: MAPREDUCE-5124 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5124 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.0.3-alpha, 0.23.5 >Reporter: Jason Lowe >Assignee: Peter Bacsko > Attachments: MAPREDUCE-5124-001.patch, > MAPREDUCE-5124-CoalescingPOC-1.patch, MAPREDUCE-5124-CoalescingPOC2.patch, > MAPREDUCE-5124-CoalescingPOC3.patch, MAPREDUCE-5124-proto.2.txt, > MAPREDUCE-5124-prototype.txt > > > The AM does not have any flow control to limit the incoming rate of events > from tasks. If the AM is unable to keep pace with the rate of incoming > events for a sufficient period of time then it will eventually exhaust the > heap and crash. MAPREDUCE-5043 addressed a major bottleneck for event > processing, but the AM could still get behind if it's starved for CPU and/or > handling a very large job with tens of thousands of active tasks. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-5124) AM lacks flow control for task events
[
https://issues.apache.org/jira/browse/MAPREDUCE-5124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261050#comment-16261050
]
Jason Lowe commented on MAPREDUCE-5124:
---
Thanks for updating the patch! Sorry for the delay.
bq. We already know that counters are always sent from MR tasks, but what about
Tez?
Sorry for the confusion. I was referring to Tez as an example of a framework
that has counters but doesn't send them on every task update. I didn't mean to
imply Tez is calling the MR code directly. We can remove the code that checks
if the counters are null to simplify things since you found they can't be null
in practice for other reasons.
This comment was not addressed which will cover the case of a rogue task trying
to heartbeat:
{quote}
The listener should check if we fail to lookup the ref for the specified
attempt ID and throw an exception with a useful message.
{quote}
The EventHandler import added in TaskAttemptListenerImpl is unused.
Nit: This whitespace change is more harmful than helpful to readability, IMO.
{noformat}
@@ -427,6 +437,8 @@ public AMFeedback statusUpdate(TaskAttemptID taskAttemptID,
}
}
+
+
// Task sends the information about the nextRecordRange to the TT
//TODO: The following are not needed here, but needed to be set somewhere
inside AppMaster.
{noformat}
Nit: In StatusUpdater#transition newReportedStatus does not need a separate
declaration as the code can declare it when it is assigned.
Normally listener would be set to null after closing to avoid a potential
double-close:
{code}
public void after() throws IOException {
if (listener != null) {
listener.close();
}
}
{code}
> AM lacks flow control for task events
> -
>
> Key: MAPREDUCE-5124
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5124
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am
>Affects Versions: 2.0.3-alpha, 0.23.5
>Reporter: Jason Lowe
>Assignee: Peter Bacsko
> Attachments: MAPREDUCE-5124-001.patch,
> MAPREDUCE-5124-CoalescingPOC-1.patch, MAPREDUCE-5124-CoalescingPOC2.patch,
> MAPREDUCE-5124-CoalescingPOC3.patch, MAPREDUCE-5124-proto.2.txt,
> MAPREDUCE-5124-prototype.txt
>
>
> The AM does not have any flow control to limit the incoming rate of events
> from tasks. If the AM is unable to keep pace with the rate of incoming
> events for a sufficient period of time then it will eventually exhaust the
> heap and crash. MAPREDUCE-5043 addressed a major bottleneck for event
> processing, but the AM could still get behind if it's starved for CPU and/or
> handling a very large job with tens of thousands of active tasks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-5124) AM lacks flow control for task events
[ https://issues.apache.org/jira/browse/MAPREDUCE-5124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16260961#comment-16260961 ] Peter Bacsko commented on MAPREDUCE-5124: - ping [~jlowe] > AM lacks flow control for task events > - > > Key: MAPREDUCE-5124 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5124 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.0.3-alpha, 0.23.5 >Reporter: Jason Lowe >Assignee: Peter Bacsko > Attachments: MAPREDUCE-5124-001.patch, > MAPREDUCE-5124-CoalescingPOC-1.patch, MAPREDUCE-5124-CoalescingPOC2.patch, > MAPREDUCE-5124-CoalescingPOC3.patch, MAPREDUCE-5124-proto.2.txt, > MAPREDUCE-5124-prototype.txt > > > The AM does not have any flow control to limit the incoming rate of events > from tasks. If the AM is unable to keep pace with the rate of incoming > events for a sufficient period of time then it will eventually exhaust the > heap and crash. MAPREDUCE-5043 addressed a major bottleneck for event > processing, but the AM could still get behind if it's starved for CPU and/or > handling a very large job with tens of thousands of active tasks. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-5907) Improve getSplits() performance for fs implementations that can utilize performance gains from recursive listing
[
https://issues.apache.org/jira/browse/MAPREDUCE-5907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16260506#comment-16260506
]
Hadoop QA commented on MAPREDUCE-5907:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} MAPREDUCE-5907 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | MAPREDUCE-5907 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12648040/MAPREDUCE-5907-3.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7230/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Improve getSplits() performance for fs implementations that can utilize
> performance gains from recursive listing
>
>
> Key: MAPREDUCE-5907
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5907
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Affects Versions: 2.4.0
>Reporter: Sumit Kumar
>Assignee: Sumit Kumar
> Labels: BB2015-05-TBR
> Attachments: MAPREDUCE-5907-2.patch, MAPREDUCE-5907-3.patch,
> MAPREDUCE-5907.patch
>
>
> FileInputFormat (both mapreduce and mapred implementations) use recursive
> listing while calculating splits. They however do this by doing listing level
> by level. That means to discover files in /foo/bar means they do listing at
> /foo/bar first to get the immediate children, then make the same call on all
> immediate children for /foo/bar to discover their immediate children and so
> on. This doesn't scale well for object store based fs implementations like s3
> and swift because every listStatus call ends up being a webservice call to
> backend. In cases where large number of files are considered for input, this
> makes getSplits() call slow.
> This patch adds a new set of recursive list apis that gives opportunity to
> the fs implementations to optimize. The behavior remains the same for other
> implementations (that is a default implementation is provided for other fs so
> they don't have to implement anything new). However for objectstore based fs
> implementations it provides a simple change to include recursive flag as true
> (as shown in the patch) to improve listing performance.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
