[jira] [Commented] (MAPREDUCE-6728) Give fetchers hint when ShuffleHandler rejects a shuffling connection
[
https://issues.apache.org/jira/browse/MAPREDUCE-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15756066#comment-15756066
]
Hadoop QA commented on MAPREDUCE-6728:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 27s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
38s {color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 6s
{color} | {color:green} branch-2.8 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 15s
{color} | {color:green} branch-2.8 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 48s
{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
29s {color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} branch-2.8 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} branch-2.8 passed with JDK v1.7.0_121 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 3s
{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 15s
{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 15s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 22s
{color} | {color:red} hadoop-mapreduce-project/hadoop-mapreduce-client: The
patch generated 2 new + 146 unchanged - 2 fixed = 148 total (was 148) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 4s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.8.0_111. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 19s
{color} | {color:green} hadoop-mapreduce-client-shuffle in the patch passed
with JDK v1.8.0_111. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 21s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 21s
{color} | {color:green} hadoop-mapreduce-client-shuffle in the patch passed
with JDK v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
[jira] [Updated] (MAPREDUCE-6728) Give fetchers hint when ShuffleHandler rejects a shuffling connection
[ https://issues.apache.org/jira/browse/MAPREDUCE-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-6728: - Status: Patch Available (was: Reopened) > Give fetchers hint when ShuffleHandler rejects a shuffling connection > - > > Key: MAPREDUCE-6728 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6728 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Haibo Chen >Assignee: Haibo Chen > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: MAPREDUCE-6728-branch-2.8.06.patch, > mapreduce6728.001.patch, mapreduce6728.002.patch, mapreduce6728.003.patch, > mapreduce6728.004.patch, mapreduce6728.005.patch, mapreduce6728.006.patch, > mapreduce6728.branch-2.8.patch, mapreduce6728.prelim.patch > > > If # of open shuffle connection to a node goes over the max, ShuffleHandler > closes the connection immediately without giving fetchers any hint of the > reason, which causes fetchers to fail due to exceptions > java.net.SocketException: Unexpected end of file from server > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:772) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323) > at > java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:323) > at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193) > OR > java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:196) > at java.net.SocketInputStream.read(SocketInputStream.java:122) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) > at java.io.BufferedInputStream.read1(BufferedInputStream.java:275) > at java.io.BufferedInputStream.read(BufferedInputStream.java:334) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323) > at > java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java > Such failures are counted as fetcher failures -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Reopened] (MAPREDUCE-6728) Give fetchers hint when ShuffleHandler rejects a shuffling connection
[ https://issues.apache.org/jira/browse/MAPREDUCE-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter reopened MAPREDUCE-6728: -- I'm reopening so that Jenkins can pick up the 2.8 patch > Give fetchers hint when ShuffleHandler rejects a shuffling connection > - > > Key: MAPREDUCE-6728 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6728 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Haibo Chen >Assignee: Haibo Chen > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: MAPREDUCE-6728-branch-2.8.06.patch, > mapreduce6728.001.patch, mapreduce6728.002.patch, mapreduce6728.003.patch, > mapreduce6728.004.patch, mapreduce6728.005.patch, mapreduce6728.006.patch, > mapreduce6728.branch-2.8.patch, mapreduce6728.prelim.patch > > > If # of open shuffle connection to a node goes over the max, ShuffleHandler > closes the connection immediately without giving fetchers any hint of the > reason, which causes fetchers to fail due to exceptions > java.net.SocketException: Unexpected end of file from server > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:772) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323) > at > java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:323) > at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193) > OR > java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:196) > at java.net.SocketInputStream.read(SocketInputStream.java:122) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) > at java.io.BufferedInputStream.read1(BufferedInputStream.java:275) > at java.io.BufferedInputStream.read(BufferedInputStream.java:334) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323) > at > java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java > Such failures are counted as fetcher failures -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache
[ https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755575#comment-15755575 ] Chris Trezzo commented on MAPREDUCE-5951: - I have filed MAPREDUCE-6825 and MAPREDUCE-6824 to address the checkstyle issues around methods that are too long. > Add support for the YARN Shared Cache > - > > Key: MAPREDUCE-5951 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Chris Trezzo >Assignee: Chris Trezzo > Labels: BB2015-05-TBR > Attachments: MAPREDUCE-5951-trunk-v1.patch, > MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, > MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v13.patch, > MAPREDUCE-5951-trunk-v14.patch, MAPREDUCE-5951-trunk-v15.patch, > MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch, > MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, > MAPREDUCE-5951-trunk-v6.patch, MAPREDUCE-5951-trunk-v7.patch, > MAPREDUCE-5951-trunk-v8.patch, MAPREDUCE-5951-trunk-v9.patch, > MAPREDUCE-5951-trunk.016.patch, MAPREDUCE-5951-trunk.017.patch, > MAPREDUCE-5951-trunk.018.patch > > > Implement the necessary changes so that the MapReduce application can > leverage the new YARN shared cache (i.e. YARN-1492). > Specifically, allow per-job configuration so that MapReduce jobs can specify > which set of resources they would like to cache (i.e. jobjar, libjars, > archives, files). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6825) YARNRunner#createApplicationSubmissionContext method is longer than 150 lines
Chris Trezzo created MAPREDUCE-6825:
---
Summary: YARNRunner#createApplicationSubmissionContext method is
longer than 150 lines
Key: MAPREDUCE-6825
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6825
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Chris Trezzo
Priority: Trivial
bq.
./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java:341:
public ApplicationSubmissionContext createApplicationSubmissionContext(:3:
Method length is 249 lines (max allowed is 150).
{{YARNRunner#createApplicationSubmissionContext}} is longer than 150 lines and
needs to be refactored.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6824) TaskAttemptImpl#createCommonContainerLaunchContext is longer than 150 lines
Chris Trezzo created MAPREDUCE-6824:
---
Summary: TaskAttemptImpl#createCommonContainerLaunchContext is
longer than 150 lines
Key: MAPREDUCE-6824
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6824
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Chris Trezzo
Priority: Trivial
bq.
./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java:752:
private static ContainerLaunchContext createCommonContainerLaunchContext(:3:
Method length is 172 lines (max allowed is 150).
{{TaskAttemptImpl#createCommonContainerLaunchContext}} is longer than 150 lines
and needs to be refactored.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache
[
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755545#comment-15755545
]
Hadoop QA commented on MAPREDUCE-5951:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 16s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 9s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 0s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 26s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 3s
{color} | {color:green} hadoop-mapreduce-project_hadoop-mapreduce-client
generated 0 new + 359 unchanged - 5 fixed = 359 total (was 364) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 42s
{color} | {color:red} hadoop-mapreduce-project/hadoop-mapreduce-client: The
patch generated 3 new + 1086 unchanged - 11 fixed = 1089 total (was 1097)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
3s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 3s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 33s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 55s
{color} | {color:green} hadoop-mapreduce-client-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 52s
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 110m 54s
{color} | {color:green} hadoop-mapreduce-client-jobclient in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 167m 29s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12843632/MAPREDUCE-5951-trunk.018.patch
|
| JIRA Issue | MAPREDUCE-5951 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 618f3b149fde 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
[jira] [Commented] (MAPREDUCE-6823) FileOutputFormat to support configurable FileOutputCommitter factory
[
https://issues.apache.org/jira/browse/MAPREDUCE-6823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755515#comment-15755515
]
Hadoop QA commented on MAPREDUCE-6823:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 17m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 45s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
39s {color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 39s
{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
30s {color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 57s
{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s {color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
16s {color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} HADOOP-13345 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 17s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 7m 17s {color}
| {color:red} root generated 2 new + 716 unchanged - 0 fixed = 718 total (was
716) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 35s
{color} | {color:red} root: The patch generated 48 new + 66 unchanged - 1 fixed
= 114 total (was 67) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
34s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 43 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 16s
{color} | {color:red} hadoop-aws in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 52s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 38s
{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 57m 54s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12843640/HADOOP-13786-HADOOP-13345-001.patch
|
| JIRA Issue | MAPREDUCE-6823 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 313569e92b73 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24
10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HADOOP-13345 / c7885de |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| javac |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6848/artifact/patchprocess/diff-compile-javac-root.txt
|
| checkstyle |
https://builds.apache.org/job/PreCo
[jira] [Comment Edited] (MAPREDUCE-5951) Add support for the YARN Shared Cache
[ https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755192#comment-15755192 ] Chris Trezzo edited comment on MAPREDUCE-5951 at 12/16/16 8:55 PM: --- Attached is v18 to fix the one checkstyle issue. There are three outstanding checkstyle issues that I am leaning towards not fixing as part of the patch. Please let me know your thoughts. They are the following: bq. ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java:752: private static ContainerLaunchContext createCommonContainerLaunchContext(:3: Method length is 172 lines (max allowed is 150). This patch barely touches this method so it seems wrong to refactor the method as part of this jira. I can file a separate jira to fix this. bq. ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java:341: public ApplicationSubmissionContext createApplicationSubmissionContext(:3: Method length is 249 lines (max allowed is 150). The same reasoning applies to this warning as the previous issue. I can file a separate jira to fix this. bq. ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java:572: private static void parseDistributedCacheArtifacts(:23: More than 7 parameters (found 8). This issue was caused by this patch adding an additional parameter to this method. I can fix the number of parameter issues, but that forces me to touch three existing calls to the deprecated DistributedCache api, which would fix 1 warning but create 3 new ones. It is a larger change to not use the deprecated api because the existing code is not set up to use it, furthermore use of the deprecated api is currently widespread in this code. My thoughts are that I will leave this warning as is. was (Author: ctrezzo): Attached is v18 to fix the one checkstyle issue. There are three outstanding checkstyle issues that I am leaning towards not fixing as part of the patch. Please let me know your thoughts. They are the following: bq. ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java:752: private static ContainerLaunchContext createCommonContainerLaunchContext(:3: Method length is 172 lines (max allowed is 150). This patch barely touches this method so it seems wrong to refactor the patch as part of this jira. I can file a separate jira to fix this. bq. ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java:341: public ApplicationSubmissionContext createApplicationSubmissionContext(:3: Method length is 249 lines (max allowed is 150). The same reasoning applies to this warning as the previous issue. I can file a separate jira to fix this. bq. ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java:572: private static void parseDistributedCacheArtifacts(:23: More than 7 parameters (found 8). This issue was caused by this patch adding an additional parameter to this method. I can fix the number of parameter issues, but that forces me to touch three existing calls to the deprecated DistributedCache api, which would fix 1 warning but create 3 new ones. It is a larger change to not use the deprecated api because the existing code is not set up to use it, furthermore use of the deprecated api is currently widespread in this code. My thoughts are that I will leave this warning as is. > Add support for the YARN Shared Cache > - > > Key: MAPREDUCE-5951 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Chris Trezzo >Assignee: Chris Trezzo > Labels: BB2015-05-TBR > Attachments: MAPREDUCE-5951-trunk-v1.patch, > MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, > MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v13.patch, > MAPREDUCE-5951-trunk-v14.patch, MAPREDUCE-5951-trunk-v15.patch, > MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch, > MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, > MAPREDUCE-5951-trunk-v6.patch, MAPREDUCE-5951-trunk-v7.patch, > MAPREDUCE-5951-trunk-v8.patch, MAPREDUCE-5951-trunk-v9.patch, > MAPREDUCE-5951-trunk.016.patch, MAPREDUCE-5951-trunk.017.patch, > MAPREDUCE-5951-trunk.018.patch > > > Implement the necessary changes so that the MapReduce application can > l
[jira] [Updated] (MAPREDUCE-6823) FileOutputFormat to support configurable FileOutputCommitter factory
[
https://issues.apache.org/jira/browse/MAPREDUCE-6823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated MAPREDUCE-6823:
--
Attachment: HADOOP-13786-HADOOP-13345-001.patch
This is the initial HADOOP-13786 001 PoC patch, to give the MR bit of the code
some testing too. It adds a new factory for FileOutputFormatter to use when
creating committers; the default one returns {{FileOutputCommitter}} instances
as normal; a special S3a one in {{hadoop-aws}} to handle S3a specific
operations.
Now, the other way to do this (given the need to keep the s3a code in the s3a
module) would be to allow a notion of a new algorithm, one which relayed to an
implementation of an interface. That would hand a problem not addressed here:
how to address subclasses of {{FileOutputFormat}} with custom subclasses of
{{FileOutputCommitter}}, and make it easier to add committers for other
non-FS-destinations, namely the other object stores.
However, it would be a more significant change to {{FileOutputCommitter}}; I
could go that way, but it'd need support before I started.
> FileOutputFormat to support configurable FileOutputCommitter factory
>
>
> Key: MAPREDUCE-6823
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6823
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 3.0.0-alpha2
> Environment: Targeting S3 as the output of work
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13786-HADOOP-13345-001.patch
>
>
> In HADOOP-13786 I'm adding a custom subclass for FileOutputFormat, one which
> can talk direct to the S3A Filesystem for more efficient operations, better
> failure modes, and, most critically, as part of HADOOP-13345, atomic commit
> of output. The normal committer relies on directory rename() being atomic for
> this; for S3 we don't have that luxury.
> To support a custom committer, we need to be able to tell FileOutputFormat
> (and implicitly, all subclasses which don't have their own custom committer),
> to use our new {{S3AOutputCommitter}}.
> I propose:
> # {{FileOutputFormat}} takes a factory to create committers.
> # The factory to take a URI and {{TaskAttemptContext}} and return a committer
> # the default implementation always returns a {{FileOutputCommitter}}
> # A configuration option allows a new factory to be named
> # An {{S3AOutputCommitterFactory}} to return a {{FileOutputCommitter}} or
> new {{S3AOutputCommitter}} depending upon the URI of the destination.
> Note that MRv1 already supports configurable committers; this is only the V2
> API
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6823) FileOutputFormat to support configurable FileOutputCommitter factory
[
https://issues.apache.org/jira/browse/MAPREDUCE-6823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated MAPREDUCE-6823:
--
Status: Patch Available (was: Open)
> FileOutputFormat to support configurable FileOutputCommitter factory
>
>
> Key: MAPREDUCE-6823
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6823
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 3.0.0-alpha2
> Environment: Targeting S3 as the output of work
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13786-HADOOP-13345-001.patch
>
>
> In HADOOP-13786 I'm adding a custom subclass for FileOutputFormat, one which
> can talk direct to the S3A Filesystem for more efficient operations, better
> failure modes, and, most critically, as part of HADOOP-13345, atomic commit
> of output. The normal committer relies on directory rename() being atomic for
> this; for S3 we don't have that luxury.
> To support a custom committer, we need to be able to tell FileOutputFormat
> (and implicitly, all subclasses which don't have their own custom committer),
> to use our new {{S3AOutputCommitter}}.
> I propose:
> # {{FileOutputFormat}} takes a factory to create committers.
> # The factory to take a URI and {{TaskAttemptContext}} and return a committer
> # the default implementation always returns a {{FileOutputCommitter}}
> # A configuration option allows a new factory to be named
> # An {{S3AOutputCommitterFactory}} to return a {{FileOutputCommitter}} or
> new {{S3AOutputCommitter}} depending upon the URI of the destination.
> Note that MRv1 already supports configurable committers; this is only the V2
> API
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6704) Container fail to launch for mapred application
[
https://issues.apache.org/jira/browse/MAPREDUCE-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755241#comment-15755241
]
Akira Ajisaka commented on MAPREDUCE-6704:
--
+1 again.
> Container fail to launch for mapred application
> ---
>
> Key: MAPREDUCE-6704
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6704
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Blocker
> Attachments: 0001-MAPREDUCE-6704.patch, 0001-YARN-5026.patch,
> ClusterSetup.html, MAPREDUCE-6704.0002.patch, MAPREDUCE-6704.0002.patch,
> MR-6704-branch2.8.tar.gz, MR-6704-trunk-tempPatch.tar.gz,
> MR-6704-trunk.tar.gz, SingleCluster.html, container-whitelist-env-wip.patch,
> temp.patch
>
>
> Container fail to launch for mapred application.
> As part for launch script {{HADOOP_MAPRED_HOME}} default value is not set
> .After
> https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c
>{{HADOOP_MAPRED_HOME}} is not able to get from {{builder.environment()}}
> since {{DefaultContainerExecutor#buildCommandExecutor}} sets inherit to false.
> {noformat}
> 16/05/02 09:16:05 INFO mapreduce.Job: Job job_1462155939310_0004 failed with
> state FAILED due to: Application application_1462155939310_0004 failed 2
> times due to AM Container for appattempt_1462155939310_0004_02 exited
> with exitCode: 1
> Failing this attempt.Diagnostics: Exception from container-launch.
> Container id: container_1462155939310_0004_02_01
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:946)
> at org.apache.hadoop.util.Shell.run(Shell.java:850)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:227)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:385)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:281)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:89)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> Error: Could not find or load main class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-5951) Add support for the YARN Shared Cache
[ https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Trezzo updated MAPREDUCE-5951: Attachment: MAPREDUCE-5951-trunk.018.patch Attached is v18 to fix the one checkstyle issue. There are three outstanding checkstyle issues that I am leaning towards not fixing as part of the patch. Please let me know your thoughts. They are the following: bq. ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java:752: private static ContainerLaunchContext createCommonContainerLaunchContext(:3: Method length is 172 lines (max allowed is 150). This patch barely touches this method so it seems wrong to refactor the patch as part of this jira. I can file a separate jira to fix this. bq. ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java:341: public ApplicationSubmissionContext createApplicationSubmissionContext(:3: Method length is 249 lines (max allowed is 150). The same reasoning applies to this warning as the previous issue. I can file a separate jira to fix this. bq. ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java:572: private static void parseDistributedCacheArtifacts(:23: More than 7 parameters (found 8). This issue was caused by this patch adding an additional parameter to this method. I can fix the number of parameter issues, but that forces me to touch three existing calls to the deprecated DistributedCache api, which would fix 1 warning but create 3 new ones. It is a larger change to not use the deprecated api because the existing code is not set up to use it, furthermore use of the deprecated api is currently widespread in this code. My thoughts are that I will leave this warning as is. > Add support for the YARN Shared Cache > - > > Key: MAPREDUCE-5951 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Chris Trezzo >Assignee: Chris Trezzo > Labels: BB2015-05-TBR > Attachments: MAPREDUCE-5951-trunk-v1.patch, > MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, > MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v13.patch, > MAPREDUCE-5951-trunk-v14.patch, MAPREDUCE-5951-trunk-v15.patch, > MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch, > MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, > MAPREDUCE-5951-trunk-v6.patch, MAPREDUCE-5951-trunk-v7.patch, > MAPREDUCE-5951-trunk-v8.patch, MAPREDUCE-5951-trunk-v9.patch, > MAPREDUCE-5951-trunk.016.patch, MAPREDUCE-5951-trunk.017.patch, > MAPREDUCE-5951-trunk.018.patch > > > Implement the necessary changes so that the MapReduce application can > leverage the new YARN shared cache (i.e. YARN-1492). > Specifically, allow per-job configuration so that MapReduce jobs can specify > which set of resources they would like to cache (i.e. jobjar, libjars, > archives, files). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
