date:20161006

[jira] [Commented] (MAPREDUCE-6728) Give fetchers hint when ShuffleHandler rejects a shuffling connection

2016-10-06 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15553396#comment-15553396
 ] 

Hadoop QA commented on MAPREDUCE-6728:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 49s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 
0s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 49s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
31s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 51s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s 
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 49s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 49s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 29s 
{color} | {color:red} hadoop-mapreduce-project/hadoop-mapreduce-client: The 
patch generated 1 new + 278 unchanged - 3 fixed = 279 total (was 281) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 49s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 20s 
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 19s 
{color} | {color:green} hadoop-mapreduce-client-shuffle in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
23s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 53s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12832045/mapreduce6728.005.patch
 |
| JIRA Issue | MAPREDUCE-6728 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 77600b4adbb5 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 48b9d5f |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6756/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6756/testReport/ |
| modules | C: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapredu

[jira] [Updated] (MAPREDUCE-6728) Give fetchers hint when ShuffleHandler rejects a shuffling connection

2016-10-06 Thread Haibo Chen (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haibo Chen updated MAPREDUCE-6728:
--
Status: Patch Available  (was: Open)

> Give fetchers hint when ShuffleHandler rejects a shuffling connection
> -
>
> Key: MAPREDUCE-6728
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6728
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: mrv2
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: mapreduce6728.001.patch, mapreduce6728.002.patch, 
> mapreduce6728.003.patch, mapreduce6728.004.patch, mapreduce6728.005.patch, 
> mapreduce6728.prelim.patch
>
>
> If # of open shuffle connection to a node goes over the max, ShuffleHandler 
> closes the connection immediately without giving fetchers any hint of the 
> reason, which causes fetchers to fail due to exceptions 
> java.net.SocketException: Unexpected end of file from server
>   at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:772)
>   at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
>   at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769)
>   at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
>   at 
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
>   at 
> java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:323)
>   at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193)
> OR 
> java.net.SocketException: Connection reset
>   at java.net.SocketInputStream.read(SocketInputStream.java:196)
>   at java.net.SocketInputStream.read(SocketInputStream.java:122)
>   at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>   at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
>   at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>   at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687)
>   at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
>   at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769)
>   at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
>   at 
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
>   at 
> java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java
> Such failures are counted as fetcher failures



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6728) Give fetchers hint when ShuffleHandler rejects a shuffling connection

2016-10-06 Thread Haibo Chen (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haibo Chen updated MAPREDUCE-6728:
--
Attachment: mapreduce6728.005.patch

Adding another patch to address Daniel's latest comments.

> Give fetchers hint when ShuffleHandler rejects a shuffling connection
> -
>
> Key: MAPREDUCE-6728
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6728
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: mrv2
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: mapreduce6728.001.patch, mapreduce6728.002.patch, 
> mapreduce6728.003.patch, mapreduce6728.004.patch, mapreduce6728.005.patch, 
> mapreduce6728.prelim.patch
>
>
> If # of open shuffle connection to a node goes over the max, ShuffleHandler 
> closes the connection immediately without giving fetchers any hint of the 
> reason, which causes fetchers to fail due to exceptions 
> java.net.SocketException: Unexpected end of file from server
>   at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:772)
>   at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
>   at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769)
>   at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
>   at 
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
>   at 
> java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:323)
>   at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193)
> OR 
> java.net.SocketException: Connection reset
>   at java.net.SocketInputStream.read(SocketInputStream.java:196)
>   at java.net.SocketInputStream.read(SocketInputStream.java:122)
>   at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>   at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
>   at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>   at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687)
>   at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
>   at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769)
>   at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
>   at 
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
>   at 
> java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java
> Such failures are counted as fetcher failures



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6790) Update jackson from 1.9.13 to 2.x in hadoop-mapreduce

2016-10-06 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15552752#comment-15552752
 ] 

Hadoop QA commented on MAPREDUCE-6790:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 
57s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 39s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
31s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 28s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
44s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
52s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s 
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
5s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 36s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 36s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
30s {color} | {color:green} hadoop-mapreduce-project/hadoop-mapreduce-client: 
The patch generated 0 new + 413 unchanged - 2 fixed = 413 total (was 415) 
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 17s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s 
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 2s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 9s 
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 50s 
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 110m 42s 
{color} | {color:green} hadoop-mapreduce-client-jobclient in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
26s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 147m 22s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12831477/HADOOP-13676.01.patch 
|
| JIRA Issue | MAPREDUCE-6790 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  xml  findbugs  checkstyle  |
| uname | Linux 1293ed95fc3e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 2cc841f |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6755/testRepo

[jira] [Commented] (MAPREDUCE-6728) Give fetchers hint when ShuffleHandler rejects a shuffling connection

2016-10-06 Thread Daniel Templeton (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15552572#comment-15552572
 ] 

Daniel Templeton commented on MAPREDUCE-6728:
-

Sorry to string this out.  I just caught two more items that should be 
addressed:

# This comment: {code}  /**
   * This should be kept in sync with ShuffleHandler.FETCH_RETRY_DELAY.
   */
  private static final long FETCH_RETRY_DELAY_DEFAULT = 1000L;{code} either 
should not be a javadoc comment or should start with a sentence that is an 
appropriate javadoc comment.
# The {{TestShuffleHandler}} should test that the Retry-after header is set to 
a non-negative integer.

> Give fetchers hint when ShuffleHandler rejects a shuffling connection
> -
>
> Key: MAPREDUCE-6728
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6728
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: mrv2
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: mapreduce6728.001.patch, mapreduce6728.002.patch, 
> mapreduce6728.003.patch, mapreduce6728.004.patch, mapreduce6728.prelim.patch
>
>
> If # of open shuffle connection to a node goes over the max, ShuffleHandler 
> closes the connection immediately without giving fetchers any hint of the 
> reason, which causes fetchers to fail due to exceptions 
> java.net.SocketException: Unexpected end of file from server
>   at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:772)
>   at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
>   at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769)
>   at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
>   at 
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
>   at 
> java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:323)
>   at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193)
> OR 
> java.net.SocketException: Connection reset
>   at java.net.SocketInputStream.read(SocketInputStream.java:196)
>   at java.net.SocketInputStream.read(SocketInputStream.java:122)
>   at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>   at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
>   at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>   at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687)
>   at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
>   at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769)
>   at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
>   at 
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
>   at 
> java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java
> Such failures are counted as fetcher failures



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6728) Give fetchers hint when ShuffleHandler rejects a shuffling connection

2016-10-06 Thread Daniel Templeton (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Templeton updated MAPREDUCE-6728:

Status: Open  (was: Patch Available)

> Give fetchers hint when ShuffleHandler rejects a shuffling connection
> -
>
> Key: MAPREDUCE-6728
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6728
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: mrv2
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: mapreduce6728.001.patch, mapreduce6728.002.patch, 
> mapreduce6728.003.patch, mapreduce6728.004.patch, mapreduce6728.prelim.patch
>
>
> If # of open shuffle connection to a node goes over the max, ShuffleHandler 
> closes the connection immediately without giving fetchers any hint of the 
> reason, which causes fetchers to fail due to exceptions 
> java.net.SocketException: Unexpected end of file from server
>   at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:772)
>   at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
>   at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769)
>   at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
>   at 
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
>   at 
> java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:323)
>   at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193)
> OR 
> java.net.SocketException: Connection reset
>   at java.net.SocketInputStream.read(SocketInputStream.java:196)
>   at java.net.SocketInputStream.read(SocketInputStream.java:122)
>   at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>   at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
>   at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>   at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687)
>   at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
>   at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769)
>   at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
>   at 
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
>   at 
> java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java
> Such failures are counted as fetcher failures



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6750) TestHSAdminServer.testRefreshSuperUserGroups is failing

2016-10-06 Thread Ray Chiang (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15552497#comment-15552497
 ] 

Ray Chiang commented on MAPREDUCE-6750:
---

Just adding a note.  Before this fix, the test breaks with the following error:

java.lang.AssertionError: Unexpected exception thrown: 
org.apache.hadoop.security.authorize.AuthorizationException: User: superuser is 
not allowed to impersonate regularUser expected null, but 
was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotNull(Assert.java:664)
at org.junit.Assert.assertNull(Assert.java:646)
at 
org.apache.hadoop.mapreduce.v2.hs.server.TestHSAdminServer.testRefreshSuperUserGroups(TestHSAdminServer.java:236)


> TestHSAdminServer.testRefreshSuperUserGroups is failing
> ---
>
> Key: MAPREDUCE-6750
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6750
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: test
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
>Priority: Minor
> Fix For: 2.8.0, 3.0.0-alpha1
>
> Attachments: MAPREDUCE-6750.patch
>
>
> HADOOP-13442 changed {{AccessControlList}} to call {{getGroups()}} instead of 
> {{getGroupNames()}}. It should work if the mocks are updated to stub the 
> right method and return the right type.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6773) Implement RM Container Reuse Requestor to handle the reuse containers for resource requests

2016-10-06 Thread Devaraj K (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15552367#comment-15552367
 ] 

Devaraj K commented on MAPREDUCE-6773:
--

The failed test and javac warnings are not related to this patch.

> Implement RM Container Reuse Requestor to handle the reuse containers for 
> resource requests
> ---
>
> Key: MAPREDUCE-6773
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6773
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>  Components: applicationmaster, mrv2
>Reporter: Devaraj K
>Assignee: Devaraj K
> Attachments: MAPREDUCE-6773-v0.patch, MAPREDUCE-6773-v1.patch, 
> MAPREDUCE-6773-v2.patch
>
>
> Add RM Container Reuse Requestor which handles the reuse containers against 
> the Job reource requests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Moved] (MAPREDUCE-6790) Update jackson from 1.9.13 to 2.x in hadoop-mapreduce

2016-10-06 Thread Akira Ajisaka (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka moved HADOOP-13676 to MAPREDUCE-6790:
---

Target Version/s:   (was: 3.0.0-alpha2)
 Component/s: (was: build)
  build
 Key: MAPREDUCE-6790  (was: HADOOP-13676)
 Project: Hadoop Map/Reduce  (was: Hadoop Common)

> Update jackson from 1.9.13 to 2.x in hadoop-mapreduce
> -
>
> Key: MAPREDUCE-6790
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6790
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: build
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
> Attachments: HADOOP-13676.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6789) Fix TestAMWebApp failure

2016-10-06 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15551178#comment-15551178
 ] 

Hudson commented on MAPREDUCE-6789:
---

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10552 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10552/])
MAPREDUCE-6789. Fix TestAMWebApp failure. Contributed by Daniel (aajisaka: rev 
272a21747e8a89b6daccc19b71c21de3d17b8d62)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/ProxyUriUtils.java
* (edit) 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAMWebApp.java


> Fix TestAMWebApp failure
> 
>
> Key: MAPREDUCE-6789
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6789
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: test
>Reporter: Akira Ajisaka
>Assignee: Daniel Templeton
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: MAPREDUCE-6789.001.patch
>
>
> TestAMWebApp.testMRWebAppRedirection fails.
> {noformat}
> Running org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebApp
> Tests run: 12, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 27.337 sec 
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebApp
> testMRWebAppRedirection(org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebApp)
>   Time elapsed: 0.854 sec  <<< FAILURE!
> org.junit.ComparisonFailure: 
> expected:<...ttp://9.9.9.9/proxy/[]application_0_/m...> but 
> was:<...ttp://9.9.9.9/proxy/[redirect/]application_0_/m...>
>   at org.junit.Assert.assertEquals(Assert.java:115)
>   at org.junit.Assert.assertEquals(Assert.java:144)
>   at 
> org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebApp.testMRWebAppRedirection(TestAMWebApp.java:253)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6740) Enforce mapreduce.task.timeout to be at least mapreduce.task.progress-report.interval

2016-10-06 Thread Karthik Kambatla (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated MAPREDUCE-6740:

   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 2.8.0
   Status: Resolved  (was: Patch Available)

Just committed to branch-2.8 as well. Thanks for posting the updated patch, 
[~haibochen]. 

> Enforce mapreduce.task.timeout to be at least 
> mapreduce.task.progress-report.interval
> -
>
> Key: MAPREDUCE-6740
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6740
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mr-am
>Affects Versions: 2.8.0
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Minor
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: mapreduce6740.001.patch, mapreduce6740.002.patch, 
> mapreduce6740.003.patch, mapreduce6740.004.patch, mapreduce6740.005.patch, 
> mapreduce6740.006.patch, mapreduce6740.007.patch, mapreduce6740.008.patch, 
> mapreduce6740.branch-2.8.patch
>
>
> MAPREDUCE-6242 makes task status update interval configurable to ease the 
> pressure on MR AM to process status updates, but it did not ensure that 
> mapreduce.task.timeout is no smaller than the configured value of task report 
> interval. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6789) Fix TestAMWebApp failure

2016-10-06 Thread Akira Ajisaka (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka updated MAPREDUCE-6789:
-
   Resolution: Fixed
Fix Version/s: 3.0.0-alpha2
   2.9.0
   Status: Resolved  (was: Patch Available)

Committed this to trunk and branch-2. Thanks [~templedf] for the quick fix!

> Fix TestAMWebApp failure
> 
>
> Key: MAPREDUCE-6789
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6789
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: test
>Reporter: Akira Ajisaka
>Assignee: Daniel Templeton
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: MAPREDUCE-6789.001.patch
>
>
> TestAMWebApp.testMRWebAppRedirection fails.
> {noformat}
> Running org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebApp
> Tests run: 12, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 27.337 sec 
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebApp
> testMRWebAppRedirection(org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebApp)
>   Time elapsed: 0.854 sec  <<< FAILURE!
> org.junit.ComparisonFailure: 
> expected:<...ttp://9.9.9.9/proxy/[]application_0_/m...> but 
> was:<...ttp://9.9.9.9/proxy/[redirect/]application_0_/m...>
>   at org.junit.Assert.assertEquals(Assert.java:115)
>   at org.junit.Assert.assertEquals(Assert.java:144)
>   at 
> org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebApp.testMRWebAppRedirection(TestAMWebApp.java:253)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6789) Fix TestAMWebApp failure

2016-10-06 Thread Akira Ajisaka (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka updated MAPREDUCE-6789:
-
Hadoop Flags: Reviewed
 Summary: Fix TestAMWebApp failure  (was: TestAMWebApp fails)

> Fix TestAMWebApp failure
> 
>
> Key: MAPREDUCE-6789
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6789
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: test
>Reporter: Akira Ajisaka
>Assignee: Daniel Templeton
> Attachments: MAPREDUCE-6789.001.patch
>
>
> TestAMWebApp.testMRWebAppRedirection fails.
> {noformat}
> Running org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebApp
> Tests run: 12, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 27.337 sec 
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebApp
> testMRWebAppRedirection(org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebApp)
>   Time elapsed: 0.854 sec  <<< FAILURE!
> org.junit.ComparisonFailure: 
> expected:<...ttp://9.9.9.9/proxy/[]application_0_/m...> but 
> was:<...ttp://9.9.9.9/proxy/[redirect/]application_0_/m...>
>   at org.junit.Assert.assertEquals(Assert.java:115)
>   at org.junit.Assert.assertEquals(Assert.java:144)
>   at 
> org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebApp.testMRWebAppRedirection(TestAMWebApp.java:253)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6728) Give fetchers hint when ShuffleHandler rejects a shuffling connection

[jira] [Updated] (MAPREDUCE-6728) Give fetchers hint when ShuffleHandler rejects a shuffling connection

[jira] [Updated] (MAPREDUCE-6728) Give fetchers hint when ShuffleHandler rejects a shuffling connection

[jira] [Commented] (MAPREDUCE-6790) Update jackson from 1.9.13 to 2.x in hadoop-mapreduce

[jira] [Commented] (MAPREDUCE-6728) Give fetchers hint when ShuffleHandler rejects a shuffling connection

[jira] [Updated] (MAPREDUCE-6728) Give fetchers hint when ShuffleHandler rejects a shuffling connection

[jira] [Commented] (MAPREDUCE-6750) TestHSAdminServer.testRefreshSuperUserGroups is failing

[jira] [Commented] (MAPREDUCE-6773) Implement RM Container Reuse Requestor to handle the reuse containers for resource requests

[jira] [Moved] (MAPREDUCE-6790) Update jackson from 1.9.13 to 2.x in hadoop-mapreduce

[jira] [Commented] (MAPREDUCE-6789) Fix TestAMWebApp failure

[jira] [Updated] (MAPREDUCE-6740) Enforce mapreduce.task.timeout to be at least mapreduce.task.progress-report.interval

[jira] [Updated] (MAPREDUCE-6789) Fix TestAMWebApp failure

[jira] [Updated] (MAPREDUCE-6789) Fix TestAMWebApp failure

13 matches

Site Navigation

Mail list logo

Footer information