[jira] [Commented] (MAPREDUCE-6958) Shuffle audit logger should log size of shuffle transfer

2017-09-19 Thread Eric Payne (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171858#comment-16171858
 ] 

Eric Payne commented on MAPREDUCE-6958:
---

Thanks [~jlowe]
The branch-2.8 patch LGTM.
+1

> Shuffle audit logger should log size of shuffle transfer
> 
>
> Key: MAPREDUCE-6958
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6958
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Minor
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: MAPREDUCE-6958.001.patch, MAPREDUCE-6958.002.patch, 
> MAPREDUCE-6958.003.patch, MAPREDUCE-6958-branch-2.002.patch, 
> MAPREDUCE-6958-branch-2.8.002.patch
>
>
> The shuffle audit logger currently logs the job ID and reducer ID but nothing 
> about the size of the requested transfer.  It calculates this as part of the 
> HTTP response headers, so it would be trivial to log the response size.  This 
> would be very valuable for debugging network traffic storms from the shuffle 
> handler.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6958) Shuffle audit logger should log size of shuffle transfer

2017-09-19 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171828#comment-16171828
 ] 

Hadoop QA commented on MAPREDUCE-6958:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} branch-2.8 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  9m 
 7s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
18s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
16s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
24s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
30s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
16s{color} | {color:green} branch-2.8 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
11s{color} | {color:red} 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle:
 The patch generated 1 new + 58 unchanged - 2 fixed = 59 total (was 60) {color} 
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
21s{color} | {color:green} hadoop-mapreduce-client-shuffle in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
17s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 23s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:c2d96dd |
| JIRA Issue | MAPREDUCE-6958 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12887902/MAPREDUCE-6958-branch-2.8.002.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 6ad070db71e2 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | branch-2.8 / b1e1f75 |
| Default Java | 1.7.0_151 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7144/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-shuffle.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7144/testReport/ |
| modules | C: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle
 U: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle
 |
| Console output | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7144/console |
| Powered by | Apache Yetus 0.5.0   http://yetus.apache.org |


This message was automatically generated.



> Shuffle audit logger should log size of shuffle transfer
> 
>
> Key: MAPREDUCE-6958
> URL:

[jira] [Commented] (MAPREDUCE-6958) Shuffle audit logger should log size of shuffle transfer

2017-09-19 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171800#comment-16171800
 ] 

Hudson commented on MAPREDUCE-6958:
---

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12912 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/12912/])
MAPREDUCE-6958. Shuffle audit logger should log size of shuffle (jlowe: rev 
3a20debddeac69596ceb5b36f8413529ea8570e6)
* (edit) 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java


> Shuffle audit logger should log size of shuffle transfer
> 
>
> Key: MAPREDUCE-6958
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6958
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Minor
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: MAPREDUCE-6958.001.patch, MAPREDUCE-6958.002.patch, 
> MAPREDUCE-6958.003.patch, MAPREDUCE-6958-branch-2.002.patch, 
> MAPREDUCE-6958-branch-2.8.002.patch
>
>
> The shuffle audit logger currently logs the job ID and reducer ID but nothing 
> about the size of the requested transfer.  It calculates this as part of the 
> HTTP response headers, so it would be trivial to log the response size.  This 
> would be very valuable for debugging network traffic storms from the shuffle 
> handler.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6958) Shuffle audit logger should log size of shuffle transfer

2017-09-19 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171764#comment-16171764
 ] 

Hudson commented on MAPREDUCE-6958:
---

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12910 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/12910/])
Revert "MAPREDUCE-6958. Shuffle audit logger should log size of shuffle (jlowe: 
rev ea845ba58c585647c4be8d30d9b814f098e34a12)
* (edit) 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java


> Shuffle audit logger should log size of shuffle transfer
> 
>
> Key: MAPREDUCE-6958
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6958
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Minor
> Fix For: 3.0.0-beta1
>
> Attachments: MAPREDUCE-6958.001.patch, MAPREDUCE-6958.002.patch, 
> MAPREDUCE-6958.003.patch, MAPREDUCE-6958-branch-2.002.patch
>
>
> The shuffle audit logger currently logs the job ID and reducer ID but nothing 
> about the size of the requested transfer.  It calculates this as part of the 
> HTTP response headers, so it would be trivial to log the response size.  This 
> would be very valuable for debugging network traffic storms from the shuffle 
> handler.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6958) Shuffle audit logger should log size of shuffle transfer

2017-09-18 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170856#comment-16170856
 ] 

Hadoop QA commented on MAPREDUCE-6958:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m 
45s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
34s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
15s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
11s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
18s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
28s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
12s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
 9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
17s{color} | {color:green} hadoop-mapreduce-client-shuffle in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 34s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:eaf5c66 |
| JIRA Issue | MAPREDUCE-6958 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12887770/MAPREDUCE-6958-branch-2.002.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 92227d8c29f4 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | branch-2 / 6aaca3d |
| Default Java | 1.7.0_151 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7143/testReport/ |
| modules | C: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle
 U: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle
 |
| Console output | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7143/console |
| Powered by | Apache Yetus 0.5.0   http://yetus.apache.org |


This message was automatically generated.



> Shuffle audit logger should log size of shuffle transfer
> 
>
> Key: MAPREDUCE-6958
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6958
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority:

[jira] [Commented] (MAPREDUCE-6958) Shuffle audit logger should log size of shuffle transfer

2017-09-18 Thread Chris Douglas (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170837#comment-16170837
 ] 

Chris Douglas commented on MAPREDUCE-6958:
--

bq. Since 3.0 hasn't officially shipped yet, I propose to revert the 003 patch 
I committed to trunk and branch-3.0 and instead commit patch version 002 which 
preserves the job-then-reducer ordering already established in the 2.x line. 
Objections?

Nope, that sounds reasonable. Thanks for the extra audit

> Shuffle audit logger should log size of shuffle transfer
> 
>
> Key: MAPREDUCE-6958
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6958
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Minor
> Fix For: 3.0.0-beta1
>
> Attachments: MAPREDUCE-6958.001.patch, MAPREDUCE-6958.002.patch, 
> MAPREDUCE-6958.003.patch, MAPREDUCE-6958-branch-2.002.patch
>
>
> The shuffle audit logger currently logs the job ID and reducer ID but nothing 
> about the size of the requested transfer.  It calculates this as part of the 
> HTTP response headers, so it would be trivial to log the response size.  This 
> would be very valuable for debugging network traffic storms from the shuffle 
> handler.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6958) Shuffle audit logger should log size of shuffle transfer

2017-09-18 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170810#comment-16170810
 ] 

Hudson commented on MAPREDUCE-6958:
---

FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #12903 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/12903/])
MAPREDUCE-6958. Shuffle audit logger should log size of shuffle (jlowe: rev 
b3d61304f2fa4a99526f7a60ccaac9f262083079)
* (edit) 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java


> Shuffle audit logger should log size of shuffle transfer
> 
>
> Key: MAPREDUCE-6958
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6958
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Minor
> Fix For: 3.0.0-beta1
>
> Attachments: MAPREDUCE-6958.001.patch, MAPREDUCE-6958.002.patch, 
> MAPREDUCE-6958.003.patch
>
>
> The shuffle audit logger currently logs the job ID and reducer ID but nothing 
> about the size of the requested transfer.  It calculates this as part of the 
> HTTP response headers, so it would be trivial to log the response size.  This 
> would be very valuable for debugging network traffic storms from the shuffle 
> handler.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6958) Shuffle audit logger should log size of shuffle transfer

2017-09-18 Thread Jason Lowe (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170796#comment-16170796
 ] 

Jason Lowe commented on MAPREDUCE-6958:
---

Thanks for the review!  Committing this.

> Shuffle audit logger should log size of shuffle transfer
> 
>
> Key: MAPREDUCE-6958
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6958
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Minor
> Attachments: MAPREDUCE-6958.001.patch, MAPREDUCE-6958.002.patch, 
> MAPREDUCE-6958.003.patch
>
>
> The shuffle audit logger currently logs the job ID and reducer ID but nothing 
> about the size of the requested transfer.  It calculates this as part of the 
> HTTP response headers, so it would be trivial to log the response size.  This 
> would be very valuable for debugging network traffic storms from the shuffle 
> handler.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6958) Shuffle audit logger should log size of shuffle transfer

2017-09-18 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170525#comment-16170525
 ] 

Hadoop QA commented on MAPREDUCE-6958:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
14s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
 9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
21s{color} | {color:green} hadoop-mapreduce-client-shuffle in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
15s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 53s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:71bbb86 |
| JIRA Issue | MAPREDUCE-6958 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12887718/MAPREDUCE-6958.003.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux a423f3a9b770 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 29dd551 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7141/testReport/ |
| modules | C: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle
 U: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle
 |
| Console output | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7141/console |
| Powered by | Apache Yetus 0.5.0   http://yetus.apache.org |


This message was automatically generated.



> Shuffle audit logger should log size of shuffle transfer
> 
>
> Key: MAPREDUCE-6958
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6958
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Minor
> Attachments:

[jira] [Commented] (MAPREDUCE-6958) Shuffle audit logger should log size of shuffle transfer

2017-09-18 Thread Chris Douglas (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170487#comment-16170487
 ] 

Chris Douglas commented on MAPREDUCE-6958:
--

Thanks for updating the patch, [~jlowe]. +1

> Shuffle audit logger should log size of shuffle transfer
> 
>
> Key: MAPREDUCE-6958
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6958
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Minor
> Attachments: MAPREDUCE-6958.001.patch, MAPREDUCE-6958.002.patch, 
> MAPREDUCE-6958.003.patch
>
>
> The shuffle audit logger currently logs the job ID and reducer ID but nothing 
> about the size of the requested transfer.  It calculates this as part of the 
> HTTP response headers, so it would be trivial to log the response size.  This 
> would be very valuable for debugging network traffic storms from the shuffle 
> handler.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6958) Shuffle audit logger should log size of shuffle transfer

2017-09-15 Thread Jason Lowe (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168588#comment-16168588
 ] 

Jason Lowe commented on MAPREDUCE-6958:
---

Thanks for the review, Chris!  I'll update the patch to preserve the existing 
format as best as I can.

bq. The shuffle sizes used to be available in the clienttrace log. Was that 
removed from the ShuffleHandler at some point?

It does log the content length in the normal logger, but it's on a separate log 
line which isn't very friendly given the multithreaded nature of the netty 
processing and also not available in the audit log itself.


> Shuffle audit logger should log size of shuffle transfer
> 
>
> Key: MAPREDUCE-6958
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6958
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Minor
> Attachments: MAPREDUCE-6958.001.patch, MAPREDUCE-6958.002.patch
>
>
> The shuffle audit logger currently logs the job ID and reducer ID but nothing 
> about the size of the requested transfer.  It calculates this as part of the 
> HTTP response headers, so it would be trivial to log the response size.  This 
> would be very valuable for debugging network traffic storms from the shuffle 
> handler.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6958) Shuffle audit logger should log size of shuffle transfer

2017-09-15 Thread Chris Douglas (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168577#comment-16168577
 ] 

Chris Douglas commented on MAPREDUCE-6958:
--

Sorry to ask for revs on this kind of patch, but this changes the format of the 
audit log in a way that might break downstream consumers. The mapIds are 
printed after the reducer in the revised version. Could this keep the format 
as-is, with the length appended?

The shuffle sizes used to be available in the clienttrace log. Was that removed 
from the ShuffleHandler at some point?

> Shuffle audit logger should log size of shuffle transfer
> 
>
> Key: MAPREDUCE-6958
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6958
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Minor
> Attachments: MAPREDUCE-6958.001.patch, MAPREDUCE-6958.002.patch
>
>
> The shuffle audit logger currently logs the job ID and reducer ID but nothing 
> about the size of the requested transfer.  It calculates this as part of the 
> HTTP response headers, so it would be trivial to log the response size.  This 
> would be very valuable for debugging network traffic storms from the shuffle 
> handler.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6958) Shuffle audit logger should log size of shuffle transfer

2017-09-15 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168511#comment-16168511
 ] 

Hadoop QA commented on MAPREDUCE-6958:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
28s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
14s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
20s{color} | {color:green} hadoop-mapreduce-client-shuffle in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 28s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:71bbb86 |
| JIRA Issue | MAPREDUCE-6958 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12887420/MAPREDUCE-6958.002.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux cfd259bb0f16 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / fbe06b5 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7140/testReport/ |
| modules | C: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle
 U: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle
 |
| Console output | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7140/console |
| Powered by | Apache Yetus 0.5.0   http://yetus.apache.org |


This message was automatically generated.



> Shuffle audit logger should log size of shuffle transfer
> 
>
> Key: MAPREDUCE-6958
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6958
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Minor
> Attachments:

[jira] [Commented] (MAPREDUCE-6958) Shuffle audit logger should log size of shuffle transfer

2017-09-15 Thread Rushabh S Shah (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168504#comment-16168504
 ] 

Rushabh S Shah commented on MAPREDUCE-6958:
---

+1 ltgm non-binding.

> Shuffle audit logger should log size of shuffle transfer
> 
>
> Key: MAPREDUCE-6958
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6958
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Minor
> Attachments: MAPREDUCE-6958.001.patch, MAPREDUCE-6958.002.patch
>
>
> The shuffle audit logger currently logs the job ID and reducer ID but nothing 
> about the size of the requested transfer.  It calculates this as part of the 
> HTTP response headers, so it would be trivial to log the response size.  This 
> would be very valuable for debugging network traffic storms from the shuffle 
> handler.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6958) Shuffle audit logger should log size of shuffle transfer

2017-09-15 Thread Rushabh S Shah (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168013#comment-16168013
 ] 

Rushabh S Shah commented on MAPREDUCE-6958:
---

Overall the patch looks good.
Just one very minor nit.
{quote}
+for (String mapId : mapIds) {
+  sb.append(" ");
+  sb.append(mapId);
+}
{quote}
Instead of going through {{mapId}}, we can just print {{mapIds}}. It will be 
just less code.

> Shuffle audit logger should log size of shuffle transfer
> 
>
> Key: MAPREDUCE-6958
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6958
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Minor
> Attachments: MAPREDUCE-6958.001.patch
>
>
> The shuffle audit logger currently logs the job ID and reducer ID but nothing 
> about the size of the requested transfer.  It calculates this as part of the 
> HTTP response headers, so it would be trivial to log the response size.  This 
> would be very valuable for debugging network traffic storms from the shuffle 
> handler.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6958) Shuffle audit logger should log size of shuffle transfer

2017-09-14 Thread Kuhu Shukla (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167251#comment-16167251
 ] 

Kuhu Shukla commented on MAPREDUCE-6958:


Thank you [~jlowe] for the patch!
+1 (non-binding) LGTM.

> Shuffle audit logger should log size of shuffle transfer
> 
>
> Key: MAPREDUCE-6958
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6958
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Minor
> Attachments: MAPREDUCE-6958.001.patch
>
>
> The shuffle audit logger currently logs the job ID and reducer ID but nothing 
> about the size of the requested transfer.  It calculates this as part of the 
> HTTP response headers, so it would be trivial to log the response size.  This 
> would be very valuable for debugging network traffic storms from the shuffle 
> handler.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6958) Shuffle audit logger should log size of shuffle transfer

2017-09-14 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167109#comment-16167109
 ] 

Hadoop QA commented on MAPREDUCE-6958:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 27m 
51s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
27s{color} | {color:green} hadoop-mapreduce-client-shuffle in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
42s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 54m 16s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:71bbb86 |
| JIRA Issue | MAPREDUCE-6958 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12887206/MAPREDUCE-6958.001.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux fddfdc03bc32 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 390c2b5 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7135/testReport/ |
| modules | C: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle
 U: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle
 |
| Console output | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7135/console |
| Powered by | Apache Yetus 0.5.0   http://yetus.apache.org |


This message was automatically generated.



> Shuffle audit logger should log size of shuffle transfer
> 
>
> Key: MAPREDUCE-6958
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6958
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Minor
> Attachments: