[jira] [Updated] (MAPREDUCE-6728) Give fetchers hint when ShuffleHandler rejects a shuffling connection

2016-09-23 Thread Haibo Chen (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haibo Chen updated MAPREDUCE-6728:
--
Attachment: mapreduce6728.003.patch

Uploading a new patch to address the warnings.

> Give fetchers hint when ShuffleHandler rejects a shuffling connection
> -
>
> Key: MAPREDUCE-6728
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6728
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: mrv2
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: mapreduce6728.001.patch, mapreduce6728.002.patch, 
> mapreduce6728.003.patch, mapreduce6728.prelim.patch
>
>
> If # of open shuffle connection to a node goes over the max, ShuffleHandler 
> closes the connection immediately without giving fetchers any hint of the 
> reason, which causes fetchers to fail due to exceptions 
> java.net.SocketException: Unexpected end of file from server
>   at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:772)
>   at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
>   at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769)
>   at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
>   at 
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
>   at 
> java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:323)
>   at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193)
> OR 
> java.net.SocketException: Connection reset
>   at java.net.SocketInputStream.read(SocketInputStream.java:196)
>   at java.net.SocketInputStream.read(SocketInputStream.java:122)
>   at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>   at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
>   at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>   at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687)
>   at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
>   at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769)
>   at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
>   at 
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
>   at 
> java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java
> Such failures are counted as fetcher failures



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6740) Enforce mapreduce.task.timeout to be at least mapreduce.task.progress-report.interval

2016-09-23 Thread Haibo Chen (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15518407#comment-15518407
 ] 

Haibo Chen commented on MAPREDUCE-6740:
---

[~kasha] Turns out MR-6741 is the dependency and it is not committed to 2.8. 
What's right action here? 

> Enforce mapreduce.task.timeout to be at least 
> mapreduce.task.progress-report.interval
> -
>
> Key: MAPREDUCE-6740
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6740
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mr-am
>Affects Versions: 2.8.0
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Minor
> Fix For: 3.0.0-alpha2
>
> Attachments: mapreduce6740.001.patch, mapreduce6740.002.patch, 
> mapreduce6740.003.patch, mapreduce6740.004.patch, mapreduce6740.005.patch, 
> mapreduce6740.006.patch, mapreduce6740.007.patch, mapreduce6740.008.patch
>
>
> MAPREDUCE-6242 makes task status update interval configurable to ease the 
> pressure on MR AM to process status updates, but it did not ensure that 
> mapreduce.task.timeout is no smaller than the configured value of task report 
> interval. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6638) Do not attempt to recover jobs if encrypted spill is enabled

2016-09-23 Thread Haibo Chen (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517217#comment-15517217
 ] 

Haibo Chen commented on MAPREDUCE-6638:
---

Hi [~hitesh] Thanks very much for your comments! Sorry for the confusing phase 
in there. I guess a better way to put it is probably something like 
Avoid recovering progress made in previous attempts if encrypted spill is 
enabled. 

If recovery and spill encryption are both enabled, the job attempt does not 
fail. The job attempt will simply start from scratch because no history file is 
parsed. If the naming in the patch is too misleading, I can create a new follow 
up jira to rename the shouldAttemptRecovery. How about 
shouldAttemptToRecoveryProgress?

> Do not attempt to recover jobs if encrypted spill is enabled
> 
>
> Key: MAPREDUCE-6638
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6638
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: applicationmaster
>Affects Versions: 2.7.2
>Reporter: Karthik Kambatla
>Assignee: Haibo Chen
> Attachments: mapreduce6638.001.patch, mapreduce6638.002.patch, 
> mapreduce6638.003.patch, mapreduce6638.004.patch, mapreduce6683.005.patch
>
>
> Post the fix to CVE-2015-1776, jobs with ecrypted spills enabled cannot be 
> recovered if the AM fails. We should store the key some place safe so they 
> can actually be recovered. If there is no "safe" place, at least we should 
> restart the job by re-running all mappers/reducers. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6782) JHS task page search based on each individual column not working

2016-09-23 Thread Brahma Reddy Battula (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brahma Reddy Battula updated MAPREDUCE-6782:

Assignee: gu-chi  (was: Bibin A Chundatt)

> JHS task page search based on each individual column not working
> 
>
> Key: MAPREDUCE-6782
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6782
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: gu-chi
>
> Submit mapreduce pi job with 10 maps
> In Jobs history server selection completed job
> Select maps to Task Page for job
> Search in individual column fields
> *Expected*
> Search should be working fine in task page for individual columns
> *Actual*
> Search not working for individual column in task page
> In Attempts page the same search is working fine
> {noformat}
> jquery.dataTables.min.js:109
>  Uncaught TypeError: Cannot read property 'oFeatures' of null
> fnFilter @ jquery.dataTables.min.js:109(anonymous function) @ m:49dispatch
>  @ jquery-1.8.2.min.js:2h @ jquery-1.8.2.min.js:2
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6782) JHS task page search based on each individual column not working

2016-09-23 Thread Brahma Reddy Battula (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516342#comment-15516342
 ] 

Brahma Reddy Battula commented on MAPREDUCE-6782:
-

Assigned to you.

> JHS task page search based on each individual column not working
> 
>
> Key: MAPREDUCE-6782
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6782
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: gu-chi
>
> Submit mapreduce pi job with 10 maps
> In Jobs history server selection completed job
> Select maps to Task Page for job
> Search in individual column fields
> *Expected*
> Search should be working fine in task page for individual columns
> *Actual*
> Search not working for individual column in task page
> In Attempts page the same search is working fine
> {noformat}
> jquery.dataTables.min.js:109
>  Uncaught TypeError: Cannot read property 'oFeatures' of null
> fnFilter @ jquery.dataTables.min.js:109(anonymous function) @ m:49dispatch
>  @ jquery-1.8.2.min.js:2h @ jquery-1.8.2.min.js:2
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6782) JHS task page search based on each individual column not working

2016-09-23 Thread gu-chi (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516335#comment-15516335
 ] 

gu-chi commented on MAPREDUCE-6782:
---

Please assign to me

> JHS task page search based on each individual column not working
> 
>
> Key: MAPREDUCE-6782
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6782
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>
> Submit mapreduce pi job with 10 maps
> In Jobs history server selection completed job
> Select maps to Task Page for job
> Search in individual column fields
> *Expected*
> Search should be working fine in task page for individual columns
> *Actual*
> Search not working for individual column in task page
> In Attempts page the same search is working fine
> {noformat}
> jquery.dataTables.min.js:109
>  Uncaught TypeError: Cannot read property 'oFeatures' of null
> fnFilter @ jquery.dataTables.min.js:109(anonymous function) @ m:49dispatch
>  @ jquery-1.8.2.min.js:2h @ jquery-1.8.2.min.js:2
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6782) JHS task page search based on each individual column not working

2016-09-23 Thread gu-chi (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516038#comment-15516038
 ] 

gu-chi commented on MAPREDUCE-6782:
---

[~bibinchundatt] As I checked the cause, this issue comes because the table 
element id does not match what the script want.

In {{HsTasksPage.java}} set the table id as *'tasks'*, but in 
{{HsTasksBlock.java}} which render the table, the table id is set to {noformat} 
   if(type != null)
  thead =
  html.table("#" + app.getJob().getID()
+ type).$class("dt-tasks").thead();
else
  thead = html.table("#tasks").thead();{noformat}

So, the table id could be 'jobid...MAP'. For this point, I not see what is the 
concern of this design, in AppMaster task page, AM just use table id of 
*'tasks'*

> JHS task page search based on each individual column not working
> 
>
> Key: MAPREDUCE-6782
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6782
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>
> Submit mapreduce pi job with 10 maps
> In Jobs history server selection completed job
> Select maps to Task Page for job
> Search in individual column fields
> *Expected*
> Search should be working fine in task page for individual columns
> *Actual*
> Search not working for individual column in task page
> In Attempts page the same search is working fine
> {noformat}
> jquery.dataTables.min.js:109
>  Uncaught TypeError: Cannot read property 'oFeatures' of null
> fnFilter @ jquery.dataTables.min.js:109(anonymous function) @ m:49dispatch
>  @ jquery-1.8.2.min.js:2h @ jquery-1.8.2.min.js:2
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Assigned] (MAPREDUCE-6782) JHS task page search based on each individual column not working

2016-09-23 Thread Bibin A Chundatt (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bibin A Chundatt reassigned MAPREDUCE-6782:
---

Assignee: Bibin A Chundatt

> JHS task page search based on each individual column not working
> 
>
> Key: MAPREDUCE-6782
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6782
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>
> Submit mapreduce pi job with 10 maps
> In Jobs history server selection completed job
> Select maps to Task Page for job
> Search in individual column fields
> *Expected*
> Search should be working fine in task page for individual columns
> *Actual*
> Search not working for individual column in task page
> In Attempts page the same search is working fine
> {noformat}
> jquery.dataTables.min.js:109
>  Uncaught TypeError: Cannot read property 'oFeatures' of null
> fnFilter @ jquery.dataTables.min.js:109(anonymous function) @ m:49dispatch
>  @ jquery-1.8.2.min.js:2h @ jquery-1.8.2.min.js:2
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-6782) JHS task page search based on each individual column not working

2016-09-23 Thread Bibin A Chundatt (JIRA) 
Bibin A Chundatt created MAPREDUCE-6782:
---

 Summary: JHS task page search based on each individual column not 
working
 Key: MAPREDUCE-6782
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6782
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Bibin A Chundatt


Submit mapreduce pi job with 10 maps
In Jobs history server selection completed job
Select maps to Task Page for job
Search in individual column fields


*Expected*
Search should be working fine in task page for individual columns

*Actual*
Search not working for individual column in task page
In Attempts page the same search is working fine



{noformat}
jquery.dataTables.min.js:109
 Uncaught TypeError: Cannot read property 'oFeatures' of null
fnFilter @ jquery.dataTables.min.js:109(anonymous function) @ m:49dispatch
 @ jquery-1.8.2.min.js:2h @ jquery-1.8.2.min.js:2
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org