[jira] [Commented] (MAPREDUCE-6638) Do not attempt to recover progress from previous job attempts if spill encryption is enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15542955#comment-15542955 ] Hudson commented on MAPREDUCE-6638: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10532 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10532/]) MAPREDUCE-6638. Do not attempt to recover progress from previous job (kasha: rev de7a0a92ca1983b35ca4beb7ab712fd700a9e6e0) * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRecovery.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java > Do not attempt to recover progress from previous job attempts if spill > encryption is enabled > > > Key: MAPREDUCE-6638 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6638 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster >Affects Versions: 2.7.2 >Reporter: Karthik Kambatla >Assignee: Haibo Chen > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: mapreduce6638.001.patch, mapreduce6638.002.patch, > mapreduce6638.003.patch, mapreduce6638.004.patch, mapreduce6683.005.patch > > > Post the fix to CVE-2015-1776, jobs with ecrypted spills enabled cannot be > recovered if the AM fails. We should store the key some place safe so they > can actually be recovered. If there is no "safe" place, at least we should > restart the job by re-running all mappers/reducers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6638) Do not attempt to recover progress from previous job attempts if spill encryption is enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-6638: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha2 2.9.0 Status: Resolved (was: Patch Available) Just committed to trunk and branch-2. Thanks [~haibochen] for cleaning this up, and [~hitesh] for chiming in on the appropriateness of the title/commit message. > Do not attempt to recover progress from previous job attempts if spill > encryption is enabled > > > Key: MAPREDUCE-6638 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6638 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster >Affects Versions: 2.7.2 >Reporter: Karthik Kambatla >Assignee: Haibo Chen > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: mapreduce6638.001.patch, mapreduce6638.002.patch, > mapreduce6638.003.patch, mapreduce6638.004.patch, mapreduce6683.005.patch > > > Post the fix to CVE-2015-1776, jobs with ecrypted spills enabled cannot be > recovered if the AM fails. We should store the key some place safe so they > can actually be recovered. If there is no "safe" place, at least we should > restart the job by re-running all mappers/reducers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6728) Give fetchers hint when ShuffleHandler rejects a shuffling connection
[
https://issues.apache.org/jira/browse/MAPREDUCE-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15542927#comment-15542927
]
Daniel Templeton commented on MAPREDUCE-6728:
-
Thanks, [~haibochen]! The patch looks great. Two tiny nits:
# The colon in {{ShuffleHandler}} needs a space before it: {code} for
(Map.Entry header: headers.entrySet()) {{code}
# It would be nice to add a comment where you send the error code to say what
the header means and why you're sending it.
> Give fetchers hint when ShuffleHandler rejects a shuffling connection
> -
>
> Key: MAPREDUCE-6728
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6728
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: mapreduce6728.001.patch, mapreduce6728.002.patch,
> mapreduce6728.003.patch, mapreduce6728.prelim.patch
>
>
> If # of open shuffle connection to a node goes over the max, ShuffleHandler
> closes the connection immediately without giving fetchers any hint of the
> reason, which causes fetchers to fail due to exceptions
> java.net.SocketException: Unexpected end of file from server
> at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:772)
> at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
> at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769)
> at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
> at
> java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:323)
> at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193)
> OR
> java.net.SocketException: Connection reset
> at java.net.SocketInputStream.read(SocketInputStream.java:196)
> at java.net.SocketInputStream.read(SocketInputStream.java:122)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687)
> at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
> at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769)
> at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
> at
> java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java
> Such failures are counted as fetcher failures
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6638) Do not attempt to recover progress from previous job attempts if spill encryption is enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15542926#comment-15542926 ] Karthik Kambatla commented on MAPREDUCE-6638: - Thanks Hitesh. Checking this in with the updated title. > Do not attempt to recover progress from previous job attempts if spill > encryption is enabled > > > Key: MAPREDUCE-6638 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6638 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster >Affects Versions: 2.7.2 >Reporter: Karthik Kambatla >Assignee: Haibo Chen > Attachments: mapreduce6638.001.patch, mapreduce6638.002.patch, > mapreduce6638.003.patch, mapreduce6638.004.patch, mapreduce6683.005.patch > > > Post the fix to CVE-2015-1776, jobs with ecrypted spills enabled cannot be > recovered if the AM fails. We should store the key some place safe so they > can actually be recovered. If there is no "safe" place, at least we should > restart the job by re-running all mappers/reducers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6638) Do not attempt to recover progress from previous job attempts if spill encryption is enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated MAPREDUCE-6638: -- Summary: Do not attempt to recover progress from previous job attempts if spill encryption is enabled (was: Do not attempt to recover jobs if encrypted spill is enabled) > Do not attempt to recover progress from previous job attempts if spill > encryption is enabled > > > Key: MAPREDUCE-6638 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6638 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster >Affects Versions: 2.7.2 >Reporter: Karthik Kambatla >Assignee: Haibo Chen > Attachments: mapreduce6638.001.patch, mapreduce6638.002.patch, > mapreduce6638.003.patch, mapreduce6638.004.patch, mapreduce6683.005.patch > > > Post the fix to CVE-2015-1776, jobs with ecrypted spills enabled cannot be > recovered if the AM fails. We should store the key some place safe so they > can actually be recovered. If there is no "safe" place, at least we should > restart the job by re-running all mappers/reducers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-6638) Do not attempt to recover jobs if encrypted spill is enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15542845#comment-15542845 ] Hitesh Shah edited comment on MAPREDUCE-6638 at 10/3/16 4:51 PM: - Havent looked at the patch in detail but [~haibochen]'s clarifying comments make sense. Jira title could be modified accordingly. +0 from my side. was (Author: hitesh): Havent looked at the patch in detail but [~haibochen]'s clarifying comments make sense. +0 from my side. > Do not attempt to recover jobs if encrypted spill is enabled > > > Key: MAPREDUCE-6638 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6638 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster >Affects Versions: 2.7.2 >Reporter: Karthik Kambatla >Assignee: Haibo Chen > Attachments: mapreduce6638.001.patch, mapreduce6638.002.patch, > mapreduce6638.003.patch, mapreduce6638.004.patch, mapreduce6683.005.patch > > > Post the fix to CVE-2015-1776, jobs with ecrypted spills enabled cannot be > recovered if the AM fails. We should store the key some place safe so they > can actually be recovered. If there is no "safe" place, at least we should > restart the job by re-running all mappers/reducers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6638) Do not attempt to recover jobs if encrypted spill is enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15542845#comment-15542845 ] Hitesh Shah commented on MAPREDUCE-6638: Havent looked at the patch in detail but [~haibochen]'s clarifying comments make sense. +0 from my side. > Do not attempt to recover jobs if encrypted spill is enabled > > > Key: MAPREDUCE-6638 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6638 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster >Affects Versions: 2.7.2 >Reporter: Karthik Kambatla >Assignee: Haibo Chen > Attachments: mapreduce6638.001.patch, mapreduce6638.002.patch, > mapreduce6638.003.patch, mapreduce6638.004.patch, mapreduce6683.005.patch > > > Post the fix to CVE-2015-1776, jobs with ecrypted spills enabled cannot be > recovered if the AM fails. We should store the key some place safe so they > can actually be recovered. If there is no "safe" place, at least we should > restart the job by re-running all mappers/reducers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6638) Do not attempt to recover jobs if encrypted spill is enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15542462#comment-15542462 ] Karthik Kambatla commented on MAPREDUCE-6638: - [~hitesh] - are you comfortable with checking this in? > Do not attempt to recover jobs if encrypted spill is enabled > > > Key: MAPREDUCE-6638 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6638 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster >Affects Versions: 2.7.2 >Reporter: Karthik Kambatla >Assignee: Haibo Chen > Attachments: mapreduce6638.001.patch, mapreduce6638.002.patch, > mapreduce6638.003.patch, mapreduce6638.004.patch, mapreduce6683.005.patch > > > Post the fix to CVE-2015-1776, jobs with ecrypted spills enabled cannot be > recovered if the AM fails. We should store the key some place safe so they > can actually be recovered. If there is no "safe" place, at least we should > restart the job by re-running all mappers/reducers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
