[jira] [Commented] (MAPREDUCE-4669) MRAM web UI does not work with HTTPS
[
https://issues.apache.org/jira/browse/MAPREDUCE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16657444#comment-16657444
]
Robert Kanter commented on MAPREDUCE-4669:
--
Thanks [~haibochen] for the review.
The 004 patch:
- Rebased on latest trunk
- Renames {{withNeedsClientAuth}} to just {{needsClientAuth}}
-- [~haibochen] I went with this instead of {{withClientAuth}} to keep it
consistent with HttpServer2 and SSL, while still making it more normal sounding
- Now only adds the truststore if {{needsClientAuth}} is {{true}}
- Updated {{TestAMWebApp#testMRWebAppSSLEnabledWithClientAuth}} to also try a
wrong client cert
-- I kept this as part of the same test because that way it shows that a client
with the right cert works, while with the wrong cert does not
- Minor refactoring in {{TestAMWebApp}}
> MRAM web UI does not work with HTTPS
>
>
> Key: MAPREDUCE-4669
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4669
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am
>Affects Versions: 2.0.3-alpha
>Reporter: Alejandro Abdelnur
>Assignee: Robert Kanter
>Priority: Major
> Attachments: MAPREDUCE-4669.001.patch, MAPREDUCE-4669.002.patch,
> MAPREDUCE-4669.003.patch, MAPREDUCE-4669.004.patch
>
>
> With Kerberos enable, the MRAM runs as the user that submitted the job, thus
> the MRAM process cannot read the cluster keystore files to get the
> certificates to start its HttpServer using HTTPS.
> We need to decouple the keystore used by RM/NM/NN/DN (which are cluster
> provided) from the keystore used by AMs (which ought to be user provided).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-4669) MRAM web UI does not work with HTTPS
[ https://issues.apache.org/jira/browse/MAPREDUCE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-4669: - Attachment: MAPREDUCE-4669.004.patch > MRAM web UI does not work with HTTPS > > > Key: MAPREDUCE-4669 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4669 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.0.3-alpha >Reporter: Alejandro Abdelnur >Assignee: Robert Kanter >Priority: Major > Attachments: MAPREDUCE-4669.001.patch, MAPREDUCE-4669.002.patch, > MAPREDUCE-4669.003.patch, MAPREDUCE-4669.004.patch > > > With Kerberos enable, the MRAM runs as the user that submitted the job, thus > the MRAM process cannot read the cluster keystore files to get the > certificates to start its HttpServer using HTTPS. > We need to decouple the keystore used by RM/NM/NN/DN (which are cluster > provided) from the keystore used by AMs (which ought to be user provided). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-4669) MRAM web UI does not work with HTTPS
[ https://issues.apache.org/jira/browse/MAPREDUCE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-4669: - Status: Patch Available (was: Open) > MRAM web UI does not work with HTTPS > > > Key: MAPREDUCE-4669 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4669 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.0.3-alpha >Reporter: Alejandro Abdelnur >Assignee: Robert Kanter >Priority: Major > Attachments: MAPREDUCE-4669.001.patch, MAPREDUCE-4669.002.patch, > MAPREDUCE-4669.003.patch > > > With Kerberos enable, the MRAM runs as the user that submitted the job, thus > the MRAM process cannot read the cluster keystore files to get the > certificates to start its HttpServer using HTTPS. > We need to decouple the keystore used by RM/NM/NN/DN (which are cluster > provided) from the keystore used by AMs (which ought to be user provided). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-4669) MRAM web UI does not work with HTTPS
[ https://issues.apache.org/jira/browse/MAPREDUCE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-4669: - Attachment: MAPREDUCE-4669.003.patch > MRAM web UI does not work with HTTPS > > > Key: MAPREDUCE-4669 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4669 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.0.3-alpha >Reporter: Alejandro Abdelnur >Assignee: Robert Kanter >Priority: Major > Attachments: MAPREDUCE-4669.001.patch, MAPREDUCE-4669.002.patch, > MAPREDUCE-4669.003.patch > > > With Kerberos enable, the MRAM runs as the user that submitted the job, thus > the MRAM process cannot read the cluster keystore files to get the > certificates to start its HttpServer using HTTPS. > We need to decouple the keystore used by RM/NM/NN/DN (which are cluster > provided) from the keystore used by AMs (which ought to be user provided). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-4669) MRAM web UI does not work with HTTPS
[ https://issues.apache.org/jira/browse/MAPREDUCE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652670#comment-16652670 ] Robert Kanter commented on MAPREDUCE-4669: -- The 003 patch: - Rebased on latest trunk (including YARN-8448) - Added configs to mapred-default > MRAM web UI does not work with HTTPS > > > Key: MAPREDUCE-4669 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4669 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.0.3-alpha >Reporter: Alejandro Abdelnur >Assignee: Robert Kanter >Priority: Major > Attachments: MAPREDUCE-4669.001.patch, MAPREDUCE-4669.002.patch, > MAPREDUCE-4669.003.patch > > > With Kerberos enable, the MRAM runs as the user that submitted the job, thus > the MRAM process cannot read the cluster keystore files to get the > certificates to start its HttpServer using HTTPS. > We need to decouple the keystore used by RM/NM/NN/DN (which are cluster > provided) from the keystore used by AMs (which ought to be user provided). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-4669) MRAM web UI does not work with HTTPS
[ https://issues.apache.org/jira/browse/MAPREDUCE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647088#comment-16647088 ] Robert Kanter commented on MAPREDUCE-4669: -- The 002 patch: * Rebased on the latest version of YARN-8448 (the 008 patch) * Renamed OFF, OPTIONAL, REQUIRED to NONE, LENIENT, and STRICT as per YARN-8448 > MRAM web UI does not work with HTTPS > > > Key: MAPREDUCE-4669 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4669 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.0.3-alpha >Reporter: Alejandro Abdelnur >Assignee: Robert Kanter >Priority: Major > Attachments: MAPREDUCE-4669.001.patch, MAPREDUCE-4669.002.patch > > > With Kerberos enable, the MRAM runs as the user that submitted the job, thus > the MRAM process cannot read the cluster keystore files to get the > certificates to start its HttpServer using HTTPS. > We need to decouple the keystore used by RM/NM/NN/DN (which are cluster > provided) from the keystore used by AMs (which ought to be user provided). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-4669) MRAM web UI does not work with HTTPS
[ https://issues.apache.org/jira/browse/MAPREDUCE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-4669: - Attachment: MAPREDUCE-4669.002.patch > MRAM web UI does not work with HTTPS > > > Key: MAPREDUCE-4669 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4669 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.0.3-alpha >Reporter: Alejandro Abdelnur >Assignee: Robert Kanter >Priority: Major > Attachments: MAPREDUCE-4669.001.patch, MAPREDUCE-4669.002.patch > > > With Kerberos enable, the MRAM runs as the user that submitted the job, thus > the MRAM process cannot read the cluster keystore files to get the > certificates to start its HttpServer using HTTPS. > We need to decouple the keystore used by RM/NM/NN/DN (which are cluster > provided) from the keystore used by AMs (which ought to be user provided). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-4669) MRAM web UI does not work with HTTPS
[ https://issues.apache.org/jira/browse/MAPREDUCE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-4669: - Target Version/s: 3.3.0 > MRAM web UI does not work with HTTPS > > > Key: MAPREDUCE-4669 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4669 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.0.3-alpha >Reporter: Alejandro Abdelnur >Assignee: Robert Kanter >Priority: Major > Attachments: MAPREDUCE-4669.001.patch > > > With Kerberos enable, the MRAM runs as the user that submitted the job, thus > the MRAM process cannot read the cluster keystore files to get the > certificates to start its HttpServer using HTTPS. > We need to decouple the keystore used by RM/NM/NN/DN (which are cluster > provided) from the keystore used by AMs (which ought to be user provided). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-4669) MRAM web UI does not work with HTTPS
[
https://issues.apache.org/jira/browse/MAPREDUCE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556366#comment-16556366
]
Robert Kanter edited comment on MAPREDUCE-4669 at 7/25/18 10:44 PM:
See this comment in YARN-8448 and the design doc in YARN-6586 for more
background on the patch. This patch (MAPREDUCE-4669.001.patch) contains the MR
changes that rely on YARN-8448.001.patch.
Some notes on the patch:
- The {{yarn.app.mapreduce.am.webapp.https.enabled}} property controls if the
MR AM should try to use the Yarn-provided keystore (when set to {{true}}); this
will also cause it to provide an HTTPS tracking URL to the RM. It defaults to
{{false}}.
- The {{yarn.app.mapreduce.am.webapp.https.client.auth}} property controls if
the MR AM should require client authentication (when set to {{true}}). It
defaults to {{false}}. In this case, the MR AM is the server and the RM is the
client, so this requires that the RM present its certificate to the AM when it
connects to the AM - the AM can then verify this certificate with the
Yarn-provided truststore.
- It won't compile without the YARN-8448 patch.
was (Author: rkanter):
See [this
comment|https://issues.apache.org/jira/browse/YARN-8448?focusedCommentId=16556364=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16556364]
in YARN-8448 and the design doc in YARN-6586 for more background on the patch.
This patch (MAPREDUCE-4669.001.patch) contains the MR changes that rely on
YARN-8448.001.patch.
Some notes on the patch:
- The {{yarn.app.mapreduce.am.webapp.https.enabled}} property controls if the
MR AM should try to use the Yarn-provided keystore (when set to {{true}}); this
will also cause it to provide an HTTPS tracking URL to the RM. It defaults to
{{false}}.
- The {{yarn.app.mapreduce.am.webapp.https.client.auth}} property controls if
the MR AM should require client authentication (when set to {{true}}). It
defaults to {{false}}. In this case, the MR AM is the server and the RM is the
client, so this requires that the RM present its certificate to the AM when it
connects to the AM - the AM can then verify this certificate with the
Yarn-provided truststore.
> MRAM web UI does not work with HTTPS
>
>
> Key: MAPREDUCE-4669
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4669
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am
>Affects Versions: 2.0.3-alpha
>Reporter: Alejandro Abdelnur
>Assignee: Robert Kanter
>Priority: Major
> Attachments: MAPREDUCE-4669.001.patch
>
>
> With Kerberos enable, the MRAM runs as the user that submitted the job, thus
> the MRAM process cannot read the cluster keystore files to get the
> certificates to start its HttpServer using HTTPS.
> We need to decouple the keystore used by RM/NM/NN/DN (which are cluster
> provided) from the keystore used by AMs (which ought to be user provided).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-4669) MRAM web UI does not work with HTTPS
[ https://issues.apache.org/jira/browse/MAPREDUCE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter reassigned MAPREDUCE-4669: Assignee: Robert Kanter > MRAM web UI does not work with HTTPS > > > Key: MAPREDUCE-4669 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4669 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.0.3-alpha >Reporter: Alejandro Abdelnur >Assignee: Robert Kanter >Priority: Major > Attachments: MAPREDUCE-4669.001.patch > > > With Kerberos enable, the MRAM runs as the user that submitted the job, thus > the MRAM process cannot read the cluster keystore files to get the > certificates to start its HttpServer using HTTPS. > We need to decouple the keystore used by RM/NM/NN/DN (which are cluster > provided) from the keystore used by AMs (which ought to be user provided). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-4669) MRAM web UI does not work with HTTPS
[
https://issues.apache.org/jira/browse/MAPREDUCE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556366#comment-16556366
]
Robert Kanter commented on MAPREDUCE-4669:
--
See [this
comment|https://issues.apache.org/jira/browse/YARN-8448?focusedCommentId=16556364=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16556364]
in YARN-8448 and the design doc in YARN-6586 for more background on the patch.
This patch (MAPREDUCE-4669.001.patch) contains the MR changes that rely on
YARN-8448.001.patch.
Some notes on the patch:
- The {{yarn.app.mapreduce.am.webapp.https.enabled}} property controls if the
MR AM should try to use the Yarn-provided keystore (when set to {{true}}); this
will also cause it to provide an HTTPS tracking URL to the RM. It defaults to
{{false}}.
- The {{yarn.app.mapreduce.am.webapp.https.client.auth}} property controls if
the MR AM should require client authentication (when set to {{true}}). It
defaults to {{false}}. In this case, the MR AM is the server and the RM is the
client, so this requires that the RM present its certificate to the AM when it
connects to the AM - the AM can then verify this certificate with the
Yarn-provided truststore.
> MRAM web UI does not work with HTTPS
>
>
> Key: MAPREDUCE-4669
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4669
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am
>Affects Versions: 2.0.3-alpha
>Reporter: Alejandro Abdelnur
>Assignee: Robert Kanter
>Priority: Major
> Attachments: MAPREDUCE-4669.001.patch
>
>
> With Kerberos enable, the MRAM runs as the user that submitted the job, thus
> the MRAM process cannot read the cluster keystore files to get the
> certificates to start its HttpServer using HTTPS.
> We need to decouple the keystore used by RM/NM/NN/DN (which are cluster
> provided) from the keystore used by AMs (which ought to be user provided).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-4669) MRAM web UI does not work with HTTPS
[ https://issues.apache.org/jira/browse/MAPREDUCE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-4669: - Attachment: MAPREDUCE-4669.001.patch > MRAM web UI does not work with HTTPS > > > Key: MAPREDUCE-4669 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4669 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.0.3-alpha >Reporter: Alejandro Abdelnur >Assignee: Robert Kanter >Priority: Major > Attachments: MAPREDUCE-4669.001.patch > > > With Kerberos enable, the MRAM runs as the user that submitted the job, thus > the MRAM process cannot read the cluster keystore files to get the > certificates to start its HttpServer using HTTPS. > We need to decouple the keystore used by RM/NM/NN/DN (which are cluster > provided) from the keystore used by AMs (which ought to be user provided). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6925) CLONE - Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
[
https://issues.apache.org/jira/browse/MAPREDUCE-6925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548149#comment-16548149
]
Robert Kanter commented on MAPREDUCE-6925:
--
AFAIK, this is still an issue. I agree that the JIRAs are a bit confusing
because there were a few different ones. I believe they've all been reverted,
and this JIRA (MAPREDUCE-6925) was left open to come up with a different fix.
Unfortunately, nobody's been able to come up with an acceptable fix. I like the
simplicity of my fix, [^MAPREDUCE-6288.002.patch], on MAPREDUCE-6288, but there
were concerns about changing permissions.
> CLONE - Make Counter limits consistent across JobClient, MRAppMaster, and
> YarnChild
> ---
>
> Key: MAPREDUCE-6925
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6925
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, client, task
>Affects Versions: 2.4.0
>Reporter: Gera Shegalov
>Assignee: Gera Shegalov
>Priority: Major
>
> Currently, counter limits "mapreduce.job.counters.*" handled by
> {{org.apache.hadoop.mapreduce.counters.Limits}} are initialized
> asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
> it's taken into account in YarnChild.
> It would be good to make the Limits job-configurable, such that max
> counters/groups is only increased when needed. With the current Limits
> implementation relying on static constants, it's going to be challenging for
> tools that submit jobs concurrently without resorting to class loading
> isolation.
> The patch that I am uploading is not perfect but demonstrates the issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7072) mapred job -history prints duplicate counter in human output
[
https://issues.apache.org/jira/browse/MAPREDUCE-7072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated MAPREDUCE-7072:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.2.0
2.10.0
Status: Resolved (was: Patch Available)
Thanks [~wilfreds]. Committed to trunk and branch-2!
> mapred job -history prints duplicate counter in human output
>
>
> Key: MAPREDUCE-7072
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7072
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
>Affects Versions: 3.0.0
>Reporter: Wilfred Spiegelenburg
>Assignee: Wilfred Spiegelenburg
>Priority: Major
> Fix For: 2.10.0, 3.2.0
>
> Attachments: MAPREDUCE-7072-branch-2.02.patch,
> MAPREDUCE-7072-branch-2.03.patch, MAPREDUCE-7072.01.patch,
> MAPREDUCE-7072.02.patch
>
>
> 'mapred job -history' command prints duplicate entries for counters only for
> the human output format. It does not do this for the JSON format.
> mapred job -history /user/history/somefile.jhist -format human
> {code}
>
> |Job Counters |Total megabyte-seconds taken by all map tasks|0 |0 |268,288,000
> ...
> |Job Counters |Total megabyte-seconds taken by all map tasks|0 |0 |268,288,000
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7072) mapred job -history prints duplicate counter in human output
[
https://issues.apache.org/jira/browse/MAPREDUCE-7072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454557#comment-16454557
]
Robert Kanter commented on MAPREDUCE-7072:
--
There was some kind of problem with the Jenkins node
([https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7402/).] I've
retriggered it: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7403/
> mapred job -history prints duplicate counter in human output
>
>
> Key: MAPREDUCE-7072
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7072
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
>Affects Versions: 3.0.0
>Reporter: Wilfred Spiegelenburg
>Assignee: Wilfred Spiegelenburg
>Priority: Major
> Attachments: MAPREDUCE-7072-branch-2.02.patch,
> MAPREDUCE-7072.01.patch, MAPREDUCE-7072.02.patch
>
>
> 'mapred job -history' command prints duplicate entries for counters only for
> the human output format. It does not do this for the JSON format.
> mapred job -history /user/history/somefile.jhist -format human
> {code}
>
> |Job Counters |Total megabyte-seconds taken by all map tasks|0 |0 |268,288,000
> ...
> |Job Counters |Total megabyte-seconds taken by all map tasks|0 |0 |268,288,000
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7072) mapred job -history prints duplicate counter in human output
[
https://issues.apache.org/jira/browse/MAPREDUCE-7072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451328#comment-16451328
]
Robert Kanter commented on MAPREDUCE-7072:
--
Retriggered Jenkins at
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7398/
> mapred job -history prints duplicate counter in human output
>
>
> Key: MAPREDUCE-7072
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7072
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
>Affects Versions: 3.0.0
>Reporter: Wilfred Spiegelenburg
>Assignee: Wilfred Spiegelenburg
>Priority: Major
> Attachments: MAPREDUCE-7072.01.patch, MAPREDUCE-7072.02.patch
>
>
> 'mapred job -history' command prints duplicate entries for counters only for
> the human output format. It does not do this for the JSON format.
> mapred job -history /user/history/somefile.jhist -format human
> {code}
>
> |Job Counters |Total megabyte-seconds taken by all map tasks|0 |0 |268,288,000
> ...
> |Job Counters |Total megabyte-seconds taken by all map tasks|0 |0 |268,288,000
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7072) mapred job -history prints duplicate counter in human output
[
https://issues.apache.org/jira/browse/MAPREDUCE-7072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450360#comment-16450360
]
Robert Kanter commented on MAPREDUCE-7072:
--
I committed this to trunk, but it fails to compile on branch-2. [~wilfreds],
can you create a branch-2 version of the patch?
{noformat}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
(default-testCompile) on project hadoop-mapreduce-client-core: Compilation
failure: Compilation failure:
[ERROR]
/Users/rkanter/dev/hadoop-git/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestHistoryViewerPrinter.java:[958,8]
cannot find symbol
[ERROR] symbol: variable succeededMaps
[ERROR] location: variable job of type
org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.JobInfo
[ERROR]
/Users/rkanter/dev/hadoop-git/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestHistoryViewerPrinter.java:[959,8]
cannot find symbol
[ERROR] symbol: variable succeededReduces
[ERROR] location: variable job of type
org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.JobInfo
[ERROR] -> [Help 1]
{noformat}
> mapred job -history prints duplicate counter in human output
>
>
> Key: MAPREDUCE-7072
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7072
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
>Affects Versions: 3.0.0
>Reporter: Wilfred Spiegelenburg
>Assignee: Wilfred Spiegelenburg
>Priority: Major
> Attachments: MAPREDUCE-7072.01.patch, MAPREDUCE-7072.02.patch
>
>
> 'mapred job -history' command prints duplicate entries for counters only for
> the human output format. It does not do this for the JSON format.
> mapred job -history /user/history/somefile.jhist -format human
> {code}
>
> |Job Counters |Total megabyte-seconds taken by all map tasks|0 |0 |268,288,000
> ...
> |Job Counters |Total megabyte-seconds taken by all map tasks|0 |0 |268,288,000
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7072) mapred job -history prints duplicate counter in human output
[
https://issues.apache.org/jira/browse/MAPREDUCE-7072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450347#comment-16450347
]
Robert Kanter commented on MAPREDUCE-7072:
--
+1 LGTM
> mapred job -history prints duplicate counter in human output
>
>
> Key: MAPREDUCE-7072
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7072
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
>Affects Versions: 3.0.0
>Reporter: Wilfred Spiegelenburg
>Assignee: Wilfred Spiegelenburg
>Priority: Major
> Attachments: MAPREDUCE-7072.01.patch, MAPREDUCE-7072.02.patch
>
>
> 'mapred job -history' command prints duplicate entries for counters only for
> the human output format. It does not do this for the JSON format.
> mapred job -history /user/history/somefile.jhist -format human
> {code}
>
> |Job Counters |Total megabyte-seconds taken by all map tasks|0 |0 |268,288,000
> ...
> |Job Counters |Total megabyte-seconds taken by all map tasks|0 |0 |268,288,000
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7072) mapred job -history prints duplicate counter in human output
[
https://issues.apache.org/jira/browse/MAPREDUCE-7072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16448958#comment-16448958
]
Robert Kanter commented on MAPREDUCE-7072:
--
Great investigation [~wilfreds]! The patch looks good overall, just one thing:
# Instead of using the iterator:
{code:java}
Iterator groupNames = totalCounters.iterator();
while (groupNames.hasNext()) {
String groupName = groupNames.next().getName();
{code}
I think we can still do a for-each loop like this:
{code:java}
for (CounterGroup counterGroup : totalCounters) {
String groupName = counterGroup.getName();
{code}
> mapred job -history prints duplicate counter in human output
>
>
> Key: MAPREDUCE-7072
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7072
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
>Affects Versions: 3.0.0
>Reporter: Wilfred Spiegelenburg
>Assignee: Wilfred Spiegelenburg
>Priority: Major
> Attachments: MAPREDUCE-7072.01.patch
>
>
> 'mapred job -history' command prints duplicate entries for counters only for
> the human output format. It does not do this for the JSON format.
> mapred job -history /user/history/somefile.jhist -format human
> {code}
>
> |Job Counters |Total megabyte-seconds taken by all map tasks|0 |0 |268,288,000
> ...
> |Job Counters |Total megabyte-seconds taken by all map tasks|0 |0 |268,288,000
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6441) Improve temporary directory name generation in LocalDistributedCacheManager for concurrent processes
[
https://issues.apache.org/jira/browse/MAPREDUCE-6441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated MAPREDUCE-6441:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.2.0
Status: Resolved (was: Patch Available)
Thanks [~wattsinabox], [~rchiang], and [~haibochen]. Committed to trunk!
> Improve temporary directory name generation in LocalDistributedCacheManager
> for concurrent processes
>
>
> Key: MAPREDUCE-6441
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6441
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: William Watson
>Assignee: Haibo Chen
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HADOOP-10924.02.patch,
> HADOOP-10924.03.jobid-plus-uuid.patch, MAPREDUCE-6441.004.patch,
> MAPREDUCE-6441.005.patch, MAPREDUCE-6441.006.patch, MAPREDUCE-6441.008.patch,
> MAPREDUCE-6441.009.patch, MAPREDUCE-6441.010.patch, MAPREDUCE-6441.011.patch
>
>
> Kicking off many sqoop processes in different threads results in:
> {code}
> 2014-08-01 13:47:24 -0400: INFO - 14/08/01 13:47:22 ERROR tool.ImportTool:
> Encountered IOException running import job: java.io.IOException:
> java.util.concurrent.ExecutionException: java.io.IOException: Rename cannot
> overwrite non empty destination directory
> /tmp/hadoop-hadoop/mapred/local/1406915233073
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.mapred.LocalDistributedCacheManager.setup(LocalDistributedCacheManager.java:149)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.mapred.LocalJobRunner$Job.(LocalJobRunner.java:163)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:731)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:432)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
> 2014-08-01 13:47:24 -0400: INFO -at
> java.security.AccessController.doPrivileged(Native Method)
> 2014-08-01 13:47:24 -0400: INFO -at
> javax.security.auth.Subject.doAs(Subject.java:415)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:186)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:159)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:239)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.manager.SqlManager.importQuery(SqlManager.java:645)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:415)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.tool.ImportTool.run(ImportTool.java:502)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.Sqoop.main(Sqoop.java:238)
> {code}
> If two are kicked off in the same second. The issue is the following lines of
> code in the org.apache.hadoop.mapred.LocalDistributedCacheManager class:
> {code}
> // Generating unique numbers for FSDownload.
> AtomicLong uniqueNumberGenerator =
>new AtomicLong(System.currentTimeMillis());
> {code}
> and
> {code}
> Long.toString(uniqueNumberGenerator.incrementAndGet())),
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6441) Improve temporary directory name generation in LocalDistributedCacheManager for concurrent processes
[
https://issues.apache.org/jira/browse/MAPREDUCE-6441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16414625#comment-16414625
]
Robert Kanter commented on MAPREDUCE-6441:
--
+1 LGTM
> Improve temporary directory name generation in LocalDistributedCacheManager
> for concurrent processes
>
>
> Key: MAPREDUCE-6441
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6441
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: William Watson
>Assignee: Haibo Chen
>Priority: Major
> Attachments: HADOOP-10924.02.patch,
> HADOOP-10924.03.jobid-plus-uuid.patch, MAPREDUCE-6441.004.patch,
> MAPREDUCE-6441.005.patch, MAPREDUCE-6441.006.patch, MAPREDUCE-6441.008.patch,
> MAPREDUCE-6441.009.patch, MAPREDUCE-6441.010.patch, MAPREDUCE-6441.011.patch
>
>
> Kicking off many sqoop processes in different threads results in:
> {code}
> 2014-08-01 13:47:24 -0400: INFO - 14/08/01 13:47:22 ERROR tool.ImportTool:
> Encountered IOException running import job: java.io.IOException:
> java.util.concurrent.ExecutionException: java.io.IOException: Rename cannot
> overwrite non empty destination directory
> /tmp/hadoop-hadoop/mapred/local/1406915233073
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.mapred.LocalDistributedCacheManager.setup(LocalDistributedCacheManager.java:149)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.mapred.LocalJobRunner$Job.(LocalJobRunner.java:163)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:731)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:432)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
> 2014-08-01 13:47:24 -0400: INFO -at
> java.security.AccessController.doPrivileged(Native Method)
> 2014-08-01 13:47:24 -0400: INFO -at
> javax.security.auth.Subject.doAs(Subject.java:415)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:186)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:159)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:239)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.manager.SqlManager.importQuery(SqlManager.java:645)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:415)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.tool.ImportTool.run(ImportTool.java:502)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.Sqoop.main(Sqoop.java:238)
> {code}
> If two are kicked off in the same second. The issue is the following lines of
> code in the org.apache.hadoop.mapred.LocalDistributedCacheManager class:
> {code}
> // Generating unique numbers for FSDownload.
> AtomicLong uniqueNumberGenerator =
>new AtomicLong(System.currentTimeMillis());
> {code}
> and
> {code}
> Long.toString(uniqueNumberGenerator.incrementAndGet())),
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-6441) Improve temporary directory name generation in LocalDistributedCacheManager for concurrent processes
[
https://issues.apache.org/jira/browse/MAPREDUCE-6441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter reassigned MAPREDUCE-6441:
Assignee: Haibo Chen (was: Ray Chiang)
> Improve temporary directory name generation in LocalDistributedCacheManager
> for concurrent processes
>
>
> Key: MAPREDUCE-6441
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6441
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: William Watson
>Assignee: Haibo Chen
>Priority: Major
> Attachments: HADOOP-10924.02.patch,
> HADOOP-10924.03.jobid-plus-uuid.patch, MAPREDUCE-6441.004.patch,
> MAPREDUCE-6441.005.patch, MAPREDUCE-6441.006.patch, MAPREDUCE-6441.008.patch,
> MAPREDUCE-6441.009.patch, MAPREDUCE-6441.010.patch, MAPREDUCE-6441.011.patch
>
>
> Kicking off many sqoop processes in different threads results in:
> {code}
> 2014-08-01 13:47:24 -0400: INFO - 14/08/01 13:47:22 ERROR tool.ImportTool:
> Encountered IOException running import job: java.io.IOException:
> java.util.concurrent.ExecutionException: java.io.IOException: Rename cannot
> overwrite non empty destination directory
> /tmp/hadoop-hadoop/mapred/local/1406915233073
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.mapred.LocalDistributedCacheManager.setup(LocalDistributedCacheManager.java:149)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.mapred.LocalJobRunner$Job.(LocalJobRunner.java:163)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:731)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:432)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
> 2014-08-01 13:47:24 -0400: INFO -at
> java.security.AccessController.doPrivileged(Native Method)
> 2014-08-01 13:47:24 -0400: INFO -at
> javax.security.auth.Subject.doAs(Subject.java:415)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:186)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:159)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:239)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.manager.SqlManager.importQuery(SqlManager.java:645)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:415)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.tool.ImportTool.run(ImportTool.java:502)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
> 2014-08-01 13:47:24 -0400: INFO -at
> org.apache.sqoop.Sqoop.main(Sqoop.java:238)
> {code}
> If two are kicked off in the same second. The issue is the following lines of
> code in the org.apache.hadoop.mapred.LocalDistributedCacheManager class:
> {code}
> // Generating unique numbers for FSDownload.
> AtomicLong uniqueNumberGenerator =
>new AtomicLong(System.currentTimeMillis());
> {code}
> and
> {code}
> Long.toString(uniqueNumberGenerator.incrementAndGet())),
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7047) Make HAR tool support IndexedLogAggregtionController
[ https://issues.apache.org/jira/browse/MAPREDUCE-7047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399363#comment-16399363 ] Robert Kanter commented on MAPREDUCE-7047: -- LGTM +1 > Make HAR tool support IndexedLogAggregtionController > > > Key: MAPREDUCE-7047 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7047 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Major > Attachments: MAPREDUCE-7047.trunk.1.patch, > MAPREDUCE-7047.trunk.2.patch, MAPREDUCE-7047.trunk.3.patch > > > In https://issues.apache.org/jira/browse/MAPREDUCE-6415, we have created a > tool to combine aggregated logs into HAR files which currently only work for > TFileLogAggregationFileController. We should make it support > IndexedLogAggregtionController as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7047) Make HAR tool support IndexedLogAggregtionController
[
https://issues.apache.org/jira/browse/MAPREDUCE-7047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395559#comment-16395559
]
Robert Kanter commented on MAPREDUCE-7047:
--
Thanks [~xgong]. A few minor things:
# This code can be simplified, and should return a non-zero exit code:
{code:java}
boolean createdWorkingDirsSuccess = true;
for (Path workingDir : workingDirs) {
if (!prepareWorkingDir(fs, workingDir)) {
createdWorkingDirsSuccess = false;
LOG.error("Failed to create the workingDir:"
+ workingDir.toString());
break;
}
}
if (!createdWorkingDirsSuccess) {
return 0;
}
{code}
to
{code:java}
for (Path workingDir : workingDirs) {
if (!prepareWorkingDir(fs, workingDir)) {
createdWorkingDirsSuccess = false;
LOG.error("Failed to create the workingDir:"
+ workingDir.toString());
return 1
}
}
{code}
# When cleaning up the working directories, is it necessary to check that the
directories exist before trying to delete them? That's going to result in
additional calls. IIRC, delete will do a no-op if the directory doesn't exist.
# Looks like we accidentally lost a blank line above the
{{runDistributedShell}} method
# {{equals}} should also consider the other fields added to {{AppInfo}}. I
don't think we'll have any cases where this would cause a problem, but it's
safer if we take those into account.
# It would be better if the test used different values for the applications
(e.g. different working dirs, etc)
> Make HAR tool support IndexedLogAggregtionController
>
>
> Key: MAPREDUCE-7047
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7047
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Xuan Gong
>Assignee: Xuan Gong
>Priority: Major
> Attachments: MAPREDUCE-7047.trunk.1.patch
>
>
> In https://issues.apache.org/jira/browse/MAPREDUCE-6415, we have created a
> tool to combine aggregated logs into HAR files which currently only work for
> TFileLogAggregationFileController. We should make it support
> IndexedLogAggregtionController as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7047) Make HAR tool support IndexedLogAggregtionController
[ https://issues.apache.org/jira/browse/MAPREDUCE-7047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393890#comment-16393890 ] Robert Kanter commented on MAPREDUCE-7047: -- Sure :) I'll take a look on Monday. > Make HAR tool support IndexedLogAggregtionController > > > Key: MAPREDUCE-7047 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7047 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Major > Attachments: MAPREDUCE-7047.trunk.1.patch > > > In https://issues.apache.org/jira/browse/MAPREDUCE-6415, we have created a > tool to combine aggregated logs into HAR files which currently only work for > TFileLogAggregationFileController. We should make it support > IndexedLogAggregtionController as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7027) HadoopArchiveLogs shouldn't delete the original logs if the HAR creation fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-7027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-7027: - Status: Patch Available (was: Open) > HadoopArchiveLogs shouldn't delete the original logs if the HAR creation fails > -- > > Key: MAPREDUCE-7027 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7027 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Gergely Novák >Assignee: Gergely Novák >Priority: Critical > Attachments: MAPREDUCE-7027.001.patch > > > If the hadoop archive command fails for any reason (for example because of an > OutOfMemoryError) the HadoopArchiveLogs tool will still delete the original > log files, so all the logs will be lost. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6995) Uploader tool for Distributed Cache Deploy documentation
[ https://issues.apache.org/jira/browse/MAPREDUCE-6995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-6995: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.1.0 Status: Resolved (was: Patch Available) Thanks [~miklos.szeg...@cloudera.com]. Committed to trunk! > Uploader tool for Distributed Cache Deploy documentation > > > Key: MAPREDUCE-6995 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6995 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Major > Fix For: 3.1.0 > > Attachments: MAPREDUCE-6995.000.patch, MAPREDUCE-6995.001.patch, > MAPREDUCE-6995.002.patch, MAPREDUCE-6995.003.patch, MAPREDUCE-6995.004.patch, > MAPREDUCE-6995.005.patch, MAPREDUCE-6995.006.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6995) Uploader tool for Distributed Cache Deploy documentation
[ https://issues.apache.org/jira/browse/MAPREDUCE-6995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16333118#comment-16333118 ] Robert Kanter commented on MAPREDUCE-6995: -- +1 LGTM > Uploader tool for Distributed Cache Deploy documentation > > > Key: MAPREDUCE-6995 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6995 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Major > Attachments: MAPREDUCE-6995.000.patch, MAPREDUCE-6995.001.patch, > MAPREDUCE-6995.002.patch, MAPREDUCE-6995.003.patch, MAPREDUCE-6995.004.patch, > MAPREDUCE-6995.005.patch, MAPREDUCE-6995.006.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6995) Uploader tool for Distributed Cache Deploy documentation
[
https://issues.apache.org/jira/browse/MAPREDUCE-6995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332943#comment-16332943
]
Robert Kanter commented on MAPREDUCE-6995:
--
A few last things and I think we'll be good:
# On the DistributedCacheDeploy page and in {{mapred}}, the tool is called
frameworkuploader, but on the MapredCommands page, it's called frameworkupload.
We should be consistent.
# On the MapredCommands page, there shouldn't be {{|}} between the arguments
because they can all be used at the same time. It should be
{noformat}
mapred frameworkupload -target [-fs ] [-input ]
[-blacklist ] [-whitelist ] [-initialReplication ]
[-acceptableReplication ] [-finalReplication ] [-timeout ]
[-nosymlink]
{noformat}
# On the MapredCommands page, the {{frameworkuploader}} command should be under
the "Administration Commands" section, not the "User Commands" section.
> Uploader tool for Distributed Cache Deploy documentation
>
>
> Key: MAPREDUCE-6995
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6995
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Major
> Attachments: MAPREDUCE-6995.000.patch, MAPREDUCE-6995.001.patch,
> MAPREDUCE-6995.002.patch, MAPREDUCE-6995.003.patch, MAPREDUCE-6995.004.patch,
> MAPREDUCE-6995.005.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-6995) Uploader tool for Distributed Cache Deploy documentation
[
https://issues.apache.org/jira/browse/MAPREDUCE-6995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331177#comment-16331177
]
Robert Kanter edited comment on MAPREDUCE-6995 at 1/18/18 8:41 PM:
---
Some minor comments:
# "The tool then returns a suggestion how to set ..." should be "The tool then
returns a suggestion *of* how to set ..."
# "Defaults to the default filesystem set by fs.defaultFS." - fs.defaultFs
should have the ` to make it monospaced.
# Can you double check that paths starting with just {{hdfs:/}} work? (as
opposed to {{hdfs://}})
# The explanations of the arguments (e.g. {{-initialReplication}}) should have
the ` around the argument names to make them monospaced.
# "If this value is set to low like a constant 10 ..." is more clear and
consistent with the next sentence if it's written as "If this is set to a low
value like 10 ..."
# The description for {{-acceptableReplication}} should say what it actually
does, not just the requirements for it. Something like "The tool will wait
until the tarball has been replicated this number of times before exiting."
# "The frameworkuploader tool has the following arguments to control, which
jars end up in the framework tarball:" should not have the comma.
# "This is an input classpath that is iterated through. jars files found will
be added to the tarball. Defaults to the classpath." should be "This is *the*
input classpath to source jar files from to add to the tarball. *It defaults to
the classpath as returned by the {{hadoop classpath}} command.*"
# "This is a comma separated regex array to filter the jar file names to
include from the class path." should be "This is a comma separated regex array
to filter the jar file names to *exclude* from the class path."
# It might be good to give an example of {{-nosymlink}} as it's not clear just
from the description of what this does. Something like "For example,
{{/a/foo.jar}} and a symlink {{/a/bar.jar}} that points to {{/a/foo.jar}} would
normally add foo.jar and bar.jar to the tarball as separate files despite them
actually being the same file. This flag would make the tool exclude
{{/a/bar.jar}} so only one copy of the file is added."
# In the {{testExplicitFilesystem}} test, we should have a similar test where
{{-target}} and {{FS_DEFAULT_NAME_KEY}} have different filesystems.
# The {{frameworkupload}} command should be added to the MapredCommands.md file
was (Author: rkanter):
Some minor comments:
# "The tool then returns a suggestion how to set ..." should be "The tool then
returns a suggestion *of* how to set ..."
# "Defaults to the default filesystem set by fs.defaultFS." - fs.defaultFs
should have the ` to make it monospaced.
# Can you double check that paths starting with just {{hdfs:/}} work? (as
opposed to {{hdfs://}})
# The explanations of the arguments (e.g. {{-initialReplication}}) should have
the ` around the argument names to make them monospaced.
# "If this value is set to low like a constant 10 ..." is more clear and
consistent with the next sentence if it's written as "If this is set to a low
value like 10 ..."
# The description for {{-acceptableReplication}} should say what it actually
does, not just the requirements for it. Something like "The tool will wait
until the tarball has been replicated this number of times before exiting."
# "The frameworkuploader tool has the following arguments to control, which
jars end up in the framework tarball:" should not have the comma.
# "This is an input classpath that is iterated through. jars files found will
be added to the tarball. Defaults to the classpath." should be "This is *the*
input classpath to source jar files from to add to the tarball. *It defaults to
the classpath as returned by the {{hadoop classpath}} command.*"
# "This is a comma separated regex array to filter the jar file names to
include from the class path." should be "This is a comma separated regex array
to filter the jar file names to *exclude* from the class path."
# It might be good to give an example of {{-nosymlink}} as it's not clear just
from the description of what this does. Something like "For example,
{{/a/foo.jar}} and a symlink {{/a/bar.jar}} that points to {{/a/foo.jar}} would
normally add foo.jar and bar.jar to the tarball as separate files despite them
actually being the same file. This flag would make the tool exclude
{{/a/bar.jar}} so only one copy of the file is added."
# In the {{testExplicitFilesystem}} test, we should have a similar test where
{{-target}} and {{FS_DEFAULT_NAME_KEY}} have different filesystems.
> Uploader tool for Distributed Cache Deploy documentation
>
>
> Key: MAPREDUCE-6995
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6995
> Project: Hadoop Map/Reduce
>
[jira] [Commented] (MAPREDUCE-6995) Uploader tool for Distributed Cache Deploy documentation
[
https://issues.apache.org/jira/browse/MAPREDUCE-6995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331177#comment-16331177
]
Robert Kanter commented on MAPREDUCE-6995:
--
Some minor comments:
# "The tool then returns a suggestion how to set ..." should be "The tool then
returns a suggestion *of* how to set ..."
# "Defaults to the default filesystem set by fs.defaultFS." - fs.defaultFs
should have the ` to make it monospaced.
# Can you double check that paths starting with just {{hdfs:/}} work? (as
opposed to {{hdfs://}})
# The explanations of the arguments (e.g. {{-initialReplication}}) should have
the ` around the argument names to make them monospaced.
# "If this value is set to low like a constant 10 ..." is more clear and
consistent with the next sentence if it's written as "If this is set to a low
value like 10 ..."
# The description for {{-acceptableReplication}} should say what it actually
does, not just the requirements for it. Something like "The tool will wait
until the tarball has been replicated this number of times before exiting."
# "The frameworkuploader tool has the following arguments to control, which
jars end up in the framework tarball:" should not have the comma.
# "This is an input classpath that is iterated through. jars files found will
be added to the tarball. Defaults to the classpath." should be "This is *the*
input classpath to source jar files from to add to the tarball. *It defaults to
the classpath as returned by the {{hadoop classpath}} command.*"
# "This is a comma separated regex array to filter the jar file names to
include from the class path." should be "This is a comma separated regex array
to filter the jar file names to *exclude* from the class path."
# It might be good to give an example of {{-nosymlink}} as it's not clear just
from the description of what this does. Something like "For example,
{{/a/foo.jar}} and a symlink {{/a/bar.jar}} that points to {{/a/foo.jar}} would
normally add foo.jar and bar.jar to the tarball as separate files despite them
actually being the same file. This flag would make the tool exclude
{{/a/bar.jar}} so only one copy of the file is added."
# In the {{testExplicitFilesystem}} test, we should have a similar test where
{{-target}} and {{FS_DEFAULT_NAME_KEY}} have different filesystems.
> Uploader tool for Distributed Cache Deploy documentation
>
>
> Key: MAPREDUCE-6995
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6995
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Major
> Attachments: MAPREDUCE-6995.000.patch, MAPREDUCE-6995.001.patch,
> MAPREDUCE-6995.002.patch, MAPREDUCE-6995.003.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7032) Add the ability to specify a delayed replication count
[ https://issues.apache.org/jira/browse/MAPREDUCE-7032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-7032: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.1.0 Status: Resolved (was: Patch Available) Thanks [~miklos.szeg...@cloudera.com]. Committed to trunk! > Add the ability to specify a delayed replication count > -- > > Key: MAPREDUCE-7032 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7032 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Major > Fix For: 3.1.0 > > Attachments: MAPREDUCE-7032.000.patch, MAPREDUCE-7032.001.patch, > MAPREDUCE-7032.002.patch, MAPREDUCE-7032.003.patch > > > Setting the delayed replication count is more robust -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7032) Add the ability to specify a delayed replication count
[ https://issues.apache.org/jira/browse/MAPREDUCE-7032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327555#comment-16327555 ] Robert Kanter commented on MAPREDUCE-7032: -- +1 LGTM > Add the ability to specify a delayed replication count > -- > > Key: MAPREDUCE-7032 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7032 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Major > Attachments: MAPREDUCE-7032.000.patch, MAPREDUCE-7032.001.patch, > MAPREDUCE-7032.002.patch, MAPREDUCE-7032.003.patch > > > Setting the delayed replication count is more robust -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7032) Add the ability to specify a delayed replication count
[
https://issues.apache.org/jira/browse/MAPREDUCE-7032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324838#comment-16324838
]
Robert Kanter commented on MAPREDUCE-7032:
--
Some trivial/minor things:
- {{new HashMap();}} can be just {{new HashMap<>();}}
- The second {{// Start with 0s for each offset}} comment looks like it doesn't
belong (copy-paste)
-- In fact, we can simplify this loop because the HashMap is already populated.
You can use a foreach on an Entry here.
- I think {{blockCount}} can be a {{}} map instead of a {{}} map because we're only counting in the value, and it should be less
than the max replication, which is an integer; then also {{getReplication}}
should return an {{int}}.
- I think we should rename {{getReplication}} to
{{getSmallestReplicatedBlockCount}} or something like that
> Add the ability to specify a delayed replication count
> --
>
> Key: MAPREDUCE-7032
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7032
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Attachments: MAPREDUCE-7032.000.patch, MAPREDUCE-7032.001.patch,
> MAPREDUCE-7032.002.patch
>
>
> Setting the delayed replication count is more robust
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7032) Add the ability to specify a delayed replication count
[
https://issues.apache.org/jira/browse/MAPREDUCE-7032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324710#comment-16324710
]
Robert Kanter commented on MAPREDUCE-7032:
--
The approach looks good overall. Here's some comments:
- {{fileSystem.getFileBlockLocations(targetPath, 0, 1024).length <
acceptableReplication}} only checks for the first block (well, that sort of
depends on the block size, but it's likely the first block). We should
probably check all of the blocks to be safe in case the first block is
replicated well, but some other block is not.
-- Unfortunately, HDFS doesn't have a simple API for that. You're probably
going to have to get all the block locations and then reorganize them to figure
out if any blocks are lower than {{acceptableReplication}}
- Instead of "Operation timed out in %d seconds", we should use a more helpful
message like "Timed out after %d seconds while waiting for acceptable
replication of %d (current replication is %d)" or something like that.
- The description in the arguments for the "timeout" should say what the
timeout is for. i.e. "Desired timeout for the acceptable replication in
seconds"
- The description for the desired replications and timeout should say what
their default values are.
- It would be good to add or update the unit tests to test the replication
changes
This isn't part of this patch, but I was just thinking about the failure
scenario when the uploader fails halfway (e.g. network issue). The tool will
print an error, but the partially uploaded file is left on the system and the
tool still has a clean exit code. Obviously if the network is out the tool
can't delete the file, but we should at least try to and also return a bad exit
code so automation tools know that it failed.
> Add the ability to specify a delayed replication count
> --
>
> Key: MAPREDUCE-7032
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7032
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Attachments: MAPREDUCE-7032.000.patch
>
>
> Setting the delayed replication count is more robust
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7030) Uploader tool should ignore symlinks to the same directory
[ https://issues.apache.org/jira/browse/MAPREDUCE-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-7030: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.1.0 Status: Resolved (was: Patch Available) Thanks [~miklos.szeg...@cloudera.com] and [~grepas] for the review. Committed to trunk! > Uploader tool should ignore symlinks to the same directory > -- > > Key: MAPREDUCE-7030 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7030 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Minor > Fix For: 3.1.0 > > Attachments: MAPREDUCE-7030.000.patch, MAPREDUCE-7030.001.patch, > MAPREDUCE-7030.002.patch, MAPREDUCE-7030.003.patch, MAPREDUCE-7030.004.patch, > MAPREDUCE-7030.005.patch, MAPREDUCE-7030.006.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7030) Uploader tool should ignore symlinks to the same directory
[ https://issues.apache.org/jira/browse/MAPREDUCE-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324621#comment-16324621 ] Robert Kanter commented on MAPREDUCE-7030: -- +1 LGTM > Uploader tool should ignore symlinks to the same directory > -- > > Key: MAPREDUCE-7030 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7030 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Minor > Attachments: MAPREDUCE-7030.000.patch, MAPREDUCE-7030.001.patch, > MAPREDUCE-7030.002.patch, MAPREDUCE-7030.003.patch, MAPREDUCE-7030.004.patch, > MAPREDUCE-7030.005.patch, MAPREDUCE-7030.006.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7030) Uploader tool should ignore symlinks to the same directory
[
https://issues.apache.org/jira/browse/MAPREDUCE-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16322725#comment-16322725
]
Robert Kanter commented on MAPREDUCE-7030:
--
The patch looks good. It doesn't look like it would be too hard to write unit
tests if you use the nio API like in the code. You can use
{{Files#createSymbolicLink}} to create symlinks. Adding a few symlinks to
{{TestFrameworkUploader#testCollectPackages}} to handle the different cases
would probably be enough.
> Uploader tool should ignore symlinks to the same directory
> --
>
> Key: MAPREDUCE-7030
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7030
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Minor
> Attachments: MAPREDUCE-7030.000.patch, MAPREDUCE-7030.001.patch,
> MAPREDUCE-7030.002.patch, MAPREDUCE-7030.003.patch, MAPREDUCE-7030.004.patch,
> MAPREDUCE-7030.005.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7018) Apply erasure coding properly to framework tarball and support plain tar
[
https://issues.apache.org/jira/browse/MAPREDUCE-7018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated MAPREDUCE-7018:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.1.0
Status: Resolved (was: Patch Available)
Thanks [~miklos.szeg...@cloudera.com]. Committed to trunk!
> Apply erasure coding properly to framework tarball and support plain tar
>
>
> Key: MAPREDUCE-7018
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7018
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Fix For: 3.1.0
>
> Attachments: MAPREDUCE-7018.000.patch, MAPREDUCE-7018.001.patch
>
>
> {code}
> 2017-12-01 17:54:51,753 INFO uploader.FrameworkUploader: Disabling Erasure
> Coding for path: hdfs://machine:9000/tmp/mr-framework.tar.gz
> 2017-12-01 17:54:51,779 ERROR uploader.FrameworkUploader: Error in execution
> Attempt to set an erasure coding policy for a file /tmp/mr-framework.tar.gz
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirErasureCodingOp.setErasureCodingPolicyXAttr(FSDirErasureCodingOp.java:147)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirErasureCodingOp.setErasureCodingPolicy(FSDirErasureCodingOp.java:127)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setErasureCodingPolicy(FSNamesystem.java:7291)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setErasureCodingPolicy(NameNodeRpcServer.java:2115)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setErasureCodingPolicy(ClientNamenodeProtocolServerSideTranslatorPB.java:1552)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7018) Apply erasure coding properly to framework tarball and support plain tar
[
https://issues.apache.org/jira/browse/MAPREDUCE-7018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16286628#comment-16286628
]
Robert Kanter commented on MAPREDUCE-7018:
--
+1
> Apply erasure coding properly to framework tarball and support plain tar
>
>
> Key: MAPREDUCE-7018
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7018
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Attachments: MAPREDUCE-7018.000.patch, MAPREDUCE-7018.001.patch
>
>
> {code}
> 2017-12-01 17:54:51,753 INFO uploader.FrameworkUploader: Disabling Erasure
> Coding for path: hdfs://machine:9000/tmp/mr-framework.tar.gz
> 2017-12-01 17:54:51,779 ERROR uploader.FrameworkUploader: Error in execution
> Attempt to set an erasure coding policy for a file /tmp/mr-framework.tar.gz
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirErasureCodingOp.setErasureCodingPolicyXAttr(FSDirErasureCodingOp.java:147)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirErasureCodingOp.setErasureCodingPolicy(FSDirErasureCodingOp.java:127)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setErasureCodingPolicy(FSNamesystem.java:7291)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setErasureCodingPolicy(NameNodeRpcServer.java:2115)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setErasureCodingPolicy(ClientNamenodeProtocolServerSideTranslatorPB.java:1552)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6994) Uploader tool for Distributed Cache Deploy code changes
[ https://issues.apache.org/jira/browse/MAPREDUCE-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-6994: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.1.0 Status: Resolved (was: Patch Available) Thanks [~miklos.szeg...@cloudera.com]. Committed to trunk! > Uploader tool for Distributed Cache Deploy code changes > --- > > Key: MAPREDUCE-6994 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6994 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi > Fix For: 3.1.0 > > Attachments: MAPREDUCE-6994.000.patch, MAPREDUCE-6994.001.patch, > MAPREDUCE-6994.002.patch, MAPREDUCE-6994.003.patch, MAPREDUCE-6994.004.patch, > MAPREDUCE-6994.005.patch, MAPREDUCE-6994.006.patch, MAPREDUCE-6994.007.patch, > MAPREDUCE-6994.008.patch, MAPREDUCE-6994.009.patch > > > The proposal is to create a tool that collects all available jars in the > Hadoop classpath and adds them to a single tarball file. It then uploads the > resulting archive to an HDFS directory. This saves the cluster administrator > from having to set this up manually for Distributed Cache Deploy. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6994) Uploader tool for Distributed Cache Deploy code changes
[ https://issues.apache.org/jira/browse/MAPREDUCE-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273775#comment-16273775 ] Robert Kanter commented on MAPREDUCE-6994: -- +1 on the 009 patch pending Jenkins > Uploader tool for Distributed Cache Deploy code changes > --- > > Key: MAPREDUCE-6994 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6994 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi > Attachments: MAPREDUCE-6994.000.patch, MAPREDUCE-6994.001.patch, > MAPREDUCE-6994.002.patch, MAPREDUCE-6994.003.patch, MAPREDUCE-6994.004.patch, > MAPREDUCE-6994.005.patch, MAPREDUCE-6994.006.patch, MAPREDUCE-6994.007.patch, > MAPREDUCE-6994.008.patch, MAPREDUCE-6994.009.patch > > > The proposal is to create a tool that collects all available jars in the > Hadoop classpath and adds them to a single tarball file. It then uploads the > resulting archive to an HDFS directory. This saves the cluster administrator > from having to set this up manually for Distributed Cache Deploy. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6994) Uploader tool for Distributed Cache Deploy code changes
[
https://issues.apache.org/jira/browse/MAPREDUCE-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273697#comment-16273697
]
Robert Kanter commented on MAPREDUCE-6994:
--
The 008 patch looks good. Just a few trivial things:
- {{beginUpload}} is marked as {{\@VisibleForTesting}} but it's also private
and not used in any tests. We should remove the {{\@VisibleForTesting}}
- Should we set the default replication to 10? I believe that's what's
normally used by the distributed cache.
- In {{testArguments}}, did you mean to pass {{"C"}} for the {{"-blacklist"}}
argument?
- The check if {{targetPath == null}} in {{buildPackage}} seems redundant.
{{buildPackage}} already calls {{beginUpload}}, which sets and validates it.
In fact, instead of making this a class variable, why not simply have
{{beginUpload}} return it or accept it as an argument? It's only ever used in
these two methods.
> Uploader tool for Distributed Cache Deploy code changes
> ---
>
> Key: MAPREDUCE-6994
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6994
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Attachments: MAPREDUCE-6994.000.patch, MAPREDUCE-6994.001.patch,
> MAPREDUCE-6994.002.patch, MAPREDUCE-6994.003.patch, MAPREDUCE-6994.004.patch,
> MAPREDUCE-6994.005.patch, MAPREDUCE-6994.006.patch, MAPREDUCE-6994.007.patch,
> MAPREDUCE-6994.008.patch
>
>
> The proposal is to create a tool that collects all available jars in the
> Hadoop classpath and adds them to a single tarball file. It then uploads the
> resulting archive to an HDFS directory. This saves the cluster administrator
> from having to set this up manually for Distributed Cache Deploy.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6994) Uploader tool for Distributed Cache Deploy code changes
[
https://issues.apache.org/jira/browse/MAPREDUCE-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246377#comment-16246377
]
Robert Kanter commented on MAPREDUCE-6994:
--
Thanks for working on this [~miklos.szeg...@cloudera.com] and [~yufeigu] for
reviews so far.
Looks good overall. Here's some more comments:
# The new pom is missing the {{}} and {{}} elements for itself.
Take a look at a sibling pom like {{hadoop-mapreduce-client-shuffle}} for an
example.
# The new pom should declare all direct dependencies. I'm actually curious how
it's compiling with no dependencies defined...
# Why are we using a mix of {{LOG}} and {{System.out.print}}? We should use
{{LOG}}.
# In {{FrameworkUploader#buildPackage}}, instead of using our own {{buffer}}
with a {{while}} loop, we can just use {{IOUtils#copy}} from commons-io.
# In {{FrameworkUploader#buildPackage}}, we can use try-with-resources for
{{out}} and {{inputStream}}
# In {{FrameworkUploader}}, instead of manually formatting and printing out the
Help text, you can use the {{HelpFormatter}} from cli-commons along with the
{{Options}} you're already using. You can see an example of this in other CLI
tools, such as {{HadoopArchiveLogs}}.
# In {{TestFrameworkUploader}}, we should move {{RANDOM}} and {{TEST_DIR}} into
an {{@Before}} {{setUp}} method.
# Please file a followup JIRA for Windows support
> Uploader tool for Distributed Cache Deploy code changes
> ---
>
> Key: MAPREDUCE-6994
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6994
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Attachments: MAPREDUCE-6994.000.patch, MAPREDUCE-6994.001.patch,
> MAPREDUCE-6994.002.patch, MAPREDUCE-6994.003.patch, MAPREDUCE-6994.004.patch,
> MAPREDUCE-6994.005.patch
>
>
> The proposal is to create a tool that collects all available jars in the
> Hadoop classpath and adds them to a single tarball file. It then uploads the
> resulting archive to an HDFS directory. This saves the cluster administrator
> from having to set this up manually for Distributed Cache Deploy.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6970) archive-logs tool should throttle container requests
[
https://issues.apache.org/jira/browse/MAPREDUCE-6970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated MAPREDUCE-6970:
-
Issue Type: Improvement (was: Bug)
> archive-logs tool should throttle container requests
>
>
> Key: MAPREDUCE-6970
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6970
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Robert Kanter
>
> The {{mapred archive-logs}} command currently has no way to throttle the
> number of requested containers. For example, we recently saw a busy cluster
> where the tool hadn't been run for a while and there were about 20,000 apps
> to process. This meant that the tool tried to request 20,000 containers and
> got a ton of GC and then OOM trying to handle that.
> This problem can be mitigated by setting {{-maxEligibleApps}} to a more
> reasonable value, but doing so would require running the tool multiple times;
> plus, the default value is {{-1}} (all).
> We should add a way to throttle the max number of concurrently running
> containers that the tool manages. Something like {{-concurrency }} where
> it would only allow up to {{n}} containers at a time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6970) archive-logs tool should throttle container requests
Robert Kanter created MAPREDUCE-6970:
Summary: archive-logs tool should throttle container requests
Key: MAPREDUCE-6970
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6970
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 3.0.0-alpha1, 2.8.0
Reporter: Robert Kanter
The {{mapred archive-logs}} command currently has no way to throttle the number
of requested containers. For example, we recently saw a busy cluster where the
tool hadn't been run for a while and there were about 20,000 apps to process.
This meant that the tool tried to request 20,000 containers and got a ton of GC
and then OOM trying to handle that.
This problem can be mitigated by setting {{-maxEligibleApps}} to a more
reasonable value, but doing so would require running the tool multiple times;
plus, the default value is {{-1}} (all).
We should add a way to throttle the max number of concurrently running
containers that the tool manages. Something like {{-concurrency }} where it
would only allow up to {{n}} containers at a time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6968) Staging directory erasure coding config property has a typo
[ https://issues.apache.org/jira/browse/MAPREDUCE-6968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16181218#comment-16181218 ] Robert Kanter commented on MAPREDUCE-6968: -- Oops, missed that. Thanks for catching it. +1 > Staging directory erasure coding config property has a typo > --- > > Key: MAPREDUCE-6968 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6968 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 3.0.0-beta1 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: MAPREDUCE-6968.001.patch > > > TestMapreduceConfigFields has been failing since MAPREDUCE-6954. > MRJobConfig#MR_AM_STAGING_DIR_ERASURECODING_ENABLED is defined as > "yarn.app.mapreduce.am.staging-direrasurecoding.enabled" but the property is > listed as "yarn.app.mapreduce.am.staging-dir.erasurecoding.enabled" in > mapred-default.xml. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6954) Disable erasure coding for files that are uploaded to the MR staging area
[ https://issues.apache.org/jira/browse/MAPREDUCE-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170383#comment-16170383 ] Robert Kanter commented on MAPREDUCE-6954: -- Oops, I was committing some other stuff that day too, and accidentally put the name of one of those in the commit messages :( trunk: https://github.com/apache/hadoop/commit/3a8d57a0a2e047b34be82f602a2b6cf5593d2125 branch-3.0: https://github.com/apache/hadoop/commit/293ab432e116b93ada6ee56e518ae07aa853ab41 I've just reverted those two: trunk: https://github.com/apache/hadoop/commit/5f496683fb00ba26a6bf5a506ae87d4bc4088727 https://github.com/apache/hadoop/commit/3847b8c861205d9c85c866ec3477e8b0d19bd659 And recommitted the code with the correct message: trunk: https://github.com/apache/hadoop/commit/0adc0471d0c06f66a31060f270dcb50a7b4ffafa https://github.com/apache/hadoop/commit/917e875e511589c297971f6a14fd85de7925409f > Disable erasure coding for files that are uploaded to the MR staging area > - > > Key: MAPREDUCE-6954 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6954 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Affects Versions: 3.0.0-alpha4 >Reporter: Peter Bacsko >Assignee: Peter Bacsko > Fix For: 3.0.0-beta1 > > Attachments: MAPREDUCE-6954-001.patch, MAPREDUCE-6954-002.patch, > MAPREDUCE-6954-003.patch, MAPREDUCE-6954-004.patch > > > Depending on the encoder/decoder used and the type or MR workload, EC might > negatively affect the performance of an MR job if too many files are > localized. > In such a scenario, users might want to disable EC in the staging area to > speed up the execution. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6954) Disable erasure coding for files that are uploaded to the MR staging area
[ https://issues.apache.org/jira/browse/MAPREDUCE-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-6954: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-beta1 Status: Resolved (was: Patch Available) Thanks [~pbacsko]. Committed to trunk and branch-3.0! > Disable erasure coding for files that are uploaded to the MR staging area > - > > Key: MAPREDUCE-6954 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6954 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Affects Versions: 3.0.0-alpha4 >Reporter: Peter Bacsko >Assignee: Peter Bacsko > Fix For: 3.0.0-beta1 > > Attachments: MAPREDUCE-6954-001.patch, MAPREDUCE-6954-002.patch, > MAPREDUCE-6954-003.patch, MAPREDUCE-6954-004.patch > > > Depending on the encoder/decoder used and the type or MR workload, EC might > negatively affect the performance of an MR job if too many files are > localized. > In such a scenario, users might want to disable EC in the staging area to > speed up the execution. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6954) Disable erasure coding for files that are uploaded to the MR staging area
[ https://issues.apache.org/jira/browse/MAPREDUCE-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168352#comment-16168352 ] Robert Kanter commented on MAPREDUCE-6954: -- +1 > Disable erasure coding for files that are uploaded to the MR staging area > - > > Key: MAPREDUCE-6954 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6954 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Affects Versions: 3.0.0-alpha4 >Reporter: Peter Bacsko >Assignee: Peter Bacsko > Attachments: MAPREDUCE-6954-001.patch, MAPREDUCE-6954-002.patch, > MAPREDUCE-6954-003.patch, MAPREDUCE-6954-004.patch > > > Depending on the encoder/decoder used and the type or MR workload, EC might > negatively affect the performance of an MR job if too many files are > localized. > In such a scenario, users might want to disable EC in the staging area to > speed up the execution. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6954) Disable erasure coding for files that are uploaded to the MR staging area
[
https://issues.apache.org/jira/browse/MAPREDUCE-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166857#comment-16166857
]
Robert Kanter commented on MAPREDUCE-6954:
--
Almost there:
# I'm not a fan of double negatives in booleans - they make things hard to
read. I understand what you're doing here, relying on the default value to be
true instead of setting it (that's a good idea btw), but instead of using
{{!shouldBeDisabled}} let's use {{default}} instead. That's much easier to
follow.
#- In a similar vein, let's rename {{testErasureCodingNotDisabled}} to
{{testErasureCodingDefault}}.
# There's some new checkstyle warnings about indenting and line length.
> Disable erasure coding for files that are uploaded to the MR staging area
> -
>
> Key: MAPREDUCE-6954
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6954
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Affects Versions: 3.0.0-alpha4
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: MAPREDUCE-6954-001.patch, MAPREDUCE-6954-002.patch
>
>
> Depending on the encoder/decoder used and the type or MR workload, EC might
> negatively affect the performance of an MR job if too many files are
> localized.
> In such a scenario, users might want to disable EC in the staging area to
> speed up the execution.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6954) Disable erasure coding for files that are uploaded to the MR staging area
[
https://issues.apache.org/jira/browse/MAPREDUCE-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162283#comment-16162283
]
Robert Kanter commented on MAPREDUCE-6954:
--
Overall looks good. A couple minor things:
- Update mapred-default like you suggested
- The staging dir config is {{yarn.app.mapreduce.am.staging-dir}} and defined
in MRJobConfig as {{MR_AM_STAGING_DIR = MR_AM_PREFIX+"staging-dir"}}. I think
we should be consistent with this and make
{{MR_JOB_STAGING_ERASURECODING_ENABLED =
"mapreduce.job.staging-dir.erasurecoding.enabled"}} be
{{MR_AM_STAGING_DIR_ERASURECODING_ENABLED = MR_AM_STAGING_DIR +
"erasurecoding.enabled"}} which would resolve to
{{yarn.app.mapreduce.am.staging-dir.erasurecoding.enabled}}.
- It would be good to add a unit test.
> Disable erasure coding for files that are uploaded to the MR staging area
> -
>
> Key: MAPREDUCE-6954
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6954
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Affects Versions: 3.0.0-alpha4
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: MAPREDUCE-6954-001.patch
>
>
> Depending on the encoder/decoder used and the type or MR workload, EC might
> negatively affect the performance of an MR job if too many files are
> localized.
> In such a scenario, users might want to disable EC in the staging area to
> speed up the execution.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6953) Skip the testcase testJobWithChangePriority if FairScheduler is used
[
https://issues.apache.org/jira/browse/MAPREDUCE-6953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated MAPREDUCE-6953:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.1.0
3.0.0
2.9.0
Status: Resolved (was: Patch Available)
Thanks [~pbacsko]. Committed to trunk, branch-3.0, and branch-2!
> Skip the testcase testJobWithChangePriority if FairScheduler is used
>
>
> Key: MAPREDUCE-6953
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6953
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: client
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Fix For: 2.9.0, 3.0.0, 3.1.0
>
> Attachments: MAPREDUCE-6953-001.patch
>
>
> We run the unit tests with Fair Scheduler downstream. FS does not support
> priorities at the moment, so TestMRJobs#testJobWithChangePriority fails.
> Just add {{Assume.assumeFalse(usingFairScheduler);}} and JUnit will skip the
> test.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6953) Skip the testcase testJobWithChangePriority if FairScheduler is used
[
https://issues.apache.org/jira/browse/MAPREDUCE-6953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16159163#comment-16159163
]
Robert Kanter commented on MAPREDUCE-6953:
--
+1
> Skip the testcase testJobWithChangePriority if FairScheduler is used
>
>
> Key: MAPREDUCE-6953
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6953
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: client
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: MAPREDUCE-6953-001.patch
>
>
> We run the unit tests with Fair Scheduler downstream. FS does not support
> priorities at the moment, so TestMRJobs#testJobWithChangePriority fails.
> Just add {{Assume.assumeFalse(usingFairScheduler);}} and JUnit will skip the
> test.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6936) Remove unnecessary dependency of hadoop-yarn-server-common from hadoop-mapreduce-client-common
[ https://issues.apache.org/jira/browse/MAPREDUCE-6936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-6936: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-beta1 2.9.0 Status: Resolved (was: Patch Available) Thanks [~haibochen]. Committed to trunk and branch-2. > Remove unnecessary dependency of hadoop-yarn-server-common from > hadoop-mapreduce-client-common > --- > > Key: MAPREDUCE-6936 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6936 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Haibo Chen >Assignee: Haibo Chen > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: MAPREDUCE-6936.00.patch > > > The dependency of hadoop-yarn-server-common in hadoop-mapreduce-client-common > seems unnecessary, as > it is not using as of the classes from hadoop-yarn-server-common. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6936) Remove unnecessary dependency of hadoop-yarn-server-common from hadoop-mapreduce-client-common
[ https://issues.apache.org/jira/browse/MAPREDUCE-6936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129594#comment-16129594 ] Robert Kanter commented on MAPREDUCE-6936: -- +1 Client artifacts shouldn't be depending on server artifacts > Remove unnecessary dependency of hadoop-yarn-server-common from > hadoop-mapreduce-client-common > --- > > Key: MAPREDUCE-6936 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6936 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: MAPREDUCE-6936.00.patch > > > The dependency of hadoop-yarn-server-common in hadoop-mapreduce-client-common > seems unnecessary, as > it is not using as of the classes from hadoop-yarn-server-common. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6288) mapred job -status fails with AccessControlException
[
https://issues.apache.org/jira/browse/MAPREDUCE-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126211#comment-16126211
]
Robert Kanter commented on MAPREDUCE-6288:
--
MAPREDUCE-6925 has been created to figure out a better solution (MAPREDUCE-5875
was Closed so it couldn't be Reopened).
> mapred job -status fails with AccessControlException
> -
>
> Key: MAPREDUCE-6288
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6288
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Robert Kanter
>Priority: Blocker
> Attachments: MAPREDUCE-6288.002.patch, MAPREDUCE-6288-gera-001.patch,
> MAPREDUCE-6288.patch
>
>
> After MAPREDUCE-5875, we're seeing this Exception when trying to do {{mapred
> job -status job_1427080398288_0001}}
> {noformat}
> Exception in thread "main" org.apache.hadoop.security.AccessControlException:
> Permission denied: user=jenkins, access=EXECUTE,
> inode="/user/history/done":mapred:hadoop:drwxrwx---
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:180)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:137)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6553)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6535)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:6460)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1919)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:545)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
> at
> org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1213)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1201)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1191)
> at
> org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:299)
> at
> org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:265)
> at org.apache.hadoop.hdfs.DFSInputStream.(DFSInputStream.java:257)
> at
[jira] [Commented] (MAPREDUCE-5875) Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
[
https://issues.apache.org/jira/browse/MAPREDUCE-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16108026#comment-16108026
]
Robert Kanter commented on MAPREDUCE-5875:
--
[~djp], it's "Closed" so I don't think we can reopen it. We should probably
just open up a new JIRA to address this issue.
> Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
> ---
>
> Key: MAPREDUCE-5875
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5875
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, client, task
>Affects Versions: 2.4.0
>Reporter: Gera Shegalov
>Assignee: Gera Shegalov
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: MAPREDUCE-5875.v01.patch, MAPREDUCE-5875.v02.patch,
> MAPREDUCE-5875.v03.patch, MAPREDUCE-5875.v04.patch, MAPREDUCE-5875.v05.patch,
> MAPREDUCE-5875.v06.patch, MAPREDUCE-5875.v07.patch, MAPREDUCE-5875.v08.patch,
> MAPREDUCE-5875.v09.patch
>
>
> Currently, counter limits "mapreduce.job.counters.*" handled by
> {{org.apache.hadoop.mapreduce.counters.Limits}} are initialized
> asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
> it's taken into account in YarnChild.
> It would be good to make the Limits job-configurable, such that max
> counters/groups is only increased when needed. With the current Limits
> implementation relying on static constants, it's going to be challenging for
> tools that submit jobs concurrently without resorting to class loading
> isolation.
> The patch that I am uploading is not perfect but demonstrates the issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6288) mapred job -status fails with AccessControlException
[
https://issues.apache.org/jira/browse/MAPREDUCE-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16104168#comment-16104168
]
Robert Kanter commented on MAPREDUCE-6288:
--
Oh, I just looked at MAPREDUCE-5875 and see now that we hadn't actually shipped
it yet; I thought we had and reverting it would be a regression, even though it
also causes this issue. Anyway, we haven't shipped it, so I'm okay with
reverting it and reopening the JIRA.
> mapred job -status fails with AccessControlException
> -
>
> Key: MAPREDUCE-6288
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6288
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Robert Kanter
>Priority: Blocker
> Attachments: MAPREDUCE-6288.002.patch, MAPREDUCE-6288-gera-001.patch,
> MAPREDUCE-6288.patch
>
>
> After MAPREDUCE-5875, we're seeing this Exception when trying to do {{mapred
> job -status job_1427080398288_0001}}
> {noformat}
> Exception in thread "main" org.apache.hadoop.security.AccessControlException:
> Permission denied: user=jenkins, access=EXECUTE,
> inode="/user/history/done":mapred:hadoop:drwxrwx---
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:180)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:137)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6553)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6535)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:6460)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1919)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:545)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
> at
> org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1213)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1201)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1191)
> at
> org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:299)
> at
>
[jira] [Commented] (MAPREDUCE-6288) mapred job -status fails with AccessControlException
[
https://issues.apache.org/jira/browse/MAPREDUCE-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16104016#comment-16104016
]
Robert Kanter commented on MAPREDUCE-6288:
--
If we revert MAPREDUCE-5875, then we need to come up with a better solution to
the problem it was trying to solve: if you increase the counter limits in your
job, and then try to use a different client to look at them, it will fail
because the counter limits aren't increased.
> mapred job -status fails with AccessControlException
> -
>
> Key: MAPREDUCE-6288
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6288
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Robert Kanter
>Priority: Blocker
> Attachments: MAPREDUCE-6288.002.patch, MAPREDUCE-6288-gera-001.patch,
> MAPREDUCE-6288.patch
>
>
> After MAPREDUCE-5875, we're seeing this Exception when trying to do {{mapred
> job -status job_1427080398288_0001}}
> {noformat}
> Exception in thread "main" org.apache.hadoop.security.AccessControlException:
> Permission denied: user=jenkins, access=EXECUTE,
> inode="/user/history/done":mapred:hadoop:drwxrwx---
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:180)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:137)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6553)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6535)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:6460)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1919)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:545)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
> at
> org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1213)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1201)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1191)
> at
> org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:299)
> at
>
[jira] [Commented] (MAPREDUCE-6647) MR usage counters use the resources requested instead of the resources allocated
[
https://issues.apache.org/jira/browse/MAPREDUCE-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16067012#comment-16067012
]
Robert Kanter commented on MAPREDUCE-6647:
--
I think [~haibo.chen]'s example supports your use case then. If the minimum
allocation is 2GB but the user requests 1GB, Yarn will give them 2GB but prior
to MR-6647, they'd only be charged for 1GB, which is incorrect.
> MR usage counters use the resources requested instead of the resources
> allocated
>
>
> Key: MAPREDUCE-6647
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6647
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: mapreduce6647.001.patch, mapreduce6647.002.patch,
> mapreduce6647.003.patch, mapreduce6647.004.patch
>
>
> As can be seen in the following snippet, the MR counters for usage use the
> resources requested instead of the resources allocated. The scheduler
> increment-allocation-mb configs could lead to these values not being the
> same. We could change the counters to use the allocated resources in order to
> account for this.
> {code}
> private static void updateMillisCounters(JobCounterUpdateEvent jce,
> TaskAttemptImpl taskAttempt) {
> /***omitted**/
> long duration = (taskAttempt.getFinishTime() -
> taskAttempt.getLaunchTime());
> int mbRequired =
> taskAttempt.getMemoryRequired(taskAttempt.conf, taskType);
> int vcoresRequired = taskAttempt.getCpuRequired(taskAttempt.conf,
> taskType);
> int minSlotMemSize = taskAttempt.conf.getInt(
> YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB,
> YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB);
> int simSlotsRequired =
> minSlotMemSize == 0 ? 0 : (int) Math.ceil((float) mbRequired
> / minSlotMemSize);
> if (taskType == TaskType.MAP) {
> jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_MAPS, simSlotsRequired *
> duration);
> jce.addCounterUpdate(JobCounter.MB_MILLIS_MAPS, duration * mbRequired);
> jce.addCounterUpdate(JobCounter.VCORES_MILLIS_MAPS, duration *
> vcoresRequired);
> jce.addCounterUpdate(JobCounter.MILLIS_MAPS, duration);
> } else {
> jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_REDUCES, simSlotsRequired
> * duration);
> jce.addCounterUpdate(JobCounter.MB_MILLIS_REDUCES, duration *
> mbRequired);
> jce.addCounterUpdate(JobCounter.VCORES_MILLIS_REDUCES, duration *
> vcoresRequired);
> jce.addCounterUpdate(JobCounter.MILLIS_REDUCES, duration);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6904) HADOOP_JOB_HISTORY_OPTS should be HADOOP_JOB_HISTORYSERVER_OPTS in mapred-config.sh
[
https://issues.apache.org/jira/browse/MAPREDUCE-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated MAPREDUCE-6904:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha4
Status: Resolved (was: Patch Available)
Thanks for the review [~aw]. Committed to trunk!
> HADOOP_JOB_HISTORY_OPTS should be HADOOP_JOB_HISTORYSERVER_OPTS in
> mapred-config.sh
> ---
>
> Key: MAPREDUCE-6904
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6904
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: scripts
>Affects Versions: 3.0.0-alpha1
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Critical
> Fix For: 3.0.0-alpha4
>
> Attachments: MAPREDUCE-6904.001.patch
>
>
> After HADOOP-13341, {{HADOOP_JOB_HISTORY_OPTS}} is deprecated in favor of
> {{MAPRED_HISTORYSERVER_OPTS}}, but still works. However, the property is
> actually supposed to be {{HADOOP_JOB_HISTORYSERVER_OPTS}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6901) Remove DistributedCache from trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16061512#comment-16061512
]
Robert Kanter commented on MAPREDUCE-6901:
--
I'd vote for leaving the classes and removing the {{@deprecated}} tags -
everything is still using it, even internally in Hadoop, so it hardly seems
actually deprecated. Cleaning up the Javadoc is a good idea too.
> Remove DistributedCache from trunk
> --
>
> Key: MAPREDUCE-6901
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6901
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: distributed-cache
>Affects Versions: 3.0.0-alpha3
>Reporter: Ray Chiang
>Assignee: Ray Chiang
>Priority: Critical
>
> Doing this as part of Hadoop 3 cleanup.
> DistributedCache has been marked as deprecated forever to the point where the
> change that did it isn't in Git.
> I don't really have a preference for whether we remove it or not, but I'd
> like to have a discussion and have it properly documented as a release not
> for Hadoop 3 before we hit final release. At the very least we can have a
> Release Note that will sum up whatever discussion we have here.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6904) HADOOP_JOB_HISTORY_OPTS should be HADOOP_JOB_HISTORYSERVER_OPTS in mapred-config.sh
[
https://issues.apache.org/jira/browse/MAPREDUCE-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated MAPREDUCE-6904:
-
Status: Patch Available (was: Open)
> HADOOP_JOB_HISTORY_OPTS should be HADOOP_JOB_HISTORYSERVER_OPTS in
> mapred-config.sh
> ---
>
> Key: MAPREDUCE-6904
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6904
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha1
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Critical
> Attachments: MAPREDUCE-6904.001.patch
>
>
> After HADOOP-13341, {{HADOOP_JOB_HISTORY_OPTS}} is deprecated in favor of
> {{MAPRED_HISTORYSERVER_OPTS}}, but still works. However, the property is
> actually supposed to be {{HADOOP_JOB_HISTORYSERVER_OPTS}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6904) HADOOP_JOB_HISTORY_OPTS should be HADOOP_JOB_HISTORYSERVER_OPTS in mapred-config.sh
[
https://issues.apache.org/jira/browse/MAPREDUCE-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated MAPREDUCE-6904:
-
Attachment: MAPREDUCE-6904.001.patch
> HADOOP_JOB_HISTORY_OPTS should be HADOOP_JOB_HISTORYSERVER_OPTS in
> mapred-config.sh
> ---
>
> Key: MAPREDUCE-6904
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6904
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha1
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Critical
> Attachments: MAPREDUCE-6904.001.patch
>
>
> After HADOOP-13341, {{HADOOP_JOB_HISTORY_OPTS}} is deprecated in favor of
> {{MAPRED_HISTORYSERVER_OPTS}}, but still works. However, the property is
> actually supposed to be {{HADOOP_JOB_HISTORYSERVER_OPTS}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6904) HADOOP_JOB_HISTORY_OPTS should be HADOOP_JOB_HISTORYSERVER_OPTS in mapred-config.sh
Robert Kanter created MAPREDUCE-6904:
Summary: HADOOP_JOB_HISTORY_OPTS should be
HADOOP_JOB_HISTORYSERVER_OPTS in mapred-config.sh
Key: MAPREDUCE-6904
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6904
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 3.0.0-alpha1
Reporter: Robert Kanter
Assignee: Robert Kanter
Priority: Critical
After HADOOP-13341, {{HADOOP_JOB_HISTORY_OPTS}} is deprecated in favor of
{{MAPRED_HISTORYSERVER_OPTS}}, but still works. However, the property is
actually supposed to be {{HADOOP_JOB_HISTORYSERVER_OPTS}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6288) mapred job -status fails with AccessControlException
[
https://issues.apache.org/jira/browse/MAPREDUCE-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16037916#comment-16037916
]
Robert Kanter commented on MAPREDUCE-6288:
--
Looking into the 3 commits mentioned (MAPREDUCE-6286, MAPREDUCE-6199, and
MAPREDUCE-5875), it looks like these were properly reverted from branch-2.8 as
well. Somehow it seems like only the CHANGES.txt change made it to branch-2.
[~vinodkv] do you remember this?
> mapred job -status fails with AccessControlException
> -
>
> Key: MAPREDUCE-6288
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6288
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Robert Kanter
>Priority: Blocker
> Attachments: MAPREDUCE-6288.002.patch, MAPREDUCE-6288-gera-001.patch,
> MAPREDUCE-6288.patch
>
>
> After MAPREDUCE-5875, we're seeing this Exception when trying to do {{mapred
> job -status job_1427080398288_0001}}
> {noformat}
> Exception in thread "main" org.apache.hadoop.security.AccessControlException:
> Permission denied: user=jenkins, access=EXECUTE,
> inode="/user/history/done":mapred:hadoop:drwxrwx---
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:180)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:137)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6553)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6535)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:6460)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1919)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:545)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
> at
> org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1213)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1201)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1191)
> at
> org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:299)
> at
>
[jira] [Commented] (MAPREDUCE-6288) mapred job -status fails with AccessControlException
[
https://issues.apache.org/jira/browse/MAPREDUCE-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16037899#comment-16037899
]
Robert Kanter commented on MAPREDUCE-6288:
--
My comment was talking about this JIRA, MAPREDUCE-6288, being reverted. That's
the patch that tried to fix this issue by changing the permissions. If you
look at around March 30 comments, you can see that it was committed and then
reverted.
> mapred job -status fails with AccessControlException
> -
>
> Key: MAPREDUCE-6288
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6288
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Robert Kanter
>Priority: Blocker
> Attachments: MAPREDUCE-6288.002.patch, MAPREDUCE-6288-gera-001.patch,
> MAPREDUCE-6288.patch
>
>
> After MAPREDUCE-5875, we're seeing this Exception when trying to do {{mapred
> job -status job_1427080398288_0001}}
> {noformat}
> Exception in thread "main" org.apache.hadoop.security.AccessControlException:
> Permission denied: user=jenkins, access=EXECUTE,
> inode="/user/history/done":mapred:hadoop:drwxrwx---
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:180)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:137)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6553)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6535)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:6460)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1919)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:545)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
> at
> org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1213)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1201)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1191)
> at
> org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:299)
> at
>
[jira] [Updated] (MAPREDUCE-6871) Allow users to specify racks and nodes for strict locality for AMs
[
https://issues.apache.org/jira/browse/MAPREDUCE-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated MAPREDUCE-6871:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha3
2.9.0
Status: Resolved (was: Patch Available)
Thanks [~haibochen] and [~kasha] for reviews. Committed to branch-2 and trunk!
> Allow users to specify racks and nodes for strict locality for AMs
> --
>
> Key: MAPREDUCE-6871
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6871
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: client
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: MAPREDUCE-6871.001.patch, MAPREDUCE-6871.002.patch,
> MAPREDUCE-6871.003.patch, MAPREDUCE-6871.004.patch, MAPREDUCE-6871.005.patch,
> MAPREDUCE-6871.005.patch
>
>
> YARN-6050 fixed the YARN API to allow multiple {{ResourceRequest}}'s when
> submitting an AM so that you can actually do rack or node locality. We
> should allow MapReduce users to take advantage of this by exposing this
> functionality in some way. The raw YARN API allows for a lot of flexibility
> (e.g. different resources per request, etc), but we don't necessarily want to
> allow the user to do too much here so they don't shoot themselves in the foot
> and we don't make this overly complicated.
> I propose we allow users to specify racks and nodes for strict locality.
> This would allow users to restrict an MR AM to specific racks and/or nodes.
> We could add a new property,
> {{mapreduce.job.am.resource-request.strict.locality}}, which takes a
> comma-separated list of entries like:
> - {{/}}
> - {{//}}
> - {{}} (assumes /default-rack)
> MapReduce would then use this information to create the corresponding
> {{ResourceRequest}}'s.
> For example,
> {{mapreduce.job.am.resource-request.strict.locality=/rack1/node1}} would
> create the following {{ResourceRequest}}'s:
> - resourceName=ANY, relaxLocality=false, capability=
> - resourceName=/rack1, relaxLocality=false, capability=
> - resourceName=node1, relaxLocality=true, capability=
> By default, the property would be unset, and you'd get the normal {{ANY}}
> {{ResourceRequest}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6871) Allow users to specify racks and nodes for strict locality for AMs
[
https://issues.apache.org/jira/browse/MAPREDUCE-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977310#comment-15977310
]
Robert Kanter commented on MAPREDUCE-6871:
--
Test failure unrelated
> Allow users to specify racks and nodes for strict locality for AMs
> --
>
> Key: MAPREDUCE-6871
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6871
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: client
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-6871.001.patch, MAPREDUCE-6871.002.patch,
> MAPREDUCE-6871.003.patch, MAPREDUCE-6871.004.patch, MAPREDUCE-6871.005.patch,
> MAPREDUCE-6871.005.patch
>
>
> YARN-6050 fixed the YARN API to allow multiple {{ResourceRequest}}'s when
> submitting an AM so that you can actually do rack or node locality. We
> should allow MapReduce users to take advantage of this by exposing this
> functionality in some way. The raw YARN API allows for a lot of flexibility
> (e.g. different resources per request, etc), but we don't necessarily want to
> allow the user to do too much here so they don't shoot themselves in the foot
> and we don't make this overly complicated.
> I propose we allow users to specify racks and nodes for strict locality.
> This would allow users to restrict an MR AM to specific racks and/or nodes.
> We could add a new property,
> {{mapreduce.job.am.resource-request.strict.locality}}, which takes a
> comma-separated list of entries like:
> - {{/}}
> - {{//}}
> - {{}} (assumes /default-rack)
> MapReduce would then use this information to create the corresponding
> {{ResourceRequest}}'s.
> For example,
> {{mapreduce.job.am.resource-request.strict.locality=/rack1/node1}} would
> create the following {{ResourceRequest}}'s:
> - resourceName=ANY, relaxLocality=false, capability=
> - resourceName=/rack1, relaxLocality=false, capability=
> - resourceName=node1, relaxLocality=true, capability=
> By default, the property would be unset, and you'd get the normal {{ANY}}
> {{ResourceRequest}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6871) Allow users to specify racks and nodes for strict locality for AMs
[
https://issues.apache.org/jira/browse/MAPREDUCE-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated MAPREDUCE-6871:
-
Attachment: MAPREDUCE-6871.005.patch
I was talking to [~templedf] about regexes and he suggested using
{{String.format}} to make the regex as one String, which makes it easier to
read. The 005 patch does that.
> Allow users to specify racks and nodes for strict locality for AMs
> --
>
> Key: MAPREDUCE-6871
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6871
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: client
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-6871.001.patch, MAPREDUCE-6871.002.patch,
> MAPREDUCE-6871.003.patch, MAPREDUCE-6871.004.patch, MAPREDUCE-6871.005.patch,
> MAPREDUCE-6871.005.patch
>
>
> YARN-6050 fixed the YARN API to allow multiple {{ResourceRequest}}'s when
> submitting an AM so that you can actually do rack or node locality. We
> should allow MapReduce users to take advantage of this by exposing this
> functionality in some way. The raw YARN API allows for a lot of flexibility
> (e.g. different resources per request, etc), but we don't necessarily want to
> allow the user to do too much here so they don't shoot themselves in the foot
> and we don't make this overly complicated.
> I propose we allow users to specify racks and nodes for strict locality.
> This would allow users to restrict an MR AM to specific racks and/or nodes.
> We could add a new property,
> {{mapreduce.job.am.resource-request.strict.locality}}, which takes a
> comma-separated list of entries like:
> - {{/}}
> - {{//}}
> - {{}} (assumes /default-rack)
> MapReduce would then use this information to create the corresponding
> {{ResourceRequest}}'s.
> For example,
> {{mapreduce.job.am.resource-request.strict.locality=/rack1/node1}} would
> create the following {{ResourceRequest}}'s:
> - resourceName=ANY, relaxLocality=false, capability=
> - resourceName=/rack1, relaxLocality=false, capability=
> - resourceName=node1, relaxLocality=true, capability=
> By default, the property would be unset, and you'd get the normal {{ANY}}
> {{ResourceRequest}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6871) Allow users to specify racks and nodes for strict locality for AMs
[
https://issues.apache.org/jira/browse/MAPREDUCE-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated MAPREDUCE-6871:
-
Attachment: MAPREDUCE-6871.005.patch
I was talking to [~templedf] about regexes and he suggested using
{{String.format}} to make the regex as one String, which makes it easier to
read. The 005 patch does that.
> Allow users to specify racks and nodes for strict locality for AMs
> --
>
> Key: MAPREDUCE-6871
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6871
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: client
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-6871.001.patch, MAPREDUCE-6871.002.patch,
> MAPREDUCE-6871.003.patch, MAPREDUCE-6871.004.patch, MAPREDUCE-6871.005.patch,
> MAPREDUCE-6871.005.patch
>
>
> YARN-6050 fixed the YARN API to allow multiple {{ResourceRequest}}'s when
> submitting an AM so that you can actually do rack or node locality. We
> should allow MapReduce users to take advantage of this by exposing this
> functionality in some way. The raw YARN API allows for a lot of flexibility
> (e.g. different resources per request, etc), but we don't necessarily want to
> allow the user to do too much here so they don't shoot themselves in the foot
> and we don't make this overly complicated.
> I propose we allow users to specify racks and nodes for strict locality.
> This would allow users to restrict an MR AM to specific racks and/or nodes.
> We could add a new property,
> {{mapreduce.job.am.resource-request.strict.locality}}, which takes a
> comma-separated list of entries like:
> - {{/}}
> - {{//}}
> - {{}} (assumes /default-rack)
> MapReduce would then use this information to create the corresponding
> {{ResourceRequest}}'s.
> For example,
> {{mapreduce.job.am.resource-request.strict.locality=/rack1/node1}} would
> create the following {{ResourceRequest}}'s:
> - resourceName=ANY, relaxLocality=false, capability=
> - resourceName=/rack1, relaxLocality=false, capability=
> - resourceName=node1, relaxLocality=true, capability=
> By default, the property would be unset, and you'd get the normal {{ANY}}
> {{ResourceRequest}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6871) Allow users to specify racks and nodes for strict locality for AMs
[
https://issues.apache.org/jira/browse/MAPREDUCE-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated MAPREDUCE-6871:
-
Attachment: MAPREDUCE-6871.004.patch
The 004 patch:
- Renames the config to {{mapreduce.job.am.strict-locality}}
- Adds a debug logging check
- Removes the redundant label setting code in a test (I think that's a holdover
from an earlier version)
[~kasha], for your feedback about the {{rackRequests}}: it does need to be a
map. If someone specifies {{/rack1,/rack1/node1}}, we'd need to update the
already created request for rack1 to have the strict locality set to false.
> Allow users to specify racks and nodes for strict locality for AMs
> --
>
> Key: MAPREDUCE-6871
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6871
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: client
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-6871.001.patch, MAPREDUCE-6871.002.patch,
> MAPREDUCE-6871.003.patch, MAPREDUCE-6871.004.patch
>
>
> YARN-6050 fixed the YARN API to allow multiple {{ResourceRequest}}'s when
> submitting an AM so that you can actually do rack or node locality. We
> should allow MapReduce users to take advantage of this by exposing this
> functionality in some way. The raw YARN API allows for a lot of flexibility
> (e.g. different resources per request, etc), but we don't necessarily want to
> allow the user to do too much here so they don't shoot themselves in the foot
> and we don't make this overly complicated.
> I propose we allow users to specify racks and nodes for strict locality.
> This would allow users to restrict an MR AM to specific racks and/or nodes.
> We could add a new property,
> {{mapreduce.job.am.resource-request.strict.locality}}, which takes a
> comma-separated list of entries like:
> - {{/}}
> - {{//}}
> - {{}} (assumes /default-rack)
> MapReduce would then use this information to create the corresponding
> {{ResourceRequest}}'s.
> For example,
> {{mapreduce.job.am.resource-request.strict.locality=/rack1/node1}} would
> create the following {{ResourceRequest}}'s:
> - resourceName=ANY, relaxLocality=false, capability=
> - resourceName=/rack1, relaxLocality=false, capability=
> - resourceName=node1, relaxLocality=true, capability=
> By default, the property would be unset, and you'd get the normal {{ANY}}
> {{ResourceRequest}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-5641) History for failed Application Masters should be made available to the Job History Server
[ https://issues.apache.org/jira/browse/MAPREDUCE-5641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter reassigned MAPREDUCE-5641: Assignee: (was: Robert Kanter) > History for failed Application Masters should be made available to the Job > History Server > - > > Key: MAPREDUCE-5641 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5641 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster, jobhistoryserver >Affects Versions: 2.2.0 >Reporter: Robert Kanter > Attachments: MAPREDUCE-5641.patch, MAPREDUCE-5641.patch > > > Currently, the JHS has no information about jobs whose AMs have failed. This > is because the History is written by the AM to the intermediate folder just > before finishing, so when it fails for any reason, this information isn't > copied there. However, it is not lost as its in the AM's staging directory. > To make the History available in the JHS, all we need to do is have another > mechanism to move the History from the staging directory to the intermediate > directory. The AM also writes a "Summary" file before exiting normally, > which is also unavailable when the AM fails. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6871) Allow users to specify racks and nodes for strict locality for AMs
[
https://issues.apache.org/jira/browse/MAPREDUCE-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated MAPREDUCE-6871:
-
Attachment: MAPREDUCE-6871.003.patch
Thanks for taking a look [~kasha]. I've addressed your feedback.
The 003 patch:
- Moves the code to it's own method
- Replaces the iterating lookup with a hashmap
> Allow users to specify racks and nodes for strict locality for AMs
> --
>
> Key: MAPREDUCE-6871
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6871
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: client
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-6871.001.patch, MAPREDUCE-6871.002.patch,
> MAPREDUCE-6871.003.patch
>
>
> YARN-6050 fixed the YARN API to allow multiple {{ResourceRequest}}'s when
> submitting an AM so that you can actually do rack or node locality. We
> should allow MapReduce users to take advantage of this by exposing this
> functionality in some way. The raw YARN API allows for a lot of flexibility
> (e.g. different resources per request, etc), but we don't necessarily want to
> allow the user to do too much here so they don't shoot themselves in the foot
> and we don't make this overly complicated.
> I propose we allow users to specify racks and nodes for strict locality.
> This would allow users to restrict an MR AM to specific racks and/or nodes.
> We could add a new property,
> {{mapreduce.job.am.resource-request.strict.locality}}, which takes a
> comma-separated list of entries like:
> - {{/}}
> - {{//}}
> - {{}} (assumes /default-rack)
> MapReduce would then use this information to create the corresponding
> {{ResourceRequest}}'s.
> For example,
> {{mapreduce.job.am.resource-request.strict.locality=/rack1/node1}} would
> create the following {{ResourceRequest}}'s:
> - resourceName=ANY, relaxLocality=false, capability=
> - resourceName=/rack1, relaxLocality=false, capability=
> - resourceName=node1, relaxLocality=true, capability=
> By default, the property would be unset, and you'd get the normal {{ANY}}
> {{ResourceRequest}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6201) TestNetworkedJob fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated MAPREDUCE-6201:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha3
Status: Resolved (was: Patch Available)
Thanks [~pbacsko] and [~haibochen] for review. Committed to trunk!
> TestNetworkedJob fails on trunk
> ---
>
> Key: MAPREDUCE-6201
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6201
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Robert Kanter
>Assignee: Peter Bacsko
> Fix For: 3.0.0-alpha3
>
> Attachments: MAPREDUCE-6201-001.patch, MAPREDUCE-6201-002.patch,
> MAPREDUCE-6201-003.patch
>
>
> Currently, {{TestNetworkedJob}} is failing on trunk:
> {noformat}
> Running org.apache.hadoop.mapred.TestNetworkedJob
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 215.01 sec
> <<< FAILURE! - in org.apache.hadoop.mapred.TestNetworkedJob
> testNetworkedJob(org.apache.hadoop.mapred.TestNetworkedJob) Time elapsed:
> 67.363 sec <<< FAILURE!
> java.lang.AssertionError: expected:<0> but was:<2>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.mapred.TestNetworkedJob.testNetworkedJob(TestNetworkedJob.java:195)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6201) TestNetworkedJob fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960069#comment-15960069
]
Robert Kanter commented on MAPREDUCE-6201:
--
+1
> TestNetworkedJob fails on trunk
> ---
>
> Key: MAPREDUCE-6201
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6201
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Robert Kanter
>Assignee: Peter Bacsko
> Attachments: MAPREDUCE-6201-001.patch, MAPREDUCE-6201-002.patch,
> MAPREDUCE-6201-003.patch
>
>
> Currently, {{TestNetworkedJob}} is failing on trunk:
> {noformat}
> Running org.apache.hadoop.mapred.TestNetworkedJob
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 215.01 sec
> <<< FAILURE! - in org.apache.hadoop.mapred.TestNetworkedJob
> testNetworkedJob(org.apache.hadoop.mapred.TestNetworkedJob) Time elapsed:
> 67.363 sec <<< FAILURE!
> java.lang.AssertionError: expected:<0> but was:<2>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.mapred.TestNetworkedJob.testNetworkedJob(TestNetworkedJob.java:195)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6871) Allow users to specify racks and nodes for strict locality for AMs
[
https://issues.apache.org/jira/browse/MAPREDUCE-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated MAPREDUCE-6871:
-
Attachment: MAPREDUCE-6871.002.patch
Thanks [~haibochen] for taking a look. I've addressed your feedback.
The 002 patch:
- Uses a regex for parsing the config. I think this makes the code much
clearer and more readable because I'm also using named captured groups.
- Renames the config property as [~haibochen] suggested.
- Refactored the test code to split it up into different tests and added a few
more cases.
> Allow users to specify racks and nodes for strict locality for AMs
> --
>
> Key: MAPREDUCE-6871
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6871
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: client
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-6871.001.patch, MAPREDUCE-6871.002.patch
>
>
> YARN-6050 fixed the YARN API to allow multiple {{ResourceRequest}}'s when
> submitting an AM so that you can actually do rack or node locality. We
> should allow MapReduce users to take advantage of this by exposing this
> functionality in some way. The raw YARN API allows for a lot of flexibility
> (e.g. different resources per request, etc), but we don't necessarily want to
> allow the user to do too much here so they don't shoot themselves in the foot
> and we don't make this overly complicated.
> I propose we allow users to specify racks and nodes for strict locality.
> This would allow users to restrict an MR AM to specific racks and/or nodes.
> We could add a new property,
> {{mapreduce.job.am.resource-request.strict.locality}}, which takes a
> comma-separated list of entries like:
> - {{/}}
> - {{//}}
> - {{}} (assumes /default-rack)
> MapReduce would then use this information to create the corresponding
> {{ResourceRequest}}'s.
> For example,
> {{mapreduce.job.am.resource-request.strict.locality=/rack1/node1}} would
> create the following {{ResourceRequest}}'s:
> - resourceName=ANY, relaxLocality=false, capability=
> - resourceName=/rack1, relaxLocality=false, capability=
> - resourceName=node1, relaxLocality=true, capability=
> By default, the property would be unset, and you'd get the normal {{ANY}}
> {{ResourceRequest}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6871) Allow users to specify racks and nodes for strict locality for AMs
[
https://issues.apache.org/jira/browse/MAPREDUCE-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated MAPREDUCE-6871:
-
Attachment: MAPREDUCE-6871.001.patch
The 001 patch:
- Adds the new property with the behavior as described above
- Adds unit tests
- I also verified that it behaves correctly in a cluster
> Allow users to specify racks and nodes for strict locality for AMs
> --
>
> Key: MAPREDUCE-6871
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6871
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: client
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-6871.001.patch
>
>
> YARN-6050 fixed the YARN API to allow multiple {{ResourceRequest}}'s when
> submitting an AM so that you can actually do rack or node locality. We
> should allow MapReduce users to take advantage of this by exposing this
> functionality in some way. The raw YARN API allows for a lot of flexibility
> (e.g. different resources per request, etc), but we don't necessarily want to
> allow the user to do too much here so they don't shoot themselves in the foot
> and we don't make this overly complicated.
> I propose we allow users to specify racks and nodes for strict locality.
> This would allow users to restrict an MR AM to specific racks and/or nodes.
> We could add a new property,
> {{mapreduce.job.am.resource-request.strict.locality}}, which takes a
> comma-separated list of entries like:
> - {{/}}
> - {{//}}
> - {{}} (assumes /default-rack)
> MapReduce would then use this information to create the corresponding
> {{ResourceRequest}}'s.
> For example,
> {{mapreduce.job.am.resource-request.strict.locality=/rack1/node1}} would
> create the following {{ResourceRequest}}'s:
> - resourceName=ANY, relaxLocality=false, capability=
> - resourceName=/rack1, relaxLocality=false, capability=
> - resourceName=node1, relaxLocality=true, capability=
> By default, the property would be unset, and you'd get the normal {{ANY}}
> {{ResourceRequest}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6871) Allow users to specify racks and nodes for strict locality for AMs
[
https://issues.apache.org/jira/browse/MAPREDUCE-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated MAPREDUCE-6871:
-
Status: Patch Available (was: Open)
> Allow users to specify racks and nodes for strict locality for AMs
> --
>
> Key: MAPREDUCE-6871
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6871
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: client
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-6871.001.patch
>
>
> YARN-6050 fixed the YARN API to allow multiple {{ResourceRequest}}'s when
> submitting an AM so that you can actually do rack or node locality. We
> should allow MapReduce users to take advantage of this by exposing this
> functionality in some way. The raw YARN API allows for a lot of flexibility
> (e.g. different resources per request, etc), but we don't necessarily want to
> allow the user to do too much here so they don't shoot themselves in the foot
> and we don't make this overly complicated.
> I propose we allow users to specify racks and nodes for strict locality.
> This would allow users to restrict an MR AM to specific racks and/or nodes.
> We could add a new property,
> {{mapreduce.job.am.resource-request.strict.locality}}, which takes a
> comma-separated list of entries like:
> - {{/}}
> - {{//}}
> - {{}} (assumes /default-rack)
> MapReduce would then use this information to create the corresponding
> {{ResourceRequest}}'s.
> For example,
> {{mapreduce.job.am.resource-request.strict.locality=/rack1/node1}} would
> create the following {{ResourceRequest}}'s:
> - resourceName=ANY, relaxLocality=false, capability=
> - resourceName=/rack1, relaxLocality=false, capability=
> - resourceName=node1, relaxLocality=true, capability=
> By default, the property would be unset, and you'd get the normal {{ANY}}
> {{ResourceRequest}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6871) Allow users to specify racks and nodes for strict locality for AMs
Robert Kanter created MAPREDUCE-6871:
Summary: Allow users to specify racks and nodes for strict
locality for AMs
Key: MAPREDUCE-6871
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6871
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: client
Reporter: Robert Kanter
Assignee: Robert Kanter
YARN-6050 fixed the YARN API to allow multiple {{ResourceRequest}}'s when
submitting an AM so that you can actually do rack or node locality. We should
allow MapReduce users to take advantage of this by exposing this functionality
in some way. The raw YARN API allows for a lot of flexibility (e.g. different
resources per request, etc), but we don't necessarily want to allow the user to
do too much here so they don't shoot themselves in the foot and we don't make
this overly complicated.
I propose we allow users to specify racks and nodes for strict locality. This
would allow users to restrict an MR AM to specific racks and/or nodes. We
could add a new property,
{{mapreduce.job.am.resource-request.strict.locality}}, which takes a
comma-separated list of entries like:
- {{/}}
- {{//}}
- {{}} (assumes /default-rack)
MapReduce would then use this information to create the corresponding
{{ResourceRequest}}'s.
For example, {{mapreduce.job.am.resource-request.strict.locality=/rack1/node1}}
would create the following {{ResourceRequest}}'s:
- resourceName=ANY, relaxLocality=false, capability=
- resourceName=/rack1, relaxLocality=false, capability=
- resourceName=node1, relaxLocality=true, capability=
By default, the property would be unset, and you'd get the normal {{ANY}}
{{ResourceRequest}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6839) TestRecovery.testCrashed failed
[ https://issues.apache.org/jira/browse/MAPREDUCE-6839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-6839: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha3 2.9.0 Status: Resolved (was: Patch Available) Thanks [~pairg] and other for reviews. Committed to trunk and branch-2! > TestRecovery.testCrashed failed > --- > > Key: MAPREDUCE-6839 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6839 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Gergő Pásztor >Assignee: Gergő Pásztor > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: MAPREDUCE-6839_v1.patch, MAPREDUCE-6839_v2.patch, > MAPREDUCE-6839_v3.patch, MAPREDUCE-6839_v4.patch > > > TestRecovery#testCrashed is a flaky test. > Error Message: > Reduce Task state not correct expected: but was: > Stack Trace: > java.lang.AssertionError: Reduce Task state not correct expected: > but was: > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at > org.apache.hadoop.mapreduce.v2.app.TestRecovery.testCrashed(TestRecovery.java:164) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6839) TestRecovery.testCrashed failed
[ https://issues.apache.org/jira/browse/MAPREDUCE-6839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900198#comment-15900198 ] Robert Kanter commented on MAPREDUCE-6839: -- +1 will commit shortly > TestRecovery.testCrashed failed > --- > > Key: MAPREDUCE-6839 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6839 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Gergő Pásztor >Assignee: Gergő Pásztor > Attachments: MAPREDUCE-6839_v1.patch, MAPREDUCE-6839_v2.patch, > MAPREDUCE-6839_v3.patch, MAPREDUCE-6839_v4.patch > > > TestRecovery#testCrashed is a flaky test. > Error Message: > Reduce Task state not correct expected: but was: > Stack Trace: > java.lang.AssertionError: Reduce Task state not correct expected: > but was: > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at > org.apache.hadoop.mapreduce.v2.app.TestRecovery.testCrashed(TestRecovery.java:164) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6201) TestNetworkedJob fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886754#comment-15886754
]
Robert Kanter commented on MAPREDUCE-6201:
--
With 100% disk usage, the other problems I was referring to would be OS-level
problems. If you've ever tried to use Windows or a Mac with only a few
megabytes of disk space level, you'll run into all kinds of funky problems.
However, I was thinking about this more, and whoever/whatever running the test
might have a quota on the disk, which would trigger the same issue, without the
OS being affected, so I think you're right that we shouldn't do 100.
A more clear message failing the test would be best. Maybe assert that the
configured amount of disk space is free at the beginning of the test? I do
think that setting this to a higher value like 95 or 99 or something is also
reasonable to do given that the job the test runs shouldn't generate much data.
> TestNetworkedJob fails on trunk
> ---
>
> Key: MAPREDUCE-6201
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6201
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Robert Kanter
>Assignee: Peter Bacsko
> Attachments: MAPREDUCE-6201-001.patch, MAPREDUCE-6201-002.patch,
> MAPREDUCE-6201-003.patch
>
>
> Currently, {{TestNetworkedJob}} is failing on trunk:
> {noformat}
> Running org.apache.hadoop.mapred.TestNetworkedJob
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 215.01 sec
> <<< FAILURE! - in org.apache.hadoop.mapred.TestNetworkedJob
> testNetworkedJob(org.apache.hadoop.mapred.TestNetworkedJob) Time elapsed:
> 67.363 sec <<< FAILURE!
> java.lang.AssertionError: expected:<0> but was:<2>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.mapred.TestNetworkedJob.testNetworkedJob(TestNetworkedJob.java:195)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6201) TestNetworkedJob fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883610#comment-15883610
]
Robert Kanter commented on MAPREDUCE-6201:
--
I think part of the problem, and this is what we saw in Oozie too, is that
disks are much bigger than they used to be, so a percentage check on disk space
doesn't work as well. For instance, on a 500GB drive, 99% full would mean that
there's 5GB free. While you should probably do some cleanup with only 5GB
free, it should be more than enough space for anything a unit test is going to
do.
Anyway, I think 100 would be best here. If the disk if full, you'll hit all
kinds of other problems first anyway.
> TestNetworkedJob fails on trunk
> ---
>
> Key: MAPREDUCE-6201
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6201
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Robert Kanter
>Assignee: Peter Bacsko
> Attachments: MAPREDUCE-6201-001.patch, MAPREDUCE-6201-002.patch,
> MAPREDUCE-6201-003.patch
>
>
> Currently, {{TestNetworkedJob}} is failing on trunk:
> {noformat}
> Running org.apache.hadoop.mapred.TestNetworkedJob
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 215.01 sec
> <<< FAILURE! - in org.apache.hadoop.mapred.TestNetworkedJob
> testNetworkedJob(org.apache.hadoop.mapred.TestNetworkedJob) Time elapsed:
> 67.363 sec <<< FAILURE!
> java.lang.AssertionError: expected:<0> but was:<2>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.mapred.TestNetworkedJob.testNetworkedJob(TestNetworkedJob.java:195)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6728) Give fetchers hint when ShuffleHandler rejects a shuffling connection
[ https://issues.apache.org/jira/browse/MAPREDUCE-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-6728: - Status: Patch Available (was: Reopened) > Give fetchers hint when ShuffleHandler rejects a shuffling connection > - > > Key: MAPREDUCE-6728 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6728 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Haibo Chen >Assignee: Haibo Chen > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: MAPREDUCE-6728-branch-2.8.06.patch, > mapreduce6728.001.patch, mapreduce6728.002.patch, mapreduce6728.003.patch, > mapreduce6728.004.patch, mapreduce6728.005.patch, mapreduce6728.006.patch, > mapreduce6728.branch-2.8.patch, mapreduce6728.prelim.patch > > > If # of open shuffle connection to a node goes over the max, ShuffleHandler > closes the connection immediately without giving fetchers any hint of the > reason, which causes fetchers to fail due to exceptions > java.net.SocketException: Unexpected end of file from server > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:772) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323) > at > java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:323) > at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193) > OR > java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:196) > at java.net.SocketInputStream.read(SocketInputStream.java:122) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) > at java.io.BufferedInputStream.read1(BufferedInputStream.java:275) > at java.io.BufferedInputStream.read(BufferedInputStream.java:334) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323) > at > java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java > Such failures are counted as fetcher failures -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Reopened] (MAPREDUCE-6728) Give fetchers hint when ShuffleHandler rejects a shuffling connection
[ https://issues.apache.org/jira/browse/MAPREDUCE-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter reopened MAPREDUCE-6728: -- I'm reopening so that Jenkins can pick up the 2.8 patch > Give fetchers hint when ShuffleHandler rejects a shuffling connection > - > > Key: MAPREDUCE-6728 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6728 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Haibo Chen >Assignee: Haibo Chen > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: MAPREDUCE-6728-branch-2.8.06.patch, > mapreduce6728.001.patch, mapreduce6728.002.patch, mapreduce6728.003.patch, > mapreduce6728.004.patch, mapreduce6728.005.patch, mapreduce6728.006.patch, > mapreduce6728.branch-2.8.patch, mapreduce6728.prelim.patch > > > If # of open shuffle connection to a node goes over the max, ShuffleHandler > closes the connection immediately without giving fetchers any hint of the > reason, which causes fetchers to fail due to exceptions > java.net.SocketException: Unexpected end of file from server > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:772) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323) > at > java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:323) > at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193) > OR > java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:196) > at java.net.SocketInputStream.read(SocketInputStream.java:122) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) > at java.io.BufferedInputStream.read1(BufferedInputStream.java:275) > at java.io.BufferedInputStream.read(BufferedInputStream.java:334) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323) > at > java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java > Such failures are counted as fetcher failures -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6288) mapred job -status fails with AccessControlException
[
https://issues.apache.org/jira/browse/MAPREDUCE-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated MAPREDUCE-6288:
-
Assignee: (was: Robert Kanter)
The code was already reverted long ago back in March. The JIRA is still open
to address the original issue in another way. I'm unassigning myself because I
don't have time to work on this for now.
> mapred job -status fails with AccessControlException
> -
>
> Key: MAPREDUCE-6288
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6288
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Robert Kanter
>Priority: Blocker
> Attachments: MAPREDUCE-6288-gera-001.patch, MAPREDUCE-6288.002.patch,
> MAPREDUCE-6288.patch
>
>
> After MAPREDUCE-5875, we're seeing this Exception when trying to do {{mapred
> job -status job_1427080398288_0001}}
> {noformat}
> Exception in thread "main" org.apache.hadoop.security.AccessControlException:
> Permission denied: user=jenkins, access=EXECUTE,
> inode="/user/history/done":mapred:hadoop:drwxrwx---
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:180)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:137)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6553)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6535)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:6460)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1919)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:545)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
> at
> org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1213)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1201)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1191)
> at
> org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:299)
> at
> org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:265)
> at
[jira] [Updated] (MAPREDUCE-6571) JobEndNotification info logs are missing in AM container syslog
[ https://issues.apache.org/jira/browse/MAPREDUCE-6571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-6571: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha2 2.9.0 Status: Resolved (was: Patch Available) Thanks [~haibochen]. Committed to trunk and branch-2! > JobEndNotification info logs are missing in AM container syslog > --- > > Key: MAPREDUCE-6571 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6571 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 2.7.0 >Reporter: Prabhu Joseph >Assignee: Haibo Chen >Priority: Minor > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: MAPREDUCE-6571.01.patch, MAPREDUCE-6571.02.patch > > > JobEndNotification logs are not written by MRAppMaster and JobEndNotifier > classes even though Log.info is present. The reason was > MRAppMaster.this.stop() has been called before the JobEndNotification and > hence somewhere during the stop log appenders also made null. > AM container syslog is not having below logs from JobEndNotifier >Job end notification trying + urlToNotify >Job end notification to + urlToNotify + succeeded / failed -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6571) JobEndNotification info logs are missing in AM container syslog
[ https://issues.apache.org/jira/browse/MAPREDUCE-6571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723607#comment-15723607 ] Robert Kanter commented on MAPREDUCE-6571: -- +1 LGTM There's a shutdown hook that already does this, so the log should still get cleanly closed. > JobEndNotification info logs are missing in AM container syslog > --- > > Key: MAPREDUCE-6571 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6571 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 2.7.0 >Reporter: Prabhu Joseph >Assignee: Haibo Chen >Priority: Minor > Attachments: MAPREDUCE-6571.01.patch, MAPREDUCE-6571.02.patch > > > JobEndNotification logs are not written by MRAppMaster and JobEndNotifier > classes even though Log.info is present. The reason was > MRAppMaster.this.stop() has been called before the JobEndNotification and > hence somewhere during the stop log appenders also made null. > AM container syslog is not having below logs from JobEndNotifier >Job end notification trying + urlToNotify >Job end notification to + urlToNotify + succeeded / failed -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6571) JobEndNotification info logs are missing in AM container syslog
[ https://issues.apache.org/jira/browse/MAPREDUCE-6571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723216#comment-15723216 ] Robert Kanter commented on MAPREDUCE-6571: -- I just kicked one off: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6831/ > JobEndNotification info logs are missing in AM container syslog > --- > > Key: MAPREDUCE-6571 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6571 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 2.7.0 >Reporter: Prabhu Joseph >Assignee: Haibo Chen >Priority: Minor > Attachments: MAPREDUCE-6571.01.patch, MAPREDUCE-6571.02.patch > > > JobEndNotification logs are not written by MRAppMaster and JobEndNotifier > classes even though Log.info is present. The reason was > MRAppMaster.this.stop() has been called before the JobEndNotification and > hence somewhere during the stop log appenders also made null. > AM container syslog is not having below logs from JobEndNotifier >Job end notification trying + urlToNotify >Job end notification to + urlToNotify + succeeded / failed -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6787) Allow job_conf.xml to be downloadable on the job overview page in JHS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-6787: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha2 2.9.0 Status: Resolved (was: Patch Available) Thanks [~haibochen]. Committed to trunk and branch-2! > Allow job_conf.xml to be downloadable on the job overview page in JHS > - > > Key: MAPREDUCE-6787 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6787 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Reporter: Haibo Chen >Assignee: Haibo Chen > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: MAPREDUCE-6787.004.patch, MAPREDUCE-6787.004.patch, > job_1478210774848_0001.xml, mapreduce6787.001.patch, mapreduce6787.002.patch, > mapreduce6787.002.patch, mapreduce6787.003.patch > > > The job overview page in JHS provides the path to the job.xml file, but it is > not a link that users can click on to download the job xml file directly from > JHS. We could provide a download link in JHS for better usability. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6787) Allow job_conf.xml to be downloadable on the job overview page in JHS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15713653#comment-15713653 ] Robert Kanter commented on MAPREDUCE-6787: -- +1 > Allow job_conf.xml to be downloadable on the job overview page in JHS > - > > Key: MAPREDUCE-6787 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6787 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: MAPREDUCE-6787.004.patch, MAPREDUCE-6787.004.patch, > job_1478210774848_0001.xml, mapreduce6787.001.patch, mapreduce6787.002.patch, > mapreduce6787.002.patch, mapreduce6787.003.patch > > > The job overview page in JHS provides the path to the job.xml file, but it is > not a link that users can click on to download the job xml file directly from > JHS. We could provide a download link in JHS for better usability. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6787) Allow job_conf.xml to be downloadable on the job overview page in JHS
[
https://issues.apache.org/jira/browse/MAPREDUCE-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703486#comment-15703486
]
Robert Kanter commented on MAPREDUCE-6787:
--
Actually, how come the file uses {{}} instead of {{}}?
{noformat:title=file from JHS page}
hdfs://0.0.0.0:23010/tmp/hadoop-yarn/staging/history/done/2016/11/28/00/job_1480375039927_0001_conf.xml
...
{noformat}
{noformat:title=file from HDFS}
dfs.journalnode.rpc-address0.0.0.0:8485falsehdfs-default.xmljob.xml
...
{noformat}
> Allow job_conf.xml to be downloadable on the job overview page in JHS
> -
>
> Key: MAPREDUCE-6787
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6787
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobhistoryserver
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: job_1478210774848_0001.xml, mapreduce6787.001.patch,
> mapreduce6787.002.patch, mapreduce6787.002.patch, mapreduce6787.003.patch
>
>
> The job overview page in JHS provides the path to the job.xml file, but it is
> not a link that users can click on to download the job xml file directly from
> JHS. We could provide a download link in JHS for better usability.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6787) Allow job_conf.xml to be downloadable on the job overview page in JHS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703470#comment-15703470 ] Robert Kanter commented on MAPREDUCE-6787: -- +1 on the 003 patch > Allow job_conf.xml to be downloadable on the job overview page in JHS > - > > Key: MAPREDUCE-6787 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6787 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: job_1478210774848_0001.xml, mapreduce6787.001.patch, > mapreduce6787.002.patch, mapreduce6787.002.patch, mapreduce6787.003.patch > > > The job overview page in JHS provides the path to the job.xml file, but it is > not a link that users can click on to download the job xml file directly from > JHS. We could provide a download link in JHS for better usability. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6787) Allow job_conf.xml to be downloadable on the job overview page in JHS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-6787: - Status: Open (was: Patch Available) > Allow job_conf.xml to be downloadable on the job overview page in JHS > - > > Key: MAPREDUCE-6787 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6787 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: job_1478210774848_0001.xml, mapreduce6787.001.patch, > mapreduce6787.002.patch > > > The job overview page in JHS provides the path to the job.xml file, but it is > not a link that users can click on to download the job xml file directly from > JHS. We could provide a download link in JHS for better usability. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6787) Allow job_conf.xml to be downloadable on the job overview page in JHS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-6787: - Attachment: job_1478210774848_0001.xml Thanks for the patch [~haibochen]; this should be useful. I was trying it out, and the downloaded XML file has a bunch of HTML at the end that doesn't belong there (I think it's JHS /conf page). Take a look at the attached jobconf XML file. > Allow job_conf.xml to be downloadable on the job overview page in JHS > - > > Key: MAPREDUCE-6787 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6787 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: job_1478210774848_0001.xml, mapreduce6787.001.patch, > mapreduce6787.002.patch > > > The job overview page in JHS provides the path to the job.xml file, but it is > not a link that users can click on to download the job xml file directly from > JHS. We could provide a download link in JHS for better usability. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6765) MR should not schedule container requests in cases where reducer or mapper containers demand resource larger than the maximum supported
[ https://issues.apache.org/jira/browse/MAPREDUCE-6765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-6765: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha2 Status: Resolved (was: Patch Available) Thanks [~haibochen]. Committed to trunk and branch-2! > MR should not schedule container requests in cases where reducer or mapper > containers demand resource larger than the maximum supported > --- > > Key: MAPREDUCE-6765 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6765 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.7.2 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Minor > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: mapreduce6765.001.patch, mapreduce6765.002.patch, > mapreduce6765.003.patch, mapreduce6765.004.patch, mapreduce6765.005.patch > > > When mapper or reducer containers request resource larger than the > maxResourceRequest in the cluster, job is to be killed. In such cases, it is > unnecessary to still schedule container requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6704) Container fail to launch for mapred application
[
https://issues.apache.org/jira/browse/MAPREDUCE-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15616663#comment-15616663
]
Robert Kanter commented on MAPREDUCE-6704:
--
Either #3 (adding {{HADOOP_MAPRED_HOME}} to the classpath) or #5 (though maybe
not actually using {{-libjars}} argument, but doing something equivalent) sound
good to me.
> Container fail to launch for mapred application
> ---
>
> Key: MAPREDUCE-6704
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6704
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Priority: Blocker
> Attachments: 0001-MAPREDUCE-6704.patch, 0001-YARN-5026.patch,
> container-whitelist-env-wip.patch
>
>
> Container fail to launch for mapred application.
> As part for launch script {{HADOOP_MAPRED_HOME}} default value is not set
> .After
> https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c
>{{HADOOP_MAPRED_HOME}} is not able to get from {{builder.environment()}}
> since {{DefaultContainerExecutor#buildCommandExecutor}} sets inherit to false.
> {noformat}
> 16/05/02 09:16:05 INFO mapreduce.Job: Job job_1462155939310_0004 failed with
> state FAILED due to: Application application_1462155939310_0004 failed 2
> times due to AM Container for appattempt_1462155939310_0004_02 exited
> with exitCode: 1
> Failing this attempt.Diagnostics: Exception from container-launch.
> Container id: container_1462155939310_0004_02_01
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:946)
> at org.apache.hadoop.util.Shell.run(Shell.java:850)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:227)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:385)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:281)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:89)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> Error: Could not find or load main class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option UseSplitVerifier;
> support was removed in 8.0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6765) MR should not schedule container requests in cases where reducer or mapper containers demand resource larger than the maximum supported
[ https://issues.apache.org/jira/browse/MAPREDUCE-6765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610256#comment-15610256 ] Robert Kanter commented on MAPREDUCE-6765: -- [~haibochen], the patch no longer cleanly applies; can you rebase the patch? > MR should not schedule container requests in cases where reducer or mapper > containers demand resource larger than the maximum supported > --- > > Key: MAPREDUCE-6765 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6765 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.7.2 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Minor > Fix For: 2.9.0 > > Attachments: mapreduce6765.001.patch, mapreduce6765.002.patch, > mapreduce6765.003.patch, mapreduce6765.004.patch > > > When mapper or reducer containers request resource larger than the > maxResourceRequest in the cluster, job is to be killed. In such cases, it is > unnecessary to still schedule container requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
