[jira] [Commented] (MAPREDUCE-4669) MRAM web UI does not work with HTTPS
[
https://issues.apache.org/jira/browse/MAPREDUCE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652847#comment-16652847
]
Hadoop QA commented on MAPREDUCE-4669:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 22s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
56s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
21s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 55s{color} | {color:orange} root: The patch generated 4 new + 662 unchanged
- 0 fixed = 666 total (was 662) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 17s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
21s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
16s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
25s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
15s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
56s{color} | {color:green} hadoop-mapreduce-client-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
25s{color} | {color:green}
[jira] [Updated] (MAPREDUCE-4669) MRAM web UI does not work with HTTPS
[ https://issues.apache.org/jira/browse/MAPREDUCE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-4669: - Status: Patch Available (was: Open) > MRAM web UI does not work with HTTPS > > > Key: MAPREDUCE-4669 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4669 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.0.3-alpha >Reporter: Alejandro Abdelnur >Assignee: Robert Kanter >Priority: Major > Attachments: MAPREDUCE-4669.001.patch, MAPREDUCE-4669.002.patch, > MAPREDUCE-4669.003.patch > > > With Kerberos enable, the MRAM runs as the user that submitted the job, thus > the MRAM process cannot read the cluster keystore files to get the > certificates to start its HttpServer using HTTPS. > We need to decouple the keystore used by RM/NM/NN/DN (which are cluster > provided) from the keystore used by AMs (which ought to be user provided). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-4669) MRAM web UI does not work with HTTPS
[ https://issues.apache.org/jira/browse/MAPREDUCE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-4669: - Attachment: MAPREDUCE-4669.003.patch > MRAM web UI does not work with HTTPS > > > Key: MAPREDUCE-4669 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4669 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.0.3-alpha >Reporter: Alejandro Abdelnur >Assignee: Robert Kanter >Priority: Major > Attachments: MAPREDUCE-4669.001.patch, MAPREDUCE-4669.002.patch, > MAPREDUCE-4669.003.patch > > > With Kerberos enable, the MRAM runs as the user that submitted the job, thus > the MRAM process cannot read the cluster keystore files to get the > certificates to start its HttpServer using HTTPS. > We need to decouple the keystore used by RM/NM/NN/DN (which are cluster > provided) from the keystore used by AMs (which ought to be user provided). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-4669) MRAM web UI does not work with HTTPS
[ https://issues.apache.org/jira/browse/MAPREDUCE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652670#comment-16652670 ] Robert Kanter commented on MAPREDUCE-4669: -- The 003 patch: - Rebased on latest trunk (including YARN-8448) - Added configs to mapred-default > MRAM web UI does not work with HTTPS > > > Key: MAPREDUCE-4669 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4669 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.0.3-alpha >Reporter: Alejandro Abdelnur >Assignee: Robert Kanter >Priority: Major > Attachments: MAPREDUCE-4669.001.patch, MAPREDUCE-4669.002.patch, > MAPREDUCE-4669.003.patch > > > With Kerberos enable, the MRAM runs as the user that submitted the job, thus > the MRAM process cannot read the cluster keystore files to get the > certificates to start its HttpServer using HTTPS. > We need to decouple the keystore used by RM/NM/NN/DN (which are cluster > provided) from the keystore used by AMs (which ought to be user provided). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7150) Optimize collections used by MR JHS to reduce its memory
[
https://issues.apache.org/jira/browse/MAPREDUCE-7150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652467#comment-16652467
]
Hudson commented on MAPREDUCE-7150:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15231 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/15231/])
MAPREDUCE-7150. Optimize collections used by MR JHS to reduce its (haibochen:
rev babd1449bf8898f44c434c852e67240721c0eb00)
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryParser.java
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedTask.java
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedTaskAttempt.java
> Optimize collections used by MR JHS to reduce its memory
>
>
> Key: MAPREDUCE-7150
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7150
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobhistoryserver, mrv2
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-8872.01.patch, YARN-8872.02.patch,
> YARN-8872.03.patch, YARN-8872.04.patch, jhs-bad-collections.png
>
>
> We analyzed, using jxray (www.jxray.com) a heap dump of JHS running with big
> heap in a large clusters, handling large MapReduce jobs. The heap is large
> (over 32GB) and 21.4% of it is wasted due to various suboptimal Java
> collections, mostly maps and lists that are either empty or contain only one
> element. In such under-populated collections considerable amount of memory is
> still used by just the internal implementation objects. See the attached
> excerpt from the jxray report for the details. If certain collections are
> almost always empty, they should be initialized lazily. If others almost
> always have just 1 or 2 elements, they should be initialized with the
> appropriate initial capacity of 1 or 2 (the default capacity is 16 for
> HashMap and 10 for ArrayList).
> Based on the attached report, we should do the following:
> # {{FileSystemCounterGroup.map}} - initialize lazily
> # {{CompletedTask.attempts}} - initialize with capacity 2, given most tasks
> only have one or two attempts
> # {{JobHistoryParser$TaskInfo.attemptsMap}} - initialize with capacity
> # {{CompletedTaskAttempt.diagnostics}} - initialize with capacity 1 since it
> contains one diagnostic message most of the time
> # {{CompletedTask.reportDiagnostics}} - switch to ArrayList (no reason to
> use the more wasteful LinkedList here) and initialize with capacity 1.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7150) Optimize collections used by MR JHS to reduce its memory
[
https://issues.apache.org/jira/browse/MAPREDUCE-7150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibo Chen updated MAPREDUCE-7150:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.3.0
Status: Resolved (was: Patch Available)
Thanks [~mi...@cloudera.com] for the contribution and [~pbacsko] for additional
reviews!
> Optimize collections used by MR JHS to reduce its memory
>
>
> Key: MAPREDUCE-7150
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7150
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobhistoryserver, mrv2
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-8872.01.patch, YARN-8872.02.patch,
> YARN-8872.03.patch, YARN-8872.04.patch, jhs-bad-collections.png
>
>
> We analyzed, using jxray (www.jxray.com) a heap dump of JHS running with big
> heap in a large clusters, handling large MapReduce jobs. The heap is large
> (over 32GB) and 21.4% of it is wasted due to various suboptimal Java
> collections, mostly maps and lists that are either empty or contain only one
> element. In such under-populated collections considerable amount of memory is
> still used by just the internal implementation objects. See the attached
> excerpt from the jxray report for the details. If certain collections are
> almost always empty, they should be initialized lazily. If others almost
> always have just 1 or 2 elements, they should be initialized with the
> appropriate initial capacity of 1 or 2 (the default capacity is 16 for
> HashMap and 10 for ArrayList).
> Based on the attached report, we should do the following:
> # {{FileSystemCounterGroup.map}} - initialize lazily
> # {{CompletedTask.attempts}} - initialize with capacity 2, given most tasks
> only have one or two attempts
> # {{JobHistoryParser$TaskInfo.attemptsMap}} - initialize with capacity
> # {{CompletedTaskAttempt.diagnostics}} - initialize with capacity 1 since it
> contains one diagnostic message most of the time
> # {{CompletedTask.reportDiagnostics}} - switch to ArrayList (no reason to
> use the more wasteful LinkedList here) and initialize with capacity 1.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7150) Optimize collections used by MR JHS to reduce its memory
[
https://issues.apache.org/jira/browse/MAPREDUCE-7150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652407#comment-16652407
]
Haibo Chen commented on MAPREDUCE-7150:
---
+1 on the latest patch. Will check it in shortly.
> Optimize collections used by MR JHS to reduce its memory
>
>
> Key: MAPREDUCE-7150
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7150
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobhistoryserver, mrv2
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: YARN-8872.01.patch, YARN-8872.02.patch,
> YARN-8872.03.patch, YARN-8872.04.patch, jhs-bad-collections.png
>
>
> We analyzed, using jxray (www.jxray.com) a heap dump of JHS running with big
> heap in a large clusters, handling large MapReduce jobs. The heap is large
> (over 32GB) and 21.4% of it is wasted due to various suboptimal Java
> collections, mostly maps and lists that are either empty or contain only one
> element. In such under-populated collections considerable amount of memory is
> still used by just the internal implementation objects. See the attached
> excerpt from the jxray report for the details. If certain collections are
> almost always empty, they should be initialized lazily. If others almost
> always have just 1 or 2 elements, they should be initialized with the
> appropriate initial capacity of 1 or 2 (the default capacity is 16 for
> HashMap and 10 for ArrayList).
> Based on the attached report, we should do the following:
> # {{FileSystemCounterGroup.map}} - initialize lazily
> # {{CompletedTask.attempts}} - initialize with capacity 2, given most tasks
> only have one or two attempts
> # {{JobHistoryParser$TaskInfo.attemptsMap}} - initialize with capacity
> # {{CompletedTaskAttempt.diagnostics}} - initialize with capacity 1 since it
> contains one diagnostic message most of the time
> # {{CompletedTask.reportDiagnostics}} - switch to ArrayList (no reason to
> use the more wasteful LinkedList here) and initialize with capacity 1.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7132) JobSplitWriter prints unnecessary warnings if EC(RS10,4) is used
[
https://issues.apache.org/jira/browse/MAPREDUCE-7132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652143#comment-16652143
]
Hudson commented on MAPREDUCE-7132:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15226 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/15226/])
MAPREDUCE-7132. JobSplitWriter prints unnecessary warnings if EC(RS10,4) (xiao:
rev 25f8fcb06476938826cdc92858a61124b18cd98d)
* (add)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/split/TestJobSplitWriterWithEC.java
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
> JobSplitWriter prints unnecessary warnings if EC(RS10,4) is used
>
>
> Key: MAPREDUCE-7132
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7132
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, mrv2
>Affects Versions: 3.1.1
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Fix For: 3.2.0, 3.0.4, 3.1.2
>
> Attachments: MAPREDUCE-7132-001.patch, MAPREDUCE-7132-002.patch,
> MAPREDUCE-7132-003.patch, MAPREDUCE-7132-004.patch, MAPREDUCE-7132-005.patch,
> MAPREDUCE-7132-006.patch, MAPREDUCE-7132-007.patch, MAPREDUCE-7132-008.patch
>
>
> Currently, {{JobSplitWriter}} compares the number of hosts for a certain
> block against a static value that comes from
> {{mapreduce.job.max.split.locations}}. The default value of this property is
> 10.
> However, an EC schema like RS-10-4 requires at least 14 hosts. In this case,
> 14 block locations will be returned and {{JobSplitWriter}} prints a warning,
> which can confuse users.
> A possible solution could check whether EC is enabled for a block and
> increase this value dynamically if needed. A simpler one is to simply
> increase the default value to a sensible number like 15.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7132) JobSplitWriter prints unnecessary warnings if EC(RS10,4) is used
[
https://issues.apache.org/jira/browse/MAPREDUCE-7132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated MAPREDUCE-7132:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.1.2
3.0.4
3.2.0
Status: Resolved (was: Patch Available)
Committed to trunk and branch-3.[0-2].
Thank you, [~pbacsko]!
> JobSplitWriter prints unnecessary warnings if EC(RS10,4) is used
>
>
> Key: MAPREDUCE-7132
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7132
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, mrv2
>Affects Versions: 3.1.1
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Fix For: 3.2.0, 3.0.4, 3.1.2
>
> Attachments: MAPREDUCE-7132-001.patch, MAPREDUCE-7132-002.patch,
> MAPREDUCE-7132-003.patch, MAPREDUCE-7132-004.patch, MAPREDUCE-7132-005.patch,
> MAPREDUCE-7132-006.patch, MAPREDUCE-7132-007.patch, MAPREDUCE-7132-008.patch
>
>
> Currently, {{JobSplitWriter}} compares the number of hosts for a certain
> block against a static value that comes from
> {{mapreduce.job.max.split.locations}}. The default value of this property is
> 10.
> However, an EC schema like RS-10-4 requires at least 14 hosts. In this case,
> 14 block locations will be returned and {{JobSplitWriter}} prints a warning,
> which can confuse users.
> A possible solution could check whether EC is enabled for a block and
> increase this value dynamically if needed. A simpler one is to simply
> increase the default value to a sensible number like 15.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7132) JobSplitWriter prints unnecessary warnings if EC(RS10,4) is used
[
https://issues.apache.org/jira/browse/MAPREDUCE-7132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated MAPREDUCE-7132:
-
Summary: JobSplitWriter prints unnecessary warnings if EC(RS10,4) is used
(was: JobSplitWriter prints unnecessary warnings if EC is used)
> JobSplitWriter prints unnecessary warnings if EC(RS10,4) is used
>
>
> Key: MAPREDUCE-7132
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7132
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, mrv2
>Affects Versions: 3.1.1
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MAPREDUCE-7132-001.patch, MAPREDUCE-7132-002.patch,
> MAPREDUCE-7132-003.patch, MAPREDUCE-7132-004.patch, MAPREDUCE-7132-005.patch,
> MAPREDUCE-7132-006.patch, MAPREDUCE-7132-007.patch, MAPREDUCE-7132-008.patch
>
>
> Currently, {{JobSplitWriter}} compares the number of hosts for a certain
> block against a static value that comes from
> {{mapreduce.job.max.split.locations}}. The default value of this property is
> 10.
> However, an EC schema like RS-10-4 requires at least 14 hosts. In this case,
> 14 block locations will be returned and {{JobSplitWriter}} prints a warning,
> which can confuse users.
> A possible solution could check whether EC is enabled for a block and
> increase this value dynamically if needed. A simpler one is to simply
> increase the default value to a sensible number like 15.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7132) JobSplitWriter prints unnecessary warnings if EC is used
[
https://issues.apache.org/jira/browse/MAPREDUCE-7132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652103#comment-16652103
]
Xiao Chen commented on MAPREDUCE-7132:
--
checkstyle is pre-existing. +1 on patch 8. Committing
> JobSplitWriter prints unnecessary warnings if EC is used
>
>
> Key: MAPREDUCE-7132
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7132
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, mrv2
>Affects Versions: 3.1.1
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MAPREDUCE-7132-001.patch, MAPREDUCE-7132-002.patch,
> MAPREDUCE-7132-003.patch, MAPREDUCE-7132-004.patch, MAPREDUCE-7132-005.patch,
> MAPREDUCE-7132-006.patch, MAPREDUCE-7132-007.patch, MAPREDUCE-7132-008.patch
>
>
> Currently, {{JobSplitWriter}} compares the number of hosts for a certain
> block against a static value that comes from
> {{mapreduce.job.max.split.locations}}. The default value of this property is
> 10.
> However, an EC schema like RS-10-4 requires at least 14 hosts. In this case,
> 14 block locations will be returned and {{JobSplitWriter}} prints a warning,
> which can confuse users.
> A possible solution could check whether EC is enabled for a block and
> increase this value dynamically if needed. A simpler one is to simply
> increase the default value to a sensible number like 15.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7150) Optimize collections used by MR JHS to reduce its memory
[
https://issues.apache.org/jira/browse/MAPREDUCE-7150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652086#comment-16652086
]
Peter Bacsko commented on MAPREDUCE-7150:
-
+1 non-binding from me. I suppose the checkstyle stuff already existed, so IMO
it can be ignored.
> Optimize collections used by MR JHS to reduce its memory
>
>
> Key: MAPREDUCE-7150
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7150
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobhistoryserver, mrv2
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: YARN-8872.01.patch, YARN-8872.02.patch,
> YARN-8872.03.patch, YARN-8872.04.patch, jhs-bad-collections.png
>
>
> We analyzed, using jxray (www.jxray.com) a heap dump of JHS running with big
> heap in a large clusters, handling large MapReduce jobs. The heap is large
> (over 32GB) and 21.4% of it is wasted due to various suboptimal Java
> collections, mostly maps and lists that are either empty or contain only one
> element. In such under-populated collections considerable amount of memory is
> still used by just the internal implementation objects. See the attached
> excerpt from the jxray report for the details. If certain collections are
> almost always empty, they should be initialized lazily. If others almost
> always have just 1 or 2 elements, they should be initialized with the
> appropriate initial capacity of 1 or 2 (the default capacity is 16 for
> HashMap and 10 for ArrayList).
> Based on the attached report, we should do the following:
> # {{FileSystemCounterGroup.map}} - initialize lazily
> # {{CompletedTask.attempts}} - initialize with capacity 2, given most tasks
> only have one or two attempts
> # {{JobHistoryParser$TaskInfo.attemptsMap}} - initialize with capacity
> # {{CompletedTaskAttempt.diagnostics}} - initialize with capacity 1 since it
> contains one diagnostic message most of the time
> # {{CompletedTask.reportDiagnostics}} - switch to ArrayList (no reason to
> use the more wasteful LinkedList here) and initialize with capacity 1.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7153) Protect XML Entity expansion while parsing XML in AllocationFileLoaderService
[
https://issues.apache.org/jira/browse/MAPREDUCE-7153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652022#comment-16652022
]
Hadoop QA commented on MAPREDUCE-7153:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
38s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 5s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
The patch generated 0 new + 19 unchanged - 2 fixed = 19 total (was 21) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 16s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
51s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | MAPREDUCE-7153 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12944161/MAPREDUCE-7153.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 74dd1e4c1f1b 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10
07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 0bf8a11 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7510/testReport/ |
| Max. process+thread count | 447 (vs. ulimit of 1) |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
U:
[jira] [Commented] (MAPREDUCE-7153) Protect XML Entity expansion while parsing XML in AllocationFileLoaderService
[ https://issues.apache.org/jira/browse/MAPREDUCE-7153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651898#comment-16651898 ] Sunil Govindan commented on MAPREDUCE-7153: --- [~leftnoteasy] [~vinodkv] [~lmccay], could u pls check this patch. Thanks. > Protect XML Entity expansion while parsing XML in AllocationFileLoaderService > - > > Key: MAPREDUCE-7153 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7153 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: MAPREDUCE-7153.001.patch > > > This Jira will track two things > # In AllocationFileLoaderService XML read module, protect or validate XML > entity to avoid any safety issues related to XML Entity expansion pblms. > # Its better to use SecureRandom in Application.java instead of normal > Random seeds. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7153) Protect XML Entity expansion while parsing XML in AllocationFileLoaderService
[ https://issues.apache.org/jira/browse/MAPREDUCE-7153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated MAPREDUCE-7153: -- Status: Patch Available (was: Open) > Protect XML Entity expansion while parsing XML in AllocationFileLoaderService > - > > Key: MAPREDUCE-7153 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7153 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: MAPREDUCE-7153.001.patch > > > This Jira will track two things > # In AllocationFileLoaderService XML read module, protect or validate XML > entity to avoid any safety issues related to XML Entity expansion pblms. > # Its better to use SecureRandom in Application.java instead of normal > Random seeds. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7153) Protect XML Entity expansion while parsing XML in AllocationFileLoaderService
[ https://issues.apache.org/jira/browse/MAPREDUCE-7153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated MAPREDUCE-7153: -- Attachment: MAPREDUCE-7153.001.patch > Protect XML Entity expansion while parsing XML in AllocationFileLoaderService > - > > Key: MAPREDUCE-7153 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7153 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: MAPREDUCE-7153.001.patch > > > This Jira will track two things > # In AllocationFileLoaderService XML read module, protect or validate XML > entity to avoid any safety issues related to XML Entity expansion pblms. > # Its better to use SecureRandom in Application.java instead of normal > Random seeds. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7153) Protect XML Entity expansion while parsing XML in AllocationFileLoaderService
Sunil Govindan created MAPREDUCE-7153: - Summary: Protect XML Entity expansion while parsing XML in AllocationFileLoaderService Key: MAPREDUCE-7153 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7153 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Sunil Govindan Assignee: Sunil Govindan This Jira will track two things # In AllocationFileLoaderService XML read module, protect or validate XML entity to avoid any safety issues related to XML Entity expansion pblms. # Its better to use SecureRandom in Application.java instead of normal Random seeds. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7132) JobSplitWriter prints unnecessary warnings if EC is used
[
https://issues.apache.org/jira/browse/MAPREDUCE-7132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651773#comment-16651773
]
Hadoop QA commented on MAPREDUCE-7132:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 42s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 59s{color} | {color:orange}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
The patch generated 1 new + 57 unchanged - 1 fixed = 58 total (was 58) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 35s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
57s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 85m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | MAPREDUCE-7132 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12944122/MAPREDUCE-7132-008.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux d2ca4d978fbd 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 0bf8a11 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7509/artifact/out/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
|
| Test Results |
[jira] [Commented] (MAPREDUCE-7152) LD_LIBRARY_PATH is always passed from MR AM to tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-7152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651690#comment-16651690
]
Peter Bacsko commented on MAPREDUCE-7152:
-
[~jlowe] [~haibochen] which solution do you prefer? I kinda like the lazyEval
approach.
> LD_LIBRARY_PATH is always passed from MR AM to tasks
>
>
> Key: MAPREDUCE-7152
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7152
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MAPREDUCE-7152-NMAdminEnvPOC_POC01.patch,
> MAPREDUCE-7152-lazyEval_POC01.patch
>
>
> {{LD_LIBRARY_PATH}} is set to {{$HADOOP_COMMON_HOME/lib/native}} by default
> in Hadoop (as part of {{mapreduce.admin.user.env}} and
> {{yarn.app.mapreduce.am.user.env}}), and passed as an environment variable
> from AM container to task containers in the container launch context.
> In cases where {{HADOOP_COMMON_HOME}} is different in AM node and task node,
> tasks will fail to load native library. A reliable way to fix this is to add
> {{LD_LIBRARY_PATH}} in {{yarn.nodemanager.admin-env}} instead.
> Another approach is to perform a lazy evaluation of {{LD_LIBRARY_PATH}} on
> the NM side.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7152) LD_LIBRARY_PATH is always passed from MR AM to tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-7152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651649#comment-16651649
]
Hadoop QA commented on MAPREDUCE-7152:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 15s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
46s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m
40s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 28s{color} | {color:orange} root: The patch generated 3 new + 730 unchanged
- 1 fixed = 733 total (was 731) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 44s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
28s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
5s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
28s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}126m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | MAPREDUCE-7152 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12944086/MAPREDUCE-7152-NMAdminEnvPOC_POC01.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit
[jira] [Updated] (MAPREDUCE-7132) JobSplitWriter prints unnecessary warnings if EC is used
[
https://issues.apache.org/jira/browse/MAPREDUCE-7132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated MAPREDUCE-7132:
Attachment: MAPREDUCE-7132-008.patch
> JobSplitWriter prints unnecessary warnings if EC is used
>
>
> Key: MAPREDUCE-7132
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7132
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, mrv2
>Affects Versions: 3.1.1
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MAPREDUCE-7132-001.patch, MAPREDUCE-7132-002.patch,
> MAPREDUCE-7132-003.patch, MAPREDUCE-7132-004.patch, MAPREDUCE-7132-005.patch,
> MAPREDUCE-7132-006.patch, MAPREDUCE-7132-007.patch, MAPREDUCE-7132-008.patch
>
>
> Currently, {{JobSplitWriter}} compares the number of hosts for a certain
> block against a static value that comes from
> {{mapreduce.job.max.split.locations}}. The default value of this property is
> 10.
> However, an EC schema like RS-10-4 requires at least 14 hosts. In this case,
> 14 block locations will be returned and {{JobSplitWriter}} prints a warning,
> which can confuse users.
> A possible solution could check whether EC is enabled for a block and
> increase this value dynamically if needed. A simpler one is to simply
> increase the default value to a sensible number like 15.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7152) LD_LIBRARY_PATH is always passed from MR AM to tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-7152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated MAPREDUCE-7152:
Attachment: MAPREDUCE-7152-NMAdminEnvPOC_POC01.patch
> LD_LIBRARY_PATH is always passed from MR AM to tasks
>
>
> Key: MAPREDUCE-7152
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7152
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MAPREDUCE-7152-NMAdminEnvPOC_POC01.patch,
> MAPREDUCE-7152-lazyEval_POC01.patch
>
>
> {{LD_LIBRARY_PATH}} is set to {{$HADOOP_COMMON_HOME/lib/native}} by default
> in Hadoop (as part of {{mapreduce.admin.user.env}} and
> {{yarn.app.mapreduce.am.user.env}}), and passed as an environment variable
> from AM container to task containers in the container launch context.
> In cases where {{HADOOP_COMMON_HOME}} is different in AM node and task node,
> tasks will fail to load native library. A reliable way to fix this is to add
> {{LD_LIBRARY_PATH}} in {{yarn.nodemanager.admin-env}} instead.
> Another approach is to perform a lazy evaluation of {{LD_LIBRARY_PATH}} on
> the NM side.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7152) LD_LIBRARY_PATH is always passed from MR AM to tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-7152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651357#comment-16651357
]
Peter Bacsko commented on MAPREDUCE-7152:
-
Test failure is related to the POC, it should be fixed.
> LD_LIBRARY_PATH is always passed from MR AM to tasks
>
>
> Key: MAPREDUCE-7152
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7152
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MAPREDUCE-7152-lazyEval_POC01.patch
>
>
> {{LD_LIBRARY_PATH}} is set to {{$HADOOP_COMMON_HOME/lib/native}} by default
> in Hadoop (as part of {{mapreduce.admin.user.env}} and
> {{yarn.app.mapreduce.am.user.env}}), and passed as an environment variable
> from AM container to task containers in the container launch context.
> In cases where {{HADOOP_COMMON_HOME}} is different in AM node and task node,
> tasks will fail to load native library. A reliable way to fix this is to add
> {{LD_LIBRARY_PATH}} in {{yarn.nodemanager.admin-env}} instead.
> Another approach is to perform a lazy evaluation of {{LD_LIBRARY_PATH}} on
> the NM side.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
