[jira] [Commented] (MAPREDUCE-6729) Accurately compute the test execute time in DFSIO
[
https://issues.apache.org/jira/browse/MAPREDUCE-6729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374181#comment-15374181
]
Hadoop QA commented on MAPREDUCE-6729:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 32s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
33s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 29s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
25s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 126m 0s
{color} | {color:green} hadoop-mapreduce-client-jobclient in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 141m 3s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | MAPREDUCE-6729 |
| GITHUB PR | https://github.com/apache/hadoop/pull/112 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 4027af10b870 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / d5f5a64 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6612/testReport/ |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
|
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6612/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Accurately compute the test execute time in DFSIO
> -
>
> Key: MAPREDUCE-6729
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6729
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: benchmarks, performance, test
>Affects Versions: 2.9.0
>Reporter: mingleizhang
>Assignee: mingleizhang
>Priority: Minor
> Labels: performance, test
>
[jira] [Updated] (MAPREDUCE-6729) Accurately compute the test execute time in DFSIO
[
https://issues.apache.org/jira/browse/MAPREDUCE-6729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mingleizhang updated MAPREDUCE-6729:
Status: Patch Available (was: Open)
> Accurately compute the test execute time in DFSIO
> -
>
> Key: MAPREDUCE-6729
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6729
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: benchmarks, performance, test
>Affects Versions: 2.9.0
>Reporter: mingleizhang
>Assignee: mingleizhang
>Priority: Minor
> Labels: performance, test
> Attachments: MAPREDUCE-6729.001.patch, MAPREDUCE-6729.002.patch
>
>
> When doing DFSIO test as a distributed i/o benchmark tool. Then especially
> writes plenty of files to disk or read from, both can cause performance issue
> and imprecise value in a way. The question is that existing practices needs
> to delete files when before running a job and that will cause extra time
> consumption and furthermore cause performance issue, statistical time error
> and imprecise throughput while the files are lots of. So we need to replace
> or improve this hack to prevent this from happening in the future.
> {code}
> public static void testWrite() throws Exception {
> FileSystem fs = cluster.getFileSystem();
> long tStart = System.currentTimeMillis();
> bench.writeTest(fs); // this line of code will cause extra time
> consumption because of fs.delete(*,*) by the writeTest method
> long execTime = System.currentTimeMillis() - tStart;
> bench.analyzeResult(fs, TestType.TEST_TYPE_WRITE, execTime);
> }
> private void writeTest(FileSystem fs) throws IOException {
> Path writeDir = getWriteDir(config);
> fs.delete(getDataDir(config), true);
> fs.delete(writeDir, true);
> runIOTest(WriteMapper.class, writeDir);
> }
> {code}
> [https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/TestDFSIO.java]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6729) Accurately compute the test execute time in DFSIO
[
https://issues.apache.org/jira/browse/MAPREDUCE-6729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mingleizhang updated MAPREDUCE-6729:
Status: Open (was: Patch Available)
> Accurately compute the test execute time in DFSIO
> -
>
> Key: MAPREDUCE-6729
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6729
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: benchmarks, performance, test
>Affects Versions: 2.9.0
>Reporter: mingleizhang
>Assignee: mingleizhang
>Priority: Minor
> Labels: performance, test
> Attachments: MAPREDUCE-6729.001.patch, MAPREDUCE-6729.002.patch
>
>
> When doing DFSIO test as a distributed i/o benchmark tool. Then especially
> writes plenty of files to disk or read from, both can cause performance issue
> and imprecise value in a way. The question is that existing practices needs
> to delete files when before running a job and that will cause extra time
> consumption and furthermore cause performance issue, statistical time error
> and imprecise throughput while the files are lots of. So we need to replace
> or improve this hack to prevent this from happening in the future.
> {code}
> public static void testWrite() throws Exception {
> FileSystem fs = cluster.getFileSystem();
> long tStart = System.currentTimeMillis();
> bench.writeTest(fs); // this line of code will cause extra time
> consumption because of fs.delete(*,*) by the writeTest method
> long execTime = System.currentTimeMillis() - tStart;
> bench.analyzeResult(fs, TestType.TEST_TYPE_WRITE, execTime);
> }
> private void writeTest(FileSystem fs) throws IOException {
> Path writeDir = getWriteDir(config);
> fs.delete(getDataDir(config), true);
> fs.delete(writeDir, true);
> runIOTest(WriteMapper.class, writeDir);
> }
> {code}
> [https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/TestDFSIO.java]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6652) Add configuration property to prevent JHS from loading jobs with a task count greater than X
[ https://issues.apache.org/jira/browse/MAPREDUCE-6652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373995#comment-15373995 ] Haibo Chen commented on MAPREDUCE-6652: --- Unit test failed because of time out. > Add configuration property to prevent JHS from loading jobs with a task count > greater than X > > > Key: MAPREDUCE-6652 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6652 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: mapreduce6652.001.patch, mapreduce6652.002.patch, > mapreduce6652.003.patch, mapreduce6652.004.patch, mapreduce6652.005.patch, > mapreduce6652.007.branch2.patch, mapreduce6652.007.patch, > mapreduce6652.008.patch > > > Jobs with large number of tasks can have job history files that are large in > size and resource-consuming(mainly memory) to parse in Job History Server. If > there are many such jobs, the job history server can very easily hang. > It would be a good usability feature if we added a new config property that > could be set to X, where the JHS wouldn't load the details for a job with > more than X tasks. The job would still show up on the list of jobs page, but > clicking on it would give a warning message that the job is too big, instead > of actually loading the job. This way we can prevent users from loading a job > that's way too big for the JHS, which currently makes the JHS hang. The > default value can be -1 so that it's disabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6652) Add configuration property to prevent JHS from loading jobs with a task count greater than X
[
https://issues.apache.org/jira/browse/MAPREDUCE-6652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373983#comment-15373983
]
Hadoop QA commented on MAPREDUCE-6652:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 55s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 23s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
44s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 0s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 32s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 25s
{color} | {color:red} hadoop-mapreduce-project/hadoop-mapreduce-client: The
patch generated 1 new + 77 unchanged - 2 fixed = 78 total (was 79) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 9s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 11s
{color} | {color:red}
hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs
generated 5 new + 198 unchanged - 0 fixed = 203 total (was 198) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 1s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 41s
{color} | {color:green} hadoop-mapreduce-client-common in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 20m 53s {color}
| {color:red} hadoop-mapreduce-client-hs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 48m 56s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.mapreduce.v2.hs.TestHistoryFileManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12817544/mapreduce6652.008.patch
|
| JIRA Issue | MAPREDUCE-6652 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 1f78ca268221 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Updated] (MAPREDUCE-6625) TestCLI#testGetJob fails occasionally
[
https://issues.apache.org/jira/browse/MAPREDUCE-6625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated MAPREDUCE-6625:
-
Fix Version/s: (was: 2.9.0)
2.8.0
> TestCLI#testGetJob fails occasionally
> -
>
> Key: MAPREDUCE-6625
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6625
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Jason Lowe
>Assignee: Haibo Chen
> Fix For: 2.8.0
>
> Attachments: mapreduce6625.001.patch, mapreduce6625.002.patch,
> mapreduce6625.003.patch
>
>
> Lately TestCLI has been failing sometimes in precommit builds:
> {noformat}
> Running org.apache.hadoop.mapreduce.tools.TestCLI
> Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.883 sec <<<
> FAILURE! - in org.apache.hadoop.mapreduce.tools.TestCLI
> testGetJob(org.apache.hadoop.mapreduce.tools.TestCLI) Time elapsed: 0.037
> sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.mapreduce.tools.TestCLI.testGetJob(TestCLI.java:175)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6625) TestCLI#testGetJob fails occasionally
[
https://issues.apache.org/jira/browse/MAPREDUCE-6625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373955#comment-15373955
]
Robert Kanter commented on MAPREDUCE-6625:
--
{quote}Is there a reason this wasn't committed to 2.8?{quote}
[~jlowe], no real reason. I've just committed it to branch-2.8 too.
> TestCLI#testGetJob fails occasionally
> -
>
> Key: MAPREDUCE-6625
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6625
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Jason Lowe
>Assignee: Haibo Chen
> Fix For: 2.8.0
>
> Attachments: mapreduce6625.001.patch, mapreduce6625.002.patch,
> mapreduce6625.003.patch
>
>
> Lately TestCLI has been failing sometimes in precommit builds:
> {noformat}
> Running org.apache.hadoop.mapreduce.tools.TestCLI
> Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.883 sec <<<
> FAILURE! - in org.apache.hadoop.mapreduce.tools.TestCLI
> testGetJob(org.apache.hadoop.mapreduce.tools.TestCLI) Time elapsed: 0.037
> sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.mapreduce.tools.TestCLI.testGetJob(TestCLI.java:175)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6729) Accurately compute the test execute time in DFSIO
[
https://issues.apache.org/jira/browse/MAPREDUCE-6729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng updated MAPREDUCE-6729:
-
Attachment: MAPREDUCE-6729.002.patch
Re-uploaded the same patch to trigger the building.
> Accurately compute the test execute time in DFSIO
> -
>
> Key: MAPREDUCE-6729
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6729
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: benchmarks, performance, test
>Affects Versions: 2.9.0
>Reporter: mingleizhang
>Assignee: mingleizhang
>Priority: Minor
> Labels: performance, test
> Attachments: MAPREDUCE-6729.001.patch, MAPREDUCE-6729.002.patch
>
>
> When doing DFSIO test as a distributed i/o benchmark tool. Then especially
> writes plenty of files to disk or read from, both can cause performance issue
> and imprecise value in a way. The question is that existing practices needs
> to delete files when before running a job and that will cause extra time
> consumption and furthermore cause performance issue, statistical time error
> and imprecise throughput while the files are lots of. So we need to replace
> or improve this hack to prevent this from happening in the future.
> {code}
> public static void testWrite() throws Exception {
> FileSystem fs = cluster.getFileSystem();
> long tStart = System.currentTimeMillis();
> bench.writeTest(fs); // this line of code will cause extra time
> consumption because of fs.delete(*,*) by the writeTest method
> long execTime = System.currentTimeMillis() - tStart;
> bench.analyzeResult(fs, TestType.TEST_TYPE_WRITE, execTime);
> }
> private void writeTest(FileSystem fs) throws IOException {
> Path writeDir = getWriteDir(config);
> fs.delete(getDataDir(config), true);
> fs.delete(writeDir, true);
> runIOTest(WriteMapper.class, writeDir);
> }
> {code}
> [https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/TestDFSIO.java]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6625) TestCLI#testGetJob fails occasionally
[
https://issues.apache.org/jira/browse/MAPREDUCE-6625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373937#comment-15373937
]
Hudson commented on MAPREDUCE-6625:
---
SUCCESS: Integrated in Hadoop-trunk-Commit #10082 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10082/])
MAPREDUCE-6625. TestCLI#testGetJob fails occasionally (haibochen via (rkanter:
rev ad256beb50e8851d5e6e5e743b00e819e75e3549)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/tools/TestCLI.java
> TestCLI#testGetJob fails occasionally
> -
>
> Key: MAPREDUCE-6625
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6625
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Jason Lowe
>Assignee: Haibo Chen
> Fix For: 2.9.0
>
> Attachments: mapreduce6625.001.patch, mapreduce6625.002.patch,
> mapreduce6625.003.patch
>
>
> Lately TestCLI has been failing sometimes in precommit builds:
> {noformat}
> Running org.apache.hadoop.mapreduce.tools.TestCLI
> Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.883 sec <<<
> FAILURE! - in org.apache.hadoop.mapreduce.tools.TestCLI
> testGetJob(org.apache.hadoop.mapreduce.tools.TestCLI) Time elapsed: 0.037
> sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.mapreduce.tools.TestCLI.testGetJob(TestCLI.java:175)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6718) add progress log to JHS during startup
[
https://issues.apache.org/jira/browse/MAPREDUCE-6718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373916#comment-15373916
]
Hadoop QA commented on MAPREDUCE-6718:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 37s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 23s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m 13s
{color} | {color:green} hadoop-mapreduce-client-hs in the patch passed. {color}
|
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 49s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12817549/mapreduce6718.002.patch
|
| JIRA Issue | MAPREDUCE-6718 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 866cbb9ad56d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / ad256be |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6610/testReport/ |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6610/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> add progress log to JHS during startup
> --
>
> Key: MAPREDUCE-6718
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6718
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobhistoryserver
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>
[jira] [Commented] (MAPREDUCE-6728) Give fetchers hint when ShuffleHandler rejects a shuffling connection
[
https://issues.apache.org/jira/browse/MAPREDUCE-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373891#comment-15373891
]
Hadoop QA commented on MAPREDUCE-6728:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 33s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 40s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 47s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s {color} | {color:green} hadoop-mapreduce-project/hadoop-mapreduce-client:
The patch generated 0 new + 280 unchanged - 1 fixed = 280 total (was 281)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 16s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 20s
{color} | {color:green} hadoop-mapreduce-client-shuffle in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 29s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12817537/mapreduce6728.001.patch
|
| JIRA Issue | MAPREDUCE-6728 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 1032caec4efb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / bf6f4a3 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6609/testReport/ |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle
U: hadoop-mapreduce-project/hadoop-mapreduce-client |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6609/console |
|
[jira] [Updated] (MAPREDUCE-6718) add progress log to JHS during startup
[ https://issues.apache.org/jira/browse/MAPREDUCE-6718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated MAPREDUCE-6718: -- Status: Patch Available (was: Open) > add progress log to JHS during startup > -- > > Key: MAPREDUCE-6718 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6718 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Minor > Labels: supportability > Attachments: mapreduce6718.001.patch, mapreduce6718.002.patch > > > lWhen the JHS starts up, it initializes the internal caches and storage via > the HistoryFileManager. If we have a large number of existing finished jobs > then we could spent minutes in this startup phase without logging progress: > 2016-03-14 10:56:01,444 INFO > org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file > system [hdfs://hadoopcdh.itnas01.ieee.org:8020] > 2016-03-14 10:56:11,455 INFO > org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager: Initializing Existing > Jobs... > 2016-03-14 12:01:36,926 INFO > org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage: CachedHistoryStorage > Init > This makes it really difficult to assess if things are working correctly (it > looks hung). We can add logs to notify users of progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6718) add progress log to JHS during startup
[ https://issues.apache.org/jira/browse/MAPREDUCE-6718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated MAPREDUCE-6718: -- Status: Open (was: Patch Available) > add progress log to JHS during startup > -- > > Key: MAPREDUCE-6718 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6718 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Minor > Labels: supportability > Attachments: mapreduce6718.001.patch, mapreduce6718.002.patch > > > lWhen the JHS starts up, it initializes the internal caches and storage via > the HistoryFileManager. If we have a large number of existing finished jobs > then we could spent minutes in this startup phase without logging progress: > 2016-03-14 10:56:01,444 INFO > org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file > system [hdfs://hadoopcdh.itnas01.ieee.org:8020] > 2016-03-14 10:56:11,455 INFO > org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager: Initializing Existing > Jobs... > 2016-03-14 12:01:36,926 INFO > org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage: CachedHistoryStorage > Init > This makes it really difficult to assess if things are working correctly (it > looks hung). We can add logs to notify users of progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6718) add progress log to JHS during startup
[ https://issues.apache.org/jira/browse/MAPREDUCE-6718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373884#comment-15373884 ] Haibo Chen commented on MAPREDUCE-6718: --- Nice catch! I added a check of maxSize for "double loadedPercent = prevCacheSize * 100.0 / jobListCache.maxSize". jobListCache.isFull() checks the size against maxSize, and returns true if maxSize is zero, so we won't come to (currCacheSize - prevCacheSize)/(double)jobListCache.maxSize in the body of the for loop. > add progress log to JHS during startup > -- > > Key: MAPREDUCE-6718 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6718 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Minor > Labels: supportability > Attachments: mapreduce6718.001.patch, mapreduce6718.002.patch > > > lWhen the JHS starts up, it initializes the internal caches and storage via > the HistoryFileManager. If we have a large number of existing finished jobs > then we could spent minutes in this startup phase without logging progress: > 2016-03-14 10:56:01,444 INFO > org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file > system [hdfs://hadoopcdh.itnas01.ieee.org:8020] > 2016-03-14 10:56:11,455 INFO > org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager: Initializing Existing > Jobs... > 2016-03-14 12:01:36,926 INFO > org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage: CachedHistoryStorage > Init > This makes it really difficult to assess if things are working correctly (it > looks hung). We can add logs to notify users of progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6625) TestCLI#testGetJob fails occasionally
[
https://issues.apache.org/jira/browse/MAPREDUCE-6625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373882#comment-15373882
]
Jason Lowe commented on MAPREDUCE-6625:
---
Thanks [~rkanter] for picking this up -- it dropped off my radar after
vacations and conferences.
Is there a reason this wasn't committed to 2.8? It looks like the original
race was added in 2.8 via MAPREDUCE-6566, so it seems appropriate to fix it
there as well. Otherwise those running the unit tests on 2.8-based releases
will occasionally see this issue.
> TestCLI#testGetJob fails occasionally
> -
>
> Key: MAPREDUCE-6625
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6625
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Jason Lowe
>Assignee: Haibo Chen
> Fix For: 2.9.0
>
> Attachments: mapreduce6625.001.patch, mapreduce6625.002.patch,
> mapreduce6625.003.patch
>
>
> Lately TestCLI has been failing sometimes in precommit builds:
> {noformat}
> Running org.apache.hadoop.mapreduce.tools.TestCLI
> Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.883 sec <<<
> FAILURE! - in org.apache.hadoop.mapreduce.tools.TestCLI
> testGetJob(org.apache.hadoop.mapreduce.tools.TestCLI) Time elapsed: 0.037
> sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.mapreduce.tools.TestCLI.testGetJob(TestCLI.java:175)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6718) add progress log to JHS during startup
[ https://issues.apache.org/jira/browse/MAPREDUCE-6718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated MAPREDUCE-6718: -- Attachment: mapreduce6718.002.patch > add progress log to JHS during startup > -- > > Key: MAPREDUCE-6718 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6718 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Minor > Labels: supportability > Attachments: mapreduce6718.001.patch, mapreduce6718.002.patch > > > lWhen the JHS starts up, it initializes the internal caches and storage via > the HistoryFileManager. If we have a large number of existing finished jobs > then we could spent minutes in this startup phase without logging progress: > 2016-03-14 10:56:01,444 INFO > org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file > system [hdfs://hadoopcdh.itnas01.ieee.org:8020] > 2016-03-14 10:56:11,455 INFO > org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager: Initializing Existing > Jobs... > 2016-03-14 12:01:36,926 INFO > org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage: CachedHistoryStorage > Init > This makes it really difficult to assess if things are working correctly (it > looks hung). We can add logs to notify users of progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6625) TestCLI#testGetJob fails occasionally
[
https://issues.apache.org/jira/browse/MAPREDUCE-6625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated MAPREDUCE-6625:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.9.0
Status: Resolved (was: Patch Available)
Thanks [~haibochen]. Committed to trunk and branch-2!
> TestCLI#testGetJob fails occasionally
> -
>
> Key: MAPREDUCE-6625
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6625
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Jason Lowe
>Assignee: Haibo Chen
> Fix For: 2.9.0
>
> Attachments: mapreduce6625.001.patch, mapreduce6625.002.patch,
> mapreduce6625.003.patch
>
>
> Lately TestCLI has been failing sometimes in precommit builds:
> {noformat}
> Running org.apache.hadoop.mapreduce.tools.TestCLI
> Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.883 sec <<<
> FAILURE! - in org.apache.hadoop.mapreduce.tools.TestCLI
> testGetJob(org.apache.hadoop.mapreduce.tools.TestCLI) Time elapsed: 0.037
> sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.mapreduce.tools.TestCLI.testGetJob(TestCLI.java:175)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6652) Add configuration property to prevent JHS from loading jobs with a task count greater than X
[ https://issues.apache.org/jira/browse/MAPREDUCE-6652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373857#comment-15373857 ] Haibo Chen commented on MAPREDUCE-6652: --- Retriggered build. Will upload branch-2 patch after the build is done. > Add configuration property to prevent JHS from loading jobs with a task count > greater than X > > > Key: MAPREDUCE-6652 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6652 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: mapreduce6652.001.patch, mapreduce6652.002.patch, > mapreduce6652.003.patch, mapreduce6652.004.patch, mapreduce6652.005.patch, > mapreduce6652.007.branch2.patch, mapreduce6652.007.patch, > mapreduce6652.008.patch > > > Jobs with large number of tasks can have job history files that are large in > size and resource-consuming(mainly memory) to parse in Job History Server. If > there are many such jobs, the job history server can very easily hang. > It would be a good usability feature if we added a new config property that > could be set to X, where the JHS wouldn't load the details for a job with > more than X tasks. The job would still show up on the list of jobs page, but > clicking on it would give a warning message that the job is too big, instead > of actually loading the job. This way we can prevent users from loading a job > that's way too big for the JHS, which currently makes the JHS hang. The > default value can be -1 so that it's disabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6652) Add configuration property to prevent JHS from loading jobs with a task count greater than X
[ https://issues.apache.org/jira/browse/MAPREDUCE-6652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated MAPREDUCE-6652: -- Attachment: (was: mapreduce6652.008.patch) > Add configuration property to prevent JHS from loading jobs with a task count > greater than X > > > Key: MAPREDUCE-6652 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6652 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: mapreduce6652.001.patch, mapreduce6652.002.patch, > mapreduce6652.003.patch, mapreduce6652.004.patch, mapreduce6652.005.patch, > mapreduce6652.007.branch2.patch, mapreduce6652.007.patch, > mapreduce6652.008.patch > > > Jobs with large number of tasks can have job history files that are large in > size and resource-consuming(mainly memory) to parse in Job History Server. If > there are many such jobs, the job history server can very easily hang. > It would be a good usability feature if we added a new config property that > could be set to X, where the JHS wouldn't load the details for a job with > more than X tasks. The job would still show up on the list of jobs page, but > clicking on it would give a warning message that the job is too big, instead > of actually loading the job. This way we can prevent users from loading a job > that's way too big for the JHS, which currently makes the JHS hang. The > default value can be -1 so that it's disabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6652) Add configuration property to prevent JHS from loading jobs with a task count greater than X
[ https://issues.apache.org/jira/browse/MAPREDUCE-6652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated MAPREDUCE-6652: -- Status: Patch Available (was: Open) > Add configuration property to prevent JHS from loading jobs with a task count > greater than X > > > Key: MAPREDUCE-6652 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6652 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: mapreduce6652.001.patch, mapreduce6652.002.patch, > mapreduce6652.003.patch, mapreduce6652.004.patch, mapreduce6652.005.patch, > mapreduce6652.007.branch2.patch, mapreduce6652.007.patch, > mapreduce6652.008.patch > > > Jobs with large number of tasks can have job history files that are large in > size and resource-consuming(mainly memory) to parse in Job History Server. If > there are many such jobs, the job history server can very easily hang. > It would be a good usability feature if we added a new config property that > could be set to X, where the JHS wouldn't load the details for a job with > more than X tasks. The job would still show up on the list of jobs page, but > clicking on it would give a warning message that the job is too big, instead > of actually loading the job. This way we can prevent users from loading a job > that's way too big for the JHS, which currently makes the JHS hang. The > default value can be -1 so that it's disabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6652) Add configuration property to prevent JHS from loading jobs with a task count greater than X
[ https://issues.apache.org/jira/browse/MAPREDUCE-6652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated MAPREDUCE-6652: -- Attachment: mapreduce6652.008.patch > Add configuration property to prevent JHS from loading jobs with a task count > greater than X > > > Key: MAPREDUCE-6652 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6652 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: mapreduce6652.001.patch, mapreduce6652.002.patch, > mapreduce6652.003.patch, mapreduce6652.004.patch, mapreduce6652.005.patch, > mapreduce6652.007.branch2.patch, mapreduce6652.007.patch, > mapreduce6652.008.patch > > > Jobs with large number of tasks can have job history files that are large in > size and resource-consuming(mainly memory) to parse in Job History Server. If > there are many such jobs, the job history server can very easily hang. > It would be a good usability feature if we added a new config property that > could be set to X, where the JHS wouldn't load the details for a job with > more than X tasks. The job would still show up on the list of jobs page, but > clicking on it would give a warning message that the job is too big, instead > of actually loading the job. This way we can prevent users from loading a job > that's way too big for the JHS, which currently makes the JHS hang. The > default value can be -1 so that it's disabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6652) Add configuration property to prevent JHS from loading jobs with a task count greater than X
[ https://issues.apache.org/jira/browse/MAPREDUCE-6652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated MAPREDUCE-6652: -- Status: Patch Available (was: Open) > Add configuration property to prevent JHS from loading jobs with a task count > greater than X > > > Key: MAPREDUCE-6652 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6652 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: mapreduce6652.001.patch, mapreduce6652.002.patch, > mapreduce6652.003.patch, mapreduce6652.004.patch, mapreduce6652.005.patch, > mapreduce6652.007.branch2.patch, mapreduce6652.007.patch, > mapreduce6652.008.patch > > > Jobs with large number of tasks can have job history files that are large in > size and resource-consuming(mainly memory) to parse in Job History Server. If > there are many such jobs, the job history server can very easily hang. > It would be a good usability feature if we added a new config property that > could be set to X, where the JHS wouldn't load the details for a job with > more than X tasks. The job would still show up on the list of jobs page, but > clicking on it would give a warning message that the job is too big, instead > of actually loading the job. This way we can prevent users from loading a job > that's way too big for the JHS, which currently makes the JHS hang. The > default value can be -1 so that it's disabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6652) Add configuration property to prevent JHS from loading jobs with a task count greater than X
[ https://issues.apache.org/jira/browse/MAPREDUCE-6652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated MAPREDUCE-6652: -- Status: Open (was: Patch Available) > Add configuration property to prevent JHS from loading jobs with a task count > greater than X > > > Key: MAPREDUCE-6652 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6652 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: mapreduce6652.001.patch, mapreduce6652.002.patch, > mapreduce6652.003.patch, mapreduce6652.004.patch, mapreduce6652.005.patch, > mapreduce6652.007.branch2.patch, mapreduce6652.007.patch, > mapreduce6652.008.patch > > > Jobs with large number of tasks can have job history files that are large in > size and resource-consuming(mainly memory) to parse in Job History Server. If > there are many such jobs, the job history server can very easily hang. > It would be a good usability feature if we added a new config property that > could be set to X, where the JHS wouldn't load the details for a job with > more than X tasks. The job would still show up on the list of jobs page, but > clicking on it would give a warning message that the job is too big, instead > of actually loading the job. This way we can prevent users from loading a job > that's way too big for the JHS, which currently makes the JHS hang. The > default value can be -1 so that it's disabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6652) Add configuration property to prevent JHS from loading jobs with a task count greater than X
[ https://issues.apache.org/jira/browse/MAPREDUCE-6652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated MAPREDUCE-6652: -- Attachment: (was: mapreduce6652.008.branch2.patch) > Add configuration property to prevent JHS from loading jobs with a task count > greater than X > > > Key: MAPREDUCE-6652 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6652 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: mapreduce6652.001.patch, mapreduce6652.002.patch, > mapreduce6652.003.patch, mapreduce6652.004.patch, mapreduce6652.005.patch, > mapreduce6652.007.branch2.patch, mapreduce6652.007.patch, > mapreduce6652.008.patch > > > Jobs with large number of tasks can have job history files that are large in > size and resource-consuming(mainly memory) to parse in Job History Server. If > there are many such jobs, the job history server can very easily hang. > It would be a good usability feature if we added a new config property that > could be set to X, where the JHS wouldn't load the details for a job with > more than X tasks. The job would still show up on the list of jobs page, but > clicking on it would give a warning message that the job is too big, instead > of actually loading the job. This way we can prevent users from loading a job > that's way too big for the JHS, which currently makes the JHS hang. The > default value can be -1 so that it's disabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6652) Add configuration property to prevent JHS from loading jobs with a task count greater than X
[ https://issues.apache.org/jira/browse/MAPREDUCE-6652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated MAPREDUCE-6652: -- Status: Open (was: Patch Available) > Add configuration property to prevent JHS from loading jobs with a task count > greater than X > > > Key: MAPREDUCE-6652 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6652 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: mapreduce6652.001.patch, mapreduce6652.002.patch, > mapreduce6652.003.patch, mapreduce6652.004.patch, mapreduce6652.005.patch, > mapreduce6652.007.branch2.patch, mapreduce6652.007.patch, > mapreduce6652.008.patch > > > Jobs with large number of tasks can have job history files that are large in > size and resource-consuming(mainly memory) to parse in Job History Server. If > there are many such jobs, the job history server can very easily hang. > It would be a good usability feature if we added a new config property that > could be set to X, where the JHS wouldn't load the details for a job with > more than X tasks. The job would still show up on the list of jobs page, but > clicking on it would give a warning message that the job is too big, instead > of actually loading the job. This way we can prevent users from loading a job > that's way too big for the JHS, which currently makes the JHS hang. The > default value can be -1 so that it's disabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6718) add progress log to JHS during startup
[ https://issues.apache.org/jira/browse/MAPREDUCE-6718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated MAPREDUCE-6718: -- Description: lWhen the JHS starts up, it initializes the internal caches and storage via the HistoryFileManager. If we have a large number of existing finished jobs then we could spent minutes in this startup phase without logging progress: 2016-03-14 10:56:01,444 INFO org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://hadoopcdh.itnas01.ieee.org:8020] 2016-03-14 10:56:11,455 INFO org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager: Initializing Existing Jobs... 2016-03-14 12:01:36,926 INFO org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage: CachedHistoryStorage Init This makes it really difficult to assess if things are working correctly (it looks hung). We can add logs to notify users of progress. was: When the JHS starts up, it initializes the internal caches and storage via the HistoryFileManager. If we have a large number of existing finished jobs then we could spent minutes in this startup phase without logging progress: 2016-03-14 10:56:01,444 INFO org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://hadoopcdh.itnas01.ieee.org:8020] 2016-03-14 10:56:11,455 INFO org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager: Initializing Existing Jobs... 2016-03-14 12:01:36,926 INFO org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage: CachedHistoryStorage Init This makes it really difficult to assess if things are working correctly (it looks hung). We can add logs to notify users of progress. > add progress log to JHS during startup > -- > > Key: MAPREDUCE-6718 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6718 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Minor > Labels: supportability > Attachments: mapreduce6718.001.patch > > > lWhen the JHS starts up, it initializes the internal caches and storage via > the HistoryFileManager. If we have a large number of existing finished jobs > then we could spent minutes in this startup phase without logging progress: > 2016-03-14 10:56:01,444 INFO > org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file > system [hdfs://hadoopcdh.itnas01.ieee.org:8020] > 2016-03-14 10:56:11,455 INFO > org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager: Initializing Existing > Jobs... > 2016-03-14 12:01:36,926 INFO > org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage: CachedHistoryStorage > Init > This makes it really difficult to assess if things are working correctly (it > looks hung). We can add logs to notify users of progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6728) Give fetchers hint when ShuffleHandler rejects a shuffling connection
[ https://issues.apache.org/jira/browse/MAPREDUCE-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated MAPREDUCE-6728: -- Status: Patch Available (was: Open) > Give fetchers hint when ShuffleHandler rejects a shuffling connection > - > > Key: MAPREDUCE-6728 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6728 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: mapreduce6728.001.patch, mapreduce6728.prelim.patch > > > If # of open shuffle connection to a node goes over the max, ShuffleHandler > closes the connection immediately without giving fetchers any hint of the > reason, which causes fetchers to fail due to exceptions > java.net.SocketException: Unexpected end of file from server > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:772) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323) > at > java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:323) > at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193) > OR > java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:196) > at java.net.SocketInputStream.read(SocketInputStream.java:122) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) > at java.io.BufferedInputStream.read1(BufferedInputStream.java:275) > at java.io.BufferedInputStream.read(BufferedInputStream.java:334) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323) > at > java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java > Such failures are counted as fetcher failures -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6728) Give fetchers hint when ShuffleHandler rejects a shuffling connection
[ https://issues.apache.org/jira/browse/MAPREDUCE-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated MAPREDUCE-6728: -- Status: Open (was: Patch Available) > Give fetchers hint when ShuffleHandler rejects a shuffling connection > - > > Key: MAPREDUCE-6728 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6728 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: mapreduce6728.001.patch, mapreduce6728.prelim.patch > > > If # of open shuffle connection to a node goes over the max, ShuffleHandler > closes the connection immediately without giving fetchers any hint of the > reason, which causes fetchers to fail due to exceptions > java.net.SocketException: Unexpected end of file from server > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:772) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323) > at > java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:323) > at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193) > OR > java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:196) > at java.net.SocketInputStream.read(SocketInputStream.java:122) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) > at java.io.BufferedInputStream.read1(BufferedInputStream.java:275) > at java.io.BufferedInputStream.read(BufferedInputStream.java:334) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323) > at > java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java > Such failures are counted as fetcher failures -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6728) Give fetchers hint when ShuffleHandler rejects a shuffling connection
[ https://issues.apache.org/jira/browse/MAPREDUCE-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated MAPREDUCE-6728: -- Attachment: mapreduce6728.001.patch New patch uploaded to accommodate the checkstyle warning > Give fetchers hint when ShuffleHandler rejects a shuffling connection > - > > Key: MAPREDUCE-6728 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6728 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: mapreduce6728.001.patch, mapreduce6728.prelim.patch > > > If # of open shuffle connection to a node goes over the max, ShuffleHandler > closes the connection immediately without giving fetchers any hint of the > reason, which causes fetchers to fail due to exceptions > java.net.SocketException: Unexpected end of file from server > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:772) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323) > at > java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:323) > at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193) > OR > java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:196) > at java.net.SocketInputStream.read(SocketInputStream.java:122) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) > at java.io.BufferedInputStream.read1(BufferedInputStream.java:275) > at java.io.BufferedInputStream.read(BufferedInputStream.java:334) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323) > at > java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:430) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:395) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:266) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java > Such failures are counted as fetcher failures -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6652) Add configuration property to prevent JHS from loading jobs with a task count greater than X
[
https://issues.apache.org/jira/browse/MAPREDUCE-6652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373825#comment-15373825
]
Hadoop QA commented on MAPREDUCE-6652:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s {color}
| {color:red} MAPREDUCE-6652 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12817535/mapreduce6652.008.branch2.patch
|
| JIRA Issue | MAPREDUCE-6652 |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6608/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Add configuration property to prevent JHS from loading jobs with a task count
> greater than X
>
>
> Key: MAPREDUCE-6652
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6652
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobhistoryserver
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: mapreduce6652.001.patch, mapreduce6652.002.patch,
> mapreduce6652.003.patch, mapreduce6652.004.patch, mapreduce6652.005.patch,
> mapreduce6652.007.branch2.patch, mapreduce6652.007.patch,
> mapreduce6652.008.branch2.patch, mapreduce6652.008.patch
>
>
> Jobs with large number of tasks can have job history files that are large in
> size and resource-consuming(mainly memory) to parse in Job History Server. If
> there are many such jobs, the job history server can very easily hang.
> It would be a good usability feature if we added a new config property that
> could be set to X, where the JHS wouldn't load the details for a job with
> more than X tasks. The job would still show up on the list of jobs page, but
> clicking on it would give a warning message that the job is too big, instead
> of actually loading the job. This way we can prevent users from loading a job
> that's way too big for the JHS, which currently makes the JHS hang. The
> default value can be -1 so that it's disabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6652) Add configuration property to prevent JHS from loading jobs with a task count greater than X
[ https://issues.apache.org/jira/browse/MAPREDUCE-6652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated MAPREDUCE-6652: -- Status: Patch Available (was: Open) > Add configuration property to prevent JHS from loading jobs with a task count > greater than X > > > Key: MAPREDUCE-6652 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6652 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: mapreduce6652.001.patch, mapreduce6652.002.patch, > mapreduce6652.003.patch, mapreduce6652.004.patch, mapreduce6652.005.patch, > mapreduce6652.007.branch2.patch, mapreduce6652.007.patch, > mapreduce6652.008.branch2.patch, mapreduce6652.008.patch > > > Jobs with large number of tasks can have job history files that are large in > size and resource-consuming(mainly memory) to parse in Job History Server. If > there are many such jobs, the job history server can very easily hang. > It would be a good usability feature if we added a new config property that > could be set to X, where the JHS wouldn't load the details for a job with > more than X tasks. The job would still show up on the list of jobs page, but > clicking on it would give a warning message that the job is too big, instead > of actually loading the job. This way we can prevent users from loading a job > that's way too big for the JHS, which currently makes the JHS hang. The > default value can be -1 so that it's disabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6652) Add configuration property to prevent JHS from loading jobs with a task count greater than X
[ https://issues.apache.org/jira/browse/MAPREDUCE-6652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated MAPREDUCE-6652: -- Attachment: mapreduce6652.008.branch2.patch mapreduce6652.008.patch > Add configuration property to prevent JHS from loading jobs with a task count > greater than X > > > Key: MAPREDUCE-6652 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6652 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: mapreduce6652.001.patch, mapreduce6652.002.patch, > mapreduce6652.003.patch, mapreduce6652.004.patch, mapreduce6652.005.patch, > mapreduce6652.007.branch2.patch, mapreduce6652.007.patch, > mapreduce6652.008.branch2.patch, mapreduce6652.008.patch > > > Jobs with large number of tasks can have job history files that are large in > size and resource-consuming(mainly memory) to parse in Job History Server. If > there are many such jobs, the job history server can very easily hang. > It would be a good usability feature if we added a new config property that > could be set to X, where the JHS wouldn't load the details for a job with > more than X tasks. The job would still show up on the list of jobs page, but > clicking on it would give a warning message that the job is too big, instead > of actually loading the job. This way we can prevent users from loading a job > that's way too big for the JHS, which currently makes the JHS hang. The > default value can be -1 so that it's disabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6652) Add configuration property to prevent JHS from loading jobs with a task count greater than X
[ https://issues.apache.org/jira/browse/MAPREDUCE-6652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated MAPREDUCE-6652: -- Status: Open (was: Patch Available) > Add configuration property to prevent JHS from loading jobs with a task count > greater than X > > > Key: MAPREDUCE-6652 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6652 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: mapreduce6652.001.patch, mapreduce6652.002.patch, > mapreduce6652.003.patch, mapreduce6652.004.patch, mapreduce6652.005.patch, > mapreduce6652.007.branch2.patch, mapreduce6652.007.patch, > mapreduce6652.008.branch2.patch, mapreduce6652.008.patch > > > Jobs with large number of tasks can have job history files that are large in > size and resource-consuming(mainly memory) to parse in Job History Server. If > there are many such jobs, the job history server can very easily hang. > It would be a good usability feature if we added a new config property that > could be set to X, where the JHS wouldn't load the details for a job with > more than X tasks. The job would still show up on the list of jobs page, but > clicking on it would give a warning message that the job is too big, instead > of actually loading the job. This way we can prevent users from loading a job > that's way too big for the JHS, which currently makes the JHS hang. The > default value can be -1 so that it's disabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6652) Add configuration property to prevent JHS from loading jobs with a task count greater than X
[ https://issues.apache.org/jira/browse/MAPREDUCE-6652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373816#comment-15373816 ] Haibo Chen commented on MAPREDUCE-6652: --- Thanks @Ray for your comments, uploading a new version including the javadoc changes. > Add configuration property to prevent JHS from loading jobs with a task count > greater than X > > > Key: MAPREDUCE-6652 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6652 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: mapreduce6652.001.patch, mapreduce6652.002.patch, > mapreduce6652.003.patch, mapreduce6652.004.patch, mapreduce6652.005.patch, > mapreduce6652.007.branch2.patch, mapreduce6652.007.patch > > > Jobs with large number of tasks can have job history files that are large in > size and resource-consuming(mainly memory) to parse in Job History Server. If > there are many such jobs, the job history server can very easily hang. > It would be a good usability feature if we added a new config property that > could be set to X, where the JHS wouldn't load the details for a job with > more than X tasks. The job would still show up on the list of jobs page, but > clicking on it would give a warning message that the job is too big, instead > of actually loading the job. This way we can prevent users from loading a job > that's way too big for the JHS, which currently makes the JHS hang. The > default value can be -1 so that it's disabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6625) TestCLI#testGetJob fails occasionally
[
https://issues.apache.org/jira/browse/MAPREDUCE-6625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373723#comment-15373723
]
Robert Kanter commented on MAPREDUCE-6625:
--
+1
> TestCLI#testGetJob fails occasionally
> -
>
> Key: MAPREDUCE-6625
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6625
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Jason Lowe
>Assignee: Haibo Chen
> Attachments: mapreduce6625.001.patch, mapreduce6625.002.patch,
> mapreduce6625.003.patch
>
>
> Lately TestCLI has been failing sometimes in precommit builds:
> {noformat}
> Running org.apache.hadoop.mapreduce.tools.TestCLI
> Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.883 sec <<<
> FAILURE! - in org.apache.hadoop.mapreduce.tools.TestCLI
> testGetJob(org.apache.hadoop.mapreduce.tools.TestCLI) Time elapsed: 0.037
> sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.mapreduce.tools.TestCLI.testGetJob(TestCLI.java:175)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-6652) Add configuration property to prevent JHS from loading jobs with a task count greater than X
[
https://issues.apache.org/jira/browse/MAPREDUCE-6652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373717#comment-15373717
]
Ray Chiang edited comment on MAPREDUCE-6652 at 7/12/16 9:19 PM:
Really minor Javadoc nits
# In HistoryFileManager.java
#* {{greater than maxTaskForLoadedJob.}} should be {{greater than
maxTasksForLoadedJob.}}
#* Phrasing and punctuation should be cleaned up here:
{quote}
@return null if the underlying file was deleted.
partially parsed job if the job has too many tasks
fully parsed job otherwise
{quote}
will read nicer:
{quote}
@return null if the underlying job history file was deleted, an \{@link
UnparsedJob\}
object representing a partially parsed job if the job tasks
exceeds the
configured maximum, or a \{@link CompletedJob\}
representing a fully
parsed job.
{quote}
was (Author: rchiang):
Really minor Javadoc nits
# In HistoryFileManager.java
#* {{greater than maxTaskForLoadedJob.}} should be {{greater than
maxTasksForLoadedJob.}}
#* Phrasing and punctuation should be cleaned up here:
{quote}
@return null if the underlying file was deleted.
partially parsed job if the job has too many tasks
fully parsed job otherwise
{quote}
will read nicer:
{quote}
@return null if the underlying job history file was deleted, an \{@link
UnparsedJob\}
object representing a partially parsed job if the job tasks
exceeds the
configured maximum, or a {@link CompletedJob} representing
a fully
parsed job.
{quote}
> Add configuration property to prevent JHS from loading jobs with a task count
> greater than X
>
>
> Key: MAPREDUCE-6652
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6652
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobhistoryserver
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: mapreduce6652.001.patch, mapreduce6652.002.patch,
> mapreduce6652.003.patch, mapreduce6652.004.patch, mapreduce6652.005.patch,
> mapreduce6652.007.branch2.patch, mapreduce6652.007.patch
>
>
> Jobs with large number of tasks can have job history files that are large in
> size and resource-consuming(mainly memory) to parse in Job History Server. If
> there are many such jobs, the job history server can very easily hang.
> It would be a good usability feature if we added a new config property that
> could be set to X, where the JHS wouldn't load the details for a job with
> more than X tasks. The job would still show up on the list of jobs page, but
> clicking on it would give a warning message that the job is too big, instead
> of actually loading the job. This way we can prevent users from loading a job
> that's way too big for the JHS, which currently makes the JHS hang. The
> default value can be -1 so that it's disabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-6652) Add configuration property to prevent JHS from loading jobs with a task count greater than X
[
https://issues.apache.org/jira/browse/MAPREDUCE-6652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373717#comment-15373717
]
Ray Chiang edited comment on MAPREDUCE-6652 at 7/12/16 9:20 PM:
Really minor Javadoc nits
In HistoryFileManager.java
1. {{greater than maxTaskForLoadedJob.}} should be {{greater than
maxTasksForLoadedJob.}}
2. Phrasing and punctuation should be cleaned up here:
{quote}
@return null if the underlying file was deleted.
partially parsed job if the job has too many tasks
fully parsed job otherwise
{quote}
will read nicer:
{quote}
@return null if the underlying job history file was deleted, an \{@link
UnparsedJob\}
object representing a partially parsed job if the job tasks
exceeds the
configured maximum, or a \{@link CompletedJob\}
representing a fully
parsed job.
{quote}
was (Author: rchiang):
Really minor Javadoc nits
# In HistoryFileManager.java
#* {{greater than maxTaskForLoadedJob.}} should be {{greater than
maxTasksForLoadedJob.}}
#* Phrasing and punctuation should be cleaned up here:
{quote}
@return null if the underlying file was deleted.
partially parsed job if the job has too many tasks
fully parsed job otherwise
{quote}
will read nicer:
{quote}
@return null if the underlying job history file was deleted, an \{@link
UnparsedJob\}
object representing a partially parsed job if the job tasks
exceeds the
configured maximum, or a \{@link CompletedJob\}
representing a fully
parsed job.
{quote}
> Add configuration property to prevent JHS from loading jobs with a task count
> greater than X
>
>
> Key: MAPREDUCE-6652
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6652
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobhistoryserver
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: mapreduce6652.001.patch, mapreduce6652.002.patch,
> mapreduce6652.003.patch, mapreduce6652.004.patch, mapreduce6652.005.patch,
> mapreduce6652.007.branch2.patch, mapreduce6652.007.patch
>
>
> Jobs with large number of tasks can have job history files that are large in
> size and resource-consuming(mainly memory) to parse in Job History Server. If
> there are many such jobs, the job history server can very easily hang.
> It would be a good usability feature if we added a new config property that
> could be set to X, where the JHS wouldn't load the details for a job with
> more than X tasks. The job would still show up on the list of jobs page, but
> clicking on it would give a warning message that the job is too big, instead
> of actually loading the job. This way we can prevent users from loading a job
> that's way too big for the JHS, which currently makes the JHS hang. The
> default value can be -1 so that it's disabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6652) Add configuration property to prevent JHS from loading jobs with a task count greater than X
[
https://issues.apache.org/jira/browse/MAPREDUCE-6652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373717#comment-15373717
]
Ray Chiang commented on MAPREDUCE-6652:
---
Really minor Javadoc nits
# In HistoryFileManager.java
#* {{greater than maxTaskForLoadedJob.}} should be {{greater than
maxTasksForLoadedJob.}}
#* Phrasing and punctuation should be cleaned up here:
{quote}
@return null if the underlying file was deleted.
partially parsed job if the job has too many tasks
fully parsed job otherwise
{quote}
will read nicer:
{quote}
@return null if the underlying job history file was deleted, an \{@link
UnparsedJob\}
object representing a partially parsed job if the job tasks
exceeds the
configured maximum, or a {@link CompletedJob} representing
a fully
parsed job.
{quote}
> Add configuration property to prevent JHS from loading jobs with a task count
> greater than X
>
>
> Key: MAPREDUCE-6652
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6652
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobhistoryserver
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: mapreduce6652.001.patch, mapreduce6652.002.patch,
> mapreduce6652.003.patch, mapreduce6652.004.patch, mapreduce6652.005.patch,
> mapreduce6652.007.branch2.patch, mapreduce6652.007.patch
>
>
> Jobs with large number of tasks can have job history files that are large in
> size and resource-consuming(mainly memory) to parse in Job History Server. If
> there are many such jobs, the job history server can very easily hang.
> It would be a good usability feature if we added a new config property that
> could be set to X, where the JHS wouldn't load the details for a job with
> more than X tasks. The job would still show up on the list of jobs page, but
> clicking on it would give a warning message that the job is too big, instead
> of actually loading the job. This way we can prevent users from loading a job
> that's way too big for the JHS, which currently makes the JHS hang. The
> default value can be -1 so that it's disabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6718) add progress log to JHS during startup
[
https://issues.apache.org/jira/browse/MAPREDUCE-6718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373701#comment-15373701
]
Robert Kanter commented on MAPREDUCE-6718:
--
One thing:
- If {{jobListCache.maxSize}}, which is loaded from a config property, is
{{0}}, then {{(currCacheSize - prevCacheSize)/(double)jobListCache.maxSize}}
will be a problem.
> add progress log to JHS during startup
> --
>
> Key: MAPREDUCE-6718
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6718
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobhistoryserver
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Minor
> Labels: supportability
> Attachments: mapreduce6718.001.patch
>
>
> When the JHS starts up, it initializes the internal caches and storage via
> the HistoryFileManager. If we have a large number of existing finished jobs
> then we could spent minutes in this startup phase without logging progress:
> 2016-03-14 10:56:01,444 INFO
> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file
> system [hdfs://hadoopcdh.itnas01.ieee.org:8020]
> 2016-03-14 10:56:11,455 INFO
> org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager: Initializing Existing
> Jobs...
> 2016-03-14 12:01:36,926 INFO
> org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage: CachedHistoryStorage
> Init
> This makes it really difficult to assess if things are working correctly (it
> looks hung). We can add logs to notify users of progress.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6652) Add configuration property to prevent JHS from loading jobs with a task count greater than X
[ https://issues.apache.org/jira/browse/MAPREDUCE-6652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373673#comment-15373673 ] Robert Kanter commented on MAPREDUCE-6652: -- +1 > Add configuration property to prevent JHS from loading jobs with a task count > greater than X > > > Key: MAPREDUCE-6652 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6652 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: mapreduce6652.001.patch, mapreduce6652.002.patch, > mapreduce6652.003.patch, mapreduce6652.004.patch, mapreduce6652.005.patch, > mapreduce6652.007.branch2.patch, mapreduce6652.007.patch > > > Jobs with large number of tasks can have job history files that are large in > size and resource-consuming(mainly memory) to parse in Job History Server. If > there are many such jobs, the job history server can very easily hang. > It would be a good usability feature if we added a new config property that > could be set to X, where the JHS wouldn't load the details for a job with > more than X tasks. The job would still show up on the list of jobs page, but > clicking on it would give a warning message that the job is too big, instead > of actually loading the job. This way we can prevent users from loading a job > that's way too big for the JHS, which currently makes the JHS hang. The > default value can be -1 so that it's disabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6731) TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling() may fail for concurrent tests
[
https://issues.apache.org/jira/browse/MAPREDUCE-6731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373103#comment-15373103
]
Hudson commented on MAPREDUCE-6731:
---
SUCCESS: Integrated in Hadoop-trunk-Commit #10079 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10079/])
MAPREDUCE-6731. (varunsaxena: rev d14e729a55c0ca4053c537290392b74f732ce939)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMRTimelineEventHandling.java
> TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling() may fail
> for concurrent tests
> -
>
> Key: MAPREDUCE-6731
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6731
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0-alpha1
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
> Fix For: 3.0.0-alpha1
>
> Attachments: MAPREDUCE-6731.01.patch, MAPREDUCE-6731.02.patch
>
>
> {{TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling()}} uses
> the default file-system storage directory, and is brittle against concurrent
> tests.
> We should use a unique storage directory for the tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6731) TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling() may fail for concurrent tests
[
https://issues.apache.org/jira/browse/MAPREDUCE-6731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373047#comment-15373047
]
Varun Saxena commented on MAPREDUCE-6731:
-
Committed to trunk.
Thanks [~sjlee0] for your contribution and [~vrushalic] for the review.
> TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling() may fail
> for concurrent tests
> -
>
> Key: MAPREDUCE-6731
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6731
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0-alpha1
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
> Fix For: 3.0.0-alpha1
>
> Attachments: MAPREDUCE-6731.01.patch, MAPREDUCE-6731.02.patch
>
>
> {{TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling()}} uses
> the default file-system storage directory, and is brittle against concurrent
> tests.
> We should use a unique storage directory for the tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6731) TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling() may fail for concurrent tests
[
https://issues.apache.org/jira/browse/MAPREDUCE-6731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated MAPREDUCE-6731:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha1
Status: Resolved (was: Patch Available)
> TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling() may fail
> for concurrent tests
> -
>
> Key: MAPREDUCE-6731
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6731
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0-alpha1
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
> Fix For: 3.0.0-alpha1
>
> Attachments: MAPREDUCE-6731.01.patch, MAPREDUCE-6731.02.patch
>
>
> {{TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling()}} uses
> the default file-system storage directory, and is brittle against concurrent
> tests.
> We should use a unique storage directory for the tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
