[jira] [Commented] (MAPREDUCE-6616) Fail to create jobhistory file if there are some multibyte characters in the job name
[
https://issues.apache.org/jira/browse/MAPREDUCE-6616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15121685#comment-15121685
]
Hadoop QA commented on MAPREDUCE-6616:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
53s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 37s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s {color} | {color:green}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common:
patch generated 0 new + 3 unchanged - 12 fixed = 3 total (was 15) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 11s
{color} | {color:green} hadoop-mapreduce-client-common in the patch passed with
JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 5s
{color} | {color:green} hadoop-mapreduce-client-common in the patch passed with
JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 55s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12784934/MAPREDUCE-6616.1.patch
|
| JIRA Issue | MAPREDUCE-6616 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 8b0cda89aa40 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (MAPREDUCE-6579) Test failure : TestNetworkedJob
[
https://issues.apache.org/jira/browse/MAPREDUCE-6579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15121817#comment-15121817
]
Hadoop QA commented on MAPREDUCE-6579:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 37s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
10s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 3s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 8s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 8s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 19s
{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
in trunk has 2 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 50s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 57s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 57s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 9s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 101m 24s
{color} | {color:red} hadoop-mapreduce-client-jobclient in the patch failed
with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 16s {color}
| {color:red} hadoop-mapreduce-client-core in the patch failed with JDK
v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 105m 19s
{color} | {color:green} hadoop-mapreduce-client-jobclient in the patch passed
with JDK v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 27s
{color} | {color:red} Patch generated 15 ASF License warnings. {color} |
|
[jira] [Commented] (MAPREDUCE-6579) Test failure : TestNetworkedJob
[
https://issues.apache.org/jira/browse/MAPREDUCE-6579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123084#comment-15123084
]
Akira AJISAKA commented on MAPREDUCE-6579:
--
Thanks [~Naganarasimha] and [~leftnoteasy] for the reviews.
Hi Wangda, could you give binding +1?
> Test failure : TestNetworkedJob
> ---
>
> Key: MAPREDUCE-6579
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6579
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Rohith Sharma K S
>Assignee: Akira AJISAKA
> Attachments: MAPREDUCE-6579.01.patch, MAPREDUCE-6579.02.patch,
> MAPREDUCE-6579.03.patch, MAPREDUCE-6579.04.patch
>
>
> From
> [https://builds.apache.org/job/PreCommit-YARN-Build/9976/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt]
> TestNetworkedJob are failed intermittently.
> {code}
> Running org.apache.hadoop.mapred.TestNetworkedJob
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 81.131 sec
> <<< FAILURE! - in org.apache.hadoop.mapred.TestNetworkedJob
> testNetworkedJob(org.apache.hadoop.mapred.TestNetworkedJob) Time elapsed:
> 30.55 sec <<< FAILURE!
> org.junit.ComparisonFailure: expected:<[[Tue Dec 15 14:02:45 + 2015]
> Application is Activated, waiting for resources to be assigned for AM.
> Details : AM Partition = ; Partition Resource =
> ; Queue's Absolute capacity = 100.0 % ; Queue's
> Absolute used capacity = 0.0 % ; Queue's Absolute max capacity = 100.0 % ; ]>
> but was:<[]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.mapred.TestNetworkedJob.testNetworkedJob(TestNetworkedJob.java:174)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6616) Fail to create jobhistory file if there are some multibyte characters in the job name
[ https://issues.apache.org/jira/browse/MAPREDUCE-6616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123122#comment-15123122 ] Akira AJISAKA commented on MAPREDUCE-6616: -- +1, committing this. > Fail to create jobhistory file if there are some multibyte characters in the > job name > - > > Key: MAPREDUCE-6616 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6616 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Reporter: Akira AJISAKA >Assignee: Kousuke Saruta > Labels: i18n > Attachments: MAPREDUCE-6616-test.patch, MAPREDUCE-6616.0.patch, > MAPREDUCE-6616.1.patch > > > When creating jobhistory file, job name is trimmed within 50 characters by > default, and the name is URL-encoded *after* the job name is trimmed. > Therefore, if there are some multibyte characters in the job name, the > encoded job name can be longer than 50 characters. Eventually it can break > the limit of the file name (Usually 255 characters). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6616) Fail to create jobhistory file if there are some multibyte characters in the job name
[ https://issues.apache.org/jira/browse/MAPREDUCE-6616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated MAPREDUCE-6616: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) Committed this to trunk, branch-2, and branch-2.8. Thanks [~sarutak] for the contribution! > Fail to create jobhistory file if there are some multibyte characters in the > job name > - > > Key: MAPREDUCE-6616 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6616 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Reporter: Akira AJISAKA >Assignee: Kousuke Saruta > Labels: i18n > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6616-test.patch, MAPREDUCE-6616.0.patch, > MAPREDUCE-6616.1.patch > > > When creating jobhistory file, job name is trimmed within 50 characters by > default, and the name is URL-encoded *after* the job name is trimmed. > Therefore, if there are some multibyte characters in the job name, the > encoded job name can be longer than 50 characters. Eventually it can break > the limit of the file name (Usually 255 characters). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6622) Add capability to set JHS job cache to a task-based limit
[ https://issues.apache.org/jira/browse/MAPREDUCE-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated MAPREDUCE-6622: -- Status: Patch Available (was: Open) > Add capability to set JHS job cache to a task-based limit > - > > Key: MAPREDUCE-6622 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6622 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Affects Versions: 2.7.2 >Reporter: Ray Chiang >Assignee: Ray Chiang > Labels: supportability > Attachments: MAPREDUCE-6622.001.patch > > > When setting the property mapreduce.jobhistory.loadedjobs.cache.size the jobs > can be of varying size. This is generally not a problem when the jobs sizes > are uniform or small, but when the job sizes can be very large (say greater > than 250k tasks), then the JHS heap size can grow tremendously. > In cases, where multiple jobs are very large, then the JHS can lock up and > spend all its time in GC. However, since the cache is holding on to all the > jobs, not much heap space can be freed up. > By setting a property that sets a cap on the number of tasks allowed in the > cache and since the total number of tasks loaded is directly proportional to > the amount of heap used, this should help prevent the JHS from locking up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6622) Add capability to set JHS job cache to a task-based limit
[ https://issues.apache.org/jira/browse/MAPREDUCE-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated MAPREDUCE-6622: -- Attachment: MAPREDUCE-6622.001.patch > Add capability to set JHS job cache to a task-based limit > - > > Key: MAPREDUCE-6622 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6622 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Affects Versions: 2.7.2 >Reporter: Ray Chiang >Assignee: Ray Chiang > Labels: supportability > Attachments: MAPREDUCE-6622.001.patch > > > When setting the property mapreduce.jobhistory.loadedjobs.cache.size the jobs > can be of varying size. This is generally not a problem when the jobs sizes > are uniform or small, but when the job sizes can be very large (say greater > than 250k tasks), then the JHS heap size can grow tremendously. > In cases, where multiple jobs are very large, then the JHS can lock up and > spend all its time in GC. However, since the cache is holding on to all the > jobs, not much heap space can be freed up. > By setting a property that sets a cap on the number of tasks allowed in the > cache and since the total number of tasks loaded is directly proportional to > the amount of heap used, this should help prevent the JHS from locking up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6616) Fail to create jobhistory file if there are some multibyte characters in the job name
[ https://issues.apache.org/jira/browse/MAPREDUCE-6616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123146#comment-15123146 ] Hudson commented on MAPREDUCE-6616: --- FAILURE: Integrated in Hadoop-trunk-Commit #9207 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9207/]) MAPREDUCE-6616. Fail to create jobhistory file if there are some (aajisaka: rev df99ea8a92d600e669606d41d3887bd004e7a3cc) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory/TestFileNameIndexUtils.java * hadoop-mapreduce-project/CHANGES.txt > Fail to create jobhistory file if there are some multibyte characters in the > job name > - > > Key: MAPREDUCE-6616 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6616 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Reporter: Akira AJISAKA >Assignee: Kousuke Saruta > Labels: i18n > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6616-test.patch, MAPREDUCE-6616.0.patch, > MAPREDUCE-6616.1.patch > > > When creating jobhistory file, job name is trimmed within 50 characters by > default, and the name is URL-encoded *after* the job name is trimmed. > Therefore, if there are some multibyte characters in the job name, the > encoded job name can be longer than 50 characters. Eventually it can break > the limit of the file name (Usually 255 characters). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6620) Jobs that did not start are shown as starting in 1969 in the JHS web UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated MAPREDUCE-6620: Assignee: Haibo Chen (was: Daniel Templeton) Passing off to [~haibochen] to finish the patch and get it committed. > Jobs that did not start are shown as starting in 1969 in the JHS web UI > --- > > Key: MAPREDUCE-6620 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6620 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 2.7.2 >Reporter: Daniel Templeton >Assignee: Haibo Chen > Attachments: MAPREDUCE-6620.001.patch > > > If a job fails, its start time is stored as -1. The RM UI correctly handles > negative start times. The JHS UI does not, blindly converting it into a date > in 1969. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6620) Jobs that did not start are shown as starting in 1969 in the JHS web UI
[
https://issues.apache.org/jira/browse/MAPREDUCE-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122416#comment-15122416
]
Hadoop QA commented on MAPREDUCE-6620:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 21s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 29s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 12s
{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs:
patch generated 1 new + 70 unchanged - 0 fixed = 71 total (was 70) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 10s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 5m 33s {color}
| {color:red} hadoop-mapreduce-client-hs in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 5m 44s {color}
| {color:red} hadoop-mapreduce-client-hs in the patch failed with JDK
v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 17s
{color} | {color:red} Patch generated 14 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 20s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesJobsQuery |
| | hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesJobs |
| | hadoop.mapreduce.v2.hs.webapp.TestBlocks |
| JDK v1.7.0_91 Failed junit tests |
hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesJobsQuery |
| | hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesJobs |
| |
[jira] [Commented] (MAPREDUCE-6563) Streaming documentation contains a stray '%' character.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122494#comment-15122494 ] Hudson commented on MAPREDUCE-6563: --- FAILURE: Integrated in Hadoop-trunk-Commit #9202 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9202/]) MAPREDUCE-6563. Streaming documentation contains a stray % character. (cnauroth: rev 41da9a07738c6d019e2467f139750b7e2f50a914) * hadoop-mapreduce-project/CHANGES.txt * hadoop-tools/hadoop-streaming/src/site/markdown/HadoopStreaming.md.vm > Streaming documentation contains a stray '%' character. > --- > > Key: MAPREDUCE-6563 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6563 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: documentation >Reporter: Chris Nauroth >Assignee: Chris Nauroth >Priority: Trivial > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6563.001.patch > > > We have an unneeded '%' character above the title in this page. > http://hadoop.apache.org/docs/r2.7.1/hadoop-streaming/HadoopStreaming.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6563) Streaming documentation contains a stray '%' character.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated MAPREDUCE-6563: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) I have committed this to trunk, branch-2 and branch-2.8. [~templedf] and [~arpitagarwal], thank you for participating in the raw excitement of this code review. :-) Seriously though, thank you. That '%' was really starting to annoy me. > Streaming documentation contains a stray '%' character. > --- > > Key: MAPREDUCE-6563 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6563 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: documentation >Reporter: Chris Nauroth >Assignee: Chris Nauroth >Priority: Trivial > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6563.001.patch > > > We have an unneeded '%' character above the title in this page. > http://hadoop.apache.org/docs/r2.7.1/hadoop-streaming/HadoopStreaming.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6620) Jobs that did not start are shown as starting in 1969 in the JHS web UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122384#comment-15122384 ] Robert Kanter commented on MAPREDUCE-6620: -- That looks right to me, though could you add a unit test? > Jobs that did not start are shown as starting in 1969 in the JHS web UI > --- > > Key: MAPREDUCE-6620 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6620 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 2.7.2 >Reporter: Daniel Templeton >Assignee: Daniel Templeton > Attachments: MAPREDUCE-6620.001.patch > > > If a job fails, its start time is stored as -1. The RM UI correctly handles > negative start times. The JHS UI does not, blindly converting it into a date > in 1969. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6618) YarnClientProtocolProvider leaking the YarnClient thread.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122324#comment-15122324
]
Xuan Gong commented on MAPREDUCE-6618:
--
[~jlowe]
Thanks for the clarification. The new patch addressed that.
> YarnClientProtocolProvider leaking the YarnClient thread.
> --
>
> Key: MAPREDUCE-6618
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6618
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: MAPREDUCE-6618.1.patch, MAPREDUCE-6618.2.patch,
> MAPREDUCE-6618.3.patch, MAPREDUCE-6618.4.patch, MAPREDUCE-6618.5.patch
>
>
> YarnClientProtocolProvider creates YarnRunner which includes
> ResourceMgrDelegate. In ResourceMgrDelegate, we would initiate and start
> yarnclient. The yarnClient thread would be leaked due to
> {code}
> @Override
> public void close(ClientProtocol clientProtocol) throws IOException {
> // nothing to do
> }
> {code} in YarnClientProtocolProvider
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6620) Jobs that did not start are shown as starting in 1969 in the JHS web UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated MAPREDUCE-6620: Status: Patch Available (was: Open) > Jobs that did not start are shown as starting in 1969 in the JHS web UI > --- > > Key: MAPREDUCE-6620 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6620 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 2.7.2 >Reporter: Daniel Templeton >Assignee: Daniel Templeton > Attachments: MAPREDUCE-6620.001.patch > > > If a job fails, its start time is stored as -1. The RM UI correctly handles > negative start times. The JHS UI does not, blindly converting it into a date > in 1969. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6618) YarnClientProtocolProvider leaking the YarnClient thread.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuan Gong updated MAPREDUCE-6618:
-
Attachment: MAPREDUCE-6618.5.patch
> YarnClientProtocolProvider leaking the YarnClient thread.
> --
>
> Key: MAPREDUCE-6618
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6618
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: MAPREDUCE-6618.1.patch, MAPREDUCE-6618.2.patch,
> MAPREDUCE-6618.3.patch, MAPREDUCE-6618.4.patch, MAPREDUCE-6618.5.patch
>
>
> YarnClientProtocolProvider creates YarnRunner which includes
> ResourceMgrDelegate. In ResourceMgrDelegate, we would initiate and start
> yarnclient. The yarnClient thread would be leaked due to
> {code}
> @Override
> public void close(ClientProtocol clientProtocol) throws IOException {
> // nothing to do
> }
> {code} in YarnClientProtocolProvider
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6620) Jobs that did not start are shown as starting in 1969 in the JHS web UI
Daniel Templeton created MAPREDUCE-6620: --- Summary: Jobs that did not start are shown as starting in 1969 in the JHS web UI Key: MAPREDUCE-6620 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6620 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 2.7.2 Reporter: Daniel Templeton Assignee: Daniel Templeton If a job fails, its start time is stored as -1. The RM UI correctly handles negative start times. The JHS UI does not, blindly converting it into a date in 1969. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6563) Streaming documentation contains a stray '%' character.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122341#comment-15122341 ] Arpit Agarwal commented on MAPREDUCE-6563: -- +1 > Streaming documentation contains a stray '%' character. > --- > > Key: MAPREDUCE-6563 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6563 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: documentation >Reporter: Chris Nauroth >Assignee: Chris Nauroth >Priority: Trivial > Attachments: MAPREDUCE-6563.001.patch > > > We have an unneeded '%' character above the title in this page. > http://hadoop.apache.org/docs/r2.7.1/hadoop-streaming/HadoopStreaming.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6620) Jobs that did not start are shown as starting in 1969 in the JHS web UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated MAPREDUCE-6620: Attachment: MAPREDUCE-6620.001.patch [~rkanter], does this look like the right solution to you? > Jobs that did not start are shown as starting in 1969 in the JHS web UI > --- > > Key: MAPREDUCE-6620 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6620 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 2.7.2 >Reporter: Daniel Templeton >Assignee: Daniel Templeton > Attachments: MAPREDUCE-6620.001.patch > > > If a job fails, its start time is stored as -1. The RM UI correctly handles > negative start times. The JHS UI does not, blindly converting it into a date > in 1969. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6617) flushTimer in JobHistoryEventHandler should purge canceled flushTimerTask
[
https://issues.apache.org/jira/browse/MAPREDUCE-6617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122695#comment-15122695
]
Jason Lowe commented on MAPREDUCE-6617:
---
Could you elaborate more on the memory leak? The javadoc for the purge method
states:
{quote}
* Most programs will have no need to call this method.
* It is designed for use by the rare application that cancels a large
* number of tasks. Calling this method trades time for space: the
* runtime of the method may be proportional to n + c log n, where n
* is the number of tasks in the queue and c is the number of cancelled
* tasks.
{quote}
I'm guessing the canceled timer tasks are reaped when they normally would
execute and the timer notices they've been marked cancelled. Is this a case
where we're queueing up tasks and canceling them faster than they normally
execute?
> flushTimer in JobHistoryEventHandler should purge canceled flushTimerTask
> -
>
> Key: MAPREDUCE-6617
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6617
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 3.0.0
>Reporter: Kousuke Saruta
>Assignee: Kousuke Saruta
> Attachments: MAPREDUCE-6617.0.patch
>
>
> In JobHistoryEventHandler, flushTask is not purged after it's canceled so GC
> never sweep flushTask.
> It can cause memory leak.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6579) Test failure : TestNetworkedJob
[
https://issues.apache.org/jira/browse/MAPREDUCE-6579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15121298#comment-15121298
]
Akira AJISAKA commented on MAPREDUCE-6579:
--
bq. Do you think we should rename JobStatus#getFailureInfo? Diagnostic message
doesn't always means failure, JobStatus's InterfaceStability marked to
"Evolving", we can update it.
I don't think we should rename {{JobStatus#getFailureInfo}} because Apache Tez
is overriding this method.
https://github.com/apache/tez/blob/master/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/client/DAGJobStatus.java
{code}
@Override
public synchronized String getFailureInfo() {
return Joiner.on(". ").join(dagStatus.getDiagnostics());
}
{code}
Insteadly, we can deprecate the method in trunk/branch-2 and remove it in
trunk. I'm thinking it should be done in a separate jira.
bq. I would suggest not compare the diagnostic info, it is evolving, only human
is supposed to read this message and we have test in YARN already.
Agree. I'll remove the assertion.
> Test failure : TestNetworkedJob
> ---
>
> Key: MAPREDUCE-6579
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6579
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Rohith Sharma K S
>Assignee: Akira AJISAKA
> Attachments: MAPREDUCE-6579.01.patch, MAPREDUCE-6579.02.patch,
> MAPREDUCE-6579.03.patch
>
>
> From
> [https://builds.apache.org/job/PreCommit-YARN-Build/9976/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt]
> TestNetworkedJob are failed intermittently.
> {code}
> Running org.apache.hadoop.mapred.TestNetworkedJob
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 81.131 sec
> <<< FAILURE! - in org.apache.hadoop.mapred.TestNetworkedJob
> testNetworkedJob(org.apache.hadoop.mapred.TestNetworkedJob) Time elapsed:
> 30.55 sec <<< FAILURE!
> org.junit.ComparisonFailure: expected:<[[Tue Dec 15 14:02:45 + 2015]
> Application is Activated, waiting for resources to be assigned for AM.
> Details : AM Partition = ; Partition Resource =
> ; Queue's Absolute capacity = 100.0 % ; Queue's
> Absolute used capacity = 0.0 % ; Queue's Absolute max capacity = 100.0 % ; ]>
> but was:<[]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.mapred.TestNetworkedJob.testNetworkedJob(TestNetworkedJob.java:174)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6579) Test failure : TestNetworkedJob
[
https://issues.apache.org/jira/browse/MAPREDUCE-6579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated MAPREDUCE-6579:
-
Attachment: MAPREDUCE-6579.04.patch
> Test failure : TestNetworkedJob
> ---
>
> Key: MAPREDUCE-6579
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6579
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Rohith Sharma K S
>Assignee: Akira AJISAKA
> Attachments: MAPREDUCE-6579.01.patch, MAPREDUCE-6579.02.patch,
> MAPREDUCE-6579.03.patch, MAPREDUCE-6579.04.patch
>
>
> From
> [https://builds.apache.org/job/PreCommit-YARN-Build/9976/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt]
> TestNetworkedJob are failed intermittently.
> {code}
> Running org.apache.hadoop.mapred.TestNetworkedJob
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 81.131 sec
> <<< FAILURE! - in org.apache.hadoop.mapred.TestNetworkedJob
> testNetworkedJob(org.apache.hadoop.mapred.TestNetworkedJob) Time elapsed:
> 30.55 sec <<< FAILURE!
> org.junit.ComparisonFailure: expected:<[[Tue Dec 15 14:02:45 + 2015]
> Application is Activated, waiting for resources to be assigned for AM.
> Details : AM Partition = ; Partition Resource =
> ; Queue's Absolute capacity = 100.0 % ; Queue's
> Absolute used capacity = 0.0 % ; Queue's Absolute max capacity = 100.0 % ; ]>
> but was:<[]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.mapred.TestNetworkedJob.testNetworkedJob(TestNetworkedJob.java:174)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6579) Test failure : TestNetworkedJob
[
https://issues.apache.org/jira/browse/MAPREDUCE-6579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15121373#comment-15121373
]
Naganarasimha G R commented on MAPREDUCE-6579:
--
Thanks [~wangda] for the comments,
bq, Do you think we should rename JobStatus#getFailureInfo?
Yes & No, *yes* because we are setting diaginfo now even if there is no failure
so it makes sense to update the name but *no* because this method was
introduced though 4.5 yrs ago as part of MAPREDUCE-2952 still the interface is
kept as evolving, so as [~ajisakaa] mentioned many other applications like tez
might be using it. Even i would support this modification to be done in trunk.
bq. I would suggest not compare the diagnostic info, it is evolving, only human
is supposed to read this message and we have test in YARN already.
I am ok even if its not there, here were just comparing there is no error
state information so it might be prone to have test case failure issues in
future if new states are added .
Thanks [~ajisakaa] for the new Patch, it LGTM !
> Test failure : TestNetworkedJob
> ---
>
> Key: MAPREDUCE-6579
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6579
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Rohith Sharma K S
>Assignee: Akira AJISAKA
> Attachments: MAPREDUCE-6579.01.patch, MAPREDUCE-6579.02.patch,
> MAPREDUCE-6579.03.patch, MAPREDUCE-6579.04.patch
>
>
> From
> [https://builds.apache.org/job/PreCommit-YARN-Build/9976/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt]
> TestNetworkedJob are failed intermittently.
> {code}
> Running org.apache.hadoop.mapred.TestNetworkedJob
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 81.131 sec
> <<< FAILURE! - in org.apache.hadoop.mapred.TestNetworkedJob
> testNetworkedJob(org.apache.hadoop.mapred.TestNetworkedJob) Time elapsed:
> 30.55 sec <<< FAILURE!
> org.junit.ComparisonFailure: expected:<[[Tue Dec 15 14:02:45 + 2015]
> Application is Activated, waiting for resources to be assigned for AM.
> Details : AM Partition = ; Partition Resource =
> ; Queue's Absolute capacity = 100.0 % ; Queue's
> Absolute used capacity = 0.0 % ; Queue's Absolute max capacity = 100.0 % ; ]>
> but was:<[]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.mapred.TestNetworkedJob.testNetworkedJob(TestNetworkedJob.java:174)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6595) Fix findbugs warnings in OutputCommitter and FileOutputCommitter
[ https://issues.apache.org/jira/browse/MAPREDUCE-6595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15121199#comment-15121199 ] Junping Du commented on MAPREDUCE-6595: --- Thanks [~ajisakaa] for reporting the issue and fix it. +1. Will commit it shortly. > Fix findbugs warnings in OutputCommitter and FileOutputCommitter > > > Key: MAPREDUCE-6595 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6595 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA > Attachments: MAPREDUCE-6595.01.patch, MAPREDUCE-6595.testing.patch, > findbugsHtml.html > > > There are 2 findbugs warnings in hadoop-mapreduce-client-core module. > https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6237/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6621) Memory Link in JobClient#submitJobInternal()
Xuan Gong created MAPREDUCE-6621: Summary: Memory Link in JobClient#submitJobInternal() Key: MAPREDUCE-6621 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6621 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Xuan Gong -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6621) Memory Link in JobClient#submitJobInternal()
[
https://issues.apache.org/jira/browse/MAPREDUCE-6621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuan Gong updated MAPREDUCE-6621:
-
Status: Patch Available (was: Open)
> Memory Link in JobClient#submitJobInternal()
>
>
> Key: MAPREDUCE-6621
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6621
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Xuan Gong
> Attachments: MAPREDUCE-6621.1.patch
>
>
> In JobClient:
> {code}
> public RunningJob submitJobInternal(final JobConf conf)
> throws FileNotFoundException, IOException {
> try {
> conf.setBooleanIfUnset("mapred.mapper.new-api", false);
> conf.setBooleanIfUnset("mapred.reducer.new-api", false);
> Job job = clientUgi.doAs(new PrivilegedExceptionAction () {
> @Override
> public Job run() throws IOException, ClassNotFoundException,
> InterruptedException {
> Job job = Job.getInstance(conf);
> job.submit();
> return job;
> }
> });
> // update our Cluster instance with the one created by Job for
> submission
> // (we can't pass our Cluster instance to Job, since Job wraps the
> config
> // instance, and the two configs would then diverge)
> cluster = job.getCluster();
> return new NetworkedJob(job);
> } catch (InterruptedException ie) {
> throw new IOException("interrupted", ie);
> }
> }
> {code}
> We will replace the cluster object with the cluster object from Job, but the
> previous old cluster object would never be closed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6622) Add capability to set JHS job cache to a task-based limit
Ray Chiang created MAPREDUCE-6622: - Summary: Add capability to set JHS job cache to a task-based limit Key: MAPREDUCE-6622 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6622 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobhistoryserver Affects Versions: 2.7.2 Reporter: Ray Chiang Assignee: Ray Chiang When setting the property mapreduce.jobhistory.loadedjobs.cache.size the jobs can be of varying size. This is generally not a problem when the jobs sizes are uniform or small, but when the job sizes can be very large (say greater than 250k tasks), then the JHS heap size can grow tremendously. In cases, where multiple jobs are very large, then the JHS can lock up and spend all its time in GC. However, since the cache is holding on to all the jobs, not much heap space can be freed up. By setting a property that sets a cap on the number of tasks allowed in the cache and since the total number of tasks loaded is directly proportional to the amount of heap used, this should help prevent the JHS from locking up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6616) Fail to create jobhistory file if there are some multibyte characters in the job name
[ https://issues.apache.org/jira/browse/MAPREDUCE-6616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated MAPREDUCE-6616: -- Attachment: MAPREDUCE-6616.1.patch Fixed style issues and brushed up. > Fail to create jobhistory file if there are some multibyte characters in the > job name > - > > Key: MAPREDUCE-6616 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6616 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Reporter: Akira AJISAKA >Assignee: Kousuke Saruta > Labels: i18n > Attachments: MAPREDUCE-6616-test.patch, MAPREDUCE-6616.0.patch, > MAPREDUCE-6616.1.patch > > > When creating jobhistory file, job name is trimmed within 50 characters by > default, and the name is URL-encoded *after* the job name is trimmed. > Therefore, if there are some multibyte characters in the job name, the > encoded job name can be longer than 50 characters. Eventually it can break > the limit of the file name (Usually 255 characters). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6595) Fix findbugs warnings in OutputCommitter and FileOutputCommitter
[ https://issues.apache.org/jira/browse/MAPREDUCE-6595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-6595: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) I have commit the patch to trunk, branch-2 and branch-2.8. Thanks [~ajisakaa] for the contribution. > Fix findbugs warnings in OutputCommitter and FileOutputCommitter > > > Key: MAPREDUCE-6595 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6595 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6595.01.patch, MAPREDUCE-6595.testing.patch, > findbugsHtml.html > > > There are 2 findbugs warnings in hadoop-mapreduce-client-core module. > https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6237/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6621) Memory Leak in JobClient#submitJobInternal()
[
https://issues.apache.org/jira/browse/MAPREDUCE-6621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuan Gong updated MAPREDUCE-6621:
-
Summary: Memory Leak in JobClient#submitJobInternal() (was: Memory Link in
JobClient#submitJobInternal())
> Memory Leak in JobClient#submitJobInternal()
>
>
> Key: MAPREDUCE-6621
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6621
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: MAPREDUCE-6621.1.patch
>
>
> In JobClient:
> {code}
> public RunningJob submitJobInternal(final JobConf conf)
> throws FileNotFoundException, IOException {
> try {
> conf.setBooleanIfUnset("mapred.mapper.new-api", false);
> conf.setBooleanIfUnset("mapred.reducer.new-api", false);
> Job job = clientUgi.doAs(new PrivilegedExceptionAction () {
> @Override
> public Job run() throws IOException, ClassNotFoundException,
> InterruptedException {
> Job job = Job.getInstance(conf);
> job.submit();
> return job;
> }
> });
> // update our Cluster instance with the one created by Job for
> submission
> // (we can't pass our Cluster instance to Job, since Job wraps the
> config
> // instance, and the two configs would then diverge)
> cluster = job.getCluster();
> return new NetworkedJob(job);
> } catch (InterruptedException ie) {
> throw new IOException("interrupted", ie);
> }
> }
> {code}
> We will replace the cluster object with the cluster object from Job, but the
> previous old cluster object would never be closed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6618) YarnClientProtocolProvider leaking the YarnClient thread.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122897#comment-15122897
]
Hadoop QA commented on MAPREDUCE-6618:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 49s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 19s
{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
patch generated 1 new + 78 unchanged - 0 fixed = 79 total (was 78) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 172m 59s
{color} | {color:red} hadoop-mapreduce-client-jobclient in the patch failed
with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 151m 59s
{color} | {color:red} hadoop-mapreduce-client-jobclient in the patch failed
with JDK v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 37s
{color} | {color:red} Patch generated 15 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 350m 46s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | hadoop.mapred.TestNetworkedJob |
| | hadoop.mapreduce.v2.TestMRJobsWithProfiler |
| | hadoop.mapreduce.v2.TestRMNMInfo |
| | hadoop.mapred.TestMRTimelineEventHandling |
| | hadoop.mapreduce.TestYarnClientProtocolProvider |
| JDK v1.8.0_66 Timed out junit tests |
[jira] [Updated] (MAPREDUCE-6621) Memory Link in JobClient#submitJobInternal()
[
https://issues.apache.org/jira/browse/MAPREDUCE-6621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuan Gong updated MAPREDUCE-6621:
-
Attachment: MAPREDUCE-6621.1.patch
> Memory Link in JobClient#submitJobInternal()
>
>
> Key: MAPREDUCE-6621
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6621
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Xuan Gong
> Attachments: MAPREDUCE-6621.1.patch
>
>
> In JobClient:
> {code}
> public RunningJob submitJobInternal(final JobConf conf)
> throws FileNotFoundException, IOException {
> try {
> conf.setBooleanIfUnset("mapred.mapper.new-api", false);
> conf.setBooleanIfUnset("mapred.reducer.new-api", false);
> Job job = clientUgi.doAs(new PrivilegedExceptionAction () {
> @Override
> public Job run() throws IOException, ClassNotFoundException,
> InterruptedException {
> Job job = Job.getInstance(conf);
> job.submit();
> return job;
> }
> });
> // update our Cluster instance with the one created by Job for
> submission
> // (we can't pass our Cluster instance to Job, since Job wraps the
> config
> // instance, and the two configs would then diverge)
> cluster = job.getCluster();
> return new NetworkedJob(job);
> } catch (InterruptedException ie) {
> throw new IOException("interrupted", ie);
> }
> }
> {code}
> We will replace the cluster object with the cluster object from Job, but the
> previous old cluster object would never be closed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6621) Memory Link in JobClient#submitJobInternal()
[
https://issues.apache.org/jira/browse/MAPREDUCE-6621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuan Gong updated MAPREDUCE-6621:
-
Description:
In JobClient:
{code}
public RunningJob submitJobInternal(final JobConf conf)
throws FileNotFoundException, IOException {
try {
conf.setBooleanIfUnset("mapred.mapper.new-api", false);
conf.setBooleanIfUnset("mapred.reducer.new-api", false);
Job job = clientUgi.doAs(new PrivilegedExceptionAction () {
@Override
public Job run() throws IOException, ClassNotFoundException,
InterruptedException {
Job job = Job.getInstance(conf);
job.submit();
return job;
}
});
// update our Cluster instance with the one created by Job for submission
// (we can't pass our Cluster instance to Job, since Job wraps the config
// instance, and the two configs would then diverge)
cluster = job.getCluster();
return new NetworkedJob(job);
} catch (InterruptedException ie) {
throw new IOException("interrupted", ie);
}
}
{code}
We will replace the cluster object with the cluster object from Job, but the
previous old cluster object would never be closed.
> Memory Link in JobClient#submitJobInternal()
>
>
> Key: MAPREDUCE-6621
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6621
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Xuan Gong
>
> In JobClient:
> {code}
> public RunningJob submitJobInternal(final JobConf conf)
> throws FileNotFoundException, IOException {
> try {
> conf.setBooleanIfUnset("mapred.mapper.new-api", false);
> conf.setBooleanIfUnset("mapred.reducer.new-api", false);
> Job job = clientUgi.doAs(new PrivilegedExceptionAction () {
> @Override
> public Job run() throws IOException, ClassNotFoundException,
> InterruptedException {
> Job job = Job.getInstance(conf);
> job.submit();
> return job;
> }
> });
> // update our Cluster instance with the one created by Job for
> submission
> // (we can't pass our Cluster instance to Job, since Job wraps the
> config
> // instance, and the two configs would then diverge)
> cluster = job.getCluster();
> return new NetworkedJob(job);
> } catch (InterruptedException ie) {
> throw new IOException("interrupted", ie);
> }
> }
> {code}
> We will replace the cluster object with the cluster object from Job, but the
> previous old cluster object would never be closed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (MAPREDUCE-6621) Memory Link in JobClient#submitJobInternal()
[
https://issues.apache.org/jira/browse/MAPREDUCE-6621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuan Gong reassigned MAPREDUCE-6621:
Assignee: Xuan Gong
> Memory Link in JobClient#submitJobInternal()
>
>
> Key: MAPREDUCE-6621
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6621
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: MAPREDUCE-6621.1.patch
>
>
> In JobClient:
> {code}
> public RunningJob submitJobInternal(final JobConf conf)
> throws FileNotFoundException, IOException {
> try {
> conf.setBooleanIfUnset("mapred.mapper.new-api", false);
> conf.setBooleanIfUnset("mapred.reducer.new-api", false);
> Job job = clientUgi.doAs(new PrivilegedExceptionAction () {
> @Override
> public Job run() throws IOException, ClassNotFoundException,
> InterruptedException {
> Job job = Job.getInstance(conf);
> job.submit();
> return job;
> }
> });
> // update our Cluster instance with the one created by Job for
> submission
> // (we can't pass our Cluster instance to Job, since Job wraps the
> config
> // instance, and the two configs would then diverge)
> cluster = job.getCluster();
> return new NetworkedJob(job);
> } catch (InterruptedException ie) {
> throw new IOException("interrupted", ie);
> }
> }
> {code}
> We will replace the cluster object with the cluster object from Job, but the
> previous old cluster object would never be closed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6621) Memory Link in JobClient#submitJobInternal()
[
https://issues.apache.org/jira/browse/MAPREDUCE-6621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122779#comment-15122779
]
Hadoop QA commented on MAPREDUCE-6621:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
44s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 32s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 9s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 5s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 10s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 48s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12785063/MAPREDUCE-6621.1.patch
|
| JIRA Issue | MAPREDUCE-6621 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux d53d7c2a00e6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (MAPREDUCE-6579) Test failure : TestNetworkedJob
[
https://issues.apache.org/jira/browse/MAPREDUCE-6579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122804#comment-15122804
]
Wangda Tan commented on MAPREDUCE-6579:
---
Agree, latest patch LGTM, thanks!
> Test failure : TestNetworkedJob
> ---
>
> Key: MAPREDUCE-6579
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6579
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Rohith Sharma K S
>Assignee: Akira AJISAKA
> Attachments: MAPREDUCE-6579.01.patch, MAPREDUCE-6579.02.patch,
> MAPREDUCE-6579.03.patch, MAPREDUCE-6579.04.patch
>
>
> From
> [https://builds.apache.org/job/PreCommit-YARN-Build/9976/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt]
> TestNetworkedJob are failed intermittently.
> {code}
> Running org.apache.hadoop.mapred.TestNetworkedJob
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 81.131 sec
> <<< FAILURE! - in org.apache.hadoop.mapred.TestNetworkedJob
> testNetworkedJob(org.apache.hadoop.mapred.TestNetworkedJob) Time elapsed:
> 30.55 sec <<< FAILURE!
> org.junit.ComparisonFailure: expected:<[[Tue Dec 15 14:02:45 + 2015]
> Application is Activated, waiting for resources to be assigned for AM.
> Details : AM Partition = ; Partition Resource =
> ; Queue's Absolute capacity = 100.0 % ; Queue's
> Absolute used capacity = 0.0 % ; Queue's Absolute max capacity = 100.0 % ; ]>
> but was:<[]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.mapred.TestNetworkedJob.testNetworkedJob(TestNetworkedJob.java:174)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6542) HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe
[
https://issues.apache.org/jira/browse/MAPREDUCE-6542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122877#comment-15122877
]
zhangyubiao commented on MAPREDUCE-6542:
[~djp],Thanks for response. Hopefully you can review it soon.
> HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe
> -
>
> Key: MAPREDUCE-6542
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6542
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 2.2.0, 2.7.1
> Environment: CentOS6.5 Hadoop
>Reporter: zhangyubiao
>Assignee: zhangyubiao
> Attachments: MAPREDUCE-6542-v10.patch, MAPREDUCE-6542-v2.patch,
> MAPREDUCE-6542-v3.patch, MAPREDUCE-6542-v4.patch, MAPREDUCE-6542-v5.patch,
> MAPREDUCE-6542-v6.patch, MAPREDUCE-6542-v7.patch, MAPREDUCE-6542-v8.patch,
> MAPREDUCE-6542-v9.patch, MAPREDUCE-6542.patch
>
>
> I use SimpleDateFormat to Parse the JobHistory File before
> {code}
> private static final SimpleDateFormat dateFormat =
> new SimpleDateFormat("-MM-dd HH:mm:ss");
> public static String getJobDetail(JobInfo job) {
> StringBuffer jobDetails = new StringBuffer("");
> SummarizedJob ts = new SummarizedJob(job);
> jobDetails.append(job.getJobId().toString().trim()).append("\t");
> jobDetails.append(job.getUsername()).append("\t");
> jobDetails.append(job.getJobname().replaceAll("\\n",
> "")).append("\t");
> jobDetails.append(job.getJobQueueName()).append("\t");
> jobDetails.append(job.getPriority()).append("\t");
> jobDetails.append(job.getJobConfPath()).append("\t");
> jobDetails.append(job.getUberized()).append("\t");
>
> jobDetails.append(dateFormat.format(job.getSubmitTime())).append("\t");
>
> jobDetails.append(dateFormat.format(job.getLaunchTime())).append("\t");
>
> jobDetails.append(dateFormat.format(job.getFinishTime())).append("\t");
>return jobDetails.toString();
> }
> {code}
> But I find I query the SubmitTime and LaunchTime in hive and compare
> JobHistory File time , I find that the submitTime and launchTime was wrong.
> Finally,I change to use the FastDateFormat to parse the time format and the
> time become right
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6616) Fail to create jobhistory file if there are some multibyte characters in the job name
[
https://issues.apache.org/jira/browse/MAPREDUCE-6616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15121505#comment-15121505
]
Akira AJISAKA commented on MAPREDUCE-6616:
--
Thanks [~sarutak] for creating the patch. Mostly looks good to me. Minor nits:
1.
{code}
// amount size of first 5 characters < 23
// 23 < amount size of first 6 characters
{code}
I'm thinking we should comment why the size is equal to 22, so the above
comment can be rewritten as follows:
{code}
// total size of the first 5 characters = 22
// 23 < total size of the first 6 characters
{code}
2.
{code}
import static java.nio.charset.StandardCharsets.UTF_8;
{code}
Would you remove unnecessary whitespace between "import" and "static"?
3.
{code}
//NumReduces
-sb.append(indexInfo.getNumReduces());
+sb.append(encodeJobHistoryFileName(
+String.valueOf(indexInfo.getNumReduces(;
sb.append(DELIMITER);
-
+
//JobStatus
-sb.append(indexInfo.getJobStatus());
+sb.append(encodeJobHistoryFileName(indexInfo.getJobStatus()));
sb.append(DELIMITER);
-
+
//QueueName
-sb.append(escapeDelimiters(getQueueName(indexInfo)));
+sb.append(escapeDelimiters(encodeJobHistoryFileName(
+getQueueName(indexInfo;
{code}
The patch includes mixed continuation indents (4 spaces and 8 spaces). Would
you make the indents consistent?
> Fail to create jobhistory file if there are some multibyte characters in the
> job name
> -
>
> Key: MAPREDUCE-6616
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6616
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Reporter: Akira AJISAKA
>Assignee: Kousuke Saruta
> Labels: i18n
> Attachments: MAPREDUCE-6616-test.patch, MAPREDUCE-6616.0.patch
>
>
> When creating jobhistory file, job name is trimmed within 50 characters by
> default, and the name is URL-encoded *after* the job name is trimmed.
> Therefore, if there are some multibyte characters in the job name, the
> encoded job name can be longer than 50 characters. Eventually it can break
> the limit of the file name (Usually 255 characters).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6595) Fix findbugs warnings in OutputCommitter and FileOutputCommitter
[ https://issues.apache.org/jira/browse/MAPREDUCE-6595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15121541#comment-15121541 ] Hudson commented on MAPREDUCE-6595: --- SUCCESS: Integrated in Hadoop-trunk-Commit #9199 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9199/]) MAPREDUCE-6595. Fix findbugs warnings in OutputCommitter and (junping_du: rev 86560a4978c498bd5e892c509a466807a812b8a7) * hadoop-mapreduce-project/dev-support/findbugs-exclude.xml * hadoop-mapreduce-project/CHANGES.txt > Fix findbugs warnings in OutputCommitter and FileOutputCommitter > > > Key: MAPREDUCE-6595 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6595 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6595.01.patch, MAPREDUCE-6595.testing.patch, > findbugsHtml.html > > > There are 2 findbugs warnings in hadoop-mapreduce-client-core module. > https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6237/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6096) SummarizedJob class NPEs with some jhist files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122977#comment-15122977 ] zhangyubiao commented on MAPREDUCE-6096: [~aw] > SummarizedJob class NPEs with some jhist files > -- > > Key: MAPREDUCE-6096 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6096 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Reporter: zhangyubiao >Assignee: zhangyubiao > Labels: easyfix, patch > Attachments: MAPREDUCE-6096-v1.patch, MAPREDUCE-6096.patch, > job_1446203652278_66705-1446308686422-dd_edw-insert+overwrite+table+bkactiv...dp%3D%27ACTIVE%27%28Stage-1446308802181-233-0-SUCCEEDED-bdp_jdw_corejob.jhist > > > When I Parse the JobHistory in the HistoryFile,I use the Hadoop System's > map-reduce-client-core project > org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser class and > HistoryViewer$SummarizedJob to Parse the JobHistoryFile(Just Like > job_1408862281971_489761-1410883171851_XXX.jhist) > and it throw an Exception Just Like > Exception in thread "pool-1-thread-1" java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.jobhistory.HistoryViewer$SummarizedJob.(HistoryViewer.java:626) > at > com.jd.hadoop.log.parse.ParseLogService.getJobDetail(ParseLogService.java:70) > After I'm see the SummarizedJob class I find that attempt.getTaskStatus() is > NULL ， So I change the order of > attempt.getTaskStatus().equals (TaskStatus.State.FAILED.toString()) to > TaskStatus.State.FAILED.toString().equals(attempt.getTaskStatus()) > and it works well . > So I wonder If we can change all attempt.getTaskStatus() after > TaskStatus.State.XXX.toString() ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
