[jira] [Updated] (MAPREDUCE-6555) TestMRAppMaster fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsuyoshi Ozawa updated MAPREDUCE-6555:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Committed this to trunk. Thanks [~djp] for the contribution and [~ajisakaa] and
[~varun_saxena] for the review and comments.
> TestMRAppMaster fails on trunk
> --
>
> Key: MAPREDUCE-6555
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6555
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: Junping Du
> Fix For: 3.0.0
>
> Attachments: MAPREDUCE-6555.patch
>
>
> Observed in QA report of YARN-3840
> {noformat}
> Running org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> Tests run: 9, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 20.699 sec
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> testMRAppMasterMidLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.474 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterMidLock(TestMRAppMaster.java:174)
> testMRAppMasterSuccessLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.175 sec <<< ERROR!
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.io.FileNotFoundException: File
> file:/home/varun/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/staging/history/done_intermediate/TestAppMasterUser/job_1317529182569_0004-1448100479292-TestAppMasterUser-%3Cmissing+job+name%3E-1448100479413-0-0-SUCCEEDED-default-1448100479292.jhist_tmp
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:640)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:866)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:630)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:372)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:513)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.moveTmpToDone(JobHistoryEventHandler.java:1346)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.processDoneFiles(JobHistoryEventHandler.java:1154)
> at
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
> at
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
> at
> org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
> at
> org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1751)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1247)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterSuccessLock(TestMRAppMaster.java:254)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6557) Some tests in mapreduce-client-app are writing outside of target
[ https://issues.apache.org/jira/browse/MAPREDUCE-6557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-6557: -- Attachment: MAPREDUCE-6557.004.patch Hi [~ajisakaa], I mark cleanup method as static in 004 patch, so tests get run successfully locally. Can you check if it sounds good? If so, I will commit it shortly. > Some tests in mapreduce-client-app are writing outside of target > > > Key: MAPREDUCE-6557 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6557 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Akira AJISAKA >Priority: Blocker > Attachments: MAPREDUCE-6555.03.patch, MAPREDUCE-6557.00.patch, > MAPREDUCE-6557.004.patch > > > There is a staging directory appearing. It should not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6557) Some tests in mapreduce-client-app are writing outside of target
[
https://issues.apache.org/jira/browse/MAPREDUCE-6557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15027147#comment-15027147
]
Hadoop QA commented on MAPREDUCE-6557:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
25s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 6s
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed with
JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 40s
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed with
JDK v1.7.0_85. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 46s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12774375/MAPREDUCE-6557.04.patch
|
| JIRA Issue | MAPREDUCE-6557 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 22d49ce0e7ae 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / bab03f3 |
| findbugs | v3.0.0 |
| JDK v1.7.0_85 Test
[jira] [Commented] (MAPREDUCE-6555) TestMRAppMaster fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026987#comment-15026987
]
Tsuyoshi Ozawa commented on MAPREDUCE-6555:
---
Jenkins CI looks to be checking another patch. +1 for MAPREDUCE-6555.patch.
Checking this in.
> TestMRAppMaster fails on trunk
> --
>
> Key: MAPREDUCE-6555
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6555
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: Junping Du
> Attachments: MAPREDUCE-6555.patch
>
>
> Observed in QA report of YARN-3840
> {noformat}
> Running org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> Tests run: 9, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 20.699 sec
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> testMRAppMasterMidLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.474 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterMidLock(TestMRAppMaster.java:174)
> testMRAppMasterSuccessLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.175 sec <<< ERROR!
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.io.FileNotFoundException: File
> file:/home/varun/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/staging/history/done_intermediate/TestAppMasterUser/job_1317529182569_0004-1448100479292-TestAppMasterUser-%3Cmissing+job+name%3E-1448100479413-0-0-SUCCEEDED-default-1448100479292.jhist_tmp
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:640)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:866)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:630)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:372)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:513)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.moveTmpToDone(JobHistoryEventHandler.java:1346)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.processDoneFiles(JobHistoryEventHandler.java:1154)
> at
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
> at
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
> at
> org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
> at
> org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1751)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1247)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterSuccessLock(TestMRAppMaster.java:254)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6557) Some tests in mapreduce-client-app are writing outside of target
[ https://issues.apache.org/jira/browse/MAPREDUCE-6557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15027059#comment-15027059 ] Junping Du commented on MAPREDUCE-6557: --- Sounds like we are updating patch at the same time. :) Anyway, 04 patch is exactly the same as 004 patch. I will commit it after Jenkins report positive. > Some tests in mapreduce-client-app are writing outside of target > > > Key: MAPREDUCE-6557 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6557 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Akira AJISAKA >Priority: Blocker > Attachments: MAPREDUCE-6555.03.patch, MAPREDUCE-6557.00.patch, > MAPREDUCE-6557.004.patch, MAPREDUCE-6557.04.patch > > > There is a staging directory appearing. It should not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6557) Some tests in mapreduce-client-app are writing outside of target
[ https://issues.apache.org/jira/browse/MAPREDUCE-6557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15027229#comment-15027229 ] Allen Wittenauer commented on MAPREDUCE-6557: - bq. I have commit 04 patch to trunk and branch-2. I've removed the 3.0.0 fix version. Please don't add 3.0.0 when you commit to an earlier branch. It messes up the autogenerated reports and it goes against the committer guidelines. Thanks. > Some tests in mapreduce-client-app are writing outside of target > > > Key: MAPREDUCE-6557 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6557 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Akira AJISAKA >Priority: Blocker > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6555.03.patch, MAPREDUCE-6557.00.patch, > MAPREDUCE-6557.004.patch, MAPREDUCE-6557.04.patch > > > There is a staging directory appearing. It should not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6542) HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe
[
https://issues.apache.org/jira/browse/MAPREDUCE-6542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026954#comment-15026954
]
Hadoop QA commented on MAPREDUCE-6542:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
23s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 33s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 42s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 33s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 48s
{color} | {color:red} hadoop-common-project/hadoop-common in trunk has 1 extant
Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 10s
{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
in trunk has 2 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 27s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 32s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 25s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 25s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 58s
{color} | {color:red} Patch generated 1 new checkstyle issues in root (total
was 299, now 299). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 10s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 26s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 58s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 33s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 6s {color} |
{color:red} hadoop-common in the patch failed with JDK v1.7.0_85. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 57s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.7.0_85. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 21s
{color} | {color:red} Patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 78m 31s {color}
| {color:black} {color} |
\\
\\
[jira] [Commented] (MAPREDUCE-6557) Some tests in mapreduce-client-app are writing outside of target
[
https://issues.apache.org/jira/browse/MAPREDUCE-6557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15027030#comment-15027030
]
Junping Du commented on MAPREDUCE-6557:
---
Oh. Forget to mention, test will run failed because code below:
{code}
+ @AfterClass
+ public void cleanup() throws IOException {
+localFS.delete(testDir, true);
+ }
{code}
cleanup() should be marked as static due to requirement for AfterClass tag.
> Some tests in mapreduce-client-app are writing outside of target
>
>
> Key: MAPREDUCE-6557
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6557
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0
>Reporter: Allen Wittenauer
>Assignee: Akira AJISAKA
>Priority: Blocker
> Attachments: MAPREDUCE-6555.03.patch, MAPREDUCE-6557.00.patch
>
>
> There is a staging directory appearing. It should not.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6557) Some tests in mapreduce-client-app are writing outside of target
[ https://issues.apache.org/jira/browse/MAPREDUCE-6557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-6557: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 3.0.0 Status: Resolved (was: Patch Available) I have commit 04 patch to trunk and branch-2. Thanks [~ajisakaa] for the contribution! > Some tests in mapreduce-client-app are writing outside of target > > > Key: MAPREDUCE-6557 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6557 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Akira AJISAKA >Priority: Blocker > Fix For: 3.0.0, 2.8.0 > > Attachments: MAPREDUCE-6555.03.patch, MAPREDUCE-6557.00.patch, > MAPREDUCE-6557.004.patch, MAPREDUCE-6557.04.patch > > > There is a staging directory appearing. It should not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6555) TestMRAppMaster fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026977#comment-15026977
]
Tsuyoshi Ozawa commented on MAPREDUCE-6555:
---
[~djp] yeah. However, we have lots intermittent test failures, so I've checked
it. +1.
> TestMRAppMaster fails on trunk
> --
>
> Key: MAPREDUCE-6555
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6555
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: Junping Du
> Attachments: MAPREDUCE-6555.patch
>
>
> Observed in QA report of YARN-3840
> {noformat}
> Running org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> Tests run: 9, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 20.699 sec
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> testMRAppMasterMidLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.474 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterMidLock(TestMRAppMaster.java:174)
> testMRAppMasterSuccessLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.175 sec <<< ERROR!
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.io.FileNotFoundException: File
> file:/home/varun/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/staging/history/done_intermediate/TestAppMasterUser/job_1317529182569_0004-1448100479292-TestAppMasterUser-%3Cmissing+job+name%3E-1448100479413-0-0-SUCCEEDED-default-1448100479292.jhist_tmp
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:640)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:866)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:630)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:372)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:513)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.moveTmpToDone(JobHistoryEventHandler.java:1346)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.processDoneFiles(JobHistoryEventHandler.java:1154)
> at
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
> at
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
> at
> org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
> at
> org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1751)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1247)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterSuccessLock(TestMRAppMaster.java:254)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6555) TestMRAppMaster fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15027017#comment-15027017
]
Hudson commented on MAPREDUCE-6555:
---
FAILURE: Integrated in Hadoop-trunk-Commit #8886 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8886/])
MAPREDUCE-6555. TestMRAppMaster fails on trunk. (Junping Du via ozawa) (ozawa:
rev 3e85542a7afb8bbbc19f2aa59da04ec824168c0e)
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java
> TestMRAppMaster fails on trunk
> --
>
> Key: MAPREDUCE-6555
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6555
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: Junping Du
> Fix For: 3.0.0
>
> Attachments: MAPREDUCE-6555.patch
>
>
> Observed in QA report of YARN-3840
> {noformat}
> Running org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> Tests run: 9, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 20.699 sec
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> testMRAppMasterMidLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.474 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterMidLock(TestMRAppMaster.java:174)
> testMRAppMasterSuccessLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.175 sec <<< ERROR!
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.io.FileNotFoundException: File
> file:/home/varun/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/staging/history/done_intermediate/TestAppMasterUser/job_1317529182569_0004-1448100479292-TestAppMasterUser-%3Cmissing+job+name%3E-1448100479413-0-0-SUCCEEDED-default-1448100479292.jhist_tmp
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:640)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:866)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:630)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:372)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:513)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.moveTmpToDone(JobHistoryEventHandler.java:1346)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.processDoneFiles(JobHistoryEventHandler.java:1154)
> at
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
> at
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
> at
> org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
> at
> org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1751)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1247)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterSuccessLock(TestMRAppMaster.java:254)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6555) TestMRAppMaster fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026986#comment-15026986
]
Akira AJISAKA commented on MAPREDUCE-6555:
--
+1
> TestMRAppMaster fails on trunk
> --
>
> Key: MAPREDUCE-6555
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6555
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: Junping Du
> Attachments: MAPREDUCE-6555.patch
>
>
> Observed in QA report of YARN-3840
> {noformat}
> Running org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> Tests run: 9, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 20.699 sec
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> testMRAppMasterMidLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.474 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterMidLock(TestMRAppMaster.java:174)
> testMRAppMasterSuccessLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.175 sec <<< ERROR!
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.io.FileNotFoundException: File
> file:/home/varun/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/staging/history/done_intermediate/TestAppMasterUser/job_1317529182569_0004-1448100479292-TestAppMasterUser-%3Cmissing+job+name%3E-1448100479413-0-0-SUCCEEDED-default-1448100479292.jhist_tmp
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:640)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:866)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:630)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:372)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:513)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.moveTmpToDone(JobHistoryEventHandler.java:1346)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.processDoneFiles(JobHistoryEventHandler.java:1154)
> at
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
> at
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
> at
> org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
> at
> org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1751)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1247)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterSuccessLock(TestMRAppMaster.java:254)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6555) TestMRAppMaster fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15027006#comment-15027006
]
Junping Du commented on MAPREDUCE-6555:
---
Thanks [~varun_saxena] for reporting it and [~ozawa] and [~ajisakaa] for review!
> TestMRAppMaster fails on trunk
> --
>
> Key: MAPREDUCE-6555
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6555
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: Junping Du
> Fix For: 3.0.0
>
> Attachments: MAPREDUCE-6555.patch
>
>
> Observed in QA report of YARN-3840
> {noformat}
> Running org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> Tests run: 9, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 20.699 sec
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> testMRAppMasterMidLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.474 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterMidLock(TestMRAppMaster.java:174)
> testMRAppMasterSuccessLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.175 sec <<< ERROR!
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.io.FileNotFoundException: File
> file:/home/varun/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/staging/history/done_intermediate/TestAppMasterUser/job_1317529182569_0004-1448100479292-TestAppMasterUser-%3Cmissing+job+name%3E-1448100479413-0-0-SUCCEEDED-default-1448100479292.jhist_tmp
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:640)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:866)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:630)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:372)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:513)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.moveTmpToDone(JobHistoryEventHandler.java:1346)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.processDoneFiles(JobHistoryEventHandler.java:1154)
> at
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
> at
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
> at
> org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
> at
> org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1751)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1247)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterSuccessLock(TestMRAppMaster.java:254)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6557) Some tests in mapreduce-client-app are writing outside of target
[ https://issues.apache.org/jira/browse/MAPREDUCE-6557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15027055#comment-15027055 ] Akira AJISAKA commented on MAPREDUCE-6557: -- bq. Can you check if it sounds good? If so, I will commit it shortly. +1 for 004 patch. Thanks [~djp]. > Some tests in mapreduce-client-app are writing outside of target > > > Key: MAPREDUCE-6557 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6557 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Akira AJISAKA >Priority: Blocker > Attachments: MAPREDUCE-6555.03.patch, MAPREDUCE-6557.00.patch, > MAPREDUCE-6557.004.patch, MAPREDUCE-6557.04.patch > > > There is a staging directory appearing. It should not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6557) Some tests in mapreduce-client-app are writing outside of target
[ https://issues.apache.org/jira/browse/MAPREDUCE-6557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated MAPREDUCE-6557: - Attachment: MAPREDUCE-6557.04.patch Thanks [~djp] for the comment. Updated the patch. > Some tests in mapreduce-client-app are writing outside of target > > > Key: MAPREDUCE-6557 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6557 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Akira AJISAKA >Priority: Blocker > Attachments: MAPREDUCE-6555.03.patch, MAPREDUCE-6557.00.patch, > MAPREDUCE-6557.004.patch, MAPREDUCE-6557.04.patch > > > There is a staging directory appearing. It should not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6557) Some tests in mapreduce-client-app are writing outside of target
[ https://issues.apache.org/jira/browse/MAPREDUCE-6557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-6557: Fix Version/s: (was: 3.0.0) > Some tests in mapreduce-client-app are writing outside of target > > > Key: MAPREDUCE-6557 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6557 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Akira AJISAKA >Priority: Blocker > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6555.03.patch, MAPREDUCE-6557.00.patch, > MAPREDUCE-6557.004.patch, MAPREDUCE-6557.04.patch > > > There is a staging directory appearing. It should not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6549) multibyte delimiters with LineRecordReader cause duplicate records
[ https://issues.apache.org/jira/browse/MAPREDUCE-6549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15027822#comment-15027822 ] Ray Chiang commented on MAPREDUCE-6549: --- +1 (nonbinding) Much easier to understand with all the unit tests. > multibyte delimiters with LineRecordReader cause duplicate records > -- > > Key: MAPREDUCE-6549 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6549 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1, mrv2 >Affects Versions: 2.7.2 >Reporter: Dustin Cote >Assignee: Wilfred Spiegelenburg > Attachments: MAPREDUCE-6549-1.patch, MAPREDUCE-6549-2.patch, > MAPREDUCE-6549.3.patch > > > LineRecorderReader currently produces duplicate records under certain > scenarios such as: > 1) input string: "abc+++def++ghi++" > delimiter string: "+++" > test passes with all sizes of the split > 2) input string: "abc++def+++ghi++" > delimiter string: "+++" > test fails with a split size of 4 > 2) input string: "abc+++def++ghi++" > delimiter string: "++" > test fails with a split size of 5 > 3) input string "abc+++defg++hij++" > delimiter string: "++" > test fails with a split size of 4 > 4) input string "abc++def+++ghi++" > delimiter string: "++" > test fails with a split size of 9 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6555) TestMRAppMaster fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15027341#comment-15027341
]
Hudson commented on MAPREDUCE-6555:
---
FAILURE: Integrated in Hadoop-Yarn-trunk #1454 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1454/])
MAPREDUCE-6555. TestMRAppMaster fails on trunk. (Junping Du via ozawa) (ozawa:
rev 3e85542a7afb8bbbc19f2aa59da04ec824168c0e)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java
* hadoop-mapreduce-project/CHANGES.txt
> TestMRAppMaster fails on trunk
> --
>
> Key: MAPREDUCE-6555
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6555
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: Junping Du
> Fix For: 3.0.0
>
> Attachments: MAPREDUCE-6555.patch
>
>
> Observed in QA report of YARN-3840
> {noformat}
> Running org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> Tests run: 9, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 20.699 sec
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> testMRAppMasterMidLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.474 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterMidLock(TestMRAppMaster.java:174)
> testMRAppMasterSuccessLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.175 sec <<< ERROR!
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.io.FileNotFoundException: File
> file:/home/varun/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/staging/history/done_intermediate/TestAppMasterUser/job_1317529182569_0004-1448100479292-TestAppMasterUser-%3Cmissing+job+name%3E-1448100479413-0-0-SUCCEEDED-default-1448100479292.jhist_tmp
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:640)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:866)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:630)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:372)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:513)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.moveTmpToDone(JobHistoryEventHandler.java:1346)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.processDoneFiles(JobHistoryEventHandler.java:1154)
> at
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
> at
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
> at
> org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
> at
> org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1751)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1247)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterSuccessLock(TestMRAppMaster.java:254)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6549) multibyte delimiters with LineRecordReader cause duplicate records
[ https://issues.apache.org/jira/browse/MAPREDUCE-6549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-6549: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) Thanks Wilfred and everyone who helped out on this. Committed to trunk and branch-2! > multibyte delimiters with LineRecordReader cause duplicate records > -- > > Key: MAPREDUCE-6549 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6549 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1, mrv2 >Affects Versions: 2.7.2 >Reporter: Dustin Cote >Assignee: Wilfred Spiegelenburg > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6549-1.patch, MAPREDUCE-6549-2.patch, > MAPREDUCE-6549.3.patch > > > LineRecorderReader currently produces duplicate records under certain > scenarios such as: > 1) input string: "abc+++def++ghi++" > delimiter string: "+++" > test passes with all sizes of the split > 2) input string: "abc++def+++ghi++" > delimiter string: "+++" > test fails with a split size of 4 > 2) input string: "abc+++def++ghi++" > delimiter string: "++" > test fails with a split size of 5 > 3) input string "abc+++defg++hij++" > delimiter string: "++" > test fails with a split size of 4 > 4) input string "abc++def+++ghi++" > delimiter string: "++" > test fails with a split size of 9 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6550) archive-logs tool changes log ownership to the Yarn user when using DefaultContainerExecutor
[
https://issues.apache.org/jira/browse/MAPREDUCE-6550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15028038#comment-15028038
]
Hudson commented on MAPREDUCE-6550:
---
FAILURE: Integrated in Hadoop-trunk-Commit #8894 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8894/])
MAPREDUCE-6550. archive-logs tool changes log ownership to the Yarn user
(rkanter: rev 6d84cc16b3e0685fef01d0e3526b0f7556ceff51)
* hadoop-mapreduce-project/CHANGES.txt
* hadoop-tools/hadoop-archive-logs/src/site/markdown/HadoopArchiveLogs.md
*
hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogs.java
*
hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogs.java
*
hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogsRunner.java
*
hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogsRunner.java
> archive-logs tool changes log ownership to the Yarn user when using
> DefaultContainerExecutor
>
>
> Key: MAPREDUCE-6550
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6550
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: 2.8.0
>
> Attachments: MAPREDUCE-6550.001.patch, MAPREDUCE-6550.002.patch,
> MAPREDUCE-6550.003.patch, MAPREDUCE-6550.004.patch, MAPREDUCE-6550.005.patch
>
>
> The archive-logs tool added in MAPREDUCE-6415 leverages the Distributed Shell
> app. When using the DefaultContainerExecutor, this means that the job will
> actually run as the Yarn user, so the resulting har files are owned by the
> Yarn user instead of the original owner. The permissions are also now
> world-readable.
> In the below example, the archived logs are owned by 'yarn' instead of 'paul'
> and are now world-readable:
> {noformat}
> [root@gs28-centos66-5 ~]# sudo -u hdfs hdfs dfs -ls -R /tmp/logs
> ...
> drwxrwx--- - paul hadoop 0 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005
> drwxr-xr-x - yarn hadoop 0 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har
> -rw-r--r-- 3 yarn hadoop 0 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/_SUCCESS
> -rw-r--r-- 3 yarn hadoop 1256 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/_index
> -rw-r--r-- 3 yarn hadoop 24 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/_masterindex
> -rw-r--r-- 3 yarn hadoop8451177 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/part-0
> drwxrwx--- - paul hadoop 0 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0006
> -rw-r- 3 paul hadoop 1155 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0006/gs-centos66-2.vpc.cloudera.com_8041
> -rw-r- 3 paul hadoop 4880 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0006/gs28-centos66-3.vpc.cloudera.com_8041
> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6549) multibyte delimiters with LineRecordReader cause duplicate records
[ https://issues.apache.org/jira/browse/MAPREDUCE-6549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15028039#comment-15028039 ] Hudson commented on MAPREDUCE-6549: --- FAILURE: Integrated in Hadoop-trunk-Commit #8894 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8894/]) MAPREDUCE-6549. multibyte delimiters with LineRecordReader cause (rkanter: rev 7fd00b3db4b7d73afd41276ba9a06ec06a0e1762) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/UncompressedSplitLineReader.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestLineRecordReader.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestLineRecordReader.java > multibyte delimiters with LineRecordReader cause duplicate records > -- > > Key: MAPREDUCE-6549 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6549 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1, mrv2 >Affects Versions: 2.7.2 >Reporter: Dustin Cote >Assignee: Wilfred Spiegelenburg > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6549-1.patch, MAPREDUCE-6549-2.patch, > MAPREDUCE-6549.3.patch > > > LineRecorderReader currently produces duplicate records under certain > scenarios such as: > 1) input string: "abc+++def++ghi++" > delimiter string: "+++" > test passes with all sizes of the split > 2) input string: "abc++def+++ghi++" > delimiter string: "+++" > test fails with a split size of 4 > 2) input string: "abc+++def++ghi++" > delimiter string: "++" > test fails with a split size of 5 > 3) input string "abc+++defg++hij++" > delimiter string: "++" > test fails with a split size of 4 > 4) input string "abc++def+++ghi++" > delimiter string: "++" > test fails with a split size of 9 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6550) archive-logs tool changes log ownership to the Yarn user when using DefaultContainerExecutor
[
https://issues.apache.org/jira/browse/MAPREDUCE-6550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated MAPREDUCE-6550:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.8.0
Status: Resolved (was: Patch Available)
Committed to trunk and branch-2!
> archive-logs tool changes log ownership to the Yarn user when using
> DefaultContainerExecutor
>
>
> Key: MAPREDUCE-6550
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6550
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: 2.8.0
>
> Attachments: MAPREDUCE-6550.001.patch, MAPREDUCE-6550.002.patch,
> MAPREDUCE-6550.003.patch, MAPREDUCE-6550.004.patch, MAPREDUCE-6550.005.patch
>
>
> The archive-logs tool added in MAPREDUCE-6415 leverages the Distributed Shell
> app. When using the DefaultContainerExecutor, this means that the job will
> actually run as the Yarn user, so the resulting har files are owned by the
> Yarn user instead of the original owner. The permissions are also now
> world-readable.
> In the below example, the archived logs are owned by 'yarn' instead of 'paul'
> and are now world-readable:
> {noformat}
> [root@gs28-centos66-5 ~]# sudo -u hdfs hdfs dfs -ls -R /tmp/logs
> ...
> drwxrwx--- - paul hadoop 0 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005
> drwxr-xr-x - yarn hadoop 0 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har
> -rw-r--r-- 3 yarn hadoop 0 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/_SUCCESS
> -rw-r--r-- 3 yarn hadoop 1256 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/_index
> -rw-r--r-- 3 yarn hadoop 24 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/_masterindex
> -rw-r--r-- 3 yarn hadoop8451177 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/part-0
> drwxrwx--- - paul hadoop 0 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0006
> -rw-r- 3 paul hadoop 1155 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0006/gs-centos66-2.vpc.cloudera.com_8041
> -rw-r- 3 paul hadoop 4880 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0006/gs28-centos66-3.vpc.cloudera.com_8041
> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6550) archive-logs tool changes log ownership to the Yarn user when using DefaultContainerExecutor
[
https://issues.apache.org/jira/browse/MAPREDUCE-6550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15027401#comment-15027401
]
Robert Kanter commented on MAPREDUCE-6550:
--
Thanks for the reviews Jason. I'll commit this later today if nobody has any
other comments.
> archive-logs tool changes log ownership to the Yarn user when using
> DefaultContainerExecutor
>
>
> Key: MAPREDUCE-6550
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6550
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-6550.001.patch, MAPREDUCE-6550.002.patch,
> MAPREDUCE-6550.003.patch, MAPREDUCE-6550.004.patch, MAPREDUCE-6550.005.patch
>
>
> The archive-logs tool added in MAPREDUCE-6415 leverages the Distributed Shell
> app. When using the DefaultContainerExecutor, this means that the job will
> actually run as the Yarn user, so the resulting har files are owned by the
> Yarn user instead of the original owner. The permissions are also now
> world-readable.
> In the below example, the archived logs are owned by 'yarn' instead of 'paul'
> and are now world-readable:
> {noformat}
> [root@gs28-centos66-5 ~]# sudo -u hdfs hdfs dfs -ls -R /tmp/logs
> ...
> drwxrwx--- - paul hadoop 0 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005
> drwxr-xr-x - yarn hadoop 0 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har
> -rw-r--r-- 3 yarn hadoop 0 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/_SUCCESS
> -rw-r--r-- 3 yarn hadoop 1256 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/_index
> -rw-r--r-- 3 yarn hadoop 24 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/_masterindex
> -rw-r--r-- 3 yarn hadoop8451177 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/part-0
> drwxrwx--- - paul hadoop 0 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0006
> -rw-r- 3 paul hadoop 1155 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0006/gs-centos66-2.vpc.cloudera.com_8041
> -rw-r- 3 paul hadoop 4880 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0006/gs28-centos66-3.vpc.cloudera.com_8041
> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6555) TestMRAppMaster fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15027276#comment-15027276
]
Hudson commented on MAPREDUCE-6555:
---
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #733 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/733/])
MAPREDUCE-6555. TestMRAppMaster fails on trunk. (Junping Du via ozawa) (ozawa:
rev 3e85542a7afb8bbbc19f2aa59da04ec824168c0e)
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java
> TestMRAppMaster fails on trunk
> --
>
> Key: MAPREDUCE-6555
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6555
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: Junping Du
> Fix For: 3.0.0
>
> Attachments: MAPREDUCE-6555.patch
>
>
> Observed in QA report of YARN-3840
> {noformat}
> Running org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> Tests run: 9, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 20.699 sec
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> testMRAppMasterMidLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.474 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterMidLock(TestMRAppMaster.java:174)
> testMRAppMasterSuccessLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.175 sec <<< ERROR!
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.io.FileNotFoundException: File
> file:/home/varun/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/staging/history/done_intermediate/TestAppMasterUser/job_1317529182569_0004-1448100479292-TestAppMasterUser-%3Cmissing+job+name%3E-1448100479413-0-0-SUCCEEDED-default-1448100479292.jhist_tmp
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:640)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:866)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:630)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:372)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:513)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.moveTmpToDone(JobHistoryEventHandler.java:1346)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.processDoneFiles(JobHistoryEventHandler.java:1154)
> at
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
> at
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
> at
> org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
> at
> org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1751)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1247)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterSuccessLock(TestMRAppMaster.java:254)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6555) TestMRAppMaster fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15027343#comment-15027343
]
Hudson commented on MAPREDUCE-6555:
---
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #722 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/722/])
MAPREDUCE-6555. TestMRAppMaster fails on trunk. (Junping Du via ozawa) (ozawa:
rev 3e85542a7afb8bbbc19f2aa59da04ec824168c0e)
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java
> TestMRAppMaster fails on trunk
> --
>
> Key: MAPREDUCE-6555
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6555
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: Junping Du
> Fix For: 3.0.0
>
> Attachments: MAPREDUCE-6555.patch
>
>
> Observed in QA report of YARN-3840
> {noformat}
> Running org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> Tests run: 9, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 20.699 sec
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> testMRAppMasterMidLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.474 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterMidLock(TestMRAppMaster.java:174)
> testMRAppMasterSuccessLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.175 sec <<< ERROR!
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.io.FileNotFoundException: File
> file:/home/varun/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/staging/history/done_intermediate/TestAppMasterUser/job_1317529182569_0004-1448100479292-TestAppMasterUser-%3Cmissing+job+name%3E-1448100479413-0-0-SUCCEEDED-default-1448100479292.jhist_tmp
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:640)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:866)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:630)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:372)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:513)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.moveTmpToDone(JobHistoryEventHandler.java:1346)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.processDoneFiles(JobHistoryEventHandler.java:1154)
> at
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
> at
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
> at
> org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
> at
> org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1751)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1247)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterSuccessLock(TestMRAppMaster.java:254)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6549) multibyte delimiters with LineRecordReader cause duplicate records
[ https://issues.apache.org/jira/browse/MAPREDUCE-6549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15027383#comment-15027383 ] Robert Kanter commented on MAPREDUCE-6549: -- +1 LGTM Will commit later today if nobody has any other comments. > multibyte delimiters with LineRecordReader cause duplicate records > -- > > Key: MAPREDUCE-6549 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6549 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1, mrv2 >Affects Versions: 2.7.2 >Reporter: Dustin Cote >Assignee: Wilfred Spiegelenburg > Attachments: MAPREDUCE-6549-1.patch, MAPREDUCE-6549-2.patch, > MAPREDUCE-6549.3.patch > > > LineRecorderReader currently produces duplicate records under certain > scenarios such as: > 1) input string: "abc+++def++ghi++" > delimiter string: "+++" > test passes with all sizes of the split > 2) input string: "abc++def+++ghi++" > delimiter string: "+++" > test fails with a split size of 4 > 2) input string: "abc+++def++ghi++" > delimiter string: "++" > test fails with a split size of 5 > 3) input string "abc+++defg++hij++" > delimiter string: "++" > test fails with a split size of 4 > 4) input string "abc++def+++ghi++" > delimiter string: "++" > test fails with a split size of 9 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6557) Some tests in mapreduce-client-app are writing outside of target
[ https://issues.apache.org/jira/browse/MAPREDUCE-6557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15027262#comment-15027262 ] Junping Du commented on MAPREDUCE-6557: --- Sure. Thanks for reminding here, [~aw]! > Some tests in mapreduce-client-app are writing outside of target > > > Key: MAPREDUCE-6557 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6557 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Akira AJISAKA >Priority: Blocker > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6555.03.patch, MAPREDUCE-6557.00.patch, > MAPREDUCE-6557.004.patch, MAPREDUCE-6557.04.patch > > > There is a staging directory appearing. It should not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6555) TestMRAppMaster fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15027625#comment-15027625
]
Hudson commented on MAPREDUCE-6555:
---
ABORTED: Integrated in Hadoop-Hdfs-trunk-Java8 #643 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/643/])
MAPREDUCE-6555. TestMRAppMaster fails on trunk. (Junping Du via ozawa) (ozawa:
rev 3e85542a7afb8bbbc19f2aa59da04ec824168c0e)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java
* hadoop-mapreduce-project/CHANGES.txt
> TestMRAppMaster fails on trunk
> --
>
> Key: MAPREDUCE-6555
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6555
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: Junping Du
> Fix For: 3.0.0
>
> Attachments: MAPREDUCE-6555.patch
>
>
> Observed in QA report of YARN-3840
> {noformat}
> Running org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> Tests run: 9, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 20.699 sec
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> testMRAppMasterMidLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.474 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterMidLock(TestMRAppMaster.java:174)
> testMRAppMasterSuccessLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.175 sec <<< ERROR!
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.io.FileNotFoundException: File
> file:/home/varun/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/staging/history/done_intermediate/TestAppMasterUser/job_1317529182569_0004-1448100479292-TestAppMasterUser-%3Cmissing+job+name%3E-1448100479413-0-0-SUCCEEDED-default-1448100479292.jhist_tmp
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:640)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:866)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:630)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:372)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:513)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.moveTmpToDone(JobHistoryEventHandler.java:1346)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.processDoneFiles(JobHistoryEventHandler.java:1154)
> at
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
> at
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
> at
> org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
> at
> org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1751)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1247)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterSuccessLock(TestMRAppMaster.java:254)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6555) TestMRAppMaster fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15027670#comment-15027670
]
Hudson commented on MAPREDUCE-6555:
---
FAILURE: Integrated in Hadoop-Hdfs-trunk #2580 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2580/])
MAPREDUCE-6555. TestMRAppMaster fails on trunk. (Junping Du via ozawa) (ozawa:
rev 3e85542a7afb8bbbc19f2aa59da04ec824168c0e)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java
* hadoop-mapreduce-project/CHANGES.txt
> TestMRAppMaster fails on trunk
> --
>
> Key: MAPREDUCE-6555
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6555
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: Junping Du
> Fix For: 3.0.0
>
> Attachments: MAPREDUCE-6555.patch
>
>
> Observed in QA report of YARN-3840
> {noformat}
> Running org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> Tests run: 9, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 20.699 sec
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> testMRAppMasterMidLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.474 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterMidLock(TestMRAppMaster.java:174)
> testMRAppMasterSuccessLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.175 sec <<< ERROR!
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.io.FileNotFoundException: File
> file:/home/varun/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/staging/history/done_intermediate/TestAppMasterUser/job_1317529182569_0004-1448100479292-TestAppMasterUser-%3Cmissing+job+name%3E-1448100479413-0-0-SUCCEEDED-default-1448100479292.jhist_tmp
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:640)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:866)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:630)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:372)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:513)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.moveTmpToDone(JobHistoryEventHandler.java:1346)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.processDoneFiles(JobHistoryEventHandler.java:1154)
> at
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
> at
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
> at
> org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
> at
> org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1751)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1247)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterSuccessLock(TestMRAppMaster.java:254)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6555) TestMRAppMaster fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15027244#comment-15027244
]
Hudson commented on MAPREDUCE-6555:
---
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2664 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2664/])
MAPREDUCE-6555. TestMRAppMaster fails on trunk. (Junping Du via ozawa) (ozawa:
rev 3e85542a7afb8bbbc19f2aa59da04ec824168c0e)
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java
> TestMRAppMaster fails on trunk
> --
>
> Key: MAPREDUCE-6555
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6555
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: Junping Du
> Fix For: 3.0.0
>
> Attachments: MAPREDUCE-6555.patch
>
>
> Observed in QA report of YARN-3840
> {noformat}
> Running org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> Tests run: 9, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 20.699 sec
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> testMRAppMasterMidLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.474 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterMidLock(TestMRAppMaster.java:174)
> testMRAppMasterSuccessLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.175 sec <<< ERROR!
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.io.FileNotFoundException: File
> file:/home/varun/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/staging/history/done_intermediate/TestAppMasterUser/job_1317529182569_0004-1448100479292-TestAppMasterUser-%3Cmissing+job+name%3E-1448100479413-0-0-SUCCEEDED-default-1448100479292.jhist_tmp
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:640)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:866)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:630)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:372)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:513)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.moveTmpToDone(JobHistoryEventHandler.java:1346)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.processDoneFiles(JobHistoryEventHandler.java:1154)
> at
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
> at
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
> at
> org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
> at
> org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1751)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1247)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterSuccessLock(TestMRAppMaster.java:254)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6549) multibyte delimiters with LineRecordReader cause duplicate records
[ https://issues.apache.org/jira/browse/MAPREDUCE-6549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15028144#comment-15028144 ] Hudson commented on MAPREDUCE-6549: --- FAILURE: Integrated in Hadoop-Yarn-trunk #1457 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1457/]) MAPREDUCE-6549. multibyte delimiters with LineRecordReader cause (rkanter: rev 7fd00b3db4b7d73afd41276ba9a06ec06a0e1762) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/UncompressedSplitLineReader.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestLineRecordReader.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestLineRecordReader.java > multibyte delimiters with LineRecordReader cause duplicate records > -- > > Key: MAPREDUCE-6549 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6549 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1, mrv2 >Affects Versions: 2.7.2 >Reporter: Dustin Cote >Assignee: Wilfred Spiegelenburg > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6549-1.patch, MAPREDUCE-6549-2.patch, > MAPREDUCE-6549.3.patch > > > LineRecorderReader currently produces duplicate records under certain > scenarios such as: > 1) input string: "abc+++def++ghi++" > delimiter string: "+++" > test passes with all sizes of the split > 2) input string: "abc++def+++ghi++" > delimiter string: "+++" > test fails with a split size of 4 > 2) input string: "abc+++def++ghi++" > delimiter string: "++" > test fails with a split size of 5 > 3) input string "abc+++defg++hij++" > delimiter string: "++" > test fails with a split size of 4 > 4) input string "abc++def+++ghi++" > delimiter string: "++" > test fails with a split size of 9 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6550) archive-logs tool changes log ownership to the Yarn user when using DefaultContainerExecutor
[
https://issues.apache.org/jira/browse/MAPREDUCE-6550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15028142#comment-15028142
]
Hudson commented on MAPREDUCE-6550:
---
FAILURE: Integrated in Hadoop-Yarn-trunk #1457 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1457/])
MAPREDUCE-6550. archive-logs tool changes log ownership to the Yarn user
(rkanter: rev 6d84cc16b3e0685fef01d0e3526b0f7556ceff51)
*
hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogs.java
* hadoop-tools/hadoop-archive-logs/src/site/markdown/HadoopArchiveLogs.md
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogsRunner.java
*
hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogs.java
*
hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogsRunner.java
> archive-logs tool changes log ownership to the Yarn user when using
> DefaultContainerExecutor
>
>
> Key: MAPREDUCE-6550
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6550
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: 2.8.0
>
> Attachments: MAPREDUCE-6550.001.patch, MAPREDUCE-6550.002.patch,
> MAPREDUCE-6550.003.patch, MAPREDUCE-6550.004.patch, MAPREDUCE-6550.005.patch
>
>
> The archive-logs tool added in MAPREDUCE-6415 leverages the Distributed Shell
> app. When using the DefaultContainerExecutor, this means that the job will
> actually run as the Yarn user, so the resulting har files are owned by the
> Yarn user instead of the original owner. The permissions are also now
> world-readable.
> In the below example, the archived logs are owned by 'yarn' instead of 'paul'
> and are now world-readable:
> {noformat}
> [root@gs28-centos66-5 ~]# sudo -u hdfs hdfs dfs -ls -R /tmp/logs
> ...
> drwxrwx--- - paul hadoop 0 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005
> drwxr-xr-x - yarn hadoop 0 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har
> -rw-r--r-- 3 yarn hadoop 0 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/_SUCCESS
> -rw-r--r-- 3 yarn hadoop 1256 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/_index
> -rw-r--r-- 3 yarn hadoop 24 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/_masterindex
> -rw-r--r-- 3 yarn hadoop8451177 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/part-0
> drwxrwx--- - paul hadoop 0 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0006
> -rw-r- 3 paul hadoop 1155 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0006/gs-centos66-2.vpc.cloudera.com_8041
> -rw-r- 3 paul hadoop 4880 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0006/gs28-centos66-3.vpc.cloudera.com_8041
> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6549) multibyte delimiters with LineRecordReader cause duplicate records
[ https://issues.apache.org/jira/browse/MAPREDUCE-6549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15028168#comment-15028168 ] Hudson commented on MAPREDUCE-6549: --- SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #726 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/726/]) MAPREDUCE-6549. multibyte delimiters with LineRecordReader cause (rkanter: rev 7fd00b3db4b7d73afd41276ba9a06ec06a0e1762) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestLineRecordReader.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestLineRecordReader.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/UncompressedSplitLineReader.java > multibyte delimiters with LineRecordReader cause duplicate records > -- > > Key: MAPREDUCE-6549 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6549 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1, mrv2 >Affects Versions: 2.7.2 >Reporter: Dustin Cote >Assignee: Wilfred Spiegelenburg > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6549-1.patch, MAPREDUCE-6549-2.patch, > MAPREDUCE-6549.3.patch > > > LineRecorderReader currently produces duplicate records under certain > scenarios such as: > 1) input string: "abc+++def++ghi++" > delimiter string: "+++" > test passes with all sizes of the split > 2) input string: "abc++def+++ghi++" > delimiter string: "+++" > test fails with a split size of 4 > 2) input string: "abc+++def++ghi++" > delimiter string: "++" > test fails with a split size of 5 > 3) input string "abc+++defg++hij++" > delimiter string: "++" > test fails with a split size of 4 > 4) input string "abc++def+++ghi++" > delimiter string: "++" > test fails with a split size of 9 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6550) archive-logs tool changes log ownership to the Yarn user when using DefaultContainerExecutor
[
https://issues.apache.org/jira/browse/MAPREDUCE-6550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15028167#comment-15028167
]
Hudson commented on MAPREDUCE-6550:
---
SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #726 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/726/])
MAPREDUCE-6550. archive-logs tool changes log ownership to the Yarn user
(rkanter: rev 6d84cc16b3e0685fef01d0e3526b0f7556ceff51)
*
hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogsRunner.java
*
hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogsRunner.java
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogs.java
* hadoop-tools/hadoop-archive-logs/src/site/markdown/HadoopArchiveLogs.md
*
hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogs.java
> archive-logs tool changes log ownership to the Yarn user when using
> DefaultContainerExecutor
>
>
> Key: MAPREDUCE-6550
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6550
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: 2.8.0
>
> Attachments: MAPREDUCE-6550.001.patch, MAPREDUCE-6550.002.patch,
> MAPREDUCE-6550.003.patch, MAPREDUCE-6550.004.patch, MAPREDUCE-6550.005.patch
>
>
> The archive-logs tool added in MAPREDUCE-6415 leverages the Distributed Shell
> app. When using the DefaultContainerExecutor, this means that the job will
> actually run as the Yarn user, so the resulting har files are owned by the
> Yarn user instead of the original owner. The permissions are also now
> world-readable.
> In the below example, the archived logs are owned by 'yarn' instead of 'paul'
> and are now world-readable:
> {noformat}
> [root@gs28-centos66-5 ~]# sudo -u hdfs hdfs dfs -ls -R /tmp/logs
> ...
> drwxrwx--- - paul hadoop 0 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005
> drwxr-xr-x - yarn hadoop 0 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har
> -rw-r--r-- 3 yarn hadoop 0 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/_SUCCESS
> -rw-r--r-- 3 yarn hadoop 1256 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/_index
> -rw-r--r-- 3 yarn hadoop 24 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/_masterindex
> -rw-r--r-- 3 yarn hadoop8451177 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/part-0
> drwxrwx--- - paul hadoop 0 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0006
> -rw-r- 3 paul hadoop 1155 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0006/gs-centos66-2.vpc.cloudera.com_8041
> -rw-r- 3 paul hadoop 4880 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0006/gs28-centos66-3.vpc.cloudera.com_8041
> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6553) Replace '\u2b05' with '<-' in rendering job configuration
[ https://issues.apache.org/jira/browse/MAPREDUCE-6553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026529#comment-15026529 ] Hudson commented on MAPREDUCE-6553: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2663 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2663/]) MAPREDUCE-6553. Replace '\u2b05' with '<-' in rendering job (aajisaka: rev 5ec44fc35c129d266edaebbc6fa17754a89b2ad6) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/ConfBlock.java > Replace '\u2b05' with '<-' in rendering job configuration > - > > Key: MAPREDUCE-6553 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6553 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 2.7.1 > Environment: Google Chrome 46.0.2490.86m with Windows 7 >Reporter: Akira AJISAKA >Assignee: Gabor Liptak >Priority: Minor > Labels: newbie > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6553.1.patch, Screen Shot 2015-11-25 at > 16.19.16.png, jobhistory.png > > > U+2B05 (leftwards black arrow) is not supported in some browsers, so we > should replace it with "<-". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6553) Replace '\u2b05' with '<-' in rendering job configuration
[ https://issues.apache.org/jira/browse/MAPREDUCE-6553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026395#comment-15026395 ] Hudson commented on MAPREDUCE-6553: --- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #731 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/731/]) MAPREDUCE-6553. Replace '\u2b05' with '<-' in rendering job (aajisaka: rev 5ec44fc35c129d266edaebbc6fa17754a89b2ad6) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/ConfBlock.java * hadoop-mapreduce-project/CHANGES.txt > Replace '\u2b05' with '<-' in rendering job configuration > - > > Key: MAPREDUCE-6553 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6553 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 2.7.1 > Environment: Google Chrome 46.0.2490.86m with Windows 7 >Reporter: Akira AJISAKA >Assignee: Gabor Liptak >Priority: Minor > Labels: newbie > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6553.1.patch, Screen Shot 2015-11-25 at > 16.19.16.png, jobhistory.png > > > U+2B05 (leftwards black arrow) is not supported in some browsers, so we > should replace it with "<-". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6553) Replace '\u2b05' with '<-' in rendering job configuration
[ https://issues.apache.org/jira/browse/MAPREDUCE-6553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026502#comment-15026502 ] Hudson commented on MAPREDUCE-6553: --- FAILURE: Integrated in Hadoop-Yarn-trunk #1453 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1453/]) MAPREDUCE-6553. Replace '\u2b05' with '<-' in rendering job (aajisaka: rev 5ec44fc35c129d266edaebbc6fa17754a89b2ad6) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/ConfBlock.java * hadoop-mapreduce-project/CHANGES.txt > Replace '\u2b05' with '<-' in rendering job configuration > - > > Key: MAPREDUCE-6553 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6553 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 2.7.1 > Environment: Google Chrome 46.0.2490.86m with Windows 7 >Reporter: Akira AJISAKA >Assignee: Gabor Liptak >Priority: Minor > Labels: newbie > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6553.1.patch, Screen Shot 2015-11-25 at > 16.19.16.png, jobhistory.png > > > U+2B05 (leftwards black arrow) is not supported in some browsers, so we > should replace it with "<-". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6553) Replace '\u2b05' with '<-' in rendering job configuration
[ https://issues.apache.org/jira/browse/MAPREDUCE-6553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026389#comment-15026389 ] Hudson commented on MAPREDUCE-6553: --- FAILURE: Integrated in Hadoop-trunk-Commit #8884 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8884/]) MAPREDUCE-6553. Replace '\u2b05' with '<-' in rendering job (aajisaka: rev 5ec44fc35c129d266edaebbc6fa17754a89b2ad6) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/ConfBlock.java * hadoop-mapreduce-project/CHANGES.txt > Replace '\u2b05' with '<-' in rendering job configuration > - > > Key: MAPREDUCE-6553 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6553 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 2.7.1 > Environment: Google Chrome 46.0.2490.86m with Windows 7 >Reporter: Akira AJISAKA >Assignee: Gabor Liptak >Priority: Minor > Labels: newbie > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6553.1.patch, Screen Shot 2015-11-25 at > 16.19.16.png, jobhistory.png > > > U+2B05 (leftwards black arrow) is not supported in some browsers, so we > should replace it with "<-". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6553) Replace '\u2b05' with '<-' in rendering job configuration
[ https://issues.apache.org/jira/browse/MAPREDUCE-6553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026446#comment-15026446 ] Hudson commented on MAPREDUCE-6553: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #721 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/721/]) MAPREDUCE-6553. Replace '\u2b05' with '<-' in rendering job (aajisaka: rev 5ec44fc35c129d266edaebbc6fa17754a89b2ad6) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/ConfBlock.java > Replace '\u2b05' with '<-' in rendering job configuration > - > > Key: MAPREDUCE-6553 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6553 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 2.7.1 > Environment: Google Chrome 46.0.2490.86m with Windows 7 >Reporter: Akira AJISAKA >Assignee: Gabor Liptak >Priority: Minor > Labels: newbie > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6553.1.patch, Screen Shot 2015-11-25 at > 16.19.16.png, jobhistory.png > > > U+2B05 (leftwards black arrow) is not supported in some browsers, so we > should replace it with "<-". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6555) TestMRAppMaster fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026732#comment-15026732
]
Junping Du commented on MAPREDUCE-6555:
---
bq. TestMRAppMaster is using directory other than test.build.data, so the
intermediate files are checked by Apache Rat. We should fix it in a separate
jira.
Agree. The worse thing is the test doesn't cleanup the directory in the end
because the cleanup() get called before every tests rather than after. Let's
file a separate JIRA to fix them.
> TestMRAppMaster fails on trunk
> --
>
> Key: MAPREDUCE-6555
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6555
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: Junping Du
> Attachments: MAPREDUCE-6555.patch
>
>
> Observed in QA report of YARN-3840
> {noformat}
> Running org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> Tests run: 9, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 20.699 sec
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> testMRAppMasterMidLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.474 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterMidLock(TestMRAppMaster.java:174)
> testMRAppMasterSuccessLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.175 sec <<< ERROR!
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.io.FileNotFoundException: File
> file:/home/varun/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/staging/history/done_intermediate/TestAppMasterUser/job_1317529182569_0004-1448100479292-TestAppMasterUser-%3Cmissing+job+name%3E-1448100479413-0-0-SUCCEEDED-default-1448100479292.jhist_tmp
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:640)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:866)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:630)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:372)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:513)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.moveTmpToDone(JobHistoryEventHandler.java:1346)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.processDoneFiles(JobHistoryEventHandler.java:1154)
> at
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
> at
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
> at
> org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
> at
> org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1751)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1247)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterSuccessLock(TestMRAppMaster.java:254)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6555) TestMRAppMaster fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026767#comment-15026767
]
Junping Du commented on MAPREDUCE-6555:
---
bq. I'm checking by running the test case. Please wait a moment.
The Jenkins test report above already show it. Isn't it?
> TestMRAppMaster fails on trunk
> --
>
> Key: MAPREDUCE-6555
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6555
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: Junping Du
> Attachments: MAPREDUCE-6555.patch
>
>
> Observed in QA report of YARN-3840
> {noformat}
> Running org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> Tests run: 9, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 20.699 sec
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> testMRAppMasterMidLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.474 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterMidLock(TestMRAppMaster.java:174)
> testMRAppMasterSuccessLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.175 sec <<< ERROR!
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.io.FileNotFoundException: File
> file:/home/varun/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/staging/history/done_intermediate/TestAppMasterUser/job_1317529182569_0004-1448100479292-TestAppMasterUser-%3Cmissing+job+name%3E-1448100479413-0-0-SUCCEEDED-default-1448100479292.jhist_tmp
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:640)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:866)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:630)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:372)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:513)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.moveTmpToDone(JobHistoryEventHandler.java:1346)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.processDoneFiles(JobHistoryEventHandler.java:1154)
> at
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
> at
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
> at
> org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
> at
> org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1751)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1247)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterSuccessLock(TestMRAppMaster.java:254)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6557) Some tests in mapreduce-client-app are writing outside of target
[ https://issues.apache.org/jira/browse/MAPREDUCE-6557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026770#comment-15026770 ] Junping Du commented on MAPREDUCE-6557: --- 03 patch LGTM. +1 pending on Jenkins report. > Some tests in mapreduce-client-app are writing outside of target > > > Key: MAPREDUCE-6557 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6557 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Akira AJISAKA >Priority: Blocker > Attachments: MAPREDUCE-6555.03.patch, MAPREDUCE-6557.00.patch > > > There is a staging directory appearing. It should not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6553) Replace '\u2b05' with '<-' in rendering job configuration
[ https://issues.apache.org/jira/browse/MAPREDUCE-6553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026726#comment-15026726 ] Hudson commented on MAPREDUCE-6553: --- FAILURE: Integrated in Hadoop-Hdfs-trunk #2579 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2579/]) MAPREDUCE-6553. Replace '\u2b05' with '<-' in rendering job (aajisaka: rev 5ec44fc35c129d266edaebbc6fa17754a89b2ad6) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/ConfBlock.java * hadoop-mapreduce-project/CHANGES.txt > Replace '\u2b05' with '<-' in rendering job configuration > - > > Key: MAPREDUCE-6553 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6553 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 2.7.1 > Environment: Google Chrome 46.0.2490.86m with Windows 7 >Reporter: Akira AJISAKA >Assignee: Gabor Liptak >Priority: Minor > Labels: newbie > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6553.1.patch, Screen Shot 2015-11-25 at > 16.19.16.png, jobhistory.png > > > U+2B05 (leftwards black arrow) is not supported in some browsers, so we > should replace it with "<-". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6555) TestMRAppMaster fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026737#comment-15026737
]
Akira AJISAKA commented on MAPREDUCE-6555:
--
bq. We should fix it in a separate jira.
This issue is tracked by MAPREDUCE-6557.
> TestMRAppMaster fails on trunk
> --
>
> Key: MAPREDUCE-6555
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6555
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: Junping Du
> Attachments: MAPREDUCE-6555.patch
>
>
> Observed in QA report of YARN-3840
> {noformat}
> Running org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> Tests run: 9, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 20.699 sec
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> testMRAppMasterMidLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.474 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterMidLock(TestMRAppMaster.java:174)
> testMRAppMasterSuccessLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.175 sec <<< ERROR!
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.io.FileNotFoundException: File
> file:/home/varun/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/staging/history/done_intermediate/TestAppMasterUser/job_1317529182569_0004-1448100479292-TestAppMasterUser-%3Cmissing+job+name%3E-1448100479413-0-0-SUCCEEDED-default-1448100479292.jhist_tmp
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:640)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:866)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:630)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:372)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:513)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.moveTmpToDone(JobHistoryEventHandler.java:1346)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.processDoneFiles(JobHistoryEventHandler.java:1154)
> at
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
> at
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
> at
> org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
> at
> org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1751)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1247)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterSuccessLock(TestMRAppMaster.java:254)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6557) Some tests in mapreduce-client-app are writing outside of target
[ https://issues.apache.org/jira/browse/MAPREDUCE-6557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026743#comment-15026743 ] Junping Du commented on MAPREDUCE-6557: --- Thanks for the patch, [~ajisakaa]! As I just mentioned in MAPREDUCE-6555, may be we also want to fix the issue that the directory is not cleanup after test finish? > Some tests in mapreduce-client-app are writing outside of target > > > Key: MAPREDUCE-6557 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6557 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Akira AJISAKA >Priority: Blocker > Attachments: MAPREDUCE-6557.00.patch > > > There is a staging directory appearing. It should not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6555) TestMRAppMaster fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated MAPREDUCE-6555:
-
Attachment: MAPREDUCE-6555.02.patch
Thanks [~djp] for the comment. Updated the patch to cleanup the test dir after
the tests.
> TestMRAppMaster fails on trunk
> --
>
> Key: MAPREDUCE-6555
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6555
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: Junping Du
> Attachments: MAPREDUCE-6555.02.patch, MAPREDUCE-6555.patch
>
>
> Observed in QA report of YARN-3840
> {noformat}
> Running org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> Tests run: 9, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 20.699 sec
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> testMRAppMasterMidLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.474 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterMidLock(TestMRAppMaster.java:174)
> testMRAppMasterSuccessLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.175 sec <<< ERROR!
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.io.FileNotFoundException: File
> file:/home/varun/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/staging/history/done_intermediate/TestAppMasterUser/job_1317529182569_0004-1448100479292-TestAppMasterUser-%3Cmissing+job+name%3E-1448100479413-0-0-SUCCEEDED-default-1448100479292.jhist_tmp
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:640)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:866)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:630)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:372)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:513)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.moveTmpToDone(JobHistoryEventHandler.java:1346)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.processDoneFiles(JobHistoryEventHandler.java:1154)
> at
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
> at
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
> at
> org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
> at
> org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1751)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1247)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterSuccessLock(TestMRAppMaster.java:254)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (MAPREDUCE-6555) TestMRAppMaster fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated MAPREDUCE-6555:
-
Comment: was deleted
(was: My mistake. I attached the patch to the wrong jira. Deleting the patch.)
> TestMRAppMaster fails on trunk
> --
>
> Key: MAPREDUCE-6555
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6555
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: Junping Du
> Attachments: MAPREDUCE-6555.patch
>
>
> Observed in QA report of YARN-3840
> {noformat}
> Running org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> Tests run: 9, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 20.699 sec
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> testMRAppMasterMidLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.474 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterMidLock(TestMRAppMaster.java:174)
> testMRAppMasterSuccessLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.175 sec <<< ERROR!
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.io.FileNotFoundException: File
> file:/home/varun/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/staging/history/done_intermediate/TestAppMasterUser/job_1317529182569_0004-1448100479292-TestAppMasterUser-%3Cmissing+job+name%3E-1448100479413-0-0-SUCCEEDED-default-1448100479292.jhist_tmp
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:640)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:866)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:630)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:372)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:513)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.moveTmpToDone(JobHistoryEventHandler.java:1346)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.processDoneFiles(JobHistoryEventHandler.java:1154)
> at
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
> at
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
> at
> org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
> at
> org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1751)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1247)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterSuccessLock(TestMRAppMaster.java:254)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6557) Some tests in mapreduce-client-app are writing outside of target
[ https://issues.apache.org/jira/browse/MAPREDUCE-6557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated MAPREDUCE-6557: - Attachment: MAPREDUCE-6555.03.patch 03 * Removed unused import. > Some tests in mapreduce-client-app are writing outside of target > > > Key: MAPREDUCE-6557 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6557 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Akira AJISAKA >Priority: Blocker > Attachments: MAPREDUCE-6555.03.patch, MAPREDUCE-6557.00.patch > > > There is a staging directory appearing. It should not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6555) TestMRAppMaster fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026762#comment-15026762
]
Akira AJISAKA commented on MAPREDUCE-6555:
--
My mistake. I attached the patch to the wrong jira. Deleting the patch.
> TestMRAppMaster fails on trunk
> --
>
> Key: MAPREDUCE-6555
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6555
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: Junping Du
> Attachments: MAPREDUCE-6555.patch
>
>
> Observed in QA report of YARN-3840
> {noformat}
> Running org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> Tests run: 9, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 20.699 sec
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> testMRAppMasterMidLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.474 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterMidLock(TestMRAppMaster.java:174)
> testMRAppMasterSuccessLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.175 sec <<< ERROR!
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.io.FileNotFoundException: File
> file:/home/varun/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/staging/history/done_intermediate/TestAppMasterUser/job_1317529182569_0004-1448100479292-TestAppMasterUser-%3Cmissing+job+name%3E-1448100479413-0-0-SUCCEEDED-default-1448100479292.jhist_tmp
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:640)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:866)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:630)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:372)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:513)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.moveTmpToDone(JobHistoryEventHandler.java:1346)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.processDoneFiles(JobHistoryEventHandler.java:1154)
> at
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
> at
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
> at
> org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
> at
> org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1751)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1247)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterSuccessLock(TestMRAppMaster.java:254)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6555) TestMRAppMaster fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026761#comment-15026761
]
Akira AJISAKA commented on MAPREDUCE-6555:
--
My mistake. I attached the patch to the wrong jira. Deleting the patch.
> TestMRAppMaster fails on trunk
> --
>
> Key: MAPREDUCE-6555
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6555
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: Junping Du
> Attachments: MAPREDUCE-6555.patch
>
>
> Observed in QA report of YARN-3840
> {noformat}
> Running org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> Tests run: 9, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 20.699 sec
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> testMRAppMasterMidLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.474 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterMidLock(TestMRAppMaster.java:174)
> testMRAppMasterSuccessLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.175 sec <<< ERROR!
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.io.FileNotFoundException: File
> file:/home/varun/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/staging/history/done_intermediate/TestAppMasterUser/job_1317529182569_0004-1448100479292-TestAppMasterUser-%3Cmissing+job+name%3E-1448100479413-0-0-SUCCEEDED-default-1448100479292.jhist_tmp
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:640)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:866)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:630)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:372)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:513)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.moveTmpToDone(JobHistoryEventHandler.java:1346)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.processDoneFiles(JobHistoryEventHandler.java:1154)
> at
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
> at
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
> at
> org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
> at
> org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1751)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1247)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterSuccessLock(TestMRAppMaster.java:254)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6555) TestMRAppMaster fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated MAPREDUCE-6555:
-
Attachment: (was: MAPREDUCE-6555.02.patch)
> TestMRAppMaster fails on trunk
> --
>
> Key: MAPREDUCE-6555
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6555
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: Junping Du
> Attachments: MAPREDUCE-6555.patch
>
>
> Observed in QA report of YARN-3840
> {noformat}
> Running org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> Tests run: 9, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 20.699 sec
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
> testMRAppMasterMidLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.474 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterMidLock(TestMRAppMaster.java:174)
> testMRAppMasterSuccessLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)
> Time elapsed: 0.175 sec <<< ERROR!
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.io.FileNotFoundException: File
> file:/home/varun/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/staging/history/done_intermediate/TestAppMasterUser/job_1317529182569_0004-1448100479292-TestAppMasterUser-%3Cmissing+job+name%3E-1448100479413-0-0-SUCCEEDED-default-1448100479292.jhist_tmp
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:640)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:866)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:630)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:372)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:513)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.moveTmpToDone(JobHistoryEventHandler.java:1346)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.processDoneFiles(JobHistoryEventHandler.java:1154)
> at
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
> at
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
> at
> org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
> at
> org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1751)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1247)
> at
> org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterSuccessLock(TestMRAppMaster.java:254)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6557) Some tests in mapreduce-client-app are writing outside of target
[ https://issues.apache.org/jira/browse/MAPREDUCE-6557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026765#comment-15026765 ] Akira AJISAKA commented on MAPREDUCE-6557: -- Thanks [~djp] for the comment. Updated the patch to cleanup the test directory after the tests. > Some tests in mapreduce-client-app are writing outside of target > > > Key: MAPREDUCE-6557 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6557 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Akira AJISAKA >Priority: Blocker > Attachments: MAPREDUCE-6555.03.patch, MAPREDUCE-6557.00.patch > > > There is a staging directory appearing. It should not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6557) Some tests in mapreduce-client-app are writing outside of target
[
https://issues.apache.org/jira/browse/MAPREDUCE-6557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026824#comment-15026824
]
Hadoop QA commented on MAPREDUCE-6557:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
31s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 46s {color}
| {color:red} hadoop-mapreduce-client-app in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 22s {color}
| {color:red} hadoop-mapreduce-client-app in the patch failed with JDK
v1.7.0_85. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 15s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | hadoop.mapreduce.v2.app.TestMRAppMaster |
| JDK v1.7.0_85 Failed junit tests | hadoop.mapreduce.v2.app.TestMRAppMaster |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12774342/MAPREDUCE-6555.03.patch
|
| JIRA Issue | MAPREDUCE-6557 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux f93ab122ef64 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool |
[jira] [Updated] (MAPREDUCE-6542) HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe
[
https://issues.apache.org/jira/browse/MAPREDUCE-6542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangyubiao updated MAPREDUCE-6542:
---
Attachment: MAPREDUCE-6542-v6.patch
> HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe
> -
>
> Key: MAPREDUCE-6542
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6542
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 2.2.0, 2.7.1
> Environment: CentOS6.5 Hadoop
>Reporter: zhangyubiao
>Assignee: zhangyubiao
> Attachments: MAPREDUCE-6542-v2.patch, MAPREDUCE-6542-v3.patch,
> MAPREDUCE-6542-v4.patch, MAPREDUCE-6542-v5.patch, MAPREDUCE-6542-v6.patch,
> MAPREDUCE-6542.patch
>
>
> I use SimpleDateFormat to Parse the JobHistory File before
> {code}
> private static final SimpleDateFormat dateFormat =
> new SimpleDateFormat("-MM-dd HH:mm:ss");
> public static String getJobDetail(JobInfo job) {
> StringBuffer jobDetails = new StringBuffer("");
> SummarizedJob ts = new SummarizedJob(job);
> jobDetails.append(job.getJobId().toString().trim()).append("\t");
> jobDetails.append(job.getUsername()).append("\t");
> jobDetails.append(job.getJobname().replaceAll("\\n",
> "")).append("\t");
> jobDetails.append(job.getJobQueueName()).append("\t");
> jobDetails.append(job.getPriority()).append("\t");
> jobDetails.append(job.getJobConfPath()).append("\t");
> jobDetails.append(job.getUberized()).append("\t");
>
> jobDetails.append(dateFormat.format(job.getSubmitTime())).append("\t");
>
> jobDetails.append(dateFormat.format(job.getLaunchTime())).append("\t");
>
> jobDetails.append(dateFormat.format(job.getFinishTime())).append("\t");
>return jobDetails.toString();
> }
> {code}
> But I find I query the SubmitTime and LaunchTime in hive and compare
> JobHistory File time , I find that the submitTime and launchTime was wrong.
> Finally,I change to use the FastDateFormat to parse the time format and the
> time become right
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6557) Some tests in mapreduce-client-app are writing outside of target
[ https://issues.apache.org/jira/browse/MAPREDUCE-6557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026844#comment-15026844 ] Junping Du commented on MAPREDUCE-6557: --- The failed tests are unrelated and are tracked in MAPREDUCE-6555. +1. committing it in. > Some tests in mapreduce-client-app are writing outside of target > > > Key: MAPREDUCE-6557 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6557 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Akira AJISAKA >Priority: Blocker > Attachments: MAPREDUCE-6555.03.patch, MAPREDUCE-6557.00.patch > > > There is a staging directory appearing. It should not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6555) TestMRAppMaster fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026816#comment-15026816
]
Hadoop QA commented on MAPREDUCE-6555:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
57s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 31s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
49s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 10m 0s {color}
| {color:red} hadoop-mapreduce-client-app in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 10m 20s {color}
| {color:red} hadoop-mapreduce-client-app in the patch failed with JDK
v1.7.0_85. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 55s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | hadoop.mapreduce.v2.app.TestMRAppMaster |
| JDK v1.7.0_85 Failed junit tests | hadoop.mapreduce.v2.app.TestMRAppMaster |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12774341/MAPREDUCE-6555.02.patch
|
| JIRA Issue | MAPREDUCE-6555 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux e8afc7dd9088 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool |
[jira] [Commented] (MAPREDUCE-6553) Replace '\u2b05' with '<-' in rendering job configuration
[ https://issues.apache.org/jira/browse/MAPREDUCE-6553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026821#comment-15026821 ] Hudson commented on MAPREDUCE-6553: --- ABORTED: Integrated in Hadoop-Hdfs-trunk-Java8 #642 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/642/]) MAPREDUCE-6553. Replace '\u2b05' with '<-' in rendering job (aajisaka: rev 5ec44fc35c129d266edaebbc6fa17754a89b2ad6) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/ConfBlock.java > Replace '\u2b05' with '<-' in rendering job configuration > - > > Key: MAPREDUCE-6553 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6553 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 2.7.1 > Environment: Google Chrome 46.0.2490.86m with Windows 7 >Reporter: Akira AJISAKA >Assignee: Gabor Liptak >Priority: Minor > Labels: newbie > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6553.1.patch, Screen Shot 2015-11-25 at > 16.19.16.png, jobhistory.png > > > U+2B05 (leftwards black arrow) is not supported in some browsers, so we > should replace it with "<-". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6542) HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe
[
https://issues.apache.org/jira/browse/MAPREDUCE-6542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026837#comment-15026837
]
zhangyubiao commented on MAPREDUCE-6542:
[~templedf],Thanks a lot for review. I learn so much this days .
I update MAPREDUCE-6542-v6.patch
> HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe
> -
>
> Key: MAPREDUCE-6542
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6542
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 2.2.0, 2.7.1
> Environment: CentOS6.5 Hadoop
>Reporter: zhangyubiao
>Assignee: zhangyubiao
> Attachments: MAPREDUCE-6542-v2.patch, MAPREDUCE-6542-v3.patch,
> MAPREDUCE-6542-v4.patch, MAPREDUCE-6542-v5.patch, MAPREDUCE-6542.patch
>
>
> I use SimpleDateFormat to Parse the JobHistory File before
> {code}
> private static final SimpleDateFormat dateFormat =
> new SimpleDateFormat("-MM-dd HH:mm:ss");
> public static String getJobDetail(JobInfo job) {
> StringBuffer jobDetails = new StringBuffer("");
> SummarizedJob ts = new SummarizedJob(job);
> jobDetails.append(job.getJobId().toString().trim()).append("\t");
> jobDetails.append(job.getUsername()).append("\t");
> jobDetails.append(job.getJobname().replaceAll("\\n",
> "")).append("\t");
> jobDetails.append(job.getJobQueueName()).append("\t");
> jobDetails.append(job.getPriority()).append("\t");
> jobDetails.append(job.getJobConfPath()).append("\t");
> jobDetails.append(job.getUberized()).append("\t");
>
> jobDetails.append(dateFormat.format(job.getSubmitTime())).append("\t");
>
> jobDetails.append(dateFormat.format(job.getLaunchTime())).append("\t");
>
> jobDetails.append(dateFormat.format(job.getFinishTime())).append("\t");
>return jobDetails.toString();
> }
> {code}
> But I find I query the SubmitTime and LaunchTime in hive and compare
> JobHistory File time , I find that the submitTime and launchTime was wrong.
> Finally,I change to use the FastDateFormat to parse the time format and the
> time become right
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6550) archive-logs tool changes log ownership to the Yarn user when using DefaultContainerExecutor
[
https://issues.apache.org/jira/browse/MAPREDUCE-6550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15028187#comment-15028187
]
Hudson commented on MAPREDUCE-6550:
---
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2668 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2668/])
MAPREDUCE-6550. archive-logs tool changes log ownership to the Yarn user
(rkanter: rev 6d84cc16b3e0685fef01d0e3526b0f7556ceff51)
*
hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogs.java
* hadoop-tools/hadoop-archive-logs/src/site/markdown/HadoopArchiveLogs.md
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogsRunner.java
*
hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogsRunner.java
*
hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogs.java
> archive-logs tool changes log ownership to the Yarn user when using
> DefaultContainerExecutor
>
>
> Key: MAPREDUCE-6550
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6550
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: 2.8.0
>
> Attachments: MAPREDUCE-6550.001.patch, MAPREDUCE-6550.002.patch,
> MAPREDUCE-6550.003.patch, MAPREDUCE-6550.004.patch, MAPREDUCE-6550.005.patch
>
>
> The archive-logs tool added in MAPREDUCE-6415 leverages the Distributed Shell
> app. When using the DefaultContainerExecutor, this means that the job will
> actually run as the Yarn user, so the resulting har files are owned by the
> Yarn user instead of the original owner. The permissions are also now
> world-readable.
> In the below example, the archived logs are owned by 'yarn' instead of 'paul'
> and are now world-readable:
> {noformat}
> [root@gs28-centos66-5 ~]# sudo -u hdfs hdfs dfs -ls -R /tmp/logs
> ...
> drwxrwx--- - paul hadoop 0 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005
> drwxr-xr-x - yarn hadoop 0 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har
> -rw-r--r-- 3 yarn hadoop 0 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/_SUCCESS
> -rw-r--r-- 3 yarn hadoop 1256 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/_index
> -rw-r--r-- 3 yarn hadoop 24 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/_masterindex
> -rw-r--r-- 3 yarn hadoop8451177 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/part-0
> drwxrwx--- - paul hadoop 0 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0006
> -rw-r- 3 paul hadoop 1155 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0006/gs-centos66-2.vpc.cloudera.com_8041
> -rw-r- 3 paul hadoop 4880 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0006/gs28-centos66-3.vpc.cloudera.com_8041
> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6549) multibyte delimiters with LineRecordReader cause duplicate records
[ https://issues.apache.org/jira/browse/MAPREDUCE-6549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15028188#comment-15028188 ] Hudson commented on MAPREDUCE-6549: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2668 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2668/]) MAPREDUCE-6549. multibyte delimiters with LineRecordReader cause (rkanter: rev 7fd00b3db4b7d73afd41276ba9a06ec06a0e1762) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/UncompressedSplitLineReader.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestLineRecordReader.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestLineRecordReader.java > multibyte delimiters with LineRecordReader cause duplicate records > -- > > Key: MAPREDUCE-6549 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6549 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1, mrv2 >Affects Versions: 2.7.2 >Reporter: Dustin Cote >Assignee: Wilfred Spiegelenburg > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6549-1.patch, MAPREDUCE-6549-2.patch, > MAPREDUCE-6549.3.patch > > > LineRecorderReader currently produces duplicate records under certain > scenarios such as: > 1) input string: "abc+++def++ghi++" > delimiter string: "+++" > test passes with all sizes of the split > 2) input string: "abc++def+++ghi++" > delimiter string: "+++" > test fails with a split size of 4 > 2) input string: "abc+++def++ghi++" > delimiter string: "++" > test fails with a split size of 5 > 3) input string "abc+++defg++hij++" > delimiter string: "++" > test fails with a split size of 4 > 4) input string "abc++def+++ghi++" > delimiter string: "++" > test fails with a split size of 9 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6549) multibyte delimiters with LineRecordReader cause duplicate records
[ https://issues.apache.org/jira/browse/MAPREDUCE-6549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15028232#comment-15028232 ] Hudson commented on MAPREDUCE-6549: --- SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #737 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/737/]) MAPREDUCE-6549. multibyte delimiters with LineRecordReader cause (rkanter: rev 7fd00b3db4b7d73afd41276ba9a06ec06a0e1762) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/UncompressedSplitLineReader.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestLineRecordReader.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestLineRecordReader.java * hadoop-mapreduce-project/CHANGES.txt > multibyte delimiters with LineRecordReader cause duplicate records > -- > > Key: MAPREDUCE-6549 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6549 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1, mrv2 >Affects Versions: 2.7.2 >Reporter: Dustin Cote >Assignee: Wilfred Spiegelenburg > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6549-1.patch, MAPREDUCE-6549-2.patch, > MAPREDUCE-6549.3.patch > > > LineRecorderReader currently produces duplicate records under certain > scenarios such as: > 1) input string: "abc+++def++ghi++" > delimiter string: "+++" > test passes with all sizes of the split > 2) input string: "abc++def+++ghi++" > delimiter string: "+++" > test fails with a split size of 4 > 2) input string: "abc+++def++ghi++" > delimiter string: "++" > test fails with a split size of 5 > 3) input string "abc+++defg++hij++" > delimiter string: "++" > test fails with a split size of 4 > 4) input string "abc++def+++ghi++" > delimiter string: "++" > test fails with a split size of 9 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6550) archive-logs tool changes log ownership to the Yarn user when using DefaultContainerExecutor
[
https://issues.apache.org/jira/browse/MAPREDUCE-6550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15028230#comment-15028230
]
Hudson commented on MAPREDUCE-6550:
---
SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #737 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/737/])
MAPREDUCE-6550. archive-logs tool changes log ownership to the Yarn user
(rkanter: rev 6d84cc16b3e0685fef01d0e3526b0f7556ceff51)
* hadoop-tools/hadoop-archive-logs/src/site/markdown/HadoopArchiveLogs.md
*
hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogsRunner.java
*
hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogs.java
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogsRunner.java
*
hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogs.java
> archive-logs tool changes log ownership to the Yarn user when using
> DefaultContainerExecutor
>
>
> Key: MAPREDUCE-6550
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6550
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: 2.8.0
>
> Attachments: MAPREDUCE-6550.001.patch, MAPREDUCE-6550.002.patch,
> MAPREDUCE-6550.003.patch, MAPREDUCE-6550.004.patch, MAPREDUCE-6550.005.patch
>
>
> The archive-logs tool added in MAPREDUCE-6415 leverages the Distributed Shell
> app. When using the DefaultContainerExecutor, this means that the job will
> actually run as the Yarn user, so the resulting har files are owned by the
> Yarn user instead of the original owner. The permissions are also now
> world-readable.
> In the below example, the archived logs are owned by 'yarn' instead of 'paul'
> and are now world-readable:
> {noformat}
> [root@gs28-centos66-5 ~]# sudo -u hdfs hdfs dfs -ls -R /tmp/logs
> ...
> drwxrwx--- - paul hadoop 0 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005
> drwxr-xr-x - yarn hadoop 0 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har
> -rw-r--r-- 3 yarn hadoop 0 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/_SUCCESS
> -rw-r--r-- 3 yarn hadoop 1256 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/_index
> -rw-r--r-- 3 yarn hadoop 24 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/_masterindex
> -rw-r--r-- 3 yarn hadoop8451177 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/part-0
> drwxrwx--- - paul hadoop 0 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0006
> -rw-r- 3 paul hadoop 1155 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0006/gs-centos66-2.vpc.cloudera.com_8041
> -rw-r- 3 paul hadoop 4880 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0006/gs28-centos66-3.vpc.cloudera.com_8041
> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6549) multibyte delimiters with LineRecordReader cause duplicate records
[ https://issues.apache.org/jira/browse/MAPREDUCE-6549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15028162#comment-15028162 ] Hudson commented on MAPREDUCE-6549: --- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #645 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/645/]) MAPREDUCE-6549. multibyte delimiters with LineRecordReader cause (rkanter: rev 7fd00b3db4b7d73afd41276ba9a06ec06a0e1762) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/UncompressedSplitLineReader.java * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestLineRecordReader.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestLineRecordReader.java > multibyte delimiters with LineRecordReader cause duplicate records > -- > > Key: MAPREDUCE-6549 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6549 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1, mrv2 >Affects Versions: 2.7.2 >Reporter: Dustin Cote >Assignee: Wilfred Spiegelenburg > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6549-1.patch, MAPREDUCE-6549-2.patch, > MAPREDUCE-6549.3.patch > > > LineRecorderReader currently produces duplicate records under certain > scenarios such as: > 1) input string: "abc+++def++ghi++" > delimiter string: "+++" > test passes with all sizes of the split > 2) input string: "abc++def+++ghi++" > delimiter string: "+++" > test fails with a split size of 4 > 2) input string: "abc+++def++ghi++" > delimiter string: "++" > test fails with a split size of 5 > 3) input string "abc+++defg++hij++" > delimiter string: "++" > test fails with a split size of 4 > 4) input string "abc++def+++ghi++" > delimiter string: "++" > test fails with a split size of 9 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6550) archive-logs tool changes log ownership to the Yarn user when using DefaultContainerExecutor
[
https://issues.apache.org/jira/browse/MAPREDUCE-6550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15028159#comment-15028159
]
Hudson commented on MAPREDUCE-6550:
---
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #645 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/645/])
MAPREDUCE-6550. archive-logs tool changes log ownership to the Yarn user
(rkanter: rev 6d84cc16b3e0685fef01d0e3526b0f7556ceff51)
*
hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogsRunner.java
* hadoop-tools/hadoop-archive-logs/src/site/markdown/HadoopArchiveLogs.md
*
hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogs.java
*
hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogsRunner.java
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogs.java
> archive-logs tool changes log ownership to the Yarn user when using
> DefaultContainerExecutor
>
>
> Key: MAPREDUCE-6550
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6550
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: 2.8.0
>
> Attachments: MAPREDUCE-6550.001.patch, MAPREDUCE-6550.002.patch,
> MAPREDUCE-6550.003.patch, MAPREDUCE-6550.004.patch, MAPREDUCE-6550.005.patch
>
>
> The archive-logs tool added in MAPREDUCE-6415 leverages the Distributed Shell
> app. When using the DefaultContainerExecutor, this means that the job will
> actually run as the Yarn user, so the resulting har files are owned by the
> Yarn user instead of the original owner. The permissions are also now
> world-readable.
> In the below example, the archived logs are owned by 'yarn' instead of 'paul'
> and are now world-readable:
> {noformat}
> [root@gs28-centos66-5 ~]# sudo -u hdfs hdfs dfs -ls -R /tmp/logs
> ...
> drwxrwx--- - paul hadoop 0 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005
> drwxr-xr-x - yarn hadoop 0 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har
> -rw-r--r-- 3 yarn hadoop 0 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/_SUCCESS
> -rw-r--r-- 3 yarn hadoop 1256 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/_index
> -rw-r--r-- 3 yarn hadoop 24 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/_masterindex
> -rw-r--r-- 3 yarn hadoop8451177 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/part-0
> drwxrwx--- - paul hadoop 0 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0006
> -rw-r- 3 paul hadoop 1155 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0006/gs-centos66-2.vpc.cloudera.com_8041
> -rw-r- 3 paul hadoop 4880 2015-10-02 13:24
> /tmp/logs/paul/logs/application_1443805425363_0006/gs28-centos66-3.vpc.cloudera.com_8041
> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
