[jira] [Updated] (MAPREDUCE-6165) [JDK8] TestCombineFileInputFormat failed on JDK8
[
https://issues.apache.org/jira/browse/MAPREDUCE-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated MAPREDUCE-6165:
-
Fix Version/s: (was: 2.7.5)
2.7.6
> [JDK8] TestCombineFileInputFormat failed on JDK8
>
>
> Key: MAPREDUCE-6165
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6165
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Wei Yan
>Assignee: Akira Ajisaka
>Priority: Minor
> Fix For: 2.8.0, 3.0.0-alpha1, 2.7.6
>
> Attachments: MAPREDUCE-6165-001.patch, MAPREDUCE-6165-002.patch,
> MAPREDUCE-6165-003.patch, MAPREDUCE-6165-003.patch, MAPREDUCE-6165-004.patch,
> MAPREDUCE-6165-reproduce.patch
>
>
> The error msg:
> {noformat}
> testSplitPlacementForCompressedFiles(org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat)
> Time elapsed: 2.487 sec <<< FAILURE!
> junit.framework.AssertionFailedError: expected:<2> but was:<1>
> at junit.framework.Assert.fail(Assert.java:57)
> at junit.framework.Assert.failNotEquals(Assert.java:329)
> at junit.framework.Assert.assertEquals(Assert.java:78)
> at junit.framework.Assert.assertEquals(Assert.java:234)
> at junit.framework.Assert.assertEquals(Assert.java:241)
> at junit.framework.TestCase.assertEquals(TestCase.java:409)
> at
> org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testSplitPlacementForCompressedFiles(TestCombineFileInputFormat.java:911)
> testSplitPlacement(org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat)
> Time elapsed: 0.985 sec <<< FAILURE!
> junit.framework.AssertionFailedError: expected:<2> but was:<1>
> at junit.framework.Assert.fail(Assert.java:57)
> at junit.framework.Assert.failNotEquals(Assert.java:329)
> at junit.framework.Assert.assertEquals(Assert.java:78)
> at junit.framework.Assert.assertEquals(Assert.java:234)
> at junit.framework.Assert.assertEquals(Assert.java:241)
> at junit.framework.TestCase.assertEquals(TestCase.java:409)
> at
> org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testSplitPlacement(TestCombineFileInputFormat.java:368)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6165) [JDK8] TestCombineFileInputFormat failed on JDK8
[
https://issues.apache.org/jira/browse/MAPREDUCE-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated MAPREDUCE-6165:
-
Fix Version/s: 2.7.5
> [JDK8] TestCombineFileInputFormat failed on JDK8
>
>
> Key: MAPREDUCE-6165
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6165
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Wei Yan
>Assignee: Akira Ajisaka
>Priority: Minor
> Fix For: 2.8.0, 3.0.0-alpha1, 2.7.5
>
> Attachments: MAPREDUCE-6165-001.patch, MAPREDUCE-6165-002.patch,
> MAPREDUCE-6165-003.patch, MAPREDUCE-6165-003.patch, MAPREDUCE-6165-004.patch,
> MAPREDUCE-6165-reproduce.patch
>
>
> The error msg:
> {noformat}
> testSplitPlacementForCompressedFiles(org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat)
> Time elapsed: 2.487 sec <<< FAILURE!
> junit.framework.AssertionFailedError: expected:<2> but was:<1>
> at junit.framework.Assert.fail(Assert.java:57)
> at junit.framework.Assert.failNotEquals(Assert.java:329)
> at junit.framework.Assert.assertEquals(Assert.java:78)
> at junit.framework.Assert.assertEquals(Assert.java:234)
> at junit.framework.Assert.assertEquals(Assert.java:241)
> at junit.framework.TestCase.assertEquals(TestCase.java:409)
> at
> org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testSplitPlacementForCompressedFiles(TestCombineFileInputFormat.java:911)
> testSplitPlacement(org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat)
> Time elapsed: 0.985 sec <<< FAILURE!
> junit.framework.AssertionFailedError: expected:<2> but was:<1>
> at junit.framework.Assert.fail(Assert.java:57)
> at junit.framework.Assert.failNotEquals(Assert.java:329)
> at junit.framework.Assert.assertEquals(Assert.java:78)
> at junit.framework.Assert.assertEquals(Assert.java:234)
> at junit.framework.Assert.assertEquals(Assert.java:241)
> at junit.framework.TestCase.assertEquals(TestCase.java:409)
> at
> org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testSplitPlacement(TestCombineFileInputFormat.java:368)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6165) [JDK8] TestCombineFileInputFormat failed on JDK8
[
https://issues.apache.org/jira/browse/MAPREDUCE-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279201#comment-16279201
]
Zhe Zhang commented on MAPREDUCE-6165:
--
I just backported to branc-2.7.
> [JDK8] TestCombineFileInputFormat failed on JDK8
>
>
> Key: MAPREDUCE-6165
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6165
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Wei Yan
>Assignee: Akira Ajisaka
>Priority: Minor
> Fix For: 2.8.0, 3.0.0-alpha1, 2.7.5
>
> Attachments: MAPREDUCE-6165-001.patch, MAPREDUCE-6165-002.patch,
> MAPREDUCE-6165-003.patch, MAPREDUCE-6165-003.patch, MAPREDUCE-6165-004.patch,
> MAPREDUCE-6165-reproduce.patch
>
>
> The error msg:
> {noformat}
> testSplitPlacementForCompressedFiles(org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat)
> Time elapsed: 2.487 sec <<< FAILURE!
> junit.framework.AssertionFailedError: expected:<2> but was:<1>
> at junit.framework.Assert.fail(Assert.java:57)
> at junit.framework.Assert.failNotEquals(Assert.java:329)
> at junit.framework.Assert.assertEquals(Assert.java:78)
> at junit.framework.Assert.assertEquals(Assert.java:234)
> at junit.framework.Assert.assertEquals(Assert.java:241)
> at junit.framework.TestCase.assertEquals(TestCase.java:409)
> at
> org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testSplitPlacementForCompressedFiles(TestCombineFileInputFormat.java:911)
> testSplitPlacement(org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat)
> Time elapsed: 0.985 sec <<< FAILURE!
> junit.framework.AssertionFailedError: expected:<2> but was:<1>
> at junit.framework.Assert.fail(Assert.java:57)
> at junit.framework.Assert.failNotEquals(Assert.java:329)
> at junit.framework.Assert.assertEquals(Assert.java:78)
> at junit.framework.Assert.assertEquals(Assert.java:234)
> at junit.framework.Assert.assertEquals(Assert.java:241)
> at junit.framework.TestCase.assertEquals(TestCase.java:409)
> at
> org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testSplitPlacement(TestCombineFileInputFormat.java:368)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-6937) Backport MAPREDUCE-6870 to branch-2 while preserving compatibility
[ https://issues.apache.org/jira/browse/MAPREDUCE-6937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang reassigned MAPREDUCE-6937: Assignee: Peter Bacsko (was: Erik Krogen) > Backport MAPREDUCE-6870 to branch-2 while preserving compatibility > -- > > Key: MAPREDUCE-6937 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6937 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Zhe Zhang >Assignee: Peter Bacsko > Attachments: MAPREDUCE-6870-branch-2.01.patch, > MAPREDUCE-6870-branch-2.7.03.patch, MAPREDUCE-6870-branch-2.7.04.patch, > MAPREDUCE-6870-branch-2.7.05.patch, MAPREDUCE-6870_branch2.7.patch, > MAPREDUCE-6870_branch2.7v2.patch, MAPREDUCE-6870-branch-2.8.03.patch, > MAPREDUCE-6870-branch-2.8.04.patch, MAPREDUCE-6870_branch2.8.patch, > MAPREDUCE-6870_branch2.8v2.patch > > > To maintain compatibility we need to disable this by default per discussion > on MAPREDUCE-6870. > Using a separate JIRA to correctly track incompatibilities. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6937) Backport MAPREDUCE-6870 to branch-2 while preserving compatibility
Zhe Zhang created MAPREDUCE-6937: Summary: Backport MAPREDUCE-6870 to branch-2 while preserving compatibility Key: MAPREDUCE-6937 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6937 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Zhe Zhang Assignee: Erik Krogen -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6433) launchTime may be negative
[ https://issues.apache.org/jira/browse/MAPREDUCE-6433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated MAPREDUCE-6433: - Labels: (was: release-blocker) > launchTime may be negative > -- > > Key: MAPREDUCE-6433 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6433 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, mrv2 >Affects Versions: 2.4.1 >Reporter: Allen Wittenauer >Assignee: zhihai xu > Fix For: 2.8.0, 2.7.4, 3.0.0-alpha1 > > Attachments: MAPREDUCE-6433.000.patch, MAPREDUCE-6433.001.patch, > MAPREDUCE-6433-branch-2.7.001.patch, REPRODUCE.patch > > > Under extremely rare conditions (.0017% in our sample size), launchTime in > the jhist files may be set to -1. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6873) MR Job Submission Fails if MR framework application path not on defaultFS
[
https://issues.apache.org/jira/browse/MAPREDUCE-6873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated MAPREDUCE-6873:
-
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-beta1
Thanks [~xkrogen] for the fix. +1 on the patch. I just committed to trunk,
working on backporting to branch-2.7 and up.
> MR Job Submission Fails if MR framework application path not on defaultFS
> -
>
> Key: MAPREDUCE-6873
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6873
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.6.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Minor
> Fix For: 3.0.0-beta1
>
> Attachments: MAPREDUCE-6873.000.patch
>
>
> {{JobSubmitter#addMRFrameworkPathToDistributedCache()}} assumes that
> {{mapreduce.framework.application.path}} has a FS which matches
> {{fs.defaultFS}} which may not always be true. This is just a consequence of
> using {{FileSystem.get(Configuration)}} instead of {{FileSystem.get(URI,
> Configuration)}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6873) MR Job Submission Fails if MR framework application path not on defaultFS
[
https://issues.apache.org/jira/browse/MAPREDUCE-6873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated MAPREDUCE-6873:
-
Resolution: Fixed
Fix Version/s: 2.8.1
2.7.4
2.9.0
Status: Resolved (was: Patch Available)
Committed to trunk~branch-2.7. Thanks Erik for the contribution.
> MR Job Submission Fails if MR framework application path not on defaultFS
> -
>
> Key: MAPREDUCE-6873
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6873
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.6.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Minor
> Fix For: 2.9.0, 2.7.4, 2.8.1, 3.0.0-beta1
>
> Attachments: MAPREDUCE-6873.000.patch
>
>
> {{JobSubmitter#addMRFrameworkPathToDistributedCache()}} assumes that
> {{mapreduce.framework.application.path}} has a FS which matches
> {{fs.defaultFS}} which may not always be true. This is just a consequence of
> using {{FileSystem.get(Configuration)}} instead of {{FileSystem.get(URI,
> Configuration)}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-6873) MR Job Submission Fails if MR framework application path not on defaultFS
[
https://issues.apache.org/jira/browse/MAPREDUCE-6873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang reassigned MAPREDUCE-6873:
Assignee: Erik Krogen
> MR Job Submission Fails if MR framework application path not on defaultFS
> -
>
> Key: MAPREDUCE-6873
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6873
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.6.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Minor
> Attachments: MAPREDUCE-6873.000.patch
>
>
> {{JobSubmitter#addMRFrameworkPathToDistributedCache()}} assumes that
> {{mapreduce.framework.application.path}} has a FS which matches
> {{fs.defaultFS}} which may not always be true. This is just a consequence of
> using {{FileSystem.get(Configuration)}} instead of {{FileSystem.get(URI,
> Configuration)}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6870) Add configuration for MR job to finish when all reducers are complete (even with unfinished mappers)
Zhe Zhang created MAPREDUCE-6870: Summary: Add configuration for MR job to finish when all reducers are complete (even with unfinished mappers) Key: MAPREDUCE-6870 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6870 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.6.1 Reporter: Zhe Zhang Even with MAPREDUCE-5817, there could still be cases where mappers get scheduled before all reducers are complete, but those mappers run for long time, even after all reducers are complete. This could hurt the performance of large MR jobs. In some cases, mappers don't have any materialize-able outcome other than providing intermediate data to reducers. In that case, the job owner should have the config option to finish the job once all reducers are complete. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6860) User intermediate-done-dir permissions should use history file permissions configuration
[
https://issues.apache.org/jira/browse/MAPREDUCE-6860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated MAPREDUCE-6860:
-
Description: (was: Currently {{JobHistoryEventHandler}} creates the
user intermediate-done-dir directory here: {noformat} doneDirPrefixPath =
FileContext.getFileContext(conf).makeQualified(new
Path(userDoneDirStr));
mkdir(doneDirFS, doneDirPrefixPath, new FsPermission(
JobHistoryUtils.HISTORY_INTERMEDIATE_USER_DIR_PERMISSIONS));{noformat} which is
hardcoded to 770. But the summary, history, and conf files under this user dir
are configurable via
{{mapreduce.jobhistory.intermediate-done-dir.file.permission}}. So if the
configured permissions has read/write/execute permissions for "other" users,
they will still not have access to these files due to the 770 permission on the
user dir.
I see two options here:
# Reuse {{mapreduce.jobhistory.intermediate-done-dir.file.permission}} as the
permissions for the user dir
# Create a new config for the user dir permissions, using 770 as the default
The latter makes more sense to me.)
> User intermediate-done-dir permissions should use history file permissions
> configuration
>
>
> Key: MAPREDUCE-6860
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6860
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Jonathan Hung
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
