[jira] [Created] (MAPREDUCE-6113) An ability to override LocalResourceVisibility for public resources
Andrey Stepachev created MAPREDUCE-6113: --- Summary: An ability to override LocalResourceVisibility for public resources Key: MAPREDUCE-6113 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6113 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client, distributed-cache Reporter: Andrey Stepachev Assignee: Andrey Stepachev Priority: Minor Currently it is impossible to specify application level for files in distributed cache in YARN. For applications with big common data it is very crucial to have ability to share data only for application and not spam public cache. Moreover, public localiser is limited for parallel downloads, so applications with big caches affect application startup of other applications. Current logic is complicated and based on file permissions. Making proposed changes, user can override public level to any level (PUBLIC/APP/PRIVATE) without breaking anything, and even not need to bother with permission, if private cache is needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Hadoop-Mapreduce-trunk - Build # 1911 - Still Failing
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1911/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 28871 lines...] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.602 sec - in org.apache.hadoop.mapred.TestSequenceFileAsBinaryOutputFormat Results : Failed tests: TestJavaSerialization.testMapReduceJob:127 expected:[a ]1 but was:[0 1]1 TestPipeApplication.testSubmitter:310 null Tests in error: TestMRCJCFileInputFormat.testAddInputPath:55 ? IO No FileSystem for scheme: s3 Tests run: 508, Failures: 2, Errors: 1, Skipped: 11 [INFO] [INFO] Reactor Summary: [INFO] [INFO] hadoop-mapreduce-client ... SUCCESS [ 2.615 s] [INFO] hadoop-mapreduce-client-core .. SUCCESS [ 57.485 s] [INFO] hadoop-mapreduce-client-common SUCCESS [ 27.163 s] [INFO] hadoop-mapreduce-client-shuffle ... SUCCESS [ 4.412 s] [INFO] hadoop-mapreduce-client-app ... SUCCESS [08:45 min] [INFO] hadoop-mapreduce-client-hs SUCCESS [05:11 min] [INFO] hadoop-mapreduce-client-jobclient . FAILURE [ 01:32 h] [INFO] hadoop-mapreduce-client-hs-plugins SKIPPED [INFO] hadoop-mapreduce-client-nativetask SKIPPED [INFO] Apache Hadoop MapReduce Examples .. SKIPPED [INFO] hadoop-mapreduce .. SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 01:48 h [INFO] Finished at: 2014-09-29T15:11:10+00:00 [INFO] Final Memory: 41M/600M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.16:test (default-test) on project hadoop-mapreduce-client-jobclient: There are test failures. [ERROR] [ERROR] Please refer to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports for the individual test results. [ERROR] - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn goals -rf :hadoop-mapreduce-client-jobclient Build step 'Execute shell' marked build as failure [FINDBUGS] Skipping publisher since build result is FAILURE Archiving artifacts Sending artifact delta relative to Hadoop-Mapreduce-trunk #1870 Archived 2 artifacts Archive block size is 32768 Received 0 blocks and 20207052 bytes Compression is 0.0% Took 14 sec Updating HDFS-7093 Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## No tests ran.
[jira] [Created] (MAPREDUCE-6114) TestMRCJCFileInputFormat fails in trunk
Ted Yu created MAPREDUCE-6114:
-
Summary: TestMRCJCFileInputFormat fails in trunk
Key: MAPREDUCE-6114
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6114
Project: Hadoop Map/Reduce
Issue Type: Test
Reporter: Ted Yu
Priority: Minor
This can be reproduced locally:
{code}
Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.474 sec
FAILURE! - in org.apache.hadoop.mapreduce.lib.input.TestMRCJCFileInputFormat
testAddInputPath(org.apache.hadoop.mapreduce.lib.input.TestMRCJCFileInputFormat)
Time elapsed: 0.86 sec ERROR!
java.io.IOException: No FileSystem for scheme: s3
at
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2583)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2590)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2629)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2611)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
at
org.apache.hadoop.mapreduce.lib.input.TestMRCJCFileInputFormat.testAddInputPath(TestMRCJCFileInputFormat.java:55)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6115) TestPipeApplication#testSubmitter fails in trunk
Ted Yu created MAPREDUCE-6115:
-
Summary: TestPipeApplication#testSubmitter fails in trunk
Key: MAPREDUCE-6115
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6115
Project: Hadoop Map/Reduce
Issue Type: Test
Reporter: Ted Yu
Priority: Minor
This can be reproduced locally:
{code}
Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.89 sec
FAILURE! - in org.apache.hadoop.mapred.pipes.TestPipeApplication
testSubmitter(org.apache.hadoop.mapred.pipes.TestPipeApplication) Time
elapsed: 0.061 sec FAILURE!
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.mapred.pipes.TestPipeApplication.testSubmitter(TestPipeApplication.java:310)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-6114) TestMRCJCFileInputFormat#testAddInputPath fails in trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe resolved MAPREDUCE-6114.
---
Resolution: Duplicate
Dup of MAPREDUCE-6094.
TestMRCJCFileInputFormat#testAddInputPath fails in trunk
Key: MAPREDUCE-6114
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6114
Project: Hadoop Map/Reduce
Issue Type: Test
Reporter: Ted Yu
Priority: Minor
This can be reproduced locally:
{code}
Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.474 sec
FAILURE! - in org.apache.hadoop.mapreduce.lib.input.TestMRCJCFileInputFormat
testAddInputPath(org.apache.hadoop.mapreduce.lib.input.TestMRCJCFileInputFormat)
Time elapsed: 0.86 sec ERROR!
java.io.IOException: No FileSystem for scheme: s3
at
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2583)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2590)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2629)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2611)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
at
org.apache.hadoop.mapreduce.lib.input.TestMRCJCFileInputFormat.testAddInputPath(TestMRCJCFileInputFormat.java:55)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
MAPREDUCE 6099
Hi, I have added a new api to org.apache.hadoop.mapreduce.lib.input.CombinFileInputFormat getSplits(JobContext, ListFileStatus). Link to the JIRA : https://issues.apache.org/jira/browse/MAPREDUCE-6099 Could anyone please review the patch ? It is kind of a blocker for a patch to hive that I need to submit. Thanks, Pankit
