[jira] [Created] (MAPREDUCE-5395) Update Teragen algorithm
Thomas Graves created MAPREDUCE-5395: Summary: Update Teragen algorithm Key: MAPREDUCE-5395 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5395 Project: Hadoop Map/Reduce Issue Type: Improvement Components: examples Affects Versions: 0.23.7 Reporter: Thomas Graves The Teragen algorithm is no longer up to date with the sortbenchmark.org gensort tool used for the official sort benchmark. The new algorithm is supposed to generate data that isn't very compressible. Also the new version of gensort can generate skewed data so we should add that option to teragen also. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-5317) Stale files left behind for failed jobs
[
https://issues.apache.org/jira/browse/MAPREDUCE-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe reopened MAPREDUCE-5317:
---
This is causing some unit tests to hang/fail in
hadoop-mapreduce-client-jobclient. TestJobCleanup on 0.23, for example. I
missed the fact that when JobImpl short-circuits the FAIL_WAIT state and goes
straight from RUNNING to FAIL_ABORT it is failing to send the committer event
that will eventually trigger it to leave the FAIL_ABORT state. That causes the
AM to just hang in the FAIL_ABORT state.
Ravi, could you add that fix along with a test case for that scenario? In the
meantime I'll back out this change to fix the builds.
Stale files left behind for failed jobs
---
Key: MAPREDUCE-5317
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5317
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 3.0.0, 2.0.4-alpha, 0.23.8
Reporter: Ravi Prakash
Assignee: Ravi Prakash
Fix For: 3.0.0, 2.3.0, 0.23.10
Attachments: MAPREDUCE-5317.branch-0.23.patch,
MAPREDUCE-5317.branch-0.23.patch, MAPREDUCE-5317.patch, MAPREDUCE-5317.patch,
MAPREDUCE-5317.patch, MAPREDUCE-5317.patch, MAPREDUCE-5317.patch
Courtesy [~amar_kamat]!
{quote}
We are seeing _temporary files left behind in the output folder if the job
fails.
The job were failed due to hitting quota issue.
I simply ran the randomwriter (from hadoop examples) with the default setting.
That failed and left behind some stray files.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Hadoop-Mapreduce-trunk - Build # 1489 - Still Failing
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1489/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 31258 lines...] Running org.apache.hadoop.mapreduce.lib.partition.TestBinaryPartitioner Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.484 sec Running org.apache.hadoop.mapreduce.lib.partition.TestMRKeyFieldBasedPartitioner Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.416 sec Running org.apache.hadoop.mapreduce.TestChild Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 60.552 sec Running org.apache.hadoop.mapreduce.filecache.TestURIFragments Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.062 sec Running org.apache.hadoop.mapreduce.TestMapReduce Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.799 sec Results : Failed tests: testJobFail(org.apache.hadoop.mapreduce.v2.TestMROldApiJobs) Tests run: 460, Failures: 1, Errors: 0, Skipped: 11 [INFO] [INFO] Reactor Summary: [INFO] [INFO] hadoop-mapreduce-client ... SUCCESS [1.703s] [INFO] hadoop-mapreduce-client-core .. SUCCESS [36.761s] [INFO] hadoop-mapreduce-client-common SUCCESS [23.801s] [INFO] hadoop-mapreduce-client-shuffle ... SUCCESS [2.323s] [INFO] hadoop-mapreduce-client-app ... SUCCESS [5:46.161s] [INFO] hadoop-mapreduce-client-hs SUCCESS [1:25.313s] [INFO] hadoop-mapreduce-client-jobclient . FAILURE [1:09:13.868s] [INFO] hadoop-mapreduce-client-hs-plugins SKIPPED [INFO] Apache Hadoop MapReduce Examples .. SKIPPED [INFO] hadoop-mapreduce .. SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 1:17:30.570s [INFO] Finished at: Tue Jul 16 14:40:24 UTC 2013 [INFO] Final Memory: 41M/66M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test) on project hadoop-mapreduce-client-jobclient: ExecutionException; nested exception is java.util.concurrent.ExecutionException: java.lang.RuntimeException: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ? - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn goals -rf :hadoop-mapreduce-client-jobclient Build step 'Execute shell' marked build as failure [FINDBUGS] Skipping publisher since build result is FAILURE Archiving artifacts Updating HDFS-4903 Updating YARN-654 Updating HADOOP-9707 Updating YARN-909 Updating YARN-523 Updating YARN-521 Updating MAPREDUCE-5317 Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## No tests ran.
[jira] [Created] (MAPREDUCE-5397) AM crashes because Webapp failed to start on multi node cluster
Jian He created MAPREDUCE-5397:
--
Summary: AM crashes because Webapp failed to start on multi node
cluster
Key: MAPREDUCE-5397
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5397
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Jian He
Assignee: Jian He
I set up a 12 nodes cluster and tried submitting jobs but get this exception.
But job is able to succeed after AM crashes and retry a few times(2 or 3)
{code}
2013-07-12 18:56:28,438 INFO [main] org.mortbay.log: Extract
jar:file:/grid/0/dev/jhe/hadoop-2.1.0-beta/share/hadoop/yarn/hadoop-yarn-common-2.1.0-beta.jar!/webapps/mapreduce
to /tmp/Jetty_0_0_0_0_43554_mapreduceljbmlg/webapp
2013-07-12 18:56:28,528 WARN [main] org.mortbay.log: Failed startup of context
org.mortbay.jetty.webapp.WebAppContext@2726b2{/,jar:file:/grid/0/dev/jhe/hadoop-2.1.0-beta/share/hadoop/yarn/hadoop-yarn-common-2.1.0-beta.jar!/webapps/mapreduce}
java.io.FileNotFoundException:
/tmp/Jetty_0_0_0_0_43554_mapreduceljbmlg/webapp/webapps/mapreduce/.keep (No
such file or directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.init(FileOutputStream.java:194)
at java.io.FileOutputStream.init(FileOutputStream.java:145)
at org.mortbay.resource.JarResource.extract(JarResource.java:215)
at
org.mortbay.jetty.webapp.WebAppContext.resolveWebApp(WebAppContext.java:974)
at
org.mortbay.jetty.webapp.WebAppContext.getWebInf(WebAppContext.java:832)
at
org.mortbay.jetty.webapp.WebInfConfiguration.configureClassLoader(WebInfConfiguration.java:62)
at
org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:489)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at
org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
at
org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at
org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
at org.mortbay.jetty.Server.doStart(Server.java:224)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.apache.hadoop.http.HttpServer.start(HttpServer.java:684)
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:211)
at
org.apache.hadoop.mapreduce.v2.app.client.MRClientService.serviceStart(MRClientService.java:134)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:101)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1019)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1394)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1390)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5398) MR changes for YARN-513
Bikas Saha created MAPREDUCE-5398: - Summary: MR changes for YARN-513 Key: MAPREDUCE-5398 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5398 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Bikas Saha Assignee: Jian He -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5398) MR changes for YARN-513
[ https://issues.apache.org/jira/browse/MAPREDUCE-5398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha resolved MAPREDUCE-5398. --- Resolution: Fixed Fix Version/s: 2.1.0-beta Target Version/s: 2.1.0-beta Hadoop Flags: Reviewed Committed to trunk, branch-2 and branch-2.1-beta. Thanks Jian! MR changes for YARN-513 --- Key: MAPREDUCE-5398 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5398 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Bikas Saha Assignee: Jian He Fix For: 2.1.0-beta Attachments: MAPREDUCE-5398.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
