[jira] [Created] (MAPREDUCE-4573) Hadoop benckmarking for NutchIndexing
prashanthi created MAPREDUCE-4573: - Summary: Hadoop benckmarking for NutchIndexing Key: MAPREDUCE-4573 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4573 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: prashanthi Hi, i have downloaded nutch indexing program ffom GitHub and trying to run. For that the input file is of size 16 GB which am nt able to download. Ccn anyone help me on if there is any other alternative to run this program or if anyone having scledown version of the input jar. Is it possible to give a text file of some URL as the input for the NutchIndexing Thanks, Prashanthi -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4574) Fix TotalOrderParitioner to work with non-WritableComparable key types
Harsh J created MAPREDUCE-4574:
--
Summary: Fix TotalOrderParitioner to work with
non-WritableComparable key types
Key: MAPREDUCE-4574
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4574
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
The current TotalOrderPartitioner class will not work with an alternative
serialization library such as Avro.
To make it work, we may edit the readPartitions bits in it to support
non-WritableComparable keys and also remove the WritableComparable check in the
class types definition.
That is, since we do not use the values at all (NullWritable), we may as well
do:
{code}
private K[] readPartitions(FileSystem fs, Path p, ClassK keyClass,
Configuration conf) throws IOException {
…
while ((key = (K) reader.next(key)) != null) {
parts.add(key);
key = ReflectionUtils.newInstance(keyClass, conf);
}
…
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
Hadoop-Mapreduce-trunk - Build # 1174 - Still Failing
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1174/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 30206 lines...] Running org.apache.hadoop.mapreduce.TestChild Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 53.36 sec Running org.apache.hadoop.mapreduce.filecache.TestURIFragments Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.06 sec Running org.apache.hadoop.mapreduce.TestMapReduce Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.49 sec Results : Failed tests: testForEmptyFile(org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat): expected:0 but was:1 Tests in error: testMissingBlocks(org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat): org.apache.hadoop.fs.BlockLocation Tests run: 384, Failures: 1, Errors: 1, Skipped: 14 [INFO] [INFO] Reactor Summary: [INFO] [INFO] hadoop-mapreduce-client ... SUCCESS [2.390s] [INFO] hadoop-mapreduce-client-core .. SUCCESS [1:26.592s] [INFO] hadoop-mapreduce-client-common SUCCESS [21.231s] [INFO] hadoop-mapreduce-client-shuffle ... SUCCESS [1.065s] [INFO] hadoop-mapreduce-client-app ... SUCCESS [4:26.469s] [INFO] hadoop-mapreduce-client-hs SUCCESS [1:06.386s] [INFO] hadoop-mapreduce-client-jobclient . FAILURE [35:47.379s] [INFO] Apache Hadoop MapReduce Examples .. SKIPPED [INFO] hadoop-mapreduce .. SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 43:12.058s [INFO] Finished at: Wed Aug 22 13:56:53 UTC 2012 [INFO] Final Memory: 20M/108M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12:test (default-test) on project hadoop-mapreduce-client-jobclient: There are test failures. [ERROR] [ERROR] Please refer to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports for the individual test results. [ERROR] - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn goals -rf :hadoop-mapreduce-client-jobclient Build step 'Execute shell' marked build as failure [FINDBUGS] Skipping publisher since build result is FAILURE Archiving artifacts Updating MAPREDUCE-3506 Updating MAPREDUCE-4570 Updating HDFS-2686 Updating HDFS-3827 Updating HADOOP-8711 Updating YARN-22 Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## No tests ran.
[jira] [Created] (MAPREDUCE-4575) Add an option to drain the JobTracker jobs for upgrades
Arun C Murthy created MAPREDUCE-4575: Summary: Add an option to drain the JobTracker jobs for upgrades Key: MAPREDUCE-4575 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4575 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Arun C Murthy Assignee: Arun C Murthy Following on from MAPREDUCE-4328 it will be useful to allow an option to drain the JobTracker so that it will finish up existing jobs and not accept new ones. This is particularly useful during upgrades. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Branch created for MAPREDUCE-3902
I've created a branch to continue work on MAPREDUE-3902 since the changes are reasonably big and intrusive. A branch should also make it easier for others to contribute to the JIRA. Will send out another mail at a later point for review requests, once this is closer to completion. Meanwhile, if anyone does want to take a look at provide feedback, that's very welcome. (The initial commit to the branch is pretty big because it creates a copy of the existing AppMaster). - Sid
Re: killApplication doesn't kill AppMaster
Did you grab a stack trace of the AM? On Aug 22, 2012, at 4:01 PM, Bo Wang wrote: Hello, I have an AM listening to a port. I kill the application by sending a request via ClientRMProtocol # killApplication. In the NM log, the corresponding container of AM transitions from RUNNING to KILLING to CONTAINER_CLEANEDUP_AFTER_KILL to DONE. However, the AM is still running and the port is not released. I wonder what's going wrong here. Thanks, Bo -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/
[jira] [Created] (MAPREDUCE-4578) Handle container requests that request more resources than available in the cluster
Hitesh Shah created MAPREDUCE-4578: -- Summary: Handle container requests that request more resources than available in the cluster Key: MAPREDUCE-4578 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4578 Project: Hadoop Map/Reduce Issue Type: Bug Components: resourcemanager Affects Versions: 2.0.0-alpha, 0.23.0 Reporter: Hitesh Shah In heterogenous clusters, a simple check at the scheduler to check if the allocation request is within the max allocatable range is not enough. If there are large nodes in the cluster which are not available, there may be situations where some allocation requests will never be fulfilled. Need an approach to decide when to invalidate such requests. For application submissions, there will need to be a feedback loop for applications that could not be launched. For running AMs, AllocationResponse may need to augmented with information for invalidated/cancelled container requests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: killApplication doesn't kill AppMaster
Thanks for looking into this, Arun. I am not sure when to grab the stack trace of the AM. In the stdout/stderr of AM, no stack trace (or exception) is emitted. Btw, I am curious how NM kills a container. Does it directly kill the JVM process? Thanks, Bo On Wed, Aug 22, 2012 at 4:22 PM, Arun C Murthy a...@hortonworks.com wrote: Did you grab a stack trace of the AM? On Aug 22, 2012, at 4:01 PM, Bo Wang wrote: Hello, I have an AM listening to a port. I kill the application by sending a request via ClientRMProtocol # killApplication. In the NM log, the corresponding container of AM transitions from RUNNING to KILLING to CONTAINER_CLEANEDUP_AFTER_KILL to DONE. However, the AM is still running and the port is not released. I wonder what's going wrong here. Thanks, Bo -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/
Re: killApplication doesn't kill AppMaster
I am not sure when to grab the stack trace of the AM. In the stdout/stderr of AM, no stack trace (or exception) is emitted. You can login to the node and if the process is still alive, you can do a kill -3 which will dump the threads' status to stderr. Btw, I am curious how NM kills a container. Does it directly kill the JVM process? NM directly kills the JVM with a SIGTERM followed by a SIGKILL. BTW, please also check the corresponding NM's logs if there is some exception/error which could mean a bug in NM code. HTH, +Vinod
