[
https://issues.apache.org/jira/browse/HBASE-10264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Himanshu Vashishtha updated HBASE-10264:
----------------------------------------
Attachment: HBase-10264.patch
Two issues in the mapred phase:
a) It misses hbase specific jars, so TT doesn't get any hbase-xxx jars shipped
to them.
b) The job also requires Counter.class from high_scalability-xxx library, as TT
instantiates HRegion object.
Here is a patch to do so.
Testing: Ran the patched job on a yarn cluster.
{code}
[jenkins@tarball-target-2 hbase]$ bin/hbase
org.apache.hadoop.hbase.regionserver.CompactionTool -mapred -major
hdfs://`hostname`:8020/hbase/data/default/sampleTable_api-compat-8.ent.cloudera.com
...
2014-01-02 11:33:02,550 INFO [main] mapreduce.Job: The url to track the job:
http://tarball-target-2.ent.cloudera.com:8088/proxy/application_1388690541295_0011/
2014-01-02 11:33:02,551 INFO [main] mapreduce.Job: Running job:
job_1388690541295_0011
2014-01-02 11:33:14,018 INFO [main] mapreduce.Job: Job job_1388690541295_0011
running in uber mode : false
2014-01-02 11:33:14,020 INFO [main] mapreduce.Job: map 0% reduce 0%
2014-01-02 11:33:23,151 INFO [main] mapreduce.Job: map 100% reduce 0%
2014-01-02 11:33:23,172 INFO [main] mapreduce.Job: Job job_1388690541295_0011
completed successfully
2014-01-02 11:33:23,362 INFO [main] mapreduce.Job: Counters: 27
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=109926
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=5749
HDFS: Number of bytes written=968
HDFS: Number of read operations=22
HDFS: Number of large read operations=0
HDFS: Number of write operations=6
Job Counters
Launched map tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=7181
Total time spent by all reduces in occupied slots (ms)=0
Map-Reduce Framework
Map input records=1
Map output records=0
Input split bytes=154
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=77
CPU time spent (ms)=1940
Physical memory (bytes) snapshot=178450432
Virtual memory (bytes) snapshot=883052544
Total committed heap usage (bytes)=114360320
File Input Format Counters
Bytes Read=143
File Output Format Counters
Bytes Written=0
{code}
> [MapReduce]: CompactionTool in mapred mode is missing classes in its classpath
> ------------------------------------------------------------------------------
>
> Key: HBASE-10264
> URL: https://issues.apache.org/jira/browse/HBASE-10264
> Project: HBase
> Issue Type: Bug
> Components: Compaction, mapreduce
> Affects Versions: 0.98.0, 0.99.0
> Reporter: Aleksandr Shulman
> Assignee: Himanshu Vashishtha
> Attachments: HBase-10264.patch
>
>
> Calling o.a.h.h.regionserver.CompactionTool fails due to classpath-related
> issues in both MRv1 and MRv2.
> {code}hbase org.apache.hadoop.hbase.regionserver.CompactionTool -mapred
> -major hdfs://`hostname`:8020/hbase/data/default/orig_1388179858868{code}
> Results:
> {code}2013-12-27 13:31:49,478 INFO [main] mapreduce.Job: Task Id :
> attempt_1388179525649_0011_m_000000_2, Status : FAILED
> Error: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.TableInfoMissingException
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.apache.hadoop.hbase.regionserver.CompactionTool$CompactionWorker.compact(CompactionTool.java:115)
> at
> org.apache.hadoop.hbase.regionserver.CompactionTool$CompactionMapper.map(CompactionTool.java:231)
> at
> org.apache.hadoop.hbase.regionserver.CompactionTool$CompactionMapper.map(CompactionTool.java:207)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:165)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:160){code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
