[
https://issues.apache.org/jira/browse/HBASE-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13912375#comment-13912375
]
Nick Dimiduk commented on HBASE-10592:
--------------------------------------
Tested the 0.96 patch on a 0.96.1/secure cluster, sequentialWrite and
randomRead work fine in MR mode. Tested the 0.98 patch on a 0.98.0/secure
cluster, but get
{noformat}
2014-02-26 01:13:33,870 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error
running child : java.lang.NoClassDefFoundError:
org/apache/commons/math/stat/descriptive/DescriptiveStatistics
at
org.apache.hadoop.hbase.PerformanceEvaluation$RandomReadTest.testTakedown(PerformanceEvaluation.java:781)
at
org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:584)
at
org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1017)
at
org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:236)
at
org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:187)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: java.lang.ClassNotFoundException:
org.apache.commons.math.stat.descriptive.DescriptiveStatistics
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 13 more
{noformat}
The following patch fixes it
{noformat}
diff -u
b/hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java
b/hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java
---
b/hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java
+++
b/hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java
@@ -392,9 +392,8 @@
TextOutputFormat.setOutputPath(job, new Path(inputDir.getParent(),
"outputs"));
TableMapReduceUtil.addDependencyJars(job);
- // Add a Class from the hbase.jar so it gets registered too.
TableMapReduceUtil.addDependencyJars(job.getConfiguration(),
- org.apache.hadoop.hbase.util.Bytes.class);
+ DescriptiveStatistics.class);
TableMapReduceUtil.initCredentials(job);
{noformat}
I'm not sure why this is necessary on 0.98 but not 0.96.
> Refactor PerformanceEvaluation tool
> -----------------------------------
>
> Key: HBASE-10592
> URL: https://issues.apache.org/jira/browse/HBASE-10592
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.96.2, 0.98.1, 0.99.0
> Reporter: Nick Dimiduk
> Assignee: Nick Dimiduk
> Priority: Minor
> Fix For: 0.99.0
>
> Attachments: HBASE-10592.00-0.96.patch, HBASE-10592.00-0.98.patch,
> HBASE-10592.00.patch
>
>
> PerfEval is kind of a mess. It's painful to add new features because the test
> options are itemized and passed as parameters to internal methods.
> Serialization is hand-rolled and tedious. Ensuring support for mapreduce mode
> is a chore because of it.
> This patch refactors the tool. Options are now passed around to methods and
> such as a POJO instead of one-by-one. Get rid of accessors that don't help
> anyone. On the mapreduce side, serialization is now handled using json
> (jackson is a dependency anyway) instead of the hand-rolled regex we used
> before. Also do away with custom InputSplit and FileFormat, instead using
> Text and NLineInputFormat. On the local mode side, combine 1 client and N
> clients into the same implementation. That implementation now uses an
> ExecutorService, so we can later decouple number of client workers from
> number of client tasks. Finally, drop a bunch of confusing local state,
> instead use the new TestOptions POJO as a parameter to static methods.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
