[
https://issues.apache.org/jira/browse/HBASE-8326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630827#comment-13630827
]
Nick Dimiduk commented on HBASE-8326:
-------------------------------------
Grepping the log files around after a test, it's easy to confirm that
JarFinder-generated jars are not reused between invocations.
{noformat}
$ grep -i 'for class .* using jar'
hbase-server/target/surefire-reports/org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan-output.txt
| cut -d ' ' -f 2,8-
17:16:16,381 org.apache.hadoop.hbase.HConstants, using jar
/Users/ndimiduk/repos/hbase/hbase-server/target/test-dir/hadoop-2782314318252813628.jar
17:16:17,344 org.apache.hadoop.hbase.protobuf.generated.ClientProtos, using jar
/Users/ndimiduk/repos/hbase/hbase-server/target/test-dir/hadoop-2092688855247426374.jar
17:16:17,692 org.apache.hadoop.hbase.client.Put, using jar
/Users/ndimiduk/repos/hbase/hbase-server/target/test-dir/hadoop-4612029855302264241.jar
17:16:17,714 org.apache.hadoop.hbase.CompatibilityFactory, using jar
/Users/ndimiduk/repos/hbase/hbase-server/target/test-dir/hadoop-1054386380400806295.jar
17:16:17,714 org.apache.zookeeper.ZooKeeper, using jar
/Users/ndimiduk/.m2/repository/org/apache/zookeeper/zookeeper/3.4.5/zookeeper-3.4.5.jar
17:16:17,715 com.google.protobuf.Message, using jar
/Users/ndimiduk/.m2/repository/com/google/protobuf/protobuf-java/2.4.1/protobuf-java-2.4.1.jar
17:16:17,715 com.google.common.collect.Lists, using jar
/Users/ndimiduk/.m2/repository/com/google/guava/guava/12.0.1/guava-12.0.1.jar
17:16:17,716 org.cloudera.htrace.Trace, using jar
/Users/ndimiduk/.m2/repository/org/cloudera/htrace/htrace/1.50/htrace-1.50.jar
17:16:17,792 org.apache.hadoop.hbase.io.ImmutableBytesWritable, using jar
/Users/ndimiduk/repos/hbase/hbase-server/target/test-dir/hadoop-1759217497204054953.jar
17:16:17,859 org.apache.hadoop.hbase.io.ImmutableBytesWritable, using jar
/Users/ndimiduk/repos/hbase/hbase-server/target/test-dir/hadoop-4547156579345911573.jar
17:16:19,680 org.apache.hadoop.hbase.mapreduce.TableInputFormat, using jar
/Users/ndimiduk/repos/hbase/hbase-server/target/test-dir/hadoop-2278643074673130017.jar
17:16:19,681 org.apache.hadoop.io.LongWritable, using jar
/Users/ndimiduk/.m2/repository/org/apache/hadoop/hadoop-core/1.1.2/hadoop-core-1.1.2.jar
17:16:19,681 org.apache.hadoop.io.Text, using jar
/Users/ndimiduk/.m2/repository/org/apache/hadoop/hadoop-core/1.1.2/hadoop-core-1.1.2.jar
17:16:19,682 org.apache.hadoop.mapreduce.lib.output.TextOutputFormat, using jar
/Users/ndimiduk/.m2/repository/org/apache/hadoop/hadoop-core/1.1.2/hadoop-core-1.1.2.jar
17:16:19,682 org.apache.hadoop.mapreduce.lib.partition.HashPartitioner, using
jar
/Users/ndimiduk/.m2/repository/org/apache/hadoop/hadoop-core/1.1.2/hadoop-core-1.1.2.jar
{noformat}
However, notice the timestamps. This iteration generated 7 jars and located 8
others in less than 4 seconds. I don't think this additional IO is sufficiently
high to cause test timeouts.
bq. add additional logic to addDependencyJars such that it will not bundle a
new jar if it has already generated a jar containing the class
I have two ideas for implementing this. The first is to keep a map of
constructed jar content to jar name. As JarFinder generates jars, it'll add a
class listing of those jars to the map, and future iterations can reference
that map before performing jar build. The second is to use a
DynamicClassLoader, to which the generated jars are added. That loader can be
inspected using identical logic to the existing code which searches for jars in
the default ClassLoader.
Both implementation add further mess to this code, and all for the sake of unit
tests. I'm not excited about either of them. Let me know if you have any other
ideas.
> mapreduce.TestTableInputFormatScan times out frequently
> -------------------------------------------------------
>
> Key: HBASE-8326
> URL: https://issues.apache.org/jira/browse/HBASE-8326
> Project: HBase
> Issue Type: Bug
> Components: mapreduce, test
> Affects Versions: 0.98.0, 0.94.7, 0.95.1
> Reporter: Nick Dimiduk
> Assignee: Nick Dimiduk
>
> bq. It looks like since this change
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan has not passed a
> single time in the EC2 builds (http://54.241.6.143/job/HBase-0.94/). Change
> was introduced in Build #72, since then this tests times out (or doesn't
> finish)
> via [~lhofhansl] in [HBASE-8140
> comment|https://issues.apache.org/jira/browse/HBASE-8140?focusedCommentId=13629101&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13629101].
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
