[
https://issues.apache.org/jira/browse/CRUNCH-619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15850149#comment-15850149
]
Attila Sasvari commented on CRUNCH-619:
---------------------------------------
I applied the patch and some Spark integration tests failed.
{noformat}
Tests in error:
SparkHFileTargetIT.setUpClass:129 ? RetriesExhausted Failed after
attempts=36,...
SparkWordCountHBaseIT.setUp:110 ? RetriesExhausted Failed after attempts=36,
e...
SparkWordCountHBaseIT.setUp:110 ? RetriesExhausted Failed after attempts=36,
e...
{noformat}
I checked {{org.apache.hadoop.hbase.ipc.CallTimeoutException}} was thrown
during the execution of SparkHFileTargetIT:
{code}
org.apache.crunch.SparkHFileTargetIT Time elapsed: 67.833 sec <<< ERROR!
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
attempts=36, exceptions:
Thu Feb 02 16:55:00 CET 2017, null, java.net.SocketTimeoutException:
callTimeout=60000, callDuration=60136: Call to /192.168.1.102:64404 failed on
local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=0,
waitTime=60002, rpcTimetout=59999 row '' on table 'hbase:meta' at
region=hbase:meta,,1.1588230740, hostname=192.168.1.102,64404,1486050837780,
seqNum=0
at
org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:255)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:229)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:177)
at
org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:314)
at
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:290)
at
org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:169)
at
org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:162)
at
org.apache.hadoop.hbase.client.ClientSimpleScanner.<init>(ClientSimpleScanner.java:39)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:378)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1105)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:1057)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:929)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:911)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:898)
at
org.apache.crunch.SparkHFileTargetIT.setUpClass(SparkHFileTargetIT.java:129)
Caused by: java.net.SocketTimeoutException: callTimeout=60000,
callDuration=60136: Call to /192.168.1.102:64404 failed on local exception:
org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=0, waitTime=60002,
rpcTimetout=59999 row '' on table 'hbase:meta' at
region=hbase:meta,,1.1588230740, hostname=192.168.1.102,64404,1486050837780,
seqNum=0
at
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:144)
at
org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Call to /192.168.1.102:64404 failed on local
exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=0,
waitTime=60002, rpcTimetout=59999
at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:172)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:387)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:94)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:407)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:403)
at org.apache.hadoop.hbase.ipc.Call.setTimeout(Call.java:96)
at
org.apache.hadoop.hbase.ipc.RpcConnection$1.run(RpcConnection.java:195)
at
io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:581)
at
io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:655)
at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:367)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=0,
waitTime=60002, rpcTimetout=59999
at
org.apache.hadoop.hbase.ipc.RpcConnection$1.run(RpcConnection.java:196)
at
io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:581)
at
io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:655)
at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:367)
at java.lang.Thread.run(Thread.java:745)
{code}
HbaseMiniCluster cannot be contacted for some reason.
I also noticed the following:
{code}
44274
[VolumeScannerThread(/root/crunch/crunch-hbase/target/test-data/a3979225-61d0-46fb-9b7a-227cf12cb8c5/dfscluster_5bb4ef9f-6747-48d2-9f0a-389634b8446d/dfs/data/data2)]
ERROR org.apache.hadoop.hdfs.server.datanode.VolumeScanner -
VolumeScanner(/root/crunch/crunch-hbase/target/test-data/a3979225-61d0-46fb-9b7a-227cf12cb8c5/dfscluster_5bb4ef9f-6747-48d2-9f0a-389634b8446d/dfs/data/data2,
DS-fd97dce5-3b9a-43e8-b02f-73d0789ccb54) exiting because of exception
java.lang.NoSuchMethodError:
org.codehaus.jackson.map.ObjectMapper.writerWithDefaultPrettyPrinter()Lorg/codehaus/jackson/map/ObjectWriter;
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl$BlockIteratorImpl.save(FsVolumeImpl.java:676)
at
org.apache.hadoop.hdfs.server.datanode.VolumeScanner.saveBlockIterator(VolumeScanner.java:314)
at
org.apache.hadoop.hdfs.server.datanode.VolumeScanner.runLoop(VolumeScanner.java:535)
at
org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:619)
{code}
It is related to the hadoop update in root pom.xml (bumped to 2.7.1).
To load the proper classes, I added the following dependencies to crunch-spark
pom.xml
{code}
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>2.4.4</version>
<type>jar</type>
</dependency>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-asl</artifactId>
<version>1.9.13</version>
</dependency>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-core-lgpl</artifactId>
<version>1.9.13</version>
</dependency>
{code}
> Run on HBase 2
> --------------
>
> Key: CRUNCH-619
> URL: https://issues.apache.org/jira/browse/CRUNCH-619
> Project: Crunch
> Issue Type: Improvement
> Affects Versions: 0.14.0
> Reporter: Tom White
> Assignee: Tom White
> Attachments: CRUNCH-619.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
