[
https://issues.apache.org/jira/browse/HBASE-10634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj Das updated HBASE-10634:
--------------------------------
Attachment: 10634-1.txt
Noticed a couple of issues while running the IT test submitted in HBASE-10616.
Ran the test as (this is still without the monkey):
{noformat}
hbase
org.apache.hadoop.hbase.test.IntegrationTestTimeBoundedMultiGetRequestsWithRegionReplicas
-Dhbase.IntegrationTestTimeBoundedMultiGetRequestsWithRegionReplicas.runtime=900000
-DIntegrationTestTimeBoundedMultiGetRequestsWithRegionReplicas.num_write_threads=40
-DIntegrationTestTimeBoundedMultiGetRequestsWithRegionReplicas.num_read_threads=40
-DIntegrationTestTimeBoundedMultiGetRequestsWithRegionReplicas.multigetBatchSize=50
-DIntegrationTestTimeBoundedMultiGetRequestsWithRegionReplicas.num_keys_per_server=3000
-DIntegrationTestTimeBoundedMultiGetRequestsWithRegionReplicas.num_regions_per_server=30
-Dhbase.ipc.client.allowsInterrupt=true
-DIntegrationTestTimeBoundedMultiGetRequestsWithRegionReplicas.tn=loadtestReplication3
{noformat}
1. I see a lot of these lines (notice the negative count)
{noformat}
14/02/27 15:45:58 INFO client.AsyncProcess: #1, waiting for -47 actions to
finish
14/02/27 15:45:58 INFO client.AsyncProcess: #1, waiting for -21 actions to
finish
14/02/27 15:45:58 INFO client.AsyncProcess: #1, waiting for -28 actions to
finish
14/02/27 15:45:58 INFO client.AsyncProcess: #1, waiting for -46 actions to
finish
14/02/27 15:45:58 INFO util.MultiThreadedAction: [R:29] Keys=1750, cols=21.2 K,
time=00:00:10 Overall: [keys/s= 162, latency=31 ms] Current: [keys/s=70,
latency=28 ms], verified=1750
14/02/27 15:45:58 INFO client.AsyncProcess: #1, waiting for -26 actions to
finish
14/02/27 15:45:58 INFO client.AsyncProcess: #1, waiting for -32 actions to
finish
14/02/27 15:45:58 INFO client.AsyncProcess: #1, waiting for -43 actions to
finish
14/02/27 15:45:59 INFO client.AsyncProcess: #1, waiting for -16 actions to
finish
14/02/27 15:45:59 INFO client.AsyncProcess: #1, waiting for -24 actions to
finish
14/02/27 15:46:01 INFO client.AsyncProcess: #1, waiting for -46 actions to
finish
14/02/27 15:46:01 INFO client.AsyncProcess: #1, waiting for -59 actions to
finish
14/02/27 15:46:01 INFO client.AsyncProcess: #1, waiting for -19 actions to
finish
14/02/27 15:46:01 INFO client.AsyncProcess: #1, waiting for -51 actions to
finish
14/02/27 15:46:01 INFO client.AsyncProcess: #1, waiting for -47 actions to
finish
14/02/27 15:46:01 INFO client.AsyncProcess: #1, waiting for -35 actions to
finish
14/02/27 15:46:03 INFO util.MultiThreadedAction: [R:29] Keys=1750, cols=21.2 K,
time=00:00:15 Overall: [keys/s= 111, latency=31 ms], verified=1750
14/02/27 15:46:03 INFO client.AsyncProcess: #1, waiting for -66 actions to
finish
{noformat}
2. Exceptions of the form:
{noformat}
java.lang.AssertionError: Cannot set actions in progress to 0
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.decActionCounter(AsyncProcess.java:1258)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.setResult(AsyncProcess.java:1142)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1069)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1100(AsyncProcess.java:546)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:673)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
{noformat}
3. Exceptions of the form:
{noformat}
Exception in thread "TimeBoundedMultiThreadedReaderThread_23"
java.lang.ClassCastException:
org.apache.hadoop.hbase.client.AsyncProcess$ReplicaResultState cannot be cast
to org.apache.hadoop.hbase.client.Result
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:841)
at
org.apache.hadoop.hbase.util.MultiThreadedReader$HBaseReaderThread.queryKey(MultiThreadedReader.java:345)
at
org.apache.hadoop.hbase.util.MultiThreadedReader$HBaseReaderThread.readKey(MultiThreadedReader.java:305)
at
org.apache.hadoop.hbase.util.MultiThreadedReader$HBaseReaderThread.runReader(MultiThreadedReader.java:227)
at
org.apache.hadoop.hbase.util.MultiThreadedReader$HBaseReaderThread.run(MultiThreadedReader.java:158)
{noformat}
I am still looking at the root cause of these but if anyone else feels like
digging, welcome. Attaching the WIP/debug patch.
> Multiget doesn't fully work
> ---------------------------
>
> Key: HBASE-10634
> URL: https://issues.apache.org/jira/browse/HBASE-10634
> Project: HBase
> Issue Type: Sub-task
> Reporter: Devaraj Das
> Fix For: hbase-10070
>
> Attachments: 10634-1.txt
>
>
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
