[
https://issues.apache.org/jira/browse/PHOENIX-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16085094#comment-16085094
]
Rajeshbabu Chintaguntla edited comment on PHOENIX-3496 at 7/13/17 3:16 AM:
---------------------------------------------------------------------------
bq. Having to disable the data table when creating a local index on it too is a
non-starter IMHO since we are essentially asking for a downtime on a parent
table to create an index on it.
I too agree and may not be good option.
bq. Would an async mechanism like map reduce building the index encounter the
same issue? Maybe we should always create a local index through our map reduce
integration?.
This may happen in MR building depending on whether the job started before all
the regions reopened or after.
bq. Andrew Purtell, we do poll for updated table descriptor.
The polling is not helping here because we are modifying the table descriptor
in HMaster cache before reopening the regions so client gets the updated
descriptor from master cache. So polling see the descriptor changed and come
out but it's to be changed in all the regions. Same thing happen with add
family as well.
One more solution might be like retry rebuild scanning when we get
NoSuchColumnFamilyException by changing the exception to IOException in that
case HBase automatically retries.
was (Author: rajeshbabu):
bq. Having to disable the data table when creating a local index on it too is a
non-starter IMHO since we are essentially asking for a downtime on a parent
table to create an index on it.
I too agree and may not be good option.
bq. Would an async mechanism like map reduce building the index encounter the
same issue? Maybe we should always create a local index through our map reduce
integration?.
This may happen in MR building depending on whether the job started before all
the regions reopened or after.
bq. Andrew Purtell, we do poll for updated table descriptor.
The polling is not helping here because we are modifying the table descriptor
in HMaster cache before reopening the regions so client gets the updated
descriptor from master cache. So polling see the descriptor changed and come
out but it's to be changed in all the regions. Same thing happen with add
family as well.
One more solution might be like retry scanning when we get
NoSuchColumnFamilyException by changing the exception to IOException in that
case HBase automatically retries.
> Figure out why LocalIndexIT#testLocalIndexRoundTrip is flapping
> ---------------------------------------------------------------
>
> Key: PHOENIX-3496
> URL: https://issues.apache.org/jira/browse/PHOENIX-3496
> Project: Phoenix
> Issue Type: Bug
> Reporter: Samarth Jain
> Assignee: Rajeshbabu Chintaguntla
>
> The test has been passing consistently on the "4.*-HBase-0.98" branches.
> However, it has been flapping pretty regularly on the master branch and the
> "4.*-HBase-1.1" branches.
> I ran the test locally a few number of times and it did flap. I did notice
> that in cases where it failed, the logs also had a RegionOpeningException.
> For example:
> {code}
> org.apache.hadoop.hbase.exceptions.RegionOpeningException: Region
> TEST_TABLET000060,\x07\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1479426366023.04f765e5d906bbd193b38a9f8c20e478.
> is opening on localhost,55599,1479426313446
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2908)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1053)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2385)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33648)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> And the test failure:
> {code}
> ava.util.concurrent.ExecutionException:
> org.apache.phoenix.exception.PhoenixIOException:
> org.apache.phoenix.exception.PhoenixIOException:
> org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column
> family L#0 does not exist in region
> TEST_TABLET000060,\x09\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1479426366023.47319064397e6e25e8e7cc992ebce3e6.
> in table 'TEST_TABLET000060', {TABLE_ATTRIBUTES => {coprocessor$1 =>
> '|org.apache.phoenix.coprocessor.ScanRegionObserver|805306366|',
> coprocessor$2 =>
> '|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|805306366|',
> coprocessor$3 =>
> '|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|805306366|',
> coprocessor$4 =>
> '|org.apache.phoenix.coprocessor.ServerCachingEndpointImpl|805306366|',
> coprocessor$5 =>
> '|org.apache.phoenix.hbase.index.Indexer|805306366|index.builder=org.apache.phoenix.index.PhoenixIndexBuilder,org.apache.hadoop.hbase.index.codec.class=org.apache.phoenix.index.PhoenixIndexCodec'},
> {NAME => '0', DATA_BLOCK_ENCODING => 'FAST_DIFF', BLOOMFILTER => 'ROW',
> REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '1', TTL =>
> 'FOREVER', MIN_VERSIONS => '0', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE =>
> '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkFamily(HRegion.java:7649)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2543)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2527)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2406)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33648)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:745)
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:188)
> at
> org.apache.phoenix.end2end.index.LocalIndexIT.testLocalIndexRoundTrip(LocalIndexIT.java:166)
> {code}
> [~rajeshbabu], can you please take a look.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
