[
https://issues.apache.org/jira/browse/PHOENIX-5515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951427#comment-16951427
]
Vincent Poon commented on PHOENIX-5515:
---------------------------------------
I see the data table regionserver getting killed on an index write failure
after this patch:
2019-10-14 13:39:26,042 FATAL
[RpcServer.FifoWFPBQ.default.handler=255,queue=21,port=16201]
regionserver.HRegionServer: ABORTING region server
vincentpoon-wsm.internal.salesforce.com,16201,1571085425169: The coprocessor
org.apache.phoenix.hbase.index.IndexRegionObserver threw
java.lang.IllegalMonitorStateException
java.lang.IllegalMonitorStateException
at
java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:151)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1261)
at java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:457)
at
org.apache.phoenix.hbase.index.LockManager$RowLockImpl.release(LockManager.java:227)
at
org.apache.phoenix.hbase.index.IndexRegionObserver.postBatchMutateIndispensably(IndexRegionObserver.java:708)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$37.call(RegionCoprocessorHost.java:1052)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1693)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1771)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1727)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutateIndispensably(RegionCoprocessorHost.java:1048)
at
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3536)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3045)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2987)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:914)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:842)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2396)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:35080)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2373)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
> Able to write indexed value to data table without writing to index table
> ------------------------------------------------------------------------
>
> Key: PHOENIX-5515
> URL: https://issues.apache.org/jira/browse/PHOENIX-5515
> Project: Phoenix
> Issue Type: Bug
> Affects Versions: 4.14.3
> Reporter: Vincent Poon
> Assignee: Kadir OZDEMIR
> Priority: Major
> Fix For: 4.15.0, 5.1.0
>
> Attachments: PHOENIX-5515.master.001.patch,
> PHOENIX-5515.master.002.patch
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Using the 4.14.3 client, it still seems the IndexFailurePolicy is still
> kicking in, which disables the index on write failure. This means that while
> the index is in 'disabled' state, writes to the data table can happen without
> any writes to the index table. While in theory this might be ok since the
> rebuilder should eventually kick in and rebuild from the disable_timestamp,
> this breaks the new indexing design invariant that there should be no data
> table rows without a corresponding index row (potentially unverified), so
> this could potentially cause some unexpected behavior.
> Steps to repro:
> 1) Create data table
> 2) Create index table
> 3) "close_region" on index region from hbase shell
> 4) Upsert to data table
> Eventually after some number of retries, the index will get disabled, which
> means any other client can write to the data table without writing to the
> index table.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
