[
https://issues.apache.org/jira/browse/PHOENIX-3986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076997#comment-16076997
]
Andrew Purtell commented on PHOENIX-3986:
-----------------------------------------
bq. In UngroupedAggregateRegionObserver.commitBatchWithHTable() do we need to
check the memstore size and wait for the flush. We are using a htable to write
the mutations.
The hardcoded wait is wrong. Worst case flush times in our production are
upwards of a minute. Why isn't letting the server block the htable's write
transparently good enough? In other words, the write will wait if the server
holds up work for flushing to complete or other criteria that does not need to
be duplicated. Is that write even going to the region this wait is spinning on?
> UngroupedAggregateRegionObserver.commitBatch() should set the index metadata
> as an attribute on every mutation
> --------------------------------------------------------------------------------------------------------------
>
> Key: PHOENIX-3986
> URL: https://issues.apache.org/jira/browse/PHOENIX-3986
> Project: Phoenix
> Issue Type: Bug
> Affects Versions: 4.9.0
> Reporter: Thomas D'Silva
> Assignee: Thomas D'Silva
> Fix For: 4.12.0, 4.11.1
>
> Attachments: PHOENIX-3986.patch
>
>
> We are seeing "unable to find cached index metadata" exceptions in
> production while running UPSERT SELECT queries on a table that has mutable
> indexes. These exceptions should not happen because PHOENIX-2935 changed the
> code to set the index metadata as an attribute.
> {code}
> ERROR 2008 (INT10): ERROR 2008 (INT10): Unable to find cached index metadata.
> key={} region={}.host={} Index update failed
> at
> org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:81)
> at
> org.apache.phoenix.util.ServerUtil.throwIOException(ServerUtil.java:55)
> at
> org.apache.phoenix.index.PhoenixIndexMetaData.getIndexMetaData(PhoenixIndexMetaData.java:86)
> at
> org.apache.phoenix.index.PhoenixIndexMetaData.<init>(PhoenixIndexMetaData.java:94)
> at
> org.apache.phoenix.index.PhoenixIndexBuilder.getIndexMetaData(PhoenixIndexBuilder.java:93)
> at
> org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:122)
> at
> org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:324)
> at
> org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:249)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$35.call(RegionCoprocessorHost.java:996)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1621)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1697)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1653)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:992)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2650)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2410)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2365)
> at
> org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver.commitBatch(UngroupedAggregateRegionObserver.java:216)
> at
> org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver.commit(UngroupedAggregateRegionObserver.java:791)
> at
> org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver.doPostScannerOpen(UngroupedAggregateRegionObserver.java:721)
> at
> org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.overrideDelegate(BaseScannerRegionObserver.java:236)
> at
> org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.nextRaw(BaseScannerRegionObserver.java:287)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3361)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32492)
> at
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2210)
> at
> org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:104)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> This is what I think is happening :
> In UngroupedAggregateRegionObserver.doPostScannerOpen we commit batches of
> mutations.
> In commitBatch() since the mutations are for the same region we only set the
> index maintainer as an attribute on the first mutation of the batch. We then
> call HRegion.batchMutate() which ends up calling
> HRegion.doMiniBatchMutation() in a loop.
> doMiniBatchMutation() loops throught the mutations one at a time and tries
> to acquire the each mutation's lock. If it cannot acquire a lock it processes
> the mutations for which it has acquired locks so far and then calls
> coprocessorHost.preBatchMutate which eventually ends up calling
> PhoenixIndexBuilder.getIndexMetaData() which set the index metadata using the
> attribute map of the first mutation in the mini batch.
> It then processes the next set of mutations for which it has acquired locks.
> The index metadata is not set as an attribute for this next mini batch, so we
> try and look it up in the cache which fails
> In PhoenixIndexBuilder
> {code}
> @Override
> public IndexMetaData
> getIndexMetaData(MiniBatchOperationInProgress<Mutation> miniBatchOp) throws
> IOException {
> return new PhoenixIndexMetaData(env,
> miniBatchOp.getOperation(0).getAttributesMap());
> }
> {code}
> In MiniBatchOperationInProgress
> {code}
> /**
> * @param index
> * @return The operation(Mutation) at the specified position.
> */
> public T getOperation(int index) {
> return operations[getAbsoluteIndex(index)];
> }
> private int getAbsoluteIndex(int index) {
> if (index < 0 || this.firstIndex + index >= this.lastIndexExclusive) {
> throw new ArrayIndexOutOfBoundsException(index);
> }
> return this.firstIndex + index;
> }
> {code}
> [~ankit.singhal] [~jamestaylor] Does this make sense?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
