[
https://issues.apache.org/jira/browse/AMBARI-13525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969384#comment-14969384
]
Dmytro Sen commented on AMBARI-13525:
-------------------------------------
Fix:
1. Set value of phoenix.coprocessor.maxServerCacheTimeToLiveMs to 60 seconds
instead of 30 by default (all metrics in METRIC_RECORD are updated every 60
seconds)
2. Catch DoNotRetryIOException when executing the JOIN query on ams collector,
log like "The cache might have expired and have been removed. Try to increase
the cache size by setting bigger value for
phoenix.coprocessor.maxMetaDataCacheSize in ams-hbase-site config. Falling back
to sort-merge join algorithm." Run all the following JOIN queries with /+
USE_SORT_MERGE_JOIN NO_CACHE/ hints, sort-merge join algorithm doesn't use the
hash cache.
> Exception in collector logs for JOIN queries
> --------------------------------------------
>
> Key: AMBARI-13525
> URL: https://issues.apache.org/jira/browse/AMBARI-13525
> Project: Ambari
> Issue Type: Bug
> Components: ambari-metrics
> Affects Versions: 2.1.1
> Reporter: Dmytro Sen
> Assignee: Dmytro Sen
> Priority: Critical
> Fix For: 2.1.3
>
> Attachments: AMBARI-13525.patch
>
>
> Exception:
> {code}
> 04:17:00,487 WARN [1836615260@qtp-1368925364-4941]
> GenericExceptionHandler:98 - INTERNAL_SERVER_ERROR
> javax.ws.rs.WebApplicationException:
> org.apache.phoenix.exception.PhoenixIOException:
> org.apache.phoenix.exception.PhoenixIOException:
> org.apache.hadoop.hbase.DoNotRetryIOException: Could not find hash cache for
> joinId: 8��^S1�(�. The cache might have expired and have been removed.
> at
> org.apache.phoenix.coprocessor.HashJoinRegionScanner.<init>(HashJoinRegionScanner.java:96)
> at
> org.apache.phoenix.coprocessor.ScanRegionObserver.doPostScannerOpen(ScanRegionObserver.java:196)
> at
> org.apache.phoenix.coprocessor.BaseScannerRegionObserver.postScannerOpen(BaseScannerRegionObserver.java:144)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postScannerOpen(RegionCoprocessorHost.java:1931)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3173)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29994)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2078)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> - Make sure that we follow best practices for JOINs:
> https://phoenix.apache.org/joins.html
> - Verify if this results in loss of functionality
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
