[
https://issues.apache.org/jira/browse/PHOENIX-7076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Viraj Jasani updated PHOENIX-7076:
----------------------------------
Description:
For older versions upgrading to 4.7+, we perform deletion of all rows on
SYSTEM.STATS table one time. This is done only if column "UPGRADE_TO_4_7" does
not exist on SYSTEM.CATALOG.
Since this task is performed in a daemon thread as part of region#postOpen
coproc hook, it might get stuck accessing SYSTEM.CATALOG table if the region
open gets delayed for some reason.
Similar to BuildIndexScheduleTask that gets scheduled with 10s of initial delay
(by default), we should perform truncate stats as a delayed task, even though
it does not require any repetition.
{code:java}
2023-10-11 22:51:09,728 DEBUG [Thread-60] client.RpcRetryingCallerImpl -
n$ClientServiceBlockingInterfaceWrapper.doCall(ServerConnectionUtils.java:163)
at
org.apache.hadoop.hbase.client.ServerConnectionUtils$ShortCircuitingClusterConnection$ClientServiceBlockingInterfaceWrapper.get(ServerConnectionUtils.java:131)
at
org.apache.hadoop.hbase.client.ClientServiceCallable.doGet(ClientServiceCallable.java:51)
at org.apache.hadoop.hbase.client.HTable$1.rpcCall(HTable.java:364)
at org.apache.hadoop.hbase.client.HTable$1.rpcCall(HTable.java:359)
at
org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:370)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:343)
at org.apache.phoenix.util.UpgradeUtil.truncateStats(UpgradeUtil.java:2057)
at
org.apache.phoenix.coprocessor.MetaDataRegionObserver$1$1.run(MetaDataRegionObserver.java:182)
at
org.apache.phoenix.coprocessor.MetaDataRegionObserver$1$1.run(MetaDataRegionObserver.java:179)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:514)
at
org.apache.hadoop.security.SecurityUtil.doAsLoginUser(SecurityUtil.java:495)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hbase.util.Methods.call(Methods.java:39)
at org.apache.hadoop.hbase.security.User.runAsLoginUser(User.java:178)
at
org.apache.phoenix.coprocessor.MetaDataRegionObserver$1.run(MetaDataRegionObserver.java:179)
at java.lang.Thread.run(Thread.java:750) {code}
Improvements:
# Schedule truncate stats task with 10s delay (configurable) rather than
starting the thread immediately.
# Only if {{phoenix.stats.collection.enabled}} is enabled, create daemon
thread for truncate stats task.
was:
For older versions upgrading to 4.7+, we perform deletion of all rows on
SYSTEM.STATS table one time. This is done only if column "UPGRADE_TO_4_7" does
not exist on SYSTEM.CATALOG.
Since this task is performed in a daemon thread as part of region#postOpen
coproc hook, it might get stuck accessing SYSTEM.CATALOG table if the region
open gets delayed for some reason.
Similar to BuildIndexScheduleTask that gets scheduled with 10s of initial delay
(by default), we should perform truncate stats as a delayed task, even though
it does not require any repetition.
{code:java}
2023-10-11 22:51:09,728 DEBUG [Thread-60] client.RpcRetryingCallerImpl -
n$ClientServiceBlockingInterfaceWrapper.doCall(ServerConnectionUtils.java:163)
at
org.apache.hadoop.hbase.client.ServerConnectionUtils$ShortCircuitingClusterConnection$ClientServiceBlockingInterfaceWrapper.get(ServerConnectionUtils.java:131)
at
org.apache.hadoop.hbase.client.ClientServiceCallable.doGet(ClientServiceCallable.java:51)
at org.apache.hadoop.hbase.client.HTable$1.rpcCall(HTable.java:364)
at org.apache.hadoop.hbase.client.HTable$1.rpcCall(HTable.java:359)
at
org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:370)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:343)
at org.apache.phoenix.util.UpgradeUtil.truncateStats(UpgradeUtil.java:2057)
at
org.apache.phoenix.coprocessor.MetaDataRegionObserver$1$1.run(MetaDataRegionObserver.java:182)
at
org.apache.phoenix.coprocessor.MetaDataRegionObserver$1$1.run(MetaDataRegionObserver.java:179)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:514)
at
org.apache.hadoop.security.SecurityUtil.doAsLoginUser(SecurityUtil.java:495)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hbase.util.Methods.call(Methods.java:39)
at org.apache.hadoop.hbase.security.User.runAsLoginUser(User.java:178)
at
org.apache.phoenix.coprocessor.MetaDataRegionObserver$1.run(MetaDataRegionObserver.java:179)
at java.lang.Thread.run(Thread.java:750) {code}
> MetaDataRegionObserver#postOpen hook improvements
> -------------------------------------------------
>
> Key: PHOENIX-7076
> URL: https://issues.apache.org/jira/browse/PHOENIX-7076
> Project: Phoenix
> Issue Type: Bug
> Affects Versions: 5.1.3
> Reporter: Viraj Jasani
> Assignee: Divneet Kaur
> Priority: Major
> Fix For: 5.2.0, 5.1.4
>
>
> For older versions upgrading to 4.7+, we perform deletion of all rows on
> SYSTEM.STATS table one time. This is done only if column "UPGRADE_TO_4_7"
> does not exist on SYSTEM.CATALOG.
> Since this task is performed in a daemon thread as part of region#postOpen
> coproc hook, it might get stuck accessing SYSTEM.CATALOG table if the region
> open gets delayed for some reason.
> Similar to BuildIndexScheduleTask that gets scheduled with 10s of initial
> delay (by default), we should perform truncate stats as a delayed task, even
> though it does not require any repetition.
>
> {code:java}
> 2023-10-11 22:51:09,728 DEBUG [Thread-60] client.RpcRetryingCallerImpl -
> n$ClientServiceBlockingInterfaceWrapper.doCall(ServerConnectionUtils.java:163)
>
> at
> org.apache.hadoop.hbase.client.ServerConnectionUtils$ShortCircuitingClusterConnection$ClientServiceBlockingInterfaceWrapper.get(ServerConnectionUtils.java:131)
>
> at
> org.apache.hadoop.hbase.client.ClientServiceCallable.doGet(ClientServiceCallable.java:51)
>
> at org.apache.hadoop.hbase.client.HTable$1.rpcCall(HTable.java:364)
> at org.apache.hadoop.hbase.client.HTable$1.rpcCall(HTable.java:359)
> at
> org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124)
>
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104)
>
> at org.apache.hadoop.hbase.client.HTable.get(HTable.java:370)
> at org.apache.hadoop.hbase.client.HTable.get(HTable.java:343)
> at
> org.apache.phoenix.util.UpgradeUtil.truncateStats(UpgradeUtil.java:2057)
> at
> org.apache.phoenix.coprocessor.MetaDataRegionObserver$1$1.run(MetaDataRegionObserver.java:182)
>
> at
> org.apache.phoenix.coprocessor.MetaDataRegionObserver$1$1.run(MetaDataRegionObserver.java:179)
>
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
>
> at
> org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:514)
> at
> org.apache.hadoop.security.SecurityUtil.doAsLoginUser(SecurityUtil.java:495)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.hbase.util.Methods.call(Methods.java:39)
> at org.apache.hadoop.hbase.security.User.runAsLoginUser(User.java:178)
> at
> org.apache.phoenix.coprocessor.MetaDataRegionObserver$1.run(MetaDataRegionObserver.java:179)
>
> at java.lang.Thread.run(Thread.java:750) {code}
>
> Improvements:
> # Schedule truncate stats task with 10s delay (configurable) rather than
> starting the thread immediately.
> # Only if {{phoenix.stats.collection.enabled}} is enabled, create daemon
> thread for truncate stats task.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
