[
https://issues.apache.org/jira/browse/PHOENIX-6883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17691752#comment-17691752
]
Kadir Ozdemir commented on PHOENIX-6883:
----------------------------------------
The description of this Jira provides a sketch of the design. A detailed
description of the redesign still needs to be spelled out based on the feedback.
[~stoty], you are right that we also need to invalidate the base table cache
entries when the metadata for an index is updated.
I was thinking about invaliding cache entries synchronously. If invalidation
fails, we need to fail the metadata update. Identifying region servers for a
given table using HBase meta and then invalidating the cache entries only on
those region servers is the approach I am planning for. I think we can optimize
this further by tracking in memory which region server has a cache entry for a
given object and invalidate only those region servers. Whenever, a syscat
region moves, we need to rebuild this info starting with assumption that every
region server has every of its objects in the cache. Please note that region
server needs to use get table RPC to populate their caches. This will be the
input for tracking. So essentially, we would have two collections in memory :
collection of invalidated objects, and collection of cached objects and their
region servers. These are my initial thoughts on the optimization.
> Phoenix metadata caching redesign
> ---------------------------------
>
> Key: PHOENIX-6883
> URL: https://issues.apache.org/jira/browse/PHOENIX-6883
> Project: Phoenix
> Issue Type: Improvement
> Reporter: Kadir Ozdemir
> Priority: Major
>
> PHOENIX-6761 improves the client side metadata caching by eliminating the
> separate cache for each connection. This improvement results in memory and
> compute savings since it eliminates copying CQSI level cache every time a
> Phoenix connection is created, and also replaces the inefficient the CQSI
> level cache implementation with Guava Cache from Google.
> Despite this improvement, the overall metadata caching architecture begs for
> redesign. This is because every operation in Phoenix need to make multiple
> RPCs to metadata servers for the SYSTEM.CATALOG table (please see
> PHOENIX-6860) to ensure the latest metadata changes are visible to clients.
> These constant RPCs makes the region servers serving SYSTEM.CATALOG hot spot
> and thus leads to poor performance and availability issues.
> The UPDATE_CACHE_FREQUENCY configuration parameter specifies how frequently
> the client cache is updated. However, setting this parameter to a non-zero
> value results in stale caching. Stale caching can cause data integrity
> issues. For example, if an index table creation is not visible to the client,
> Phoenix would skip updating the index table in the write path. That's why is
> this parameter is typically set to zero. However, this defeats the purpose of
> client side metadata caching.
> The redesign of the metadata caching architecture is to directly address this
> issue by making sure that the client metadata caching is always used (that
> is, UPDATE_CACHE_FREQUENCY is set to NEVER) but still ensures the data
> integrity. This is achieved by three main changes.
> The first change is to introduce server side metadata caching in all region
> servers. Currently, the server side metadata caching is used on the region
> servers serving SYSTEM.CATALOG. This metadata caching should be strongly
> consistent such that the metadata updates should include invalidating the
> corresponding entries on the server side caches. This would ensure the server
> cache would not become stale.
> The second change is that the Phoenix client passes the LAST_DDL_TIMESTAMP
> table attribute along with scan and mutation operations to the server regions
> (more accurately to the Phoenix coprocessors). Then the Phoenix coprocessors
> would check the timestamp on a given operation against with the timestamp in
> its server side cache to validate that the client did not use stale metadata
> when it prepared the operation. If the client did use stale metadata then the
> coprocessor would return an exception (this exception can be called
> StaleClientMetadataCacheException) to the client.
> The third change is that upon receiving StaleClientMetadataCacheException the
> Phoenix client makes an RPC call to the metadata server to update the client
> cache, reconstruct the operation with the updated cached, and retry the
> operation.
> This redesign would require updating client and server metadata caches only
> when metadata is stale instead of updating the client metadata cache for each
> (scan or mutation) operation. This would eliminate hot spotting on the
> metadata servers and thus poor performance and availability issues caused by
> this hot spotting.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
