[
https://issues.apache.org/jira/browse/PHOENIX-2940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15319231#comment-15319231
]
Josh Elser commented on PHOENIX-2940:
-------------------------------------
{quote}
bq. If the client-side cache is too small, then the client might be querying
too often for the stats, but I'm not sure what the best way to prevent this
would be. Ideally, we really want to have a propensity to cache the stats that
are asked for most frequently. If we go the timer route, and the client-side
cache is too small, then it just becomes less and less likely that the stats
are in the cache when we need them (essentially disabling stats). It'd help if
we had PHOENIX-2675 so that tables that don't need stats wouldn't fill up the
cache.
Agreed, limiting the contents of the cache based on size is rather difficult to
work around but important so that we don't blow out the client's heap. The
normal eviction stuff will work pretty well (evicting the least-recently used
elements first), but, like you point out, that doesn't help if the cache is
woefully small to begin with. I could think up some tricky solution which would
warn the user if we were continually re-fetching the stats, but that would
still require human interaction which is "meh".
{quote}
Actually, the use of {{expireAfterWrite}} is a little nonsense, I think.
{code}
final long halfStatsUpdateFreq = config.getLong(
QueryServices.STATS_UPDATE_FREQ_MS_ATTRIB,
QueryServicesOptions.DEFAULT_STATS_UPDATE_FREQ_MS) / 2;
final long maxTableStatsCacheEntries = config.getLong(
QueryServices.STATS_MAX_CACHE_ENTRIES,
QueryServicesOptions.DEFAULT_STATS_MAX_CACHE_ENTRIES);
tableStatsCache = CacheBuilder.newBuilder()
.maximumSize(maxTableStatsCacheEntries)
.expireAfterWrite(halfStatsUpdateFreq, TimeUnit.MILLISECONDS)
.build();
{code}
So, we have some frequency at which we update stats for a table (default
15mins). After 7m30s from caching the stats for a table, we'll evict it,
regardless of how often (or not) it was accessed. This seems goofball --
{{expireAfterAccess(long)}} seems to make much more sense to me (will remove it
if it isn't accessed at all after 7m30s). This also matches what is currently
done in {{GlobalCache}}.
Trying to understand this even more, in
{{MetaDataClient#updateStatisticsInternal(...)}}, we also use half of
{{QueryServices.STATS_UPDATE_FREQ_MS_ATTRIB}} (when
{{QueryServices.MIN_STATS_UPDATE_FREQ_MS_ATTRIB}} is unset) to determine if we
should ask the server to refresh the stats. Why are we doing this at half of
the update frequency?
> Remove STATS RPCs from rowlock
> ------------------------------
>
> Key: PHOENIX-2940
> URL: https://issues.apache.org/jira/browse/PHOENIX-2940
> Project: Phoenix
> Issue Type: Improvement
> Environment: HDP 2.3 + Apache Phoenix 4.6.0
> Reporter: Nick Dimiduk
> Assignee: Josh Elser
> Fix For: 4.9.0
>
> Attachments: PHOENIX-2940.001.patch
>
>
> We have an unfortunate situation wherein we potentially execute many RPCs
> while holding a row lock. This is problem is discussed in detail on the user
> list thread ["Write path blocked by MetaDataEndpoint acquiring region
> lock"|http://search-hadoop.com/m/9UY0h2qRaBt6Tnaz1&subj=Write+path+blocked+by+MetaDataEndpoint+acquiring+region+lock].
> During some situations, the
> [MetaDataEndpoint|https://github.com/apache/phoenix/blob/10909ae502095bac775d98e6d92288c5cad9b9a6/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java#L492]
> coprocessor will attempt to refresh it's view of the schema definitions and
> statistics. This involves [taking a
> rowlock|https://github.com/apache/phoenix/blob/10909ae502095bac775d98e6d92288c5cad9b9a6/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java#L2862],
> executing a scan against the [local
> region|https://github.com/apache/phoenix/blob/10909ae502095bac775d98e6d92288c5cad9b9a6/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java#L542],
> and then a scan against a [potentially
> remote|https://github.com/apache/phoenix/blob/10909ae502095bac775d98e6d92288c5cad9b9a6/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java#L964]
> statistics table.
> This issue is apparently exacerbated by the use of user-provided timestamps
> (in my case, the use of the ROW_TIMESTAMP feature, or perhaps as in
> PHOENIX-2607). When combined with other issues (PHOENIX-2939), we end up with
> total gridlock in our handler threads -- everyone queued behind the rowlock,
> scanning and rescanning SYSTEM.STATS. Because this happens in the
> MetaDataEndpoint, the means by which all clients refresh their knowledge of
> schema, gridlock in that RS can effectively stop all forward progress on the
> cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
