[
https://issues.apache.org/jira/browse/PHOENIX-5239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16831840#comment-16831840
]
Josh Elser commented on PHOENIX-5239:
-------------------------------------
Late to the party (still working off emails after PTO). I share Lars' concern –
while this approach would work well if you're the sole consumer of a cluster or
you have a relatively small cluster (say <20 nodes), I think it really breaks
down at scale.
{quote}This means that when we first start running the query, we'll
inconsistently hit the cache until it ends up being propagated to all the
regionservers.
{quote}
{quote} this would just be for persistently cached subqueries and regular
subqueries would continue to send the cache to only the 10 relevant servers for
the query.
{quote}
{quote}if following queries occurred serially (same example query, just
substituting out the where clause in the outer query), I believe the current
code would produce this behavior:
{quote}
It seems to me that the problem is that, upon re-use of some already-computed
persistent query, the implementation doesn't ensure that it's on the RS's that
you need for the new query.
Take two queries: Q1 and Q2. Q1 and Q2 differ only by the where clause as you
describe above. They both reference some same inner query whose results are
cache in some persistent query (PQ1). Say Q1 involves a subset of RegionServers
RS1 and Q2 involves a disjoint subset RS2.
The problem is that after running Q1, RS1 has PQ1 cached, but RS2 does not.
Thus, when running Q2, the query fails/errors/is-slow because you have to cache
PQ1 on RS2. Is this correct? Why, when Q2 starts, wouldn't it make sure that
it's cached on RS2?
I do recall that persistent queries are using a separate ServerCache (which
helps avoid thrashing cache concerns), but I would still be concerned about
this implementation scaling up as you get to 10, 20, 50, 100 RegionServers.
You're going to be paying a linear cost (at best) for persistent queries WRT
your RegionServers which isn't good.
> Send persistent subquery cache to all regionservers
> ---------------------------------------------------
>
> Key: PHOENIX-5239
> URL: https://issues.apache.org/jira/browse/PHOENIX-5239
> Project: Phoenix
> Issue Type: Improvement
> Reporter: John Phillips
> Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> PHOENIX-4666 introduced a persistent subquery cache that allowed phoenix to
> cache the results from an expensive subquery (enabled with a
> {{USE_PERSISTENT_CACHE}} query hint) to speed up subsequent queries.
> More context is available on the PHOENIX-4666 ticket, but a quick example
> would be a query like:
> {code:java}
> SELECT /*+ USE_PERSISTENT_CACHE */ *
> FROM table1
> JOIN (SELECT id_1 FROM large_table WHERE x = 10) expensive_result
> ON table1.id_1 = expensive_result.id_2
> WHERE table1.id_1 = [some_id]
> {code}
> Where lots of queries are ran, differing only by {{some_id}}. Our usage
> involves first running one query over phoenix to warm the cache (which takes
> ~20 seconds), then once complete, allowing the live query to run which
> utilize the persistent subquery cache (~100ms).
> However, we noticed that when phoenix sends the cache to the regionservers,
> it looks at {{some_id}} in the outer query to figure out which regionservers
> might contain {{table1.id_1 = [some_id]}} ([code
> here|https://github.com/apache/phoenix/blob/2084a6c/phoenix-core/src/main/java/org/apache/phoenix/cache/ServerCacheClient.java#L282-L283]).
> This means that when we first start running the query, we'll inconsistently
> hit the cache until it ends up being propagated to all the regionservers.
> Basically, we'd like to have some way to warm the subquery cache and ensure
> it's on all the regionservers so subsequent queries will always find the
> cache. I think the simplest solution might be updating the [if statement in
> ServerCacheClient#addServerCache|https://github.com/apache/phoenix/blob/2084a6c/phoenix-core/src/main/java/org/apache/phoenix/cache/ServerCacheClient.java#L282-L283]
> to simply always send the cache to all the regionservers if it's a
> persistent subquery:
> {code:java}
> - if ( ! servers.contains(entry) &&
> - keyRanges.intersectRegion(regionStartKey, regionEndKey,
> - cacheUsingTable.getIndexType() == IndexType.LOCAL)) {
> + boolean keyRangesIntersect = keyRanges.intersectRegion(regionStartKey,
> regionEndKey,
> + cacheUsingTable.getIndexType() == IndexType.LOCAL);
> + if (!servers.contains(entry) && (keyRangesIntersect || usePersistentCache))
> {
> {code}
> I tested this out, and it seems to work as expected. If it sounds like an
> acceptable solution, I'd be happy to make an actual PR. Or, if anyone has any
> other suggestions on better ways to handle this, it would be much appreciated.
> FYI [~jamestaylor], [~elserj], and [~maryannxue] since it looks like you
> three handled most of the review on the [original persistent cache
> PR|https://github.com/apache/phoenix/pull/298]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
