[
https://issues.apache.org/jira/browse/PHOENIX-180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093710#comment-14093710
]
ramkrishna.s.vasudevan commented on PHOENIX-180:
------------------------------------------------
Updates on this patch. The following things have been done till now, This
work is purely based on phoenix only.
Using the github push done by Jesse on statistics collection and using James
feedback, statistics collection is now being added as part of phoenix-core. A
SYSTEM.STATs table is getting created
{code}
CREATE TABLE SYSTEM.STATS (
TABLE_NAME VARCHAR NOT NULL,
REGION_NAME VARCHAR,
GUIDEPOSTS VARBINARY[],
MIN_KEY VARBINARY,
MAX_KEY VARBINARY
CONSTRAINT pk PRIMARY KEY (TABLE_NAME.REGION_NAME)
);
{code}
The statistics collection could be triggered using an Endpoint and the same
endpoint is also a BaseRegionObserver that tries to collect the stats during
major compaction. for now there are only 3 stats - > max key, min key and
guideposts.
The collected stats are updated to the stats table created.
The MetaDAtaEndpointImpl tries to read the udpated stats periodically. The
PTable currently has the PTAbleStats which is now improvised to collect these
stats. In the existing code the ParallelIterators, during every scan, tries to
find the max key and min key and use them to use in the Bytes.split() to find
keys inside a region. The StatsManager was being used to manage these min key
and max key retrieval. Now the PTable which has a PTableStats would be
directly populated when the MetaDataEndpointImpl does a getTable() and that
would be directly used as stats for the ParallelIterators.
The WIP of this patch is
-> Created a Phoenix table for stats
-> Using the StatsCollector endpoint the stats table would be populated.
-> The MetaDataEndpointImpl would use getTable() to collect the stats.
-> The PTable.proto has been changed to accomodate max key, min key and guide
posts and all the toProto() and createFromProto() have been changed to
construct the stats and associate with the the PTable.
-> As the stats collector is an endpoint the code that tries to update the
stats in the stats table is unified at one place.
Some doubts/queries
-> Should the stats collection be per region and per CF? Because in major
compaction case we would be able to collect the stats per family (on a region).
Similarly when the StatsColletor endpoint is used per region we could issue a
scan and collect the stats per CF. So we may have to add the CF name also in
the primary key of the table and then try to group the stats like for a
CF/region/table - > what is the max key, min key and guide posts.
-> How should the guide posts information be consumed in ParallelIterators.
-> In case there are no stats collected during a scan using ParallelIterators
should we use the existing way of determining the region splits?
> Use stats to guide query parallelization
> ----------------------------------------
>
> Key: PHOENIX-180
> URL: https://issues.apache.org/jira/browse/PHOENIX-180
> Project: Phoenix
> Issue Type: Task
> Reporter: James Taylor
> Assignee: ramkrishna.s.vasudevan
> Labels: enhancement
>
> We're currently not using stats, beyond a table-wide min key/max key cached
> per client connection, to guide parallelization. If a query targets just a
> few regions, we don't know how to evenly divide the work among threads,
> because we don't know the data distribution. This other [issue]
> (https://github.com/forcedotcom/phoenix/issues/64) is targeting gather and
> maintaining the stats, while this issue is focused on using the stats.
> The main changes are:
> 1. Create a PTableStats interface that encapsulates the stats information
> (and implements the Writable interface so that it can be serialized back from
> the server).
> 2. Add a stats member variable off of PTable to hold this.
> 3. From MetaDataEndPointImpl, lookup the stats row for the table in the stats
> table. If the stats have changed, return a new PTable with the updated stats
> information. We may want to cache the stats row and have the stats gatherer
> invalidate the cache row when updated so we don't have to always do a scan
> for it. Additionally, it would be idea if we could use the same split policy
> on the stats table that we use on the system table to guarantee co-location
> of data (for the sake of caching).
> - modify the client-side parallelization (ParallelIterators.getSplits()) to
> use this information to guide how to chunk up the scans at query time.
> This should help boost query performance, especially in cases where the data
> is highly skewed. It's likely the cause for the slowness reported in this
> issue: https://github.com/forcedotcom/phoenix/issues/47.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
