[
https://issues.apache.org/jira/browse/HIVE-26313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17702796#comment-17702796
]
Alessandro Solimando commented on HIVE-26313:
---------------------------------------------
[~zabetak] there is a draft branch available
[here|https://github.com/asolimando/hive/tree/master-HIVE-26313-statistics_blob],
it's the one [~veghlaci05] is referring to.
The code is based on _Jackson_ and _Immutables_ libraries and has the building
blocks to serialize statistics to _json_ and deserialize them.
What I was aiming for, was to keep both individual columns and the blob, test
that everything was working end-to-end (comparing both), then remove and clean
up the individual columns version once I was happy with the result.
That's the reason why you see many unnecessary serialization and
deserialization of the json blob.
At the same time the idea was to also simplify the subclasses of _
ColumnStatsMerger_ and push more complexity into each class (in line with
HIVE-27000 but pushed even further).
I am not currently working on it, so if you are interested feel free to pick
this up and use the branch if it's useful.
> Aggregate all column statistics into a single field in metastore
> ----------------------------------------------------------------
>
> Key: HIVE-26313
> URL: https://issues.apache.org/jira/browse/HIVE-26313
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore, Statistics
> Affects Versions: 4.0.0-alpha-2
> Reporter: Alessandro Solimando
> Priority: Major
> Labels: backward-incompatible
>
> At the moment, column statistics tables in the metastore schema look like
> this (it's similar for _PART_COL_STATS_):
> {noformat}
> CREATE TABLE "APP"."TAB_COL_STATS"(
> "CAT_NAME" VARCHAR(256) NOT NULL,
> "DB_NAME" VARCHAR(128) NOT NULL,
> "TABLE_NAME" VARCHAR(256) NOT NULL,
> "COLUMN_NAME" VARCHAR(767) NOT NULL,
> "COLUMN_TYPE" VARCHAR(128) NOT NULL,
> "LONG_LOW_VALUE" BIGINT,
> "LONG_HIGH_VALUE" BIGINT,
> "DOUBLE_LOW_VALUE" DOUBLE,
> "DOUBLE_HIGH_VALUE" DOUBLE,
> "BIG_DECIMAL_LOW_VALUE" VARCHAR(4000),
> "BIG_DECIMAL_HIGH_VALUE" VARCHAR(4000),
> "NUM_DISTINCTS" BIGINT,
> "NUM_NULLS" BIGINT NOT NULL,
> "AVG_COL_LEN" DOUBLE,
> "MAX_COL_LEN" BIGINT,
> "NUM_TRUES" BIGINT,
> "NUM_FALSES" BIGINT,
> "LAST_ANALYZED" BIGINT,
> "CS_ID" BIGINT NOT NULL,
> "TBL_ID" BIGINT NOT NULL,
> "BIT_VECTOR" BLOB,
> "ENGINE" VARCHAR(128) NOT NULL
> );
> {noformat}
> The idea is to have a single blob named _STATISTICS_ to replace them, as
> follows:
> {noformat}
> CREATE TABLE "APP"."TAB_COL_STATS"(
> "CAT_NAME" VARCHAR(256) NOT NULL,
> "DB_NAME" VARCHAR(128) NOT NULL,
> "TABLE_NAME" VARCHAR(256) NOT NULL,
> "COLUMN_NAME" VARCHAR(767) NOT NULL,
> "COLUMN_TYPE" VARCHAR(128) NOT NULL,
> "STATISTICS" BLOB,
> "LAST_ANALYZED" BIGINT,
> "CS_ID" BIGINT NOT NULL,
> "TBL_ID" BIGINT NOT NULL,
> "ENGINE" VARCHAR(128) NOT NULL
> );
> {noformat}
> The _STATISTICS_ column could be the serialization of a Json-encoded string,
> which will be consumed in a "schema-on-read" fashion.
> At first at least the removed column statistics will be encoded in the
> _STATISTICS_ column, but since each "consumer" will read the portion of the
> schema it is interested into, multiple engines (see the _ENGINE_ column) can
> read and write statistics as they deem fit.
> Another advantage is that, if we plan to add more statistics in the future,
> we won't need to change the thrift interface for the metastore again.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
