[
https://issues.apache.org/jira/browse/HIVE-30176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-30176:
----------------------------------
Description:
With variant.shredding.enabled, Hive rewrites variant_get(col, '$.path', type)
<op> literal into an Iceberg
extract(col, "$.path", type) predicate. Iceberg 1.12 cannot plan with it:
- ExpressionUtil.describe throws on extract terms;
- InclusiveMetricsEvaluator reads the manifest variant bounds in big-endian
order.
So Hive strips these predicates from the planning filter, and every data file
is planned.
apache/iceberg#18518 fixes both but is unreleased. Until Hive's Iceberg version
includes it, prune in split generation: evaluate the top-level extract
conjuncts, or the NOT of one, against each file's variant bounds with
InclusiveMetricsEvaluator on little-endian views of the bound buffers. Once
#18518 is in Hive's Iceberg version, keep extract in the planning filter and
remove the bridge.
Known limitation, as in Iceberg's own planning: variant string bounds written
by Iceberg 1.10/1.11 can be inverted (apache/iceberg#17397) and are trusted.
was:
With variant.shredding.enabled, Hive rewrites variant_get(col, '$.path', type)
<op> literal into an Iceberg
extract(col, "$.path", type) predicate. Iceberg 1.12 cannot plan with it:
- ExpressionUtil.describe throws on extract terms;
- InclusiveMetricsEvaluator reads the manifest variant bounds in big-endian
order.
So Hive strips these predicates from the planning filter, and every data file
is planned.
apache/iceberg#18518 fixes both but is unreleased. Until Hive's Iceberg version
includes it, prune in split
generation: evaluate the top-level extract conjuncts, or the NOT of one,
against each file's variant bounds with
InclusiveMetricsEvaluator on little-endian views of the bound buffers. Once
#18518 is in Hive's Iceberg version, keep
extract in the planning filter and remove the bridge.
Known limitation, as in Iceberg's own planning: variant string bounds written
by Iceberg 1.10/1.11 can be inverted
(apache/iceberg#17397) and are trusted.
> Iceberg: file-level pruning of shredded VARIANT values
> ----------------------------------------------------------
>
> Key: HIVE-30176
> URL: https://issues.apache.org/jira/browse/HIVE-30176
> Project: Hive
> Issue Type: Sub-task
> Reporter: Denys Kuzmenko
> Priority: Major
> Labels: pull-request-available
>
> With variant.shredding.enabled, Hive rewrites variant_get(col, '$.path',
> type) <op> literal into an Iceberg
> extract(col, "$.path", type) predicate. Iceberg 1.12 cannot plan with it:
> - ExpressionUtil.describe throws on extract terms;
> - InclusiveMetricsEvaluator reads the manifest variant bounds in big-endian
> order.
> So Hive strips these predicates from the planning filter, and every data file
> is planned.
> apache/iceberg#18518 fixes both but is unreleased. Until Hive's Iceberg
> version includes it, prune in split generation: evaluate the top-level
> extract conjuncts, or the NOT of one, against each file's variant bounds with
> InclusiveMetricsEvaluator on little-endian views of the bound buffers. Once
> #18518 is in Hive's Iceberg version, keep extract in the planning filter and
> remove the bridge.
> Known limitation, as in Iceberg's own planning: variant string bounds written
> by Iceberg 1.10/1.11 can be inverted (apache/iceberg#17397) and are trusted.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
