[
https://issues.apache.org/jira/browse/HIVE-30171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-30171:
----------------------------------
Description:
=, IN, GROUP BY, DISTINCT and joins compare raw bytes. On master, v =
parse_json('{"a":1,"b":"x"}') matches no rows
even when the stored value is identical.
The review of HIVE-30170 (#6881) found more places that compare or hash a
variant byte-wise. Each must follow the
semantic rule, or be kept byte-wise on purpose:
- KeyWrapperFactory.ListKeyWrapper (row-mode hash GROUP BY, DISTINCT,
count(DISTINCT)): hashes the raw key
objects. Iceberg's VariantData has no equals/hashCode, so map-side
aggregation never merges equal Iceberg variants
and switches itself off after 100k rows. Results stay correct; the reducer
groups by BinarySortable bytes.
- GenericUDAFApproximateDistinct: hashes the Java-serialized list of the
metadata and value arrays.
- HiveVariantComparator (merge join): its own byte-wise order, identical
today to ObjectInspectorUtils.compare.
Merge join must agree with the BinarySortable key order, so decide whether
it stays byte-wise.
- Array set functions (array_distinct, array_intersect, array_except,
array_union, array_remove) compare elements
with equals on Variant objects, so they must follow the same rule as =.
was:
=, IN, GROUP BY, DISTINCT and joins compare raw bytes. Task 0 showed on base
that v = parse_json('{"a":1,"b":"x"}') matches no rows even when the stored
value is identical.
The review of HIVE-30170 (#6881) found more places that compare or hash a
variant byte-wise. Each must follow the
semantic rule, or be kept byte-wise on purpose:
- KeyWrapperFactory.ListKeyWrapper (row-mode hash GROUP BY, DISTINCT,
count(DISTINCT)): hashes the raw key
objects. Iceberg's VariantData has no equals/hashCode, so map-side
aggregation never merges equal Iceberg variants
and switches itself off after 100k rows. Results stay correct; the reducer
groups by BinarySortable bytes.
- GenericUDAFApproximateDistinct: hashes the Java-serialized list of the
metadata and value arrays.
- HiveVariantComparator (merge join): its own byte-wise order, identical today
to ObjectInspectorUtils.compare.
Merge join must agree with the BinarySortable key order, so decide whether it
stays byte-wise.
- Array set functions use Object.equals on the raw element: array_distinct,
array_intersect, array_except,
array_union, array_remove. The answer depends on the source of the array
(standard values match since
HIVE-30170; Iceberg, text and LazyBinary values never match). array_contains
and array_position already go through
ObjectInspectorUtils.compare.
- Array set functions (array_distinct, array_intersect, array_except,
array_union, array_remove) compare elements
with equals on Variant objects, so they must follow the same rule as =.
> Semantic equality, hashing and grouping for VARIANT
> ---------------------------------------------------
>
> Key: HIVE-30171
> URL: https://issues.apache.org/jira/browse/HIVE-30171
> Project: Hive
> Issue Type: Sub-task
> Reporter: Denys Kuzmenko
> Priority: Major
>
> =, IN, GROUP BY, DISTINCT and joins compare raw bytes. On master, v =
> parse_json('{"a":1,"b":"x"}') matches no rows
>
> even when the stored value is identical.
>
>
>
>
>
> The review of HIVE-30170 (#6881) found more places that compare or hash a
> variant byte-wise. Each must follow the
>
> semantic rule, or be kept byte-wise on purpose:
>
>
> - KeyWrapperFactory.ListKeyWrapper (row-mode hash GROUP BY, DISTINCT,
> count(DISTINCT)): hashes the raw key
>
> objects. Iceberg's VariantData has no equals/hashCode, so map-side
> aggregation never merges equal Iceberg variants
>
> and switches itself off after 100k rows. Results stay correct; the
> reducer groups by BinarySortable bytes.
>
> - GenericUDAFApproximateDistinct: hashes the Java-serialized list of the
> metadata and value arrays.
>
> - HiveVariantComparator (merge join): its own byte-wise order, identical
> today to ObjectInspectorUtils.compare.
>
> Merge join must agree with the BinarySortable key order, so decide
> whether it stays byte-wise.
>
> - Array set functions (array_distinct, array_intersect, array_except,
> array_union, array_remove) compare elements
>
> with equals on Variant objects, so they must follow the same rule as =.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
