Mihaly Szjatinya has uploaded this change for review. (
http://gerrit.cloudera.org:8080/24590
Change subject: IMPALA-9821: Change DataSketches functions to return BINARY
......................................................................
IMPALA-9821: Change DataSketches functions to return BINARY
When Hive writes DataSketches to ORC tables, it correctly stores
them as BINARY. Impala's ds_*_sketch() functions historically
returned STRING, causing this error when reading Hive-created ORC
tables:
Type mismatch: table column STRING is map to column binary in
ORC file
After IMPALA-9482 fixed the ORC scanner to accept BINARY columns,
a frontend analysis error remained (if using sketches with consumer
functions):
AnalysisException: No matching function with signature:
ds_hll_estimate(BINARY)
This patch completes the fix by:
1. Changed return types to BINARY:
- ds_hll_sketch(), ds_cpc_sketch(), ds_theta_sketch(),
ds_kll_sketch()
2. Added BINARY overloads for all consumer functions:
- HLL: ds_hll_estimate, ds_hll_estimate_bounds_as_string,
ds_hll_union_f, ds_hll_stringify, ds_hll_union (UDAF)
- CPC: ds_cpc_estimate, ds_cpc_stringify, ds_cpc_union_f,
ds_cpc_union (UDAF)
- Theta: ds_theta_estimate, ds_theta_exclude,
ds_theta_union_f, ds_theta_intersect_f, ds_theta_union
(UDAF), ds_theta_intersect (UDAF)
- KLL: ds_kll_quantile, ds_kll_n, ds_kll_rank,
ds_kll_quantiles_as_string, ds_kll_pmf_as_string,
ds_kll_cdf_as_string, ds_kll_stringify, ds_kll_union
(UDAF)
3. Updated test expectations (TYPES: STRING → BINARY, storage
columns STRING → BINARY)
Thanks to IMPALA-9482 (BINARY type support), this only requires
FE changes — no backend modifications needed. The C++ UDFs already
treat StringVal as raw bytes.
Backward compatibility: All STRING overloads are preserved.
Existing queries using STRING sketch columns continue to work. New
tables should use BINARY columns for sketch storage.
Interop caveat: This fix enables reading Hive-written ORC sketch
columns. A separate interop hazard remains (IMPALA-9939): when the
*source column* being sketched is of STRING type, Impala hashes
the values as UTF-8 bytes while Hive uses UTF-16 chars. The
resulting sketches are not mergeable across engines regardless of
how they are stored (BINARY or STRING). To sketch string data
compatibly, cast the source to BINARY in Hive:
ds_hll_sketch(CAST(col AS BINARY))
Testing:
- Added datasketches-hll-hive-orc.test demonstrating the ORC
interop fix
- Updated all four .test files for BINARY return types
- All test_datasketches.py tests pass (5/5)
Change-Id: Id4a6b54089dd356e37257bc24adeb1eb98e82c25
Assisted-by: Claude Sonnet 4.6 (GitHub Copilot)
---
M common/function-registry/impala_functions.py
M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java
M testdata/workloads/functional-query/queries/QueryTest/datasketches-cpc.test
A
testdata/workloads/functional-query/queries/QueryTest/datasketches-hll-hive-orc.test
M testdata/workloads/functional-query/queries/QueryTest/datasketches-hll.test
M testdata/workloads/functional-query/queries/QueryTest/datasketches-kll.test
M testdata/workloads/functional-query/queries/QueryTest/datasketches-theta.test
M tests/query_test/test_datasketches.py
8 files changed, 196 insertions(+), 27 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24590/1
--
To view, visit http://gerrit.cloudera.org:8080/24590
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Id4a6b54089dd356e37257bc24adeb1eb98e82c25
Gerrit-Change-Number: 24590
Gerrit-PatchSet: 1
Gerrit-Owner: Mihaly Szjatinya <[email protected]>
