Mihaly Szjatinya has uploaded a new patch set (#4). (
http://gerrit.cloudera.org:8080/24590 )
Change subject: IMPALA-9821: Change DataSketches functions to return BINARY
......................................................................
IMPALA-9821: Change DataSketches functions to return BINARY
Currently Impala returns a STRING value for ds_hll_sketch() and
ds_hll_union() functions. Among other things, that causes interop issues
with e.g. Hive, which returns BINARY, and particularly when dealing with
ORC tables with its strict type system.
When Hive writes DataSketches to ORC tables, it correctly stores them as
BINARY. Impala's ds_*_sketch() functions historically returned STRING,
causing this error when reading Hive-created ORC tables:
Type mismatch: table column STRING is map to column binary in
ORC file
After IMPALA-9482 fixed the ORC scanner to accept BINARY columns,
a frontend analysis error remained (if using sketches with consumer
functions):
AnalysisException: No matching function with signature:
ds_hll_estimate(BINARY)
This patch completes the fix by:
1. Changed return types to BINARY:
- ds_hll_sketch(), ds_cpc_sketch(), ds_theta_sketch(),
ds_kll_sketch()
2. Changed parameter types to BINARY for all consumer functions:
- HLL: ds_hll_estimate, ds_hll_estimate_bounds_as_string,
ds_hll_union_f, ds_hll_stringify, ds_hll_union (UDAF)
- CPC: ds_cpc_estimate, ds_cpc_stringify, ds_cpc_union_f,
ds_cpc_union (UDAF)
- Theta: ds_theta_estimate, ds_theta_exclude,
ds_theta_union_f, ds_theta_intersect_f, ds_theta_union
(UDAF), ds_theta_intersect (UDAF)
- KLL: ds_kll_quantile, ds_kll_n, ds_kll_rank,
ds_kll_quantiles_as_string, ds_kll_pmf_as_string,
ds_kll_cdf_as_string, ds_kll_stringify, ds_kll_union
(UDAF)
Thanks to IMPALA-9482 (BINARY type support), this only requires
FE changes — no backend modifications needed. The C++ UDFs already
treat StringVal as raw bytes.
Backward compatibility: All STRING sketch columns require migration.
Existing queries passing STRING sketches to consumer functions will
fail at analysis time with a type mismatch. Migrate by storing
sketches in BINARY columns (or using CAST(col AS BINARY) at the
call site). Queries that only call ds_*_sketch() to create sketches
are unaffected — the return type changes silently from STRING to
BINARY, which only matters when INSERTing into a typed column.
Interop caveat: This fix enables reading Hive-written ORC sketch
columns. A separate interop hazard remains (IMPALA-9939): when the
*source column* being sketched is of STRING type, Impala hashes
the values as UTF-8 bytes while Hive uses UTF-16 chars. The
resulting sketches are not mergeable across engines regardless of
how they are stored (BINARY or STRING). To sketch string data
compatibly, cast the source to BINARY in Hive:
ds_hll_sketch(CAST(col AS BINARY))
Testing:
- Added testdata/workloads/functional-query/queries/QueryTest/
datasketches-hll-hive-orc.test demonstrating the ORC interop fix
- Updated the four existing sketch test files for BINARY return
types for tests/query_test/test_datasketches.py
Change-Id: Id4a6b54089dd356e37257bc24adeb1eb98e82c25
Assisted-by: Claude Sonnet 4.6 (GitHub Copilot)
---
M common/function-registry/impala_functions.py
M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java
M testdata/workloads/functional-query/queries/QueryTest/datasketches-cpc.test
A
testdata/workloads/functional-query/queries/QueryTest/datasketches-hll-hive-orc.test
M testdata/workloads/functional-query/queries/QueryTest/datasketches-hll.test
M testdata/workloads/functional-query/queries/QueryTest/datasketches-kll.test
M testdata/workloads/functional-query/queries/QueryTest/datasketches-theta.test
M tests/query_test/test_datasketches.py
8 files changed, 413 insertions(+), 148 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/24590/4
--
To view, visit http://gerrit.cloudera.org:8080/24590
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id4a6b54089dd356e37257bc24adeb1eb98e82c25
Gerrit-Change-Number: 24590
Gerrit-PatchSet: 4
Gerrit-Owner: Mihaly Szjatinya <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Mihaly Szjatinya <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
