[
https://issues.apache.org/jira/browse/IMPALA-9482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18118986#comment-18118986
]
ASF subversion and git services commented on IMPALA-9482:
---------------------------------------------------------
Commit 6a8a5a52ba77b644e4b8a25f38c4c0a30217431b in impala's branch
refs/heads/master from Mihaly Szjatinya
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=6a8a5a52b ]
IMPALA-9821: Change DataSketches functions to return BINARY
DataSketches functions were added before BINARY type support
(IMPALA-9482), leading to the current STRING based solution, which is
incompatible with other systems (like Hive). After adding BINARY, the
changing of signatures (IMPALA-9821) was delayed, waiting for a major
version, because it is not a backward compatible. Now it is added in
preparation for Impala 5.0.
When Hive writes DataSketches to ORC tables, it correctly stores them as
BINARY. Impala's ds_*_sketch() functions, however, historically returned
STRING when reading Hive-created ORC tables, thus causing a type
mismatch error — the original issue in the Jira description, which was
solved by IMPALA-9482 without adding an explicit test for that exact
scenario. 'datasketches-hll-hive-orc.test' adds a regression test for
that.
After IMPALA-9482 fixed the ORC scanner to accept BINARY columns,
a frontend analysis error remained (if using sketches with consumer
functions):
AnalysisException: No matching function with signature:
ds_hll_estimate(BINARY)
This patch completes the fix by:
1. Changed return types to BINARY:
- ds_hll_sketch(), ds_cpc_sketch(), ds_theta_sketch(),
ds_kll_sketch()
2. Changed parameter types to BINARY for all consumer functions:
- HLL: ds_hll_estimate, ds_hll_estimate_bounds_as_string,
ds_hll_union_f, ds_hll_stringify, ds_hll_union (UDAF)
- CPC: ds_cpc_estimate, ds_cpc_stringify, ds_cpc_union_f,
ds_cpc_union (UDAF)
- Theta: ds_theta_estimate, ds_theta_exclude,
ds_theta_union_f, ds_theta_intersect_f, ds_theta_union
(UDAF), ds_theta_intersect (UDAF)
- KLL: ds_kll_quantile, ds_kll_n, ds_kll_rank,
ds_kll_quantiles_as_string, ds_kll_pmf_as_string,
ds_kll_cdf_as_string, ds_kll_stringify, ds_kll_union
(UDAF)
Thanks to IMPALA-9482 (BINARY type support), this only requires
FE changes — no backend modifications needed. The C++ UDFs already
treat StringVal as raw bytes.
Backward compatibility: All STRING sketch columns require migration.
Existing queries passing STRING sketches to consumer functions will
fail at analysis time with a type mismatch. Migrate by retyping the
sketch column in place with ALTER TABLE ... CHANGE COLUMN from STRING
to BINARY; the on-disk bytes are unchanged. Alternatively, store
sketches in BINARY columns or wrap them with CAST(col AS BINARY) at
the call site. Queries that only call ds_*_sketch() to create sketches
are unaffected — the return type changes silently from STRING to
BINARY, which only matters when INSERTing into a typed column.
Interop caveat: This fix enables reading Hive-written ORC sketch
columns. A separate interop hazard remains (IMPALA-9939): when the
*source column* being sketched is of STRING type, Impala hashes
the values as UTF-8 bytes while Hive uses UTF-16 chars. The
resulting sketches are not mergeable across engines regardless of
how they are stored (BINARY or STRING). To sketch string data
compatibly, cast the source to BINARY in Hive:
ds_hll_sketch(CAST(col AS BINARY))
Testing:
- Added testdata/workloads/functional-query/queries/QueryTest/
datasketches-hll-hive-orc.test demonstrating the ORC interop fix
- Updated the four existing sketch test files for BINARY return
types for tests/query_test/test_datasketches.py
Change-Id: Id4a6b54089dd356e37257bc24adeb1eb98e82c25
Assisted-by: Claude Sonnet 4.6 (GitHub Copilot)
Reviewed-on: http://gerrit.cloudera.org:8080/24590
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Support BINARY columns
> ----------------------
>
> Key: IMPALA-9482
> URL: https://issues.apache.org/jira/browse/IMPALA-9482
> Project: IMPALA
> Issue Type: New Feature
> Components: Backend, Frontend
> Reporter: Csaba Ringhofer
> Assignee: Csaba Ringhofer
> Priority: Major
> Fix For: Impala 4.2.0
>
>
> Hive's binary data types seem very similar to strings with some exceptions:
> - no implicit/explicit cast to any type except string
> - bit different built-in functions than for string, but some functions accept
> both string and binary (e.g. concat)
> - written to text files with base64 encoding
> (see https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF )
> I didn't look around about client support yet.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
