[
https://issues.apache.org/jira/browse/IMPALA-10020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17185348#comment-17185348
]
ASF subversion and git services commented on IMPALA-10020:
----------------------------------------------------------
Commit 28d94851b1418ddd57e4c046646f7a2df0e702d7 in impala's branch
refs/heads/master from Gabor Kaszab
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=28d9485 ]
IMPALA-10020: Implement ds_kll_cdf_as_string() function
This is the support for Cumulative Distribution Function (CDF) from
Apache DataSketches KLL algorithm collection. It receives a serialized
KLL sketch and one or more float values to represent ranges in the
sketched values.
E.g. [1, 5, 10] will mean the following ranges:
(-inf, 1), (-inf, 5), (-inf, 10), (-inf, +inf)
Returns a comma separated string where each value in the string is a
number in the range of [0,1] and shows that what percentage of the
data is in the particular ranges.
Note, ds_kll_cdf() should return an Array of doubles as the result but
with that we have to wait for the complex type support. Until, we
provide ds_kll_cdf_as_string() that can be deprecated once we
have array support. Tracking Jira for returning complex types from
functions is IMPALA-9520.
Example:
select ds_kll_cdf_as_string(ds_kll_sketch(float_col), 2, 4, 10)
from alltypes;
+----------------------------------------------------------+
| ds_kll_cdf_as_string(ds_kll_sketch(float_col), 2, 4, 10) |
+----------------------------------------------------------+
| 0.2,0.401644,1,1 |
+----------------------------------------------------------+
Change-Id: I77e6afc4556ad05a295b89f6d06c2e4a6bb2cf82
Reviewed-on: http://gerrit.cloudera.org:8080/16359
Reviewed-by: Gabor Kaszab <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Implement ds_kll_cdf() function
> -------------------------------
>
> Key: IMPALA-10020
> URL: https://issues.apache.org/jira/browse/IMPALA-10020
> Project: IMPALA
> Issue Type: New Feature
> Components: Backend, Frontend
> Reporter: Gabor Kaszab
> Assignee: Gabor Kaszab
> Priority: Major
> Fix For: Impala 4.0
>
>
> Requirements for ds_kll_cdf() (Cumulative Distribution Function):
> - Receives a serialized KLL sketch in BINARY type (in Impala it can be
> STRING as long as we don't have BINARY) as first parameter.
> - Receives one or more float values to create ranges from the sketched data.
> - In Hive the return type is an array of doubles. However, Impala can't
> return complex types from functions at this point so we have to find some
> alternative approaches to implement this function. Follow whatever solution
> came up in https://issues.apache.org/jira/browse/IMPALA-9962
> An example:
> {code:java}
> select ds_kll_cdf(sketch_col, 1, 2, 3, 4) from sketches_table;
> {code}
> This will generate the following ranges: (-inf, 1), (-inf,2), (-inf,3),
> (-inf,4), (-inf,+inf)
> In Hive, the result would have an array of 5 doubles for the 5 ranges, where
> each number gives the probability between [0,1] that an item will fall into
> the particular range. Or in other words a ratio of items belonging to that
> range.
> Taking input values such as: 1,2,3,4,5
> {code:java}
> select ds_kll_cdf(f, 1, 3, 4, 5, 10) from kll_sketches;
> +----------------------------+
> | _c0 |
> +----------------------------+
> | [0.0,0.4,0.6,0.8,1.0,1.0] |
> +----------------------------+
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
