[
https://issues.apache.org/jira/browse/CALCITE-1787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16037436#comment-16037436
]
Julian Hyde commented on CALCITE-1787:
--------------------------------------
[~zhumayun], I reviewed your pull request. I think we would be making a mistake
to allow {{count(distinct sketchColumn)}} in SQL; the syntax should be
{{count(distinct column)}} and Calcite should automatically rewrite to a sketch
if possible. In particular the 3 test cases
https://github.com/apache/calcite/pull/455/files#diff-88c776102776d0dfedaf74e4be27854fR2126
are
{noformat}
select count(distinct \"user_unique\") as users from \"wiki\"
select count(distinct \"added\") as \"added\" from \"wiki\"
select count(distinct \"user_unique\") as users from \"foodmart\"
{noformat}
but I think they should be
{noformat}
select count(distinct \"user_id\") as users from \"wiki\"
select count(distinct \"added\") as \"added\" from \"wiki\"
select count(distinct \"customer_id\") as users from \"foodmart\"
{noformat}
This means adding a "user_id" column to the "wiki" table, and Druid (or
Calcite's Druid adapter) needs to know that the "user_unique" sketch is a
"hyperUnique" sketch of "user_id".
It is possible that "user_id" is never used directly; I'm no expert on
designing Druid schemas, but I imagine it will use fewer resources if it is
declared as a dimension rather than a metric.
In "foodmart", the "customer_id" column already exists; we will need to make
the Druid adapter know that "user_unique" is a "thetaSketch" of "customer_id".
Lastly, all of these rewrites would be enabled only if the
{{approximateDistinctCount}} property (added in CALCITE-1587) is true. Thus
{{sql(sql)}} in your tests becomes something like this:
{code}CalciteAssert.that()
.enable(enabled())
.with(ImmutableMap.of("model", FOODMART.getPath()))
.with(CalciteConnectionProperty.APPROXIMATE_DISTINCT_COUNT.name(), true)
.query(sql){code}
> thetaSketch Support for Druid Adapter
> -------------------------------------
>
> Key: CALCITE-1787
> URL: https://issues.apache.org/jira/browse/CALCITE-1787
> Project: Calcite
> Issue Type: New Feature
> Components: druid
> Affects Versions: 1.12.0
> Reporter: Zain Humayun
> Assignee: Zain Humayun
> Priority: Minor
>
> Currently, the Druid adapter does not support the
> [thetaSketch|http://druid.io/docs/latest/development/extensions-core/datasketches-aggregators.html]
> aggregate type, which is used to measure the cardinality of a column
> quickly. Many Druid instances support theta sketches, so I think it would be
> a nice feature to have.
> I've been looking at the Druid adapter, and propose we add a new DruidType
> called {{thetaSketch}} and then add logic in the {{getJsonAggregation}}
> method in class {{DruidQuery}} to generate the {{thetaSketch}} aggregate.
> This will require accessing information about the columns (what data type
> they are) so that the thetaSketch aggregate is only produced if the column's
> type is {{thetaSketch}}.
> Also, I've noticed that a {{hyperUnique}} DruidType is currently defined, but
> a {{hyperUnique}} aggregate is never produced. Since both are approximate
> aggregators, I could also couple in the logic for {{hyperUnique}}.
> I'd love to hear your thoughts on my approach, and any suggestions you have
> for this feature.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
