[jira] [Commented] (CALCITE-1787) thetaSketch Support for Druid Adapter

[Julian Hyde (JIRA)]

    [ 
https://issues.apache.org/jira/browse/CALCITE-1787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043293#comment-16043293
 ] 

Julian Hyde commented on CALCITE-1787:
--------------------------------------

I was thinking of adding a new collection alongside "dimensions" and "metrics":

{code}
  "abstractMetrics": [
    "user"
  ]
{code}

and "userUnique" continues to be a metric:

{code}
  {
    "name" : "user_unique",
    "type" : "hyperUnique",
    "fieldName" : "user"
  }
{code}

Then if you write "select count(distinct user) ..." the adapter will use the 
"user_unique" sketch in the background; if you write "select user ..." the 
adapter will give an error because the raw values of "user" are not stored.

If in future you add a histogram sketch:

{code}
  {
    "name" : "user_histogram",
    "type" : "histogram",
    "fieldName" : "user"
  }
{code}

then you will be able to do approximate median queries: "select count(distinct 
user), median(user), count(user) filter (where user > 50), ..." and the adapter 
will use the appropriate histogram for each aggregate function.

But notice that "user_histogram" and "user_unique" do not appear in queries. 
(You could use them in queries, but why would you?) 

> thetaSketch Support for Druid Adapter
> -------------------------------------
>
>                 Key: CALCITE-1787
>                 URL: https://issues.apache.org/jira/browse/CALCITE-1787
>             Project: Calcite
>          Issue Type: New Feature
>          Components: druid
>    Affects Versions: 1.12.0
>            Reporter: Zain Humayun
>            Assignee: Zain Humayun
>            Priority: Minor
>
> Currently, the Druid adapter does not support the 
> [thetaSketch|http://druid.io/docs/latest/development/extensions-core/datasketches-aggregators.html]
>  aggregate type, which is used to measure the cardinality of a column 
> quickly. Many Druid instances support theta sketches, so I think it would be 
> a nice feature to have.
> I've been looking at the Druid adapter, and propose we add a new DruidType 
> called {{thetaSketch}} and then add logic in the {{getJsonAggregation}} 
> method in class {{DruidQuery}} to generate the {{thetaSketch}} aggregate. 
> This will require accessing information about the columns (what data type 
> they are) so that the thetaSketch aggregate is only produced if the column's 
> type is {{thetaSketch}}. 
> Also, I've noticed that a {{hyperUnique}} DruidType is currently defined, but 
> a {{hyperUnique}} aggregate is never produced. Since both are approximate 
> aggregators, I could also couple in the logic for {{hyperUnique}}.
> I'd love to hear your thoughts on my approach, and any suggestions you have 
> for this feature.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)