Re: Mondrian + Calcite + Druid

Do you have any docs or examples or suggestions around how that could be done? It does sound like it'd be useful but I don't know where to start. Gian On Mon, Jan 23, 2017 at 12:19 PM, Julian Hyde <jh...@apache.org> wrote: > > > On Jan 23, 2017, at 12:14 PM, Gian Merlino <g.

Re: Mondrian + Calcite + Druid

Date dimension tables don't work well with Druid. Druid works best if you store the timestamp in Druid's native timestamp column and then use Druid's native time functions to do date filtering and grouping. Instead of: select "datasource"."username_0" as "c0", "dimdate"."Date" as "c1",

Re: Druid multi-value columns + Calcite

2017 at 12:10 PM, Julian Hyde <jh...@apache.org> wrote: > How about modeling them as a collection (ARRAY or MULTISET)? This document > describes them pretty well: http://farrago.sourceforge. > net/design/CollectionTypes.html <http://farrago.sourceforge. > net/design/CollectionTy

Re: [ANNOUNCE] New committer: Gian Merlino

. So far I have it has been a pleasure working with other folks in the Calcite community and I am glad to be part of it. Gian On Fri, Feb 17, 2017 at 8:30 AM, Josh Elser <els...@apache.org> wrote: > Welcome, Gian! > > > Julian Hyde wrote: > >> On behalf of the PMC I

Druid multi-value columns + Calcite

Hey Calcites, I'm hoping for some feedback on how to best handle Druid multi-value columns in SQL. Background: Druid has a "multi-value column" feature that lets you have string columns with more than one value per row. Details and examples of how this works are here:

Re: What type is DATE + INTERVAL?

> > FLOOR( .. TO timeUnit) is non-standard, if that matters. > > Have you seen DateRangeRules (and DateRangeRulesTest)? The work you are > doing would fit in perfectly. > > Julian > > > > > On Feb 16, 2017, at 1:28 PM, Gian Merlino <g...@imply.io> wrote: > > >

Re: Embedding Calcite, adjusting convertlets

, and will commit shortly. > > Julian > > > On Nov 23, 2016, at 12:32 PM, Gian Merlino <g...@imply.io> wrote: > > > > Do you know examples of projects that use Planner or PlannerImpl > currently > > (from "outside")? As far as I can tell, wi

Re: Embedding Calcite, adjusting convertlets

ass of CalcitePrepare that > works for your environment, one with an explicit convertletTable rather > than just using the default. > > Julian > > > > On Nov 17, 2016, at 5:01 PM, Gian Merlino <g...@imply.io> wrote: > > > > Hey Julian, > > > &g

Re: Embedding Calcite, adjusting convertlets

ta and runtime engine. > > As long as you build the valuable stuff into planner rules, new relational > operators (if necessary) and use the schema SPI, you should be able to > change packaging in the future. > > Julian > > > > > > On Nov 17, 2016, at 1:59

Re: Embedding Calcite, adjusting convertlets

Rules(myRules)) .executor(new RexExecutorImpl(Schemas.createDataContext(null))) .context(Contexts.EMPTY_CONTEXT) .build(); return Frameworks.getPlanner(frameworkConfig); Gian On Sat, Dec 3, 2016 at 5:53 PM, Gian Merlino <g...@imply.io> wrote: > Sure, I added those

Re: Embedding Calcite, adjusting convertlets

ten). > > > On Dec 14, 2016, at 11:19 AM, Gian Merlino <g...@imply.io> wrote: > > > > Ah, thanks. So if that sort of thing is not a smoking gun, do you have an > > idea about where I should look next? If not I'll keep poking around. > > > > Gian > &

Re: Embedding Calcite, adjusting convertlets

; we introduced collation > as a trait, and that made subsets non-disjoint (a RelNode can be sorted on > (x, y), and also on (x), and also on (), and also on (z)) so we made > RelSubset just a view onto a RelSet, filtering the list of RelNodes > according to the ones that have (“subsume”

Re: Column Ordinal Support in GROUP BY

om Microsoft Power BI > > On Wed, Dec 14, 2016 at 9:00 PM, Gian Merlino <g...@imply.io> wrote: > > > With Tableau you can use a TDC file to tweak the SQL generation. If you > set > > CAP_QUERY_GROUP_BY_ALIAS and CAP_QUERY_GROUP_BY_DEGREE to "no" then > Tablea

Re: Column Ordinal Support in GROUP BY

With Tableau you can use a TDC file to tweak the SQL generation. If you set CAP_QUERY_GROUP_BY_ALIAS and CAP_QUERY_GROUP_BY_DEGREE to "no" then Tableau will avoid using aliases and ordinals for GROUP BY. See http://kb.tableau.com/articles/knowledgebase/customizing-odbc-connections for more info.

Re: Embedding Calcite, adjusting convertlets

This was totally user error, I was ignoring "fields" and "collation" of the RelRoot. Passing "collation" to planner.transform stopped it from removing the sort. Gian On Wed, Dec 14, 2016 at 3:21 PM, Gian Merlino <g...@imply.io> wrote: > I don't think so

Re: Embedding Calcite, adjusting convertlets

her > set-ops? Are you sorting on columns that do not appear in the final result? > > Julian > > > > On Nov 28, 2016, at 10:45 AM, Gian Merlino <g...@imply.io> wrote: > > > > I traveled a bit down the Frameworks/Planner road and got most of my > tests >

Re: Pushing order by to source

Fwiw, Druid has a built-in set of Calcite rules now, and it does support pushing down order by and limit. It can even re-order the grouping dimensions for better performance, and choose timeseries, topN, or groupBy as appropriate. Depending on what you're trying to do, this may work for you. The

Re: Pushing order by to source

e > something of a fork right now, and that’s not bad per se, but it will be > bad if one of the copies of the code starts to fall behind in functionality. > > Julian > > > > On Dec 19, 2016, at 10:57 AM, Gian Merlino <g...@imply.io> wrote: > > > > Fwiw, Druid ha

PushProjector behavior

PushProjector has some code like this in "convertProject": if ( (projRefs.cardinality() == nChildFields) && (childPreserveExprs.size() == 0)) { return null; Which, if I am following right, means when there aren't any preserve-exprs, we will only push a project past a

Re: Setting up remote JDBC Calcite Against Druid using Avatica

Hey Jack, Based on the URL http://localhost:8082/druid/v2/sql it looks like you're using the built in Druid SQL. It doesn't support authentication yet so you could put anything in for user/password (it will be ignored). This will change in the future, when there will be a server-side option to

Re: Setting up remote JDBC Calcite Against Druid using Avatica

at sqlline.SqlLine.begin(SqlLine.java:668) > at sqlline.SqlLine.start(SqlLine.java:373) > at sqlline.SqlLine.main(SqlLine.java:265) > > > > > > On Mon, Jun 26, 2017 at 3:04 PM, Gian Merlino <g...@imply.io> wrote: > > > Hey Jack, > > >

Re: Avatica and load balancers

tica :) > > > > The only real configuration that comes to mind is specifying the > > "jdbc:phoenix:thin" jdbc url prefix > > > > > > On 5/25/17 7:25 PM, Gian Merlino wrote: > >> > >> I was thinking that client retries are probably a bad thing, t

Re: Avatica and load balancers

In the first cut, we weren't planning > on doing any client retries, but that's probably a good idea to add. > > James > > On Thu, May 25, 2017 at 1:28 PM, Gian Merlino <g...@imply.io> wrote: > > > I'm asking in the context of Druid, which also uses Avatica. Alth

Re: Avatica and load balancers

Thu, May 25, 2017 at 12:02 PM, Gian Merlino <g...@imply.io> wrote: > > > Is anyone out there using Avatica with servers (that don't share > connection > > state) behind load balancers? Is that a workable configuration? I'm > > guessing it might be if sticky sessions a

Re: Starting with Apache Calcite as SQLPlanner

Druid uses Calcite for SQL parsing and planning. The code is all in https://github.com/druid-io/druid/tree/master/sql/src/main/java/io/druid/sql if you want to take a look. The user-facing docs are at http://druid.io/docs/latest/querying/sql.html if you want to see how users interact with it. The

Re: [VOTE] Release apache-calcite-1.14.0 (release candidate 0)

I built a version of Druid using an artifact I built from the calcite-1.14.0 tag in git (7426eef84) and our SQL test suite passed. So, +1! (non-binding) Gian On Wed, Sep 27, 2017 at 4:25 PM, Michael Mior wrote: > Thanks for the comments. Will make note of the history change

Re: [DISCUSS] Reviewing pull requests

We have a similar problem in the Druid community. PR review backlogs are long and not all committers help out equally. We're grateful when anyone does show an interest in developing the Druid community rather than just using Druid or contributing patches. But that interest in the community is not

Connection time zone concept

Hi Calcites, In Druid SQL we have a connection time zone concept; see the mentions of "sqlTimeZone" here: http://druid.io/docs/latest/querying/sql.html. It affects the time zone of any operation involving timestamps, including the interpretation of literals, the behavior of functions like FLOOR

Re: Embed druid-sql inside Calcite?

odule. > > I think, the hypothetical case you mentioned is also worth considering, to > ease up the development process, we can consider moving calcite-druid as a > module in druid, so that we make release of both druid-sql and > calcite-adapter together. > > On Wed, 7 Feb 2018 at 09:

Re: Embed druid-sql inside Calcite?

d adapter. The challenge will be the > transition. You will need to convince the Hive developers that the move is > worthwhile. (It will help if you can point to some quick benefits to making > the transition.) > > Julian > > > > On Feb 7, 2018, at 2:59 PM, Gian Merlino <g...

Re: Embed druid-sql inside Calcite?

es while > > the execution part and result serde which pulls in lots of druid > > dependencies can reside in separate module and calcite druid-adapter need > > not depend on that module. > > > > I think, the hypothetical case you mentioned is also worth considering, > to

Re: Setting up Calcite development with IDEA

Welcome to Calcite!! To start go to File > Open... and open up Calcite's "pom.xml" file. When IntelliJ asks if you want to open it as a project or a file, say you want a project. Also say yes when it asks if you want a new window. Then, IntelliJ's Maven project importer should handle the rest. I

Embed druid-sql inside Calcite?

Hi Calcites, I would like to raise the idea of adding druid-sql ( http://search.maven.org/#artifactdetails%7Cio.druid%7Cdruid-sql%7C0.11.0%7Cjar) as a dependency in Calcite's Druid adapter. It should reduce the size of calcite-druid substantially, since it would mostly just be calling into

Re: DISTINCT ON clause

In Druid I was hoping to eventually deal with this using FIRST and LAST aggregators. So the original query would be: SELECT a, FIRST(b), FIRST(c) FROM T GROUP BY a However, we have only implemented FIRST and LAST for numeric types, not for strings or arrays yet. And we also haven't wired it

Re: How to convert a node to another that has a different row type ?

Hi Gelbana, My understanding is that the planner rules are not meant to change row types or query behavior. They are just meant to transform into something that is equivalent, but hopefully better in some way (lower cost, proper calling convention, etc). What you can do is have your SQLiQuery's

System fields

I'm looking at adding system fields to certain tables in Druid: they should be referenceable by name but shouldn't show up through star-expansion. For example, we'd like to expose the name of the data file currently being read as "__filename". I'm trying to figure out if this is something we can

Re: System fields

t would be fairly easy > to implement because it only affects star expansion (and therefore only > affects the validator and sql-to-rel-converter, not the planner). > > Julian > > > > On Oct 27, 2023, at 12:45 PM, Gian Merlino wrote: > > > > I'm looking at

Re: Calcite jars with shaded guava

wrote: > The bug is now fixed. (I started from your PR, but I had to do quite a lot of > extra work. Despite what I said earlier, there were several places where > Calcite was using APIs added between Guava 16 and 19.) > > > On Dec 9, 2022, at 1:30 PM, Gian Merlino wr

Re: Calcite jars with shaded guava

It pains me to admit that the reason Druid still uses Guava 16 is that we still support Hadoop 2, which relies on APIs that have been removed in newer versions of Guava. From time to time it is proposed that the project drops support for Hadoop 2 in order to be free to move up Guava versions,

Re: Calcite jars with shaded guava

> > I am frustrated that the Hadoop folks get locked into a particular version of > Guava (and JDK… don’t get me started on that one) and in Calcite I would like > to do the opposite. > > Julian > > > > On Dec 8, 2022, at 9:24 AM, Gian Merlino wrote: > > > &g

Re: Calcite jars with shaded guava

to main.yml (GitHub CI), comments in > gradle.properties, and history.md (release notes for the upcoming > release). > > Julian > > On Fri, Dec 9, 2022 at 12:30 PM Gian Merlino wrote: > > > > If Calcite ends up dropping the minimum to 16, that would certainly be &g

Re: Calcite jars with shaded guava

and continue to make progress integrating this new version into Druid. On 2023/01/04 06:15:00 Gian Merlino wrote: > Thank you! I've built a copy of main and have started working on integrating > it into Druid. I'll let the group know if anything interesting comes up. > There is also a g

Re: [VOTE] Release Apache Calcite 1.35.0 (release candidate 2)

When testing this with Apache Druid we found a problem: the change in CALCITE-5747 broke a few of our test cases due to this piece: https://github.com/apache/calcite/pull/3245/files#diff-e873041549333502af52ece8a1b34301ae5a059ff4719e9bddbaef48929e7047L7338-L7341 Essentially, FLOOR and CEIL no

Re: [DISCUSS] Towards Calcite 1.35.0

Some more detail: - the PR for CALCITE-5477 is https://github.com/apache/calcite/pull/3249 - the PR for CALCITE-5479 is https://github.com/apache/calcite/pull/3030 I've just pushed up some updates to reflect the new approach we discussed for guava backwards-compat in CALCITE-5477, and to sync

Re: [DISCUSS] Towards Calcite 1.35.0

I committed the patch for CALCITE-5477. I would appreciate a review on CALCITE-5479, if someone has a chance. On 2023/06/07 09:53:08 Gian Merlino wrote: > Some more detail: > > - the PR for CALCITE-5477 is https://github.com/apache/calcite/pull/3249 > - the PR for CALCITE-5

Re: [DISCUSS] Towards Calcite 1.35.0

ld we review this pr, so that it could be merged in 1.35. then > >> the > >>>>> 1.35 > >>>>>> will almost have the full spark collection function. > >>>>>> I would very appreciate a review. if someone ha

[jira] [Created] (CALCITE-1347) Druid adapter column detection misses strings

Gian Merlino created CALCITE-1347: - Summary: Druid adapter column detection misses strings Key: CALCITE-1347 URL: https://issues.apache.org/jira/browse/CALCITE-1347 Project: Calcite Issue

[jira] [Created] (CALCITE-1658) DateRangeRules issues

Gian Merlino created CALCITE-1658: - Summary: DateRangeRules issues Key: CALCITE-1658 URL: https://issues.apache.org/jira/browse/CALCITE-1658 Project: Calcite Issue Type: Bug

[jira] [Created] (CALCITE-1488) ValuesReduceRule doesn't propagate rowType on Project of empty Values

Gian Merlino created CALCITE-1488: - Summary: ValuesReduceRule doesn't propagate rowType on Project of empty Values Key: CALCITE-1488 URL: https://issues.apache.org/jira/browse/CALCITE-1488 Project

[jira] [Created] (CALCITE-1479) AssertionError in ReduceExpressionsRule on multi-column IN subquery

Gian Merlino created CALCITE-1479: - Summary: AssertionError in ReduceExpressionsRule on multi-column IN subquery Key: CALCITE-1479 URL: https://issues.apache.org/jira/browse/CALCITE-1479 Project

[jira] [Created] (CALCITE-1437) AggregateNode NPE for aggregators other than SUM/COUNT

Gian Merlino created CALCITE-1437: - Summary: AggregateNode NPE for aggregators other than SUM/COUNT Key: CALCITE-1437 URL: https://issues.apache.org/jira/browse/CALCITE-1437 Project: Calcite

[jira] [Created] (CALCITE-1910) NPE on filtered aggregators using "IN"

Gian Merlino created CALCITE-1910: - Summary: NPE on filtered aggregators using "IN" Key: CALCITE-1910 URL: https://issues.apache.org/jira/browse/CALCITE-1910 Project: Calcite Issue

[jira] [Created] (CALCITE-1873) Validator errors when grouping by ordinals that refer to aliases

Gian Merlino created CALCITE-1873: - Summary: Validator errors when grouping by ordinals that refer to aliases Key: CALCITE-1873 URL: https://issues.apache.org/jira/browse/CALCITE-1873 Project

[jira] [Created] (CALCITE-1799) OR IN Subquery conversion wrong

Gian Merlino created CALCITE-1799: - Summary: OR IN Subquery conversion wrong Key: CALCITE-1799 URL: https://issues.apache.org/jira/browse/CALCITE-1799 Project: Calcite Issue Type: Bug

[jira] [Created] (CALCITE-2008) Fix braces in TRIM signature

Gian Merlino created CALCITE-2008: - Summary: Fix braces in TRIM signature Key: CALCITE-2008 URL: https://issues.apache.org/jira/browse/CALCITE-2008 Project: Calcite Issue Type: Bug

[jira] [Created] (CALCITE-2635) getMonotonocity is slow on wide tables

Gian Merlino created CALCITE-2635: - Summary: getMonotonocity is slow on wide tables Key: CALCITE-2635 URL: https://issues.apache.org/jira/browse/CALCITE-2635 Project: Calcite Issue Type

[jira] [Created] (CALCITE-3178) RexSimplify.simplifyOrTerms slow with large OR filters

Gian Merlino created CALCITE-3178: - Summary: RexSimplify.simplifyOrTerms slow with large OR filters Key: CALCITE-3178 URL: https://issues.apache.org/jira/browse/CALCITE-3178 Project: Calcite

[jira] [Created] (CALCITE-5428) Reduce minimum Guava version to 16.0.1.

Gian Merlino created CALCITE-5428: - Summary: Reduce minimum Guava version to 16.0.1. Key: CALCITE-5428 URL: https://issues.apache.org/jira/browse/CALCITE-5428 Project: Calcite Issue Type

[jira] [Created] (CALCITE-5479) Restore consistent handling of iFormalOperand in sequence checkers

Gian Merlino created CALCITE-5479: - Summary: Restore consistent handling of iFormalOperand in sequence checkers Key: CALCITE-5479 URL: https://issues.apache.org/jira/browse/CALCITE-5479 Project

[jira] [Created] (CALCITE-5477) Prefer Util.checkArgument over Preconditions.checkArgument

Gian Merlino created CALCITE-5477: - Summary: Prefer Util.checkArgument over Preconditions.checkArgument Key: CALCITE-5477 URL: https://issues.apache.org/jira/browse/CALCITE-5477 Project: Calcite

[jira] [Created] (CALCITE-5478) Use highest input precision for datetimes in SqlTypeFactoryImpl.leastRestrictive

Gian Merlino created CALCITE-5478: - Summary: Use highest input precision for datetimes in SqlTypeFactoryImpl.leastRestrictive Key: CALCITE-5478 URL: https://issues.apache.org/jira/browse/CALCITE-5478

[jira] [Created] (CALCITE-5865) ClassCastException with FLOOR and CEIL on conformances that are not builtin

Gian Merlino created CALCITE-5865: - Summary: ClassCastException with FLOOR and CEIL on conformances that are not builtin Key: CALCITE-5865 URL: https://issues.apache.org/jira/browse/CALCITE-5865