Re: I wrote a guide to building new query language frontends in Calcite, as an interactive Jupyter Notebook (GraphQL used as example)

[Zoltan Farkas]

Gavin, 

awesome work!

I was on a similar quest a while ago, and went “full circle" on SQL :-)…

I was feeling like I am re-implementing SQL with another syntax… a syntax that 
very few people will be familiar with… unlike with SQL…
But everyone will have their own use cases and tradeoffs.

Here is a demo of how I went  ahead to use SQL together with REST-ful resources:

https://github.com/zolyfarkas/jaxrs-spf4j-demo/wiki/AvroCalciteRest 


cheers.

—Z

> On Jan 24, 2022, at 9:51 AM, Gavin Ray  wrote:
> 
> Thank you Stamatis and Walaa!
> I believe it's only been possible because of the great community around
> Calcite =)
> 
> And actually, Walaa was of much help when I was researching the idea.
> He talked with me a bit on the Coral slack and pointed me towards some good
> resources + shared experiences.
> 
> Coral is one of the projects I took inspiration from, along with
> Dremio/Trino cited in the readme.
> 
> On this topic -- could it make sense to form a Calcite "Special Interest
> Group" for GraphQL-on-Calcite, open to whoever is interested?
> It could be good to share ideas and try to prevent doing the same
> work/making mistakes that others have already learned from.
> 
> For instance, I have designed the GraphQL layer I am building to be
> customizable/extendable via "Conventions"
> A "Convention" allows you to configure the naming of the API operators and
> to register which operations you support.
> 
> This way, I can call my operations IE "_and", "_or", but someone else may
> want to call them "AND" and "OR"
> Or perhaps someone wants to support other binary operators/special
> operators that aren't part of the most common ones. Then they can extend
> the base operators and add their own.
> 
> 
> 
> On Sun, Jan 23, 2022 at 10:53 PM Walaa Eldin Moustafa 
> wrote:
> 
>> Nice article indeed. In Coral [1], we have simplified this process to the
>> end user where those steps can be done through implementing a small number
>> of abstract methods, listed here [2].
>> 
>> [1] https://github.com/linkedin/coral
>> [2]
>> 
>> https://github.com/linkedin/coral/blob/master/coral-common/src/main/java/com/linkedin/coral/common/ToRelConverter.java
>> 
>> On Sun, Jan 23, 2022 at 12:59 PM Stamatis Zampetakis 
>> wrote:
>> 
>>> Thanks for sharing this Gavin, very nice article!
>>> 
>>> On Sun, Jan 23, 2022 at 7:00 PM Gavin Ray  wrote:
>>> 
 I thought that someone else might find it useful, since there was no
 existing guide covering building new frontends.
 It looks best on Jetbrains Datalore, but you can also view it using the
 ".ipynb" in the Github repo:
 
 https://datalore.jetbrains.com/view/notebook/JYTVfn90xYSmv6U5f2NIQR
 
 
>>> 
>> https://github.com/GavinRay97/calcite-new-frontend-tutorial/blob/master/Calcite%20Frontend%20Guide.ipynb
 
>>> 
>> 

Re: calcite-core 1.28.0 now depends on kotlin-stdlib?

[Zoltan Farkas]

According to the published pom it is a direct dependency: 
https://repo1.maven.org/maven2/org/apache/calcite/calcite-core/1.28.0/calcite-core-1.28.0.pom
 



> On Nov 29, 2021, at 11:41 PM, Vladimir Sitnikov  
> wrote:
> 
>> since core/src/kotlin does not exist yet.
> 
> I mean core/src/main/kotlin does not exist yet
> 
> Vladimir

calcite-core 1.28.0 now depends on kotlin-stdlib?

[Zoltan Farkas]

I see that the latest release of calcite-core brings in kotlin-stdlib as a 
dependency.,


  org.jetbrains.kotlin
  kotlin-stdlib-jdk8
  1.5.31


I looked over the release notes, and I cannot find anything that mentions this, 
anyone knows why is this new dependency required?
(up until 1.27.0 kotlin was only a test dependency)

thanks!

—Z

Re: Why migrate from Maven to Gradle

[Zoltan Farkas]

Vladimir, 

A new user will probably be impacted more by the IDE experience.

Some IDEs do better at handling maven than gradle... For example I use 
NetBeans, and things just work with maven.

In my experience, Maven is simply a more mature tool (use it since 2002), more 
plugins, good IDE support, with more developers being fluent with it. 

Gradle will add friction to contributors familiar with maven and unfamiliar 
with gradle...  

Hope it’s worth it...

--z

> On Jun 15, 2020, at 2:49 PM, Vladimir Sitnikov  
> wrote:
> 
> 
>> 
>> Gradle is not that user friendly for new uses
> 
> Can you please elaborate?
> 
> Gradle's command line is easier to follow as it provides help with the
> usable tasks and descriptions,
> and it requires much less ceremony.
> For instance, with Maven I often had issues like "ExtenderSqlParserImpl not
> found" (== I had to build the project before importing to the IDE),
> and Gradle fixes that.
> 
> Vladimir

Re: Quicksql

[Zoltan Farkas]

What is the ultimate goal of the Calcite Interpreter? 

To provide some context, I have been playing around with calcite + REST (see 
https://github.com/zolyfarkas/jaxrs-spf4j-demo/wiki/AvroCalciteRest 
 for 
detail of my experiments)


—Z

> On Dec 9, 2019, at 9:05 PM, Julian Hyde  wrote:
> 
> Yes, virtualization is one of Calcite’s goals. In fact, when I created 
> Calcite I was thinking about virtualization + in-memory materialized views. 
> Not only the Spark convention but any of the “engine” conventions (Drill, 
> Flink, Beam, Enumerable) could be used to create a virtual query engine.
> 
> See e.g. a talk I gave in 2013 about Optiq (precursor to Calcite) 
> https://www.slideshare.net/julianhyde/optiq-a-dynamic-data-management-framework
>  
> .
> 
> Julian
> 
> 
> 
>> On Dec 9, 2019, at 2:29 PM, Muhammad Gelbana  wrote:
>> 
>> I recently contacted one of the active contributors asking about the
>> purpose of the project and here's his reply:
>> 
>> From my understanding, Quicksql is a data virtualization platform. It can
>>> query multiple data sources altogether and in a distributed way; Say, you
>>> can write a SQL with a MySql table join with an Elasticsearch table.
>>> Quicksql can recognize that, and then generate Spark code, in which it will
>>> fetch the MySQL/ES data as a temporary table separately, and then join them
>>> in Spark. The execution is in Spark so it is totally distributed. The user
>>> doesn't need to aware of where the table is from.
>>> 
>> 
>> I understand that the Spark convention Calcite has attempts to achieve the
>> same goal, but it isn't fully implemented yet.
>> 
>> 
>> On Tue, Oct 29, 2019 at 9:43 PM Julian Hyde  wrote:
>> 
>>> Anyone know anything about Quicksql? It seems to be quite a popular
>>> project, and they have an internal fork of Calcite.
>>> 
>>> https://github.com/Qihoo360/ 
>>> 
>>> 
>>> https://github.com/Qihoo360/Quicksql/tree/master/analysis/src/main/java/org/apache/calcite
>>> <
>>> https://github.com/Qihoo360/Quicksql/tree/master/analysis/src/main/java/org/apache/calcite
 
>>> 
>>> Julian
>>> 
>>> 
> 

Re: Question about Interpreter and Corelations

[Zoltan Farkas]

Done: https://issues.apache.org/jira/browse/CALCITE-3488 
<https://issues.apache.org/jira/browse/CALCITE-3488> 
let me know if you need any more info.

thank you

—Z

> On Nov 8, 2019, at 2:17 PM, Julian Hyde  wrote:
> 
> As I said, I think you should log a bug.
> 
> On Fri, Nov 8, 2019 at 10:23 AM Haisheng Yuan  wrote:
>> 
>> I am afraid this query can't be easily decorrelated.
>> 
>> - Haisheng
>> 
>> ------
>> 发件人：Zoltan Farkas
>> 日　期：2019年11月08日 22:46:53
>> 收件人：
>> 主　题：Re: Question about Interpreter and Corelations
>> 
>> Thanks Julian,
>> 
>> Any idea how could I de-corelate the query?
>> RelDecorrelator.decorrelateQuery does not seem to alter the plan.
>> 
>> Will log a jira for the interpreter issue.
>> 
>> thank you
>> 
>> —Z
>> 
>> 
>> 
>>> On Nov 7, 2019, at 5:08 PM, Julian Hyde  wrote:
>>> 
>>> I don’t recall what the interpreter is currently capable of, but you should 
>>> log a bug.
>>> 
>>> I wonder whether if you could get the query to work in the interpreter if 
>>> you decorrelated the query first.
>>> 
>>> Julian
>>> 
>>>> On Nov 7, 2019, at 11:16, Zoltan Farkas  
>>>> wrote:
>>>> 
>>>> for a test query with the following plan:
>>>> 
>>>> LogicalProject(name=[$1], friends=[$4])
>>>> LogicalCorrelate(correlation=[$cor0], joinType=[inner], 
>>>> requiredColumns=[{0}])
>>>>  LogicalTableScan(table=[[characters]])
>>>>  Collect(field=[EXPR$0])
>>>>LogicalProject(name=[$2])
>>>>  LogicalJoin(condition=[=($0, $1)], joinType=[inner])
>>>>LogicalProject(characterId2=[$1])
>>>>  LogicalFilter(condition=[=($0, $cor0.characterId)])
>>>>   LogicalTableScan(table=[[friendships]])
>>>>LogicalProject(characterId=[$0], name=[$1])
>>>>  LogicalTableScan(table=[[characters]])
>>>> 
>>>> I get :
>>>> 
>>>> java.lang.nsupportedOperationException
>>>>  at 
>>>> o.a.c.i.JaninoRexCompiler.lambda$compile$0(JaninoRexCompiler.java:94)[calcite-core-1.21.0.jar:1.21.0]
>>>>  at o.a.c.a.e.RexToLixTranslator.translate0(RexToLixTranslator.java:714)[^]
>>>>  at ^.translate(^:199)[^]
>>>>  at ^.translate0(^:684)[^]
>>>>  at ^.translate(^:199)[^]
>>>>  at ^.translate(^:194)[^]
>>>>  ...
>>>> 
>>>> when trying to use the Interpreter.
>>>> 
>>>> Is this a current limitation?
>>>> 
>>>> the query is :
>>>> 
>>>> select name,
>>>>ARRAY(select c2.name from friendships f, characters c2
>>>>   where f.characterId1 = c.characterId and 
>>>> f.characterId2 = c2.characterId) as friends
>>>> from characters c
>>>> 
>>>> let me know
>>>> 
>>>> thank you
>>>> 
>>>> —Z

[jira] [Created] (CALCITE-3488) Interpreter does not implement LogicalCorelate.

[Zoltan Farkas (Jira)]

Zoltan Farkas created CALCITE-3488:
--

 Summary: Interpreter does not implement LogicalCorelate.
 Key: CALCITE-3488
 URL: https://issues.apache.org/jira/browse/CALCITE-3488
 Project: Calcite
  Issue Type: Bug
Reporter: Zoltan Farkas


Example query:

{code}
select name,
 ARRAY(select c2.name from friendships f, characters c2
where f.characterId1 = c.characterId and 
f.characterId2 = c2.characterId) as friends
{code}

where the 2 tables involved are:

{code}
record Character {
  string characterId;
  string name = "";
  string homeWorldPlanet;
  string speciesName;
}

record Friendship {
  string characterId1;
  string characterId2;
}
{code}


the plan:

{code}
LogicalProject(name=[$1], friends=[$4])
  LogicalCorrelate(correlation=[$cor0], joinType=[inner], requiredColumns=[{0}])
LogicalTableScan(table=[[characters]])
Collect(field=[EXPR$0])
  LogicalProject(name=[$2])
LogicalJoin(condition=[=($0, $1)], joinType=[inner])
  LogicalProject(characterId2=[$1])
LogicalFilter(condition=[=($0, $cor0.characterId)])
  LogicalTableScan(table=[[friendships]])
  LogicalProject(characterId=[$0], name=[$1])
LogicalTableScan(table=[[characters]])\n
{code}

when interpreted, results in:

{code}
java.lang.UnsupportedOperationException
at 
o.a.c.i.JaninoRexCompiler.lambda$compile$0(JaninoRexCompiler.java:94)[calcite-core-1.21.0.jar:1.21.0]
at 
o.a.c.a.e.RexToLixTranslator.translate0(RexToLixTranslator.java:714)[^]
at ^.translate(^:199)[^]
at ^.translate0(^:684)[^]
at ^.translate(^:199)[^]
at ^.translate(^:194)[^]
at ^.translateList(^:928)[^]
at ^.translateList(^:901)[^]
at o.a.c.a.e.RexImpTable.implementCall(RexImpTable.java:1163)[^]
at ^.implementNullSemantics(^:1153)[^]
at ^.implementNullSemantics0(^:1043)[^]
at ^.lambda$createImplementor$4(^:690)[^]
at 
o.a.c.a.e.RexToLixTranslator.translateCall(RexToLixTranslator.java:758)[^]
at ^.translate0(^:730)[^]
at ^.translate(^:199)[^]
at ^.translate0(^:684)[^]
at ^.translate(^:199)[^]
at ^.translate(^:194)[^]
at ^.translateList(^:928)[^]
at ^.translateProjects(^:169)[^]
at o.a.c.i.JaninoRexCompiler.compile(JaninoRexCompiler.java:101)[^]
at o.a.c.i.Interpreter$CompilerImpl.compile(Interpreter.java:487)[^]
at o.a.c.i.Nodes$CoreCompiler.compile(Nodes.java:43)[^]
at o.a.c.i.TableScanNode.createEnumerable(TableScanNode.java:266)[^]
at ^.createProjectableFilterable(^:233)[^]
at ^.create(^:81)[^]
at o.a.c.i.Nodes$CoreCompiler.visit(Nodes.java:70)[^]
at j.i.r.NativeMethodAccessorImpl.invoke0(Native Method)
at ^.invoke(^:62)[^]
at 
j.i.r.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[^]
at j.l.r.Method.invoke(Method.java:566)[^]
at 
o.a.c.u.ReflectUtil.invokeVisitorInternal(ReflectUtil.java:257)[calcite-core-1.21.0.jar:1.21.0]
at ^.invokeVisitor(^:214)[^]
at o.a.c.u.ReflectUtil$1.invokeVisitor(^:464)[^]
at o.a.c.i.Interpreter$CompilerImpl.visit(Interpreter.java:451)[^]
at o.a.c.i.Nodes$CoreCompiler.visit(Nodes.java:43)[^]
at o.a.c.r.BiRel.childrenAccept(BiRel.java:45)[^]
at o.a.c.i.Interpreter$CompilerImpl.visit(Interpreter.java:447)[^]
at o.a.c.i.Nodes$CoreCompiler.visit(Nodes.java:43)[^]
at o.a.c.r.SingleRel.childrenAccept(SingleRel.java:72)[^]
at o.a.c.i.Interpreter$CompilerImpl.visit(Interpreter.java:447)[^]
at o.a.c.i.Nodes$CoreCompiler.visit(Nodes.java:43)[^]
at o.a.c.r.SingleRel.childrenAccept(SingleRel.java:72)[^]
at o.a.c.i.Interpreter$CompilerImpl.visit(Interpreter.java:447)[^]
at o.a.c.i.Nodes$CoreCompiler.visit(Nodes.java:43)[^]
at o.a.c.r.BiRel.childrenAccept(BiRel.java:46)[^]
at o.a.c.i.Interpreter$CompilerImpl.visit(Interpreter.java:447)[^]
at o.a.c.i.Nodes$CoreCompiler.visit(Nodes.java:43)[^]
at o.a.c.r.SingleRel.childrenAccept(SingleRel.java:72)[^]
at o.a.c.i.Interpreter$CompilerImpl.visit(Interpreter.java:447)[^]
at o.a.c.i.Nodes$CoreCompiler.visit(Nodes.java:43)[^]
at o.a.c.i.Interpreter$CompilerImpl.visitRoot(Interpreter.java:405)[^]
at o.a.c.i.Interpreter.(^:88)[^]
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Question about Interpreter and Corelations

[Zoltan Farkas]

Thanks Julian,

Any idea how could I de-corelate the query?
RelDecorrelator.decorrelateQuery does not seem to alter the plan.

Will log a jira for the interpreter issue.

thank you

—Z

 

> On Nov 7, 2019, at 5:08 PM, Julian Hyde  wrote:
> 
> I don’t recall what the interpreter is currently capable of, but you should 
> log a bug.
> 
> I wonder whether if you could get the query to work in the interpreter if you 
> decorrelated the query first. 
> 
> Julian
> 
>> On Nov 7, 2019, at 11:16, Zoltan Farkas  wrote:
>> 
>> for a test query with the following plan:
>> 
>> LogicalProject(name=[$1], friends=[$4])
>> LogicalCorrelate(correlation=[$cor0], joinType=[inner], 
>> requiredColumns=[{0}])
>>   LogicalTableScan(table=[[characters]])
>>   Collect(field=[EXPR$0])
>> LogicalProject(name=[$2])
>>   LogicalJoin(condition=[=($0, $1)], joinType=[inner])
>> LogicalProject(characterId2=[$1])
>>   LogicalFilter(condition=[=($0, $cor0.characterId)])
>>LogicalTableScan(table=[[friendships]])
>> LogicalProject(characterId=[$0], name=[$1])
>>   LogicalTableScan(table=[[characters]])
>> 
>> I get :
>> 
>> java.lang.nsupportedOperationException
>>   at 
>> o.a.c.i.JaninoRexCompiler.lambda$compile$0(JaninoRexCompiler.java:94)[calcite-core-1.21.0.jar:1.21.0]
>>   at o.a.c.a.e.RexToLixTranslator.translate0(RexToLixTranslator.java:714)[^]
>>   at ^.translate(^:199)[^]
>>   at ^.translate0(^:684)[^]
>>   at ^.translate(^:199)[^]
>>   at ^.translate(^:194)[^]
>>   ...
>> 
>> when trying to use the Interpreter.
>> 
>> Is this a current limitation?  
>> 
>> the query is :
>> 
>> select name,
>> ARRAY(select c2.name from friendships f, characters c2
>>where f.characterId1 = c.characterId and 
>> f.characterId2 = c2.characterId) as friends
>> from characters c
>> 
>> let me know
>> 
>> thank you
>> 
>> —Z

Question about Interpreter and Corelations

[Zoltan Farkas]

for a test query with the following plan:

LogicalProject(name=[$1], friends=[$4])
  LogicalCorrelate(correlation=[$cor0], joinType=[inner], requiredColumns=[{0}])
LogicalTableScan(table=[[characters]])
Collect(field=[EXPR$0])
  LogicalProject(name=[$2])
LogicalJoin(condition=[=($0, $1)], joinType=[inner])
  LogicalProject(characterId2=[$1])
LogicalFilter(condition=[=($0, $cor0.characterId)])
 LogicalTableScan(table=[[friendships]])
  LogicalProject(characterId=[$0], name=[$1])
LogicalTableScan(table=[[characters]])

I get :

java.lang.nsupportedOperationException
at 
o.a.c.i.JaninoRexCompiler.lambda$compile$0(JaninoRexCompiler.java:94)[calcite-core-1.21.0.jar:1.21.0]
at 
o.a.c.a.e.RexToLixTranslator.translate0(RexToLixTranslator.java:714)[^]
at ^.translate(^:199)[^]
at ^.translate0(^:684)[^]
at ^.translate(^:199)[^]
at ^.translate(^:194)[^]
...

when trying to use the Interpreter.

Is this a current limitation?  

the query is :

select name,
  ARRAY(select c2.name from friendships f, characters c2
 where f.characterId1 = c.characterId and 
f.characterId2 = c2.characterId) as friends
from characters c

let me know

thank you

—Z