Re: graphql query to sql query with Calcite ?

[Stamatis Zampetakis]

Hi Eugen,

You can have a look at RelBuilderTest to see how to instantiate a
RelBuilder.
To create a Calcite Schema you may also take inspiration from the following
method:
org.apache.calcite.test.CalciteAssert#addSchema

Best,
Stamatis


Στις Τρί, 28 Αυγ 2018 στις 10:31 π.μ., ο/η Eugen Stan 
έγραψε:

> Hello Julian,
>
> Thank you for the pointers. It is very helpful and I have started
> working on the project. I'm currently discovering Calcite API.
>
> Any ideas on where I can find documentation to create an In memory
> schema so I can pass it to RelBuilder ?
>
> I did not find much on the website. I'll use graphql-java to parse the
> query and build an "AST".
>
> My goal is to make it work for the simple graphql example with StarWars :).
>
> Once this is done I will target PostgreSQL. One interesting feature of
> PostgreSQL is Foreign Data Wrappers [fdw] .It makes multiple databases
> visible as one, including PostgreSQL and it can make the query planning
> for me :D .
>
> Thanks,
>
> Eugen
>
> [fdw] https://wiki.postgresql.org/wiki/Foreign_data_wrappers
>
> On 24.08.2018 00:32, Julian Hyde wrote:
> > I’d call that a "GraphQL front-end for Calcite". (SQL is our main front
> end, but other front-ends include linq4j and I gather there are other query
> languages in commercial products, e.g. Stardog uses Calcite to translate
> SPARQL to SQL[1].)
> >
> > I think this is a good fit for Calcite, and would support it. Should it
> be a module in Calcite, or a standalone project that uses Calcite? Both are
> reasonable options.
> >
> > In case folks on the dev list are not familiar with GraphQL, I will
> point out that it is NOT a query language for graph databases (as are
> Cypher, SPARQL, Gremlin). But it is exceedingly good at running queries on
> data sources with nested data and producing results to power web
> applications. And it is becoming extremely popular.
> >
> > My thumbnail sketch of the architecture: write (or better, re-use) a
> GraphQL parser and semantic analyzer. Take the validated GraphQL AST and
> convert it into Calcite relational algebra (probably using RelBuilder[2]).
> Then use RelToSqlConverter to convert relational algebra to SQL.
> RelToSqlConverter handles differences in dialect, and is getting better all
> the time.
> >
> > Julian
> >
> > [1] https://www.stardog.com/blog/virtual-graphs-in-stardog-5/ <
> https://www.stardog.com/blog/virtual-graphs-in-stardog-5/>
> >
> > [2] https://calcite.apache.org/docs/algebra.html#algebra-builder <
> https://calcite.apache.org/docs/algebra.html#algebra-builder>
> >
> >
> >> On Aug 21, 2018, at 10:21 PM, Eugen Stan  wrote:
> >>
> >> Hello,
> >>
> >> TLDR:
> >>
> >> I'm wondering if I can integrate Calcite with [graphql-java] and use
> >> Calcite to transform a graphql query into an SQL query and send it
> >> directly to the database.
> >>
> >> Furthermore, I'm curious if I can use Calcite's adapters to emulate an
> >> SQL layer on top of other remote services and leverage the query planner
> >> from Calcite to build smart/optimal queries.
> >>
> >> There is prior art to this: a project called [join-monster] that does
> >> this for JS. See [join-monster-7-min] video for a short description.
> >>
> >> The process to go from graphql query to SQL query is described in
> >> [join-monster-process] and it's quite short.
> >>
> >> Longer version
> >>
> >> I'm working on a GraphQL API for a SaaS platform. Right now we are
> >> facing with a common problem in GraphQL: one query for a graph of
> >> objects will turn to N+1 queries on the back-end data-store. There is a
> >> lot of literature on this on the internet and also descibed in
> >> [data-loader] and [join-monster].
> >>
> >> Now, one solution for this problem is to use [data-loader]  - to cache
> >> objects. This works for some, and it is kind of the only solution for
> >> remote data stores (other http API endpoints).
> >>
> >> My initial objective is to transform the AST that graphql-java builds
> >> into an AST for SQL and push this SQL to one database.
> >>
> >> I believe Calcite can help with this and I'm reaching out to the
> >> community since I'm not familiar with the project and the features and
> >> limitations it has.
> >>
> >> Can Calcite help me transform the GraphQL query AST to an SQL AST?
> >>
> >> Should I look into this or should I go straight to something like ANTLR.
> >> I know there is a definition for [graphql-java-antlr] . I'm asking this
> >> to know if it has features that could help me or could block me?
> >>
> >> Features that could help I imagine is the [SQL-grammer] ?
> >>
> >>
> >> Thank you,
> >>
> >> Eugen
> >>
> >>
> >> [data-loader] https://github.com/facebook/dataloader
> >>
> >> [graphql-java] https://github.com/graphql-java/graphql-java/
> >>
> >> [join-monster] https://join-monster.readthedocs.io/en/latest/
> >>
> >> [join-monster-7-min] https://www.youtube.com/watch?v=Y7AdMIuXOgs
> >>
> >> [join-monster-process]
> >> 

Re: graphql query to sql query with Calcite ?

[Eugen Stan]

Hello Julian,

Thank you for the pointers. It is very helpful and I have started
working on the project. I'm currently discovering Calcite API.

Any ideas on where I can find documentation to create an In memory
schema so I can pass it to RelBuilder ?

I did not find much on the website. I'll use graphql-java to parse the
query and build an "AST". 

My goal is to make it work for the simple graphql example with StarWars :).

Once this is done I will target PostgreSQL. One interesting feature of
PostgreSQL is Foreign Data Wrappers [fdw] .It makes multiple databases
visible as one, including PostgreSQL and it can make the query planning
for me :D .

Thanks,

Eugen

[fdw] https://wiki.postgresql.org/wiki/Foreign_data_wrappers

On 24.08.2018 00:32, Julian Hyde wrote:
> I’d call that a "GraphQL front-end for Calcite". (SQL is our main front end, 
> but other front-ends include linq4j and I gather there are other query 
> languages in commercial products, e.g. Stardog uses Calcite to translate 
> SPARQL to SQL[1].)
>
> I think this is a good fit for Calcite, and would support it. Should it be a 
> module in Calcite, or a standalone project that uses Calcite? Both are 
> reasonable options.
>
> In case folks on the dev list are not familiar with GraphQL, I will point out 
> that it is NOT a query language for graph databases (as are Cypher, SPARQL, 
> Gremlin). But it is exceedingly good at running queries on data sources with 
> nested data and producing results to power web applications. And it is 
> becoming extremely popular. 
>
> My thumbnail sketch of the architecture: write (or better, re-use) a GraphQL 
> parser and semantic analyzer. Take the validated GraphQL AST and convert it 
> into Calcite relational algebra (probably using RelBuilder[2]). Then use 
> RelToSqlConverter to convert relational algebra to SQL. RelToSqlConverter 
> handles differences in dialect, and is getting better all the time.
>
> Julian
>
> [1] https://www.stardog.com/blog/virtual-graphs-in-stardog-5/ 
> 
>
> [2] https://calcite.apache.org/docs/algebra.html#algebra-builder 
> 
>
>
>> On Aug 21, 2018, at 10:21 PM, Eugen Stan  wrote:
>>
>> Hello,
>>
>> TLDR:
>>
>> I'm wondering if I can integrate Calcite with [graphql-java] and use
>> Calcite to transform a graphql query into an SQL query and send it
>> directly to the database.
>>
>> Furthermore, I'm curious if I can use Calcite's adapters to emulate an
>> SQL layer on top of other remote services and leverage the query planner
>> from Calcite to build smart/optimal queries.
>>
>> There is prior art to this: a project called [join-monster] that does
>> this for JS. See [join-monster-7-min] video for a short description.
>>
>> The process to go from graphql query to SQL query is described in
>> [join-monster-process] and it's quite short.
>>
>> Longer version
>>
>> I'm working on a GraphQL API for a SaaS platform. Right now we are
>> facing with a common problem in GraphQL: one query for a graph of
>> objects will turn to N+1 queries on the back-end data-store. There is a
>> lot of literature on this on the internet and also descibed in
>> [data-loader] and [join-monster].
>>
>> Now, one solution for this problem is to use [data-loader]  - to cache
>> objects. This works for some, and it is kind of the only solution for
>> remote data stores (other http API endpoints).
>>
>> My initial objective is to transform the AST that graphql-java builds
>> into an AST for SQL and push this SQL to one database.
>>
>> I believe Calcite can help with this and I'm reaching out to the
>> community since I'm not familiar with the project and the features and
>> limitations it has.
>>
>> Can Calcite help me transform the GraphQL query AST to an SQL AST? 
>>
>> Should I look into this or should I go straight to something like ANTLR.
>> I know there is a definition for [graphql-java-antlr] . I'm asking this
>> to know if it has features that could help me or could block me?
>>
>> Features that could help I imagine is the [SQL-grammer] ? 
>>
>>
>> Thank you,
>>
>> Eugen
>>
>>
>> [data-loader] https://github.com/facebook/dataloader
>>
>> [graphql-java] https://github.com/graphql-java/graphql-java/ 
>>
>> [join-monster] https://join-monster.readthedocs.io/en/latest/
>>
>> [join-monster-7-min] https://www.youtube.com/watch?v=Y7AdMIuXOgs
>>
>> [join-monster-process]
>> https://github.com/acarl005/join-monster/tree/master/src
>>
>> [graphql-java-antlr]
>> https://github.com/graphql-java/graphql-java/tree/master/src/main/antlr 
>>
>> [sql-grammmer] https://calcite.apache.org/docs/reference.html
>>
>>
>



signature.asc
Description: OpenPGP digital signature

Re: graphql query to sql query with Calcite ?

[Julian Hyde]

I’d call that a "GraphQL front-end for Calcite". (SQL is our main front end, 
but other front-ends include linq4j and I gather there are other query 
languages in commercial products, e.g. Stardog uses Calcite to translate SPARQL 
to SQL[1].)

I think this is a good fit for Calcite, and would support it. Should it be a 
module in Calcite, or a standalone project that uses Calcite? Both are 
reasonable options.

In case folks on the dev list are not familiar with GraphQL, I will point out 
that it is NOT a query language for graph databases (as are Cypher, SPARQL, 
Gremlin). But it is exceedingly good at running queries on data sources with 
nested data and producing results to power web applications. And it is becoming 
extremely popular. 

My thumbnail sketch of the architecture: write (or better, re-use) a GraphQL 
parser and semantic analyzer. Take the validated GraphQL AST and convert it 
into Calcite relational algebra (probably using RelBuilder[2]). Then use 
RelToSqlConverter to convert relational algebra to SQL. RelToSqlConverter 
handles differences in dialect, and is getting better all the time.

Julian

[1] https://www.stardog.com/blog/virtual-graphs-in-stardog-5/ 


[2] https://calcite.apache.org/docs/algebra.html#algebra-builder 



> On Aug 21, 2018, at 10:21 PM, Eugen Stan  wrote:
> 
> Hello,
> 
> TLDR:
> 
> I'm wondering if I can integrate Calcite with [graphql-java] and use
> Calcite to transform a graphql query into an SQL query and send it
> directly to the database.
> 
> Furthermore, I'm curious if I can use Calcite's adapters to emulate an
> SQL layer on top of other remote services and leverage the query planner
> from Calcite to build smart/optimal queries.
> 
> There is prior art to this: a project called [join-monster] that does
> this for JS. See [join-monster-7-min] video for a short description.
> 
> The process to go from graphql query to SQL query is described in
> [join-monster-process] and it's quite short.
> 
> Longer version
> 
> I'm working on a GraphQL API for a SaaS platform. Right now we are
> facing with a common problem in GraphQL: one query for a graph of
> objects will turn to N+1 queries on the back-end data-store. There is a
> lot of literature on this on the internet and also descibed in
> [data-loader] and [join-monster].
> 
> Now, one solution for this problem is to use [data-loader]  - to cache
> objects. This works for some, and it is kind of the only solution for
> remote data stores (other http API endpoints).
> 
> My initial objective is to transform the AST that graphql-java builds
> into an AST for SQL and push this SQL to one database.
> 
> I believe Calcite can help with this and I'm reaching out to the
> community since I'm not familiar with the project and the features and
> limitations it has.
> 
> Can Calcite help me transform the GraphQL query AST to an SQL AST? 
> 
> Should I look into this or should I go straight to something like ANTLR.
> I know there is a definition for [graphql-java-antlr] . I'm asking this
> to know if it has features that could help me or could block me?
> 
> Features that could help I imagine is the [SQL-grammer] ? 
> 
> 
> Thank you,
> 
> Eugen
> 
> 
> [data-loader] https://github.com/facebook/dataloader
> 
> [graphql-java] https://github.com/graphql-java/graphql-java/ 
> 
> [join-monster] https://join-monster.readthedocs.io/en/latest/
> 
> [join-monster-7-min] https://www.youtube.com/watch?v=Y7AdMIuXOgs
> 
> [join-monster-process]
> https://github.com/acarl005/join-monster/tree/master/src
> 
> [graphql-java-antlr]
> https://github.com/graphql-java/graphql-java/tree/master/src/main/antlr 
> 
> [sql-grammmer] https://calcite.apache.org/docs/reference.html
> 
> 

graphql query to sql query with Calcite ?

[Eugen Stan]

Hello,

TLDR:

I'm wondering if I can integrate Calcite with [graphql-java] and use
Calcite to transform a graphql query into an SQL query and send it
directly to the database.

Furthermore, I'm curious if I can use Calcite's adapters to emulate an
SQL layer on top of other remote services and leverage the query planner
from Calcite to build smart/optimal queries.

There is prior art to this: a project called [join-monster] that does
this for JS. See [join-monster-7-min] video for a short description.

The process to go from graphql query to SQL query is described in
[join-monster-process] and it's quite short.

Longer version

I'm working on a GraphQL API for a SaaS platform. Right now we are
facing with a common problem in GraphQL: one query for a graph of
objects will turn to N+1 queries on the back-end data-store. There is a
lot of literature on this on the internet and also descibed in
[data-loader] and [join-monster].

Now, one solution for this problem is to use [data-loader]  - to cache
objects. This works for some, and it is kind of the only solution for
remote data stores (other http API endpoints).

My initial objective is to transform the AST that graphql-java builds
into an AST for SQL and push this SQL to one database.

I believe Calcite can help with this and I'm reaching out to the
community since I'm not familiar with the project and the features and
limitations it has.

Can Calcite help me transform the GraphQL query AST to an SQL AST? 

Should I look into this or should I go straight to something like ANTLR.
I know there is a definition for [graphql-java-antlr] . I'm asking this
to know if it has features that could help me or could block me?

Features that could help I imagine is the [SQL-grammer] ? 


Thank you,

Eugen


[data-loader] https://github.com/facebook/dataloader

[graphql-java] https://github.com/graphql-java/graphql-java/ 

[join-monster] https://join-monster.readthedocs.io/en/latest/

[join-monster-7-min] https://www.youtube.com/watch?v=Y7AdMIuXOgs

[join-monster-process]
https://github.com/acarl005/join-monster/tree/master/src

[graphql-java-antlr]
https://github.com/graphql-java/graphql-java/tree/master/src/main/antlr 

[sql-grammmer] https://calcite.apache.org/docs/reference.html




signature.asc
Description: OpenPGP digital signature