Re: Some updated benchmarks of GraphQL on Calcite
That sounds great Gavin! Looking forward to where this leads. Regards, On 06.02.2022 02:25, Gavin Ray wrote: In case anyone is interested, I have finished the "next leg" of the GraphQL implementation. Now the "DataFetcher" functions (what I would call a "resolver") are auto-generated as well. This makes the GraphQL API actually queryable/executable, rather than what I'd shown before which was hand-running RelNode expressions from parsed queries. Using an in-memory HSQLDB, and selecting a single table I get an average execution time of ~2ms. This is on my high-end laptop though. Just trying to establish a lower-bound and understand the weight of Calcite's code generation + query planning and parsing before seeing if it's even worth continuing. So far, signs point to "Yes!" =) Code for the benchmark is here: https://gist.github.com/GavinRay97/05538fb9ce3b25aa94fe6dd9afe484cc Iteration 1: 2.099 ±(99.9%) 0.034 ms/op Iteration 2: 2.138 ±(99.9%) 0.086 ms/op Iteration 3: 2.131 ±(99.9%) 0.037 ms/op Iteration 4: 2.163 ±(99.9%) 0.039 ms/op Iteration 5: 2.202 ±(99.9%) 0.037 ms/op N = 11659 mean = 2.146 ±(99.9%) 0.023 ms/op benchmark summary: Benchmark ModeCnt Score Error querysample 11659 2.146 ± 0.023 query:query·p0.00sample 1.939 ms/op query:query·p0.50sample 2.056 ms/op query:query·p0.90sample 2.281 ms/op query:query·p0.95sample 2.404 ms/op query:query·p0.99sample 3.016 ms/op query:query·p0.999 sample 10.896 ms/op query:query·p0. sample 48.382 ms/op query:query·p1.00 sample 55.181 ms/op -- Eugen Stan +40770 941 271 / https://www.netdava.combegin:vcard fn:Eugen Stan n:Stan;Eugen email;internet:eugen.s...@netdava.com tel;cell:+40720898747 x-mozilla-html:FALSE url:https://www.netdava.com version:2.1 end:vcard
Re: Using Calcite with Python
Hi Nicola, It's a question I was asking myself the other day. I don't know the answer but I do have an exploration direction: Avatica client. There is some nice description and diagram here https://calcite.apache.org/avatica/docs/ And also a list of clients down bellow. See https://calcite.apache.org/avatica/docs/#apache-phoenix-database-adapter-for-python Please let me know how it goes and what you find out. On 31.01.2022 00:35, Nicola Vitucci wrote: Hi all, What would be the best way to use Calcite with Python? I've come up with two potential solutions: - using the jaydebeapi package, to connect via the JDBC driver directly from a JVM created via jpype; - using Apache Arrow via the pyarrow package, to connect in basically the same way but creating Arrow objects with JdbcToArrowUtils (and optionally converting them to Pandas). Although the former is more straightforward, the latter allows to achieve better performance (see [1] for instance) since it's exactly what Arrow is for. I've created two Jupyter notebooks [2] showing each solution. What would you recommend? Is there an even better approach? Thanks, Nicola [1] https://uwekorn.com/2020/12/30/fast-jdbc-revisited.html [2] https://github.com/nvitucci/calcite-sparql/tree/v0.0.2/examples/python Regards, -- Eugen Stan +40770 941 271 / https://www.netdava.combegin:vcard fn:Eugen Stan n:Stan;Eugen email;internet:eugen.s...@netdava.com tel;cell:+40720898747 x-mozilla-html:FALSE url:https://www.netdava.com version:2.1 end:vcard
Re: [DISCUSS] Apache Calcite Online Meetup January 2022
Hi Stamatis, I created a PR https://github.com/apache/calcite/pull/2704 . Thanks for the guidance. Regards, -- Eugen Stan +40770 941 271 / https://www.netdava.combegin:vcard fn:Eugen Stan n:Stan;Eugen email;internet:eugen.s...@netdava.com tel;cell:+40720898747 x-mozilla-html:FALSE url:https://www.netdava.com version:2.1 end:vcard
Re: [DISCUSS] Apache Calcite Online Meetup January 2022
Hi, Very much appreciated Gavin! I do think there is overlap and I will publish the work I do and hopefully you will find it useful as well. Regards, -- Eugen Stan +40770 941 271 / https://www.netdava.combegin:vcard fn:Eugen Stan n:Stan;Eugen email;internet:eugen.s...@netdava.com tel;cell:+40720898747 x-mozilla-html:FALSE url:https://www.netdava.com version:2.1 end:vcard
Re: [DISCUSS] Apache Calcite Online Meetup January 2022
Hi, I published my presentation here https://ieugen.github.io/calcite-clj/ for anyone who is interested. Once the video is up I will add a link to it as well. It was nice meeting with you! Regards, -- Eugen Stan +40770 941 271 / https://www.netdava.combegin:vcard fn:Eugen Stan n:Stan;Eugen email;internet:eugen.s...@netdava.com tel;cell:+40720898747 x-mozilla-html:FALSE url:https://www.netdava.com version:2.1 end:vcard
Re: Calcite-Clojure integration - use Clojure to implement calcite back-ends
Thanks!
On 12.01.2022 11:02, Stamatis Zampetakis wrote:
I added your talk to the meetup event.
The meetup is on January 19, apologies for the confusion.
Best,
Stamatis
On Tue, Jan 11, 2022 at 10:18 PM Eugen Stan <mailto:eugen.s...@netdava.com>> wrote:
Hello Stamatis,
I sent my proposal on the original meeting thread.
Forgot to ask: What is the exact day of the event?
I see January 19 mentioned here:
https://www.meetup.com/Apache-Calcite/events/282836907/
<https://www.meetup.com/Apache-Calcite/events/282836907/>
And you mentioned January 17 on this email.
Regards,
--
Eugen Stan
+40770 941 271 / https://www.netdava.com <https://www.netdava.com>
--
Eugen Stan
+40770 941 271 / https://www.netdava.combegin:vcard
fn:Eugen Stan
n:Stan;Eugen
email;internet:eugen.s...@netdava.com
tel;cell:+40720898747
x-mozilla-html:FALSE
url:https://www.netdava.com
version:2.1
end:vcard
Re: [DISCUSS] Apache Calcite Online Meetup January 2022
I would love to see GraphQL over Calcite for sure. On 12.01.2022 17:37, Gavin Ray wrote: If anyone is interested, I can spend 5-10 minutes briefly presenting the work I've been doing on automatic GraphQL API generation from Calcite data sources. GraphQL is still somewhat niche though, so I'll hold off on asking to speak about it unless there are folks attending who are interested/using it. -- Eugen Stan +40770 941 271 / https://www.netdava.combegin:vcard fn:Eugen Stan n:Stan;Eugen email;internet:eugen.s...@netdava.com tel;cell:+40720898747 x-mozilla-html:FALSE url:https://www.netdava.com version:2.1 end:vcard
Re: Re: [DISCUSS] Apache Calcite Online Meetup January 2022
Hello Stamatis, I would also like to share my work. Title: calcite-clj - First steps using Calcite with Clojure Abstract: A quick introduction on using Calcite with Clojure language. I'll go through an simple example on how to use in memory table. The example will cover REPL driven development. If I have time I will showcase Apache OFBiz entity engine on top of Apache Calcite and how to query OFBIz xml files (OFBiz data exports) using the same engine (ongoing work). Duration: 15 minutes
RE: Calcite-Clojure integration - use Clojure to implement calcite back-ends
Hello, You can now write Apache Calcite adapters in Clojure. There is a library to help you and an example code to guide you. I found some time to work on this and I published a library version of that class and also a Clojure example that showcases how you would use it in your app The code is available https://github.com/ieugen/calcite-clj . Library published on Clojars https://clojars.org/io.github.ieugen/calcite-clj . Would this be of interest for the community meeting ? Regards, -- Eugen Stan +40770 941 271 / https://www.netdava.combegin:vcard fn:Eugen Stan n:Stan;Eugen email;internet:eugen.s...@netdava.com tel;cell:+40720898747 x-mozilla-html:FALSE url:https://www.netdava.com version:2.1 end:vcard
Calcite-Clojure integration - use Clojure to implement calcite back-ends
Hello,
I would like to announce that I have ported the Calcite CSV example
(scannable table only) from Java to Clojure.
I did it mainly as an exercise: to learn how Calicte works and how it
would look with Clojure.
I learn a lot in the process and also discovered a Calcite bug
https://issues.apache.org/jira/browse/CALCITE-4966 .
If anyone is interested, my code is available here:
https://github.com/ieugen/clojure-training/tree/main/csv-clojure .
Id does need a version of Calcite with the fix for
https://issues.apache.org/jira/browse/CALCITE-4966 (I built and
deployed a local version).
It also needs a small 1 class library used as a bridge (build and
install locally):
https://github.com/ieugen/clojure-training/blob/main/csv-clojure/calcite-clj/src/main/java/ro.ieugen.calcite.clj/SchemaFactory.java
.
With the above code I was able to call code like this:
(let [db {:jdbcUrl "jdbc:calcite:model=resources/model.json"
:user "admin"
:password "admin"}
ds (jdbc/get-datasource db)]
(jdbc/execute! ds ["select * from emps where age is null or age >=
40"])))
and get back SQL results.
Part of the magic is in model.json file.
It uses ro.ieugen.calcite.clj.SchemaFactory and it has an operand property
"clojure-clj.schema-factory": "ro.ieugen.calcite-csv/csv-schema" .
This property is a reference to the Clojure namespace
(ro.ieugen.calcite-csv) and function (csv-schema) to call for generating
the Schema.
The schema factory is generic and if there is interest I would like to
contribute it upstream.
It allows the use of Clojure functions to be used as Schema factories
thus creating a bridge to Clojure in a seamless way.
Full model.json bellow:
```
{
"version": "1.0",
"defaultSchema": "SALES",
"schemas": [
{
"name": "SALES",
"type": "custom",
"factory": "ro.ieugen.calcite.clj.SchemaFactory",
"operand": {
"clojure-clj.schema-factory": "ro.ieugen.calcite-csv/csv-schema",
"directory": "resources/sales"
}
}
]
}
```
--
Eugen Stan
+40770 941 271 / https://www.netdava.combegin:vcard
fn:Eugen Stan
n:Stan;Eugen
email;internet:eugen.s...@netdava.com
tel;cell:+40720898747
x-mozilla-html:FALSE
url:https://www.netdava.com
version:2.1
end:vcard
[jira] [Created] (CALCITE-4966) org.apache.calcite.adapter.file.CsvEnumerator.RowConverter should be public
Ioan Eugen Stan created CALCITE-4966:
Summary:
org.apache.calcite.adapter.file.CsvEnumerator.RowConverter should be public
Key: CALCITE-4966
URL: https://issues.apache.org/jira/browse/CALCITE-4966
Project: Calcite
Issue Type: Bug
Components: file-adapter
Reporter: Ioan Eugen Stan
Hello,
I believe class org.apache.calcite.adapter.file.CsvEnumerator.RowConverter
should be public instead of package private since it's exposed in the public
api via
public constructor:
{code:java}
public CsvEnumerator(Source source, AtomicBoolean cancelFlag, boolean stream,
@Nullable String @Nullable [] filterValues, RowConverter rowConverter) {
{code}
public static method:
{code:java}
public static RowConverter<@Nullable Object[]> arrayConverter(
List fieldTypes, List fields, boolean stream) {
return new ArrayRowConverter(fieldTypes, fields, stream);
} {code}
Patch is trivial, add "public" in front :)
{code:java}
public abstract static class RowConverter { {code}
Would love to get this into 1.29.0 - so I can make a surprise for Calcite :D
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
Re: Re: [DISCUSS] Apache Calcite Online Meetup January 2022
Hi, Great idea with the meetup. I've watched that talk and gone through the tutorial code - submitted some feedback and here is some more: It's been VERY useful and I appreciate the work you do for Calcite and documentation. Regarding the training (youtube video) also hard to follow for a calcite/relational algebra beginner like me. I think an even gentler introduction / quick start is needed (not necessarily during meetup, but for newcomers in general). I found the online video to go quite fast into explaining Calcite classes and WHAT needs to be implemented but it was hard for me to understand WHY - or I need more intermediate steps. I had to go through the talk, the code and other code examples to better understand things. So these online resources (videos, talks, blogs, tutorials and code samples) are VERY welcomed. Although I am new to Calcite I might submit a small 5 min lightning talk if I have time to validate my idea :) . On 2021/11/23 18:34:59 Julian Hyde wrote: > +1 for Jan 19 > > It would be great if Stamatis repeated one or two of the segments of > his tutorial at BOSS. (I was co-presenter but you've all heard enough > from me.) The material about adapters is foundational to understanding > Calcite. It could happen on Jan 19 or at another meetup in a couple of > months. > > Julian > > > On Tue, Nov 23, 2021 at 1:55 AM Stamatis Zampetakis wrote: > > > > Thanks everyone for the feedback! > > > > @ Vladimir O, Vladimir S: Thanks for proposing to give a talk. > > I believe both topics will be very interesting for the community. > > When you settle up on what you would like to present please share an > > abstract & duration for the talk. > > > > Since there is interest and we already have 1-2 people willing to give a > > talk, I would suggest fixing the date/time as soon as possible. > > How about Wednesday, 19 January 2022, 17:00 UTC [1] ? > > > > If more people express interest in giving a talk we can either fit it to > > the same meetup or organise another meetup in 2-3 months after. > > > > Best, > > Stamatis > > > > [1] https://www.timeanddate.com/worldclock/fixedtime.html?iso=20220119T1700 > > > > On Tue, Nov 23, 2021 at 12:05 AM Haisheng Yuan > > wrote: > > > > > +1 for online meetup. > > > > > > - Haisheng > > > > > > -- > > > Sender:Julian Hyde > > > Date:2021/11/23 00:58:25 > > > Recipient: > > > Theme:Re: [DISCUSS] Apache Calcite Online Meetup January 2022 > > > > > > +1 > > > > > > Thanks for suggesting this, Stamatis. > > > > > > Julian > > > > > > > On Nov 22, 2021, at 7:12 AM, Zhe Hu wrote: > > > > > > > > Got it. Thanks！ > > > > | > > > > > > > > > > > > | > > > > Replied Message > > > > | From | xiong duan | > > > > | Date | 11/22/2021 23:09 | > > > > | To | dev | > > > > | Subject | Re: [DISCUSS] Apache Calcite Online Meetup January 2022 | > > > > Yes. You can find the relative talks in > > > > http://calcite.apache.org/community/#talks . > > > > > > > > Zhe Hu 于2021年11月22日周一 下午10:50写道： > > > > > > > >> +1 for online meetup. > > > >> And I’m curious about if there is any playback for this meetup, because > > > >> somebody might couldn't attend due to timezone variety or any other > > > reasons. > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> On 11/22/2021 22:26，xiong duan wrote： > > > >> Great idea. +1 for an online meetup. > > > >> > > > >> Justin Swanhart 于2021年11月20日周六 上午12:42写道： > > > >> > > > >> +1 for an online meetup > > > >> > > > >> On Fri, Nov 19, 2021, 11:32 AM Jacques Nadeau > > > wrote: > > > >> > > > >> WRT to talk structure, I propose we do 5 or 10 minute lightning talks > > > >> as > > > >> opposed to something longer. It ensures a wider variety of topics, > > > better > > > >> attendance and you never have to sit that long through something you're > > > >> not > > > >> interested in... > > > >> > > > >> On Thu, Nov 18, 2021 at 11:47 PM Vladimir Sitnikov < > > > >> sitnikov.vladi...@gmail.com> wrote: > > > >> > > > >> Are the people willing to give a talk around Calcite? > > > >> > > > >> I may present a talk regarding Gradle, and the upcoming migration to > > > >> the > > > >> idiomatic structure: > > > >> https://issues.apache.org/jira/browse/CALCITE-4832 > > > >> https://github.com/apache/calcite/pull/2566 > > > >> > > > >> > > > https://docs.gradle.org/7.3/userguide/structuring_software_products.html > > > >> > > > >> What do people think in terms of timing? > > > >> > > > >> I think the new build structure is more like 30-45 min talk > > > >> > > > >> Vladimir > > > >> > > > >> > > > >> > > > >> > > > > > > > > > > > >
Re: graphql query to sql query with Calcite ?
Hello Julian, Thank you for the pointers. It is very helpful and I have started working on the project. I'm currently discovering Calcite API. Any ideas on where I can find documentation to create an In memory schema so I can pass it to RelBuilder ? I did not find much on the website. I'll use graphql-java to parse the query and build an "AST". My goal is to make it work for the simple graphql example with StarWars :). Once this is done I will target PostgreSQL. One interesting feature of PostgreSQL is Foreign Data Wrappers [fdw] .It makes multiple databases visible as one, including PostgreSQL and it can make the query planning for me :D . Thanks, Eugen [fdw] https://wiki.postgresql.org/wiki/Foreign_data_wrappers On 24.08.2018 00:32, Julian Hyde wrote: > I’d call that a "GraphQL front-end for Calcite". (SQL is our main front end, > but other front-ends include linq4j and I gather there are other query > languages in commercial products, e.g. Stardog uses Calcite to translate > SPARQL to SQL[1].) > > I think this is a good fit for Calcite, and would support it. Should it be a > module in Calcite, or a standalone project that uses Calcite? Both are > reasonable options. > > In case folks on the dev list are not familiar with GraphQL, I will point out > that it is NOT a query language for graph databases (as are Cypher, SPARQL, > Gremlin). But it is exceedingly good at running queries on data sources with > nested data and producing results to power web applications. And it is > becoming extremely popular. > > My thumbnail sketch of the architecture: write (or better, re-use) a GraphQL > parser and semantic analyzer. Take the validated GraphQL AST and convert it > into Calcite relational algebra (probably using RelBuilder[2]). Then use > RelToSqlConverter to convert relational algebra to SQL. RelToSqlConverter > handles differences in dialect, and is getting better all the time. > > Julian > > [1] https://www.stardog.com/blog/virtual-graphs-in-stardog-5/ > <https://www.stardog.com/blog/virtual-graphs-in-stardog-5/> > > [2] https://calcite.apache.org/docs/algebra.html#algebra-builder > <https://calcite.apache.org/docs/algebra.html#algebra-builder> > > >> On Aug 21, 2018, at 10:21 PM, Eugen Stan wrote: >> >> Hello, >> >> TLDR: >> >> I'm wondering if I can integrate Calcite with [graphql-java] and use >> Calcite to transform a graphql query into an SQL query and send it >> directly to the database. >> >> Furthermore, I'm curious if I can use Calcite's adapters to emulate an >> SQL layer on top of other remote services and leverage the query planner >> from Calcite to build smart/optimal queries. >> >> There is prior art to this: a project called [join-monster] that does >> this for JS. See [join-monster-7-min] video for a short description. >> >> The process to go from graphql query to SQL query is described in >> [join-monster-process] and it's quite short. >> >> Longer version >> >> I'm working on a GraphQL API for a SaaS platform. Right now we are >> facing with a common problem in GraphQL: one query for a graph of >> objects will turn to N+1 queries on the back-end data-store. There is a >> lot of literature on this on the internet and also descibed in >> [data-loader] and [join-monster]. >> >> Now, one solution for this problem is to use [data-loader] - to cache >> objects. This works for some, and it is kind of the only solution for >> remote data stores (other http API endpoints). >> >> My initial objective is to transform the AST that graphql-java builds >> into an AST for SQL and push this SQL to one database. >> >> I believe Calcite can help with this and I'm reaching out to the >> community since I'm not familiar with the project and the features and >> limitations it has. >> >> Can Calcite help me transform the GraphQL query AST to an SQL AST? >> >> Should I look into this or should I go straight to something like ANTLR. >> I know there is a definition for [graphql-java-antlr] . I'm asking this >> to know if it has features that could help me or could block me? >> >> Features that could help I imagine is the [SQL-grammer] ? >> >> >> Thank you, >> >> Eugen >> >> >> [data-loader] https://github.com/facebook/dataloader >> >> [graphql-java] https://github.com/graphql-java/graphql-java/ >> >> [join-monster] https://join-monster.readthedocs.io/en/latest/ >> >> [join-monster-7-min] https://www.youtube.com/watch?v=Y7AdMIuXOgs >> >> [join-monster-process] >> https://github.com/acarl005/join-monster/tree/master/src >> >> [graphql-java-antlr] >> https://github.com/graphql-java/graphql-java/tree/master/src/main/antlr >> >> [sql-grammmer] https://calcite.apache.org/docs/reference.html >> >> > signature.asc Description: OpenPGP digital signature
graphql query to sql query with Calcite ?
Hello, TLDR: I'm wondering if I can integrate Calcite with [graphql-java] and use Calcite to transform a graphql query into an SQL query and send it directly to the database. Furthermore, I'm curious if I can use Calcite's adapters to emulate an SQL layer on top of other remote services and leverage the query planner from Calcite to build smart/optimal queries. There is prior art to this: a project called [join-monster] that does this for JS. See [join-monster-7-min] video for a short description. The process to go from graphql query to SQL query is described in [join-monster-process] and it's quite short. Longer version I'm working on a GraphQL API for a SaaS platform. Right now we are facing with a common problem in GraphQL: one query for a graph of objects will turn to N+1 queries on the back-end data-store. There is a lot of literature on this on the internet and also descibed in [data-loader] and [join-monster]. Now, one solution for this problem is to use [data-loader] - to cache objects. This works for some, and it is kind of the only solution for remote data stores (other http API endpoints). My initial objective is to transform the AST that graphql-java builds into an AST for SQL and push this SQL to one database. I believe Calcite can help with this and I'm reaching out to the community since I'm not familiar with the project and the features and limitations it has. Can Calcite help me transform the GraphQL query AST to an SQL AST? Should I look into this or should I go straight to something like ANTLR. I know there is a definition for [graphql-java-antlr] . I'm asking this to know if it has features that could help me or could block me? Features that could help I imagine is the [SQL-grammer] ? Thank you, Eugen [data-loader] https://github.com/facebook/dataloader [graphql-java] https://github.com/graphql-java/graphql-java/ [join-monster] https://join-monster.readthedocs.io/en/latest/ [join-monster-7-min] https://www.youtube.com/watch?v=Y7AdMIuXOgs [join-monster-process] https://github.com/acarl005/join-monster/tree/master/src [graphql-java-antlr] https://github.com/graphql-java/graphql-java/tree/master/src/main/antlr [sql-grammmer] https://calcite.apache.org/docs/reference.html signature.asc Description: OpenPGP digital signature
