Yes, virtualization is one of Calcite’s goals. In fact, when I created Calcite I was thinking about virtualization + in-memory materialized views. Not only the Spark convention but any of the “engine” conventions (Drill, Flink, Beam, Enumerable) could be used to create a virtual query engine.
See e.g. a talk I gave in 2013 about Optiq (precursor to Calcite) https://www.slideshare.net/julianhyde/optiq-a-dynamic-data-management-framework <https://www.slideshare.net/julianhyde/optiq-a-dynamic-data-management-framework>. Julian > On Dec 9, 2019, at 2:29 PM, Muhammad Gelbana <[email protected]> wrote: > > I recently contacted one of the active contributors asking about the > purpose of the project and here's his reply: > > From my understanding, Quicksql is a data virtualization platform. It can >> query multiple data sources altogether and in a distributed way; Say, you >> can write a SQL with a MySql table join with an Elasticsearch table. >> Quicksql can recognize that, and then generate Spark code, in which it will >> fetch the MySQL/ES data as a temporary table separately, and then join them >> in Spark. The execution is in Spark so it is totally distributed. The user >> doesn't need to aware of where the table is from. >> > > I understand that the Spark convention Calcite has attempts to achieve the > same goal, but it isn't fully implemented yet. > > > On Tue, Oct 29, 2019 at 9:43 PM Julian Hyde <[email protected]> wrote: > >> Anyone know anything about Quicksql? It seems to be quite a popular >> project, and they have an internal fork of Calcite. >> >> https://github.com/Qihoo360/ <https://github.com/Qihoo360/> >> >> >> https://github.com/Qihoo360/Quicksql/tree/master/analysis/src/main/java/org/apache/calcite >> < >> https://github.com/Qihoo360/Quicksql/tree/master/analysis/src/main/java/org/apache/calcite >>> >> >> Julian >> >>
