Maybe you could see something in Hive,as far as I know, Hive uses Calcite, but eventually the plan will be transferred to Hive's own execution plan,Spark and Hive themselves are also very compatible. Now I don't find any project to convert Calcite Logical Plan to Spark Logical Plan directly,but you could do this by yourself.However, there are many differences between Calcite and Spark that need to be considered, such as differences in functions, operators of relational algebra trees, differences in data types, etc. I think that Calcite => Substrait => Spark is better way to do this,you could leverage the power of the Calcite community and the Substrait community.
Best, LakeShen Julian Hyde <[email protected]> 于2023年10月13日周五 08:20写道： > There aren’t any plans, but I like the idea of transforming between the > intermediate languages (Calcite algebra, Spark algebra, SQL of any dialect, > Substrate). > > In particular I was thinking of translating Calcite algebra —> Substrate, > and Substrate —> any supported SQL dialect. The former would be done in > Calcite, the latter could be done in a sister project of Substrate. > > Julian > > > > On Oct 12, 2023, at 9:07 AM, Guillaume Masse < > [email protected]> wrote: > > > > Hi All, > > > > We use Apache Calcite to transform SQL then we want to run the logical > plan > > on our spark cluster. Currently we use RelToSqlConverter and execute the > > result with Spark SQL. I was wondering if we could just execute from a > > Spark logical plan ( > > > https://github.com/apache/spark/blob/b0576fff9b72880cd81a9d22c044dec329bc67d0/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L211-L213 > > ). > > > > Are there any project that goes from Calcite Logical Plan to Spark > Logical > > Plan directly? > > > > I know there is one possibility: Calcite => Substrait => Spark > > Substrait => Calcite via substrate-java > > <https://github.com/substrait-io/substrait-java> > > Substrait => Spark via gluten > > < > https://github.com/oap-project/gluten/tree/main/substrait/substrait-spark> > > > > > > -- > > Guillaume Massé > > [Gee-OHM] > > (马赛卫) > >
