Re: Using Spark 2.4.0 as Execution Engine with Apache Calcite

[Shubham Kumar]

Hey Julian,

Thanks, I have subscribed to the list now.

On Wed, Aug 28, 2019 at 10:58 PM Julian Hyde  wrote:

> Your message was received. I guess no one had the time to respond.
>
> Can you please subscribe to the list? I had to moderate your message
> through because you are not a subscriber.
>
> Julian
>
>
> > On Aug 28, 2019, at 9:03 AM, Shubham Kumar 
> wrote:
> >
> > Hey,
> >
> > Does anyone have any insights on this? Also wanted to confirm that it is
> > being sent on the mailing list of Calcite.
> >
> > Thanks
> > Shubham
> >
> > On Mon, Aug 26, 2019 at 6:16 PM Shubham Kumar <
> shubhamkumar1...@gmail.com>
> > wrote:
> >
> >> Hey,
> >>
> >> I am trying to optimize queries by creating materialized views in Hive
> and
> >> then exposing them to Calcite for optimized query rewrite.
> >>
> >> I wanted to know that how mature is the spark adapter of Calcite i.e.
> >> Let's say while making the connection I set parameter spark=true
> >> <
> https://calcite.apache.org/apidocs/org/apache/calcite/config/CalciteConnectionProperty.html#SPARK
> >.
> >> Will the workflow be like this?
> >>
> >> i) Calcite using Hive's MV's for query rewrites.
> >> ii) When query comes ; optimized query rewrite is done by Calcite.
> >> iii) This is converted to spark convention and successfully executed
> using
> >> Spark as the engine.
> >>
> >> I had a doubt since in many places it's mentioned that spark adapter is
> >> not production ready or powerful enough and has been developed for
> >> compatibility till spark 1.x only.
> >>
> >> Also, can we extract out the query rewrite of Calcite using some API
> >> before it is actually executed?
> >> --
> >> Thanks & Regards
> >> Shubham Kumar
> >>
> >>
> >
> > --
> > Thanks & Regards
> >
> > Shubham Kumar
>
>

-- 
Thanks & Regards

Shubham Kumar

Re: Using Spark 2.4.0 as Execution Engine with Apache Calcite

[Feng Zhu]

 Hi，Sgybham,
According to our study, it is developed for Spark 1.x and not powerful in
production.
Rather than relying on Spark adapter, I suggest you convert RelNode back to
SQL based on SparkSqlDialect, and then execute them by Spark SQL.

Shubham Kumar  于2019年8月29日周四 上午1:26写道：

> Hey,
>
> Does anyone have any insights on this? Also wanted to confirm that it is
> being sent on the mailing list of Calcite.
>
> Thanks
> Shubham
>
> On Mon, Aug 26, 2019 at 6:16 PM Shubham Kumar 
> wrote:
>
> > Hey,
> >
> > I am trying to optimize queries by creating materialized views in Hive
> and
> > then exposing them to Calcite for optimized query rewrite.
> >
> > I wanted to know that how mature is the spark adapter of Calcite i.e.
> > Let's say while making the connection I set parameter spark=true
> > <
> https://calcite.apache.org/apidocs/org/apache/calcite/config/CalciteConnectionProperty.html#SPARK
> >.
> > Will the workflow be like this?
> >
> > i) Calcite using Hive's MV's for query rewrites.
> > ii) When query comes ; optimized query rewrite is done by Calcite.
> > iii) This is converted to spark convention and successfully executed
> using
> > Spark as the engine.
> >
> > I had a doubt since in many places it's mentioned that spark adapter is
> > not production ready or powerful enough and has been developed for
> > compatibility till spark 1.x only.
> >
> > Also, can we extract out the query rewrite of Calcite using some API
> > before it is actually executed?
> > --
> > Thanks & Regards
> > Shubham Kumar
> >
> >
>
> --
> Thanks & Regards
>
> Shubham Kumar
>

Re: Using Spark 2.4.0 as Execution Engine with Apache Calcite

[Julian Hyde]

Your message was received. I guess no one had the time to respond. 

Can you please subscribe to the list? I had to moderate your message through 
because you are not a subscriber.

Julian


> On Aug 28, 2019, at 9:03 AM, Shubham Kumar  wrote:
> 
> Hey,
> 
> Does anyone have any insights on this? Also wanted to confirm that it is
> being sent on the mailing list of Calcite.
> 
> Thanks
> Shubham
> 
> On Mon, Aug 26, 2019 at 6:16 PM Shubham Kumar 
> wrote:
> 
>> Hey,
>> 
>> I am trying to optimize queries by creating materialized views in Hive and
>> then exposing them to Calcite for optimized query rewrite.
>> 
>> I wanted to know that how mature is the spark adapter of Calcite i.e.
>> Let's say while making the connection I set parameter spark=true
>> .
>> Will the workflow be like this?
>> 
>> i) Calcite using Hive's MV's for query rewrites.
>> ii) When query comes ; optimized query rewrite is done by Calcite.
>> iii) This is converted to spark convention and successfully executed using
>> Spark as the engine.
>> 
>> I had a doubt since in many places it's mentioned that spark adapter is
>> not production ready or powerful enough and has been developed for
>> compatibility till spark 1.x only.
>> 
>> Also, can we extract out the query rewrite of Calcite using some API
>> before it is actually executed?
>> --
>> Thanks & Regards
>> Shubham Kumar
>> 
>> 
> 
> -- 
> Thanks & Regards
> 
> Shubham Kumar

Re: Using Spark 2.4.0 as Execution Engine with Apache Calcite

[Shubham Kumar]

Hey,

Does anyone have any insights on this? Also wanted to confirm that it is
being sent on the mailing list of Calcite.

Thanks
Shubham

On Mon, Aug 26, 2019 at 6:16 PM Shubham Kumar 
wrote:

> Hey,
>
> I am trying to optimize queries by creating materialized views in Hive and
> then exposing them to Calcite for optimized query rewrite.
>
> I wanted to know that how mature is the spark adapter of Calcite i.e.
> Let's say while making the connection I set parameter spark=true
> .
> Will the workflow be like this?
>
> i) Calcite using Hive's MV's for query rewrites.
> ii) When query comes ; optimized query rewrite is done by Calcite.
> iii) This is converted to spark convention and successfully executed using
> Spark as the engine.
>
> I had a doubt since in many places it's mentioned that spark adapter is
> not production ready or powerful enough and has been developed for
> compatibility till spark 1.x only.
>
> Also, can we extract out the query rewrite of Calcite using some API
> before it is actually executed?
> --
> Thanks & Regards
> Shubham Kumar
>
>

-- 
Thanks & Regards

Shubham Kumar

Using Spark 2.4.0 as Execution Engine with Apache Calcite

[Shubham Kumar]

Hey,

I am trying to optimize queries by creating materialized views in Hive and
then exposing them to Calcite for optimized query rewrite.

I wanted to know that how mature is the spark adapter of Calcite i.e. Let's
say while making the connection I set parameter spark=true
.
Will the workflow be like this?

i) Calcite using Hive's MV's for query rewrites.
ii) When query comes ; optimized query rewrite is done by Calcite.
iii) This is converted to spark convention and successfully executed using
Spark as the engine.

I had a doubt since in many places it's mentioned that spark adapter is not
production ready or powerful enough and has been developed for
compatibility till spark 1.x only.

Also, can we extract out the query rewrite of Calcite using some API before
it is actually executed?
-- 
Thanks & Regards
Shubham Kumar