Re: Does Spark SQL uses Calcite?
Hi Jules, I am looking to connect to Spark via JDBC so I can run Spark SQL queries via JDBC but not use SPARK SQL to connect to other JDBC sources. Thanks! On Sat, Aug 19, 2017 at 5:54 PM, Jules Damji wrote: > Try this link to see how you may connect https://docs.databricks.com/ > spark/latest/data-sources/sql-databases.html > > Cheers > Jules > > Sent from my iPhone > Pardon the dumb thumb typos :) > > On Aug 19, 2017, at 5:27 PM, kant kodali wrote: > > Hi Russell, > > I went through this https://jaceklaskowski.gitbooks.io/mastering-apache- > spark/spark-sql-thrift-server.html and I am still a bit confused on what > hive is doing in here ? Is there any example I can look at on how to talk > to Spark using Spark SQL JDBC driver alone and not hive ? > > Thanks, > kant > > > > > > On Sat, Aug 12, 2017 at 6:07 AM, Russell Spitzer < > russell.spit...@gmail.com> wrote: > >> You don't have to go through hive. It's just spark sql. The application >> is just a forked hive thrift server. >> >> On Fri, Aug 11, 2017 at 8:53 PM kant kodali wrote: >> >>> @Ryan it looks like if I enable thrift server I need to go through hive. >>> I was talking more about having JDBC connector for Spark SQL itself other >>> words not going through hive. >>> >>> On Fri, Aug 11, 2017 at 6:50 PM, kant kodali wrote: >>> >>>> @Ryan Does it work with Spark SQL 2.1.1? >>>> >>>> On Fri, Aug 11, 2017 at 12:53 AM, Ryan wrote: >>>> >>>>> the thrift server is a jdbc server, Kanth >>>>> >>>>> On Fri, Aug 11, 2017 at 2:51 PM, wrote: >>>>> >>>>>> I also wonder why there isn't a jdbc connector for spark sql? >>>>>> >>>>>> Sent from my iPhone >>>>>> >>>>>> On Aug 10, 2017, at 2:45 PM, Jules Damji wrote: >>>>>> >>>>>> Yes, it's more used in Hive than Spark >>>>>> >>>>>> Sent from my iPhone >>>>>> Pardon the dumb thumb typos :) >>>>>> >>>>>> On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu < >>>>>> vsathishkuma...@gmail.com> wrote: >>>>>> >>>>>> I think it is for hive dependency. >>>>>> On Thu, Aug 10, 2017 at 4:14 PM kant kodali >>>>>> wrote: >>>>>> >>>>>>> Since I see a calcite dependency in Spark I wonder where Calcite is >>>>>>> being used? >>>>>>> >>>>>>> On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu < >>>>>>> vsathishkuma...@gmail.com> wrote: >>>>>>> >>>>>>>> Spark SQL doesn't use Calcite >>>>>>>> >>>>>>>> On Thu, Aug 10, 2017 at 3:14 PM kant kodali >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi All, >>>>>>>>> >>>>>>>>> Does Spark SQL uses Calcite? If so, what for? I thought the Spark >>>>>>>>> SQL has catalyst which would generate its own logical plans, physical >>>>>>>>> plans >>>>>>>>> and other optimizations. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Kant >>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>> >>> >
Re: Does Spark SQL uses Calcite?
Try this link to see how you may connect https://docs.databricks.com/spark/latest/data-sources/sql-databases.html Cheers Jules Sent from my iPhone Pardon the dumb thumb typos :) > On Aug 19, 2017, at 5:27 PM, kant kodali wrote: > > Hi Russell, > > I went through this > https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-sql-thrift-server.html > and I am still a bit confused on what hive is doing in here ? Is there any > example I can look at on how to talk to Spark using Spark SQL JDBC driver > alone and not hive ? > > Thanks, > kant > > > > > >> On Sat, Aug 12, 2017 at 6:07 AM, Russell Spitzer >> wrote: >> You don't have to go through hive. It's just spark sql. The application is >> just a forked hive thrift server. >> >>> On Fri, Aug 11, 2017 at 8:53 PM kant kodali wrote: >>> @Ryan it looks like if I enable thrift server I need to go through hive. I >>> was talking more about having JDBC connector for Spark SQL itself other >>> words not going through hive. >>> >>>> On Fri, Aug 11, 2017 at 6:50 PM, kant kodali wrote: >>>> @Ryan Does it work with Spark SQL 2.1.1? >>>> >>>>> On Fri, Aug 11, 2017 at 12:53 AM, Ryan wrote: >>>>> the thrift server is a jdbc server, Kanth >>>>> >>>>>> On Fri, Aug 11, 2017 at 2:51 PM, wrote: >>>>>> I also wonder why there isn't a jdbc connector for spark sql? >>>>>> >>>>>> Sent from my iPhone >>>>>> >>>>>>> On Aug 10, 2017, at 2:45 PM, Jules Damji wrote: >>>>>>> >>>>>>> Yes, it's more used in Hive than Spark >>>>>>> >>>>>>> Sent from my iPhone >>>>>>> Pardon the dumb thumb typos :) >>>>>>> >>>>>>>> On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu >>>>>>>> wrote: >>>>>>>> >>>>>>>> I think it is for hive dependency. >>>>>>>>> On Thu, Aug 10, 2017 at 4:14 PM kant kodali >>>>>>>>> wrote: >>>>>>>>> Since I see a calcite dependency in Spark I wonder where Calcite is >>>>>>>>> being used? >>>>>>>>> >>>>>>>>>> On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu >>>>>>>>>> wrote: >>>>>>>>>> Spark SQL doesn't use Calcite >>>>>>>>>> >>>>>>>>>>> On Thu, Aug 10, 2017 at 3:14 PM kant kodali >>>>>>>>>>> wrote: >>>>>>>>>>> Hi All, >>>>>>>>>>> >>>>>>>>>>> Does Spark SQL uses Calcite? If so, what for? I thought the Spark >>>>>>>>>>> SQL has catalyst which would generate its own logical plans, >>>>>>>>>>> physical plans and other optimizations. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Kant >>>>>>>>> >>>>> >>>> >>> >
Re: Does Spark SQL uses Calcite?
Hi Russell, I went through this https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-sql-thrift-server.html and I am still a bit confused on what hive is doing in here ? Is there any example I can look at on how to talk to Spark using Spark SQL JDBC driver alone and not hive ? Thanks, kant On Sat, Aug 12, 2017 at 6:07 AM, Russell Spitzer wrote: > You don't have to go through hive. It's just spark sql. The application is > just a forked hive thrift server. > > On Fri, Aug 11, 2017 at 8:53 PM kant kodali wrote: > >> @Ryan it looks like if I enable thrift server I need to go through hive. >> I was talking more about having JDBC connector for Spark SQL itself other >> words not going through hive. >> >> On Fri, Aug 11, 2017 at 6:50 PM, kant kodali wrote: >> >>> @Ryan Does it work with Spark SQL 2.1.1? >>> >>> On Fri, Aug 11, 2017 at 12:53 AM, Ryan wrote: >>> >>>> the thrift server is a jdbc server, Kanth >>>> >>>> On Fri, Aug 11, 2017 at 2:51 PM, wrote: >>>> >>>>> I also wonder why there isn't a jdbc connector for spark sql? >>>>> >>>>> Sent from my iPhone >>>>> >>>>> On Aug 10, 2017, at 2:45 PM, Jules Damji wrote: >>>>> >>>>> Yes, it's more used in Hive than Spark >>>>> >>>>> Sent from my iPhone >>>>> Pardon the dumb thumb typos :) >>>>> >>>>> On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu < >>>>> vsathishkuma...@gmail.com> wrote: >>>>> >>>>> I think it is for hive dependency. >>>>> On Thu, Aug 10, 2017 at 4:14 PM kant kodali >>>>> wrote: >>>>> >>>>>> Since I see a calcite dependency in Spark I wonder where Calcite is >>>>>> being used? >>>>>> >>>>>> On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu < >>>>>> vsathishkuma...@gmail.com> wrote: >>>>>> >>>>>>> Spark SQL doesn't use Calcite >>>>>>> >>>>>>> On Thu, Aug 10, 2017 at 3:14 PM kant kodali >>>>>>> wrote: >>>>>>> >>>>>>>> Hi All, >>>>>>>> >>>>>>>> Does Spark SQL uses Calcite? If so, what for? I thought the Spark >>>>>>>> SQL has catalyst which would generate its own logical plans, physical >>>>>>>> plans >>>>>>>> and other optimizations. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Kant >>>>>>>> >>>>>>> >>>>>> >>>> >>> >>
Re: Does Spark SQL uses Calcite?
You don't have to go through hive. It's just spark sql. The application is just a forked hive thrift server. On Fri, Aug 11, 2017 at 8:53 PM kant kodali wrote: > @Ryan it looks like if I enable thrift server I need to go through hive. I > was talking more about having JDBC connector for Spark SQL itself other > words not going through hive. > > On Fri, Aug 11, 2017 at 6:50 PM, kant kodali wrote: > >> @Ryan Does it work with Spark SQL 2.1.1? >> >> On Fri, Aug 11, 2017 at 12:53 AM, Ryan wrote: >> >>> the thrift server is a jdbc server, Kanth >>> >>> On Fri, Aug 11, 2017 at 2:51 PM, wrote: >>> >>>> I also wonder why there isn't a jdbc connector for spark sql? >>>> >>>> Sent from my iPhone >>>> >>>> On Aug 10, 2017, at 2:45 PM, Jules Damji wrote: >>>> >>>> Yes, it's more used in Hive than Spark >>>> >>>> Sent from my iPhone >>>> Pardon the dumb thumb typos :) >>>> >>>> On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu < >>>> vsathishkuma...@gmail.com> wrote: >>>> >>>> I think it is for hive dependency. >>>> On Thu, Aug 10, 2017 at 4:14 PM kant kodali wrote: >>>> >>>>> Since I see a calcite dependency in Spark I wonder where Calcite is >>>>> being used? >>>>> >>>>> On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu < >>>>> vsathishkuma...@gmail.com> wrote: >>>>> >>>>>> Spark SQL doesn't use Calcite >>>>>> >>>>>> On Thu, Aug 10, 2017 at 3:14 PM kant kodali >>>>>> wrote: >>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> Does Spark SQL uses Calcite? If so, what for? I thought the Spark >>>>>>> SQL has catalyst which would generate its own logical plans, physical >>>>>>> plans >>>>>>> and other optimizations. >>>>>>> >>>>>>> Thanks, >>>>>>> Kant >>>>>>> >>>>>> >>>>> >>> >> >
Re: Does Spark SQL uses Calcite?
@Ryan it looks like if I enable thrift server I need to go through hive. I was talking more about having JDBC connector for Spark SQL itself other words not going through hive. On Fri, Aug 11, 2017 at 6:50 PM, kant kodali wrote: > @Ryan Does it work with Spark SQL 2.1.1? > > On Fri, Aug 11, 2017 at 12:53 AM, Ryan wrote: > >> the thrift server is a jdbc server, Kanth >> >> On Fri, Aug 11, 2017 at 2:51 PM, wrote: >> >>> I also wonder why there isn't a jdbc connector for spark sql? >>> >>> Sent from my iPhone >>> >>> On Aug 10, 2017, at 2:45 PM, Jules Damji wrote: >>> >>> Yes, it's more used in Hive than Spark >>> >>> Sent from my iPhone >>> Pardon the dumb thumb typos :) >>> >>> On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu < >>> vsathishkuma...@gmail.com> wrote: >>> >>> I think it is for hive dependency. >>> On Thu, Aug 10, 2017 at 4:14 PM kant kodali wrote: >>> >>>> Since I see a calcite dependency in Spark I wonder where Calcite is >>>> being used? >>>> >>>> On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu < >>>> vsathishkuma...@gmail.com> wrote: >>>> >>>>> Spark SQL doesn't use Calcite >>>>> >>>>> On Thu, Aug 10, 2017 at 3:14 PM kant kodali >>>>> wrote: >>>>> >>>>>> Hi All, >>>>>> >>>>>> Does Spark SQL uses Calcite? If so, what for? I thought the Spark SQL >>>>>> has catalyst which would generate its own logical plans, physical plans >>>>>> and >>>>>> other optimizations. >>>>>> >>>>>> Thanks, >>>>>> Kant >>>>>> >>>>> >>>> >> >
Re: Does Spark SQL uses Calcite?
@Ryan Does it work with Spark SQL 2.1.1? On Fri, Aug 11, 2017 at 12:53 AM, Ryan wrote: > the thrift server is a jdbc server, Kanth > > On Fri, Aug 11, 2017 at 2:51 PM, wrote: > >> I also wonder why there isn't a jdbc connector for spark sql? >> >> Sent from my iPhone >> >> On Aug 10, 2017, at 2:45 PM, Jules Damji wrote: >> >> Yes, it's more used in Hive than Spark >> >> Sent from my iPhone >> Pardon the dumb thumb typos :) >> >> On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu < >> vsathishkuma...@gmail.com> wrote: >> >> I think it is for hive dependency. >> On Thu, Aug 10, 2017 at 4:14 PM kant kodali wrote: >> >>> Since I see a calcite dependency in Spark I wonder where Calcite is >>> being used? >>> >>> On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu < >>> vsathishkuma...@gmail.com> wrote: >>> >>>> Spark SQL doesn't use Calcite >>>> >>>> On Thu, Aug 10, 2017 at 3:14 PM kant kodali wrote: >>>> >>>>> Hi All, >>>>> >>>>> Does Spark SQL uses Calcite? If so, what for? I thought the Spark SQL >>>>> has catalyst which would generate its own logical plans, physical plans >>>>> and >>>>> other optimizations. >>>>> >>>>> Thanks, >>>>> Kant >>>>> >>>> >>> >
Re: Does Spark SQL uses Calcite?
the thrift server is a jdbc server, Kanth On Fri, Aug 11, 2017 at 2:51 PM, wrote: > I also wonder why there isn't a jdbc connector for spark sql? > > Sent from my iPhone > > On Aug 10, 2017, at 2:45 PM, Jules Damji wrote: > > Yes, it's more used in Hive than Spark > > Sent from my iPhone > Pardon the dumb thumb typos :) > > On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu < > vsathishkuma...@gmail.com> wrote: > > I think it is for hive dependency. > On Thu, Aug 10, 2017 at 4:14 PM kant kodali wrote: > >> Since I see a calcite dependency in Spark I wonder where Calcite is being >> used? >> >> On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu < >> vsathishkuma...@gmail.com> wrote: >> >>> Spark SQL doesn't use Calcite >>> >>> On Thu, Aug 10, 2017 at 3:14 PM kant kodali wrote: >>> >>>> Hi All, >>>> >>>> Does Spark SQL uses Calcite? If so, what for? I thought the Spark SQL >>>> has catalyst which would generate its own logical plans, physical plans and >>>> other optimizations. >>>> >>>> Thanks, >>>> Kant >>>> >>> >>
Re: Does Spark SQL uses Calcite?
I also wonder why there isn't a jdbc connector for spark sql? Sent from my iPhone > On Aug 10, 2017, at 2:45 PM, Jules Damji wrote: > > Yes, it's more used in Hive than Spark > > Sent from my iPhone > Pardon the dumb thumb typos :) > >> On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu >> wrote: >> >> I think it is for hive dependency. >>> On Thu, Aug 10, 2017 at 4:14 PM kant kodali wrote: >>> Since I see a calcite dependency in Spark I wonder where Calcite is being >>> used? >>> >>>> On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu >>>> wrote: >>>> Spark SQL doesn't use Calcite >>>> >>>>> On Thu, Aug 10, 2017 at 3:14 PM kant kodali wrote: >>>>> Hi All, >>>>> >>>>> Does Spark SQL uses Calcite? If so, what for? I thought the Spark SQL has >>>>> catalyst which would generate its own logical plans, physical plans and >>>>> other optimizations. >>>>> >>>>> Thanks, >>>>> Kant >>>
Re: Does Spark SQL uses Calcite?
Yes, it's more used in Hive than Spark Sent from my iPhone Pardon the dumb thumb typos :) > On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu > wrote: > > I think it is for hive dependency. >> On Thu, Aug 10, 2017 at 4:14 PM kant kodali wrote: >> Since I see a calcite dependency in Spark I wonder where Calcite is being >> used? >> >>> On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu >>> wrote: >>> Spark SQL doesn't use Calcite >>> >>>> On Thu, Aug 10, 2017 at 3:14 PM kant kodali wrote: >>>> Hi All, >>>> >>>> Does Spark SQL uses Calcite? If so, what for? I thought the Spark SQL has >>>> catalyst which would generate its own logical plans, physical plans and >>>> other optimizations. >>>> >>>> Thanks, >>>> Kant >>
Re: Does Spark SQL uses Calcite?
I think it is for hive dependency. On Thu, Aug 10, 2017 at 4:14 PM kant kodali wrote: > Since I see a calcite dependency in Spark I wonder where Calcite is being > used? > > On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu < > vsathishkuma...@gmail.com> wrote: > >> Spark SQL doesn't use Calcite >> >> On Thu, Aug 10, 2017 at 3:14 PM kant kodali wrote: >> >>> Hi All, >>> >>> Does Spark SQL uses Calcite? If so, what for? I thought the Spark SQL >>> has catalyst which would generate its own logical plans, physical plans and >>> other optimizations. >>> >>> Thanks, >>> Kant >>> >> >
Re: Does Spark SQL uses Calcite?
Since I see a calcite dependency in Spark I wonder where Calcite is being used? On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu < vsathishkuma...@gmail.com> wrote: > Spark SQL doesn't use Calcite > > On Thu, Aug 10, 2017 at 3:14 PM kant kodali wrote: > >> Hi All, >> >> Does Spark SQL uses Calcite? If so, what for? I thought the Spark SQL has >> catalyst which would generate its own logical plans, physical plans and >> other optimizations. >> >> Thanks, >> Kant >> >
Re: Does Spark SQL uses Calcite?
Spark SQL doesn't use Calcite On Thu, Aug 10, 2017 at 3:14 PM kant kodali wrote: > Hi All, > > Does Spark SQL uses Calcite? If so, what for? I thought the Spark SQL has > catalyst which would generate its own logical plans, physical plans and > other optimizations. > > Thanks, > Kant >
Does Spark SQL uses Calcite?
Hi All, Does Spark SQL uses Calcite? If so, what for? I thought the Spark SQL has catalyst which would generate its own logical plans, physical plans and other optimizations. Thanks, Kant
