Re: Does Spark SQL uses Calcite?
Hi Jules, I am looking to connect to Spark via JDBC so I can run Spark SQL queries via JDBC but not use SPARK SQL to connect to other JDBC sources. Thanks! On Sat, Aug 19, 2017 at 5:54 PM, Jules Damji wrote: > Try this link to see how you may connect https://docs.databricks.com/ > spark/latest/data-sources/sql-databases.html > > Cheers > Jules > > Sent from my iPhone > Pardon the dumb thumb typos :) > > On Aug 19, 2017, at 5:27 PM, kant kodali wrote: > > Hi Russell, > > I went through this https://jaceklaskowski.gitbooks.io/mastering-apache- > spark/spark-sql-thrift-server.html and I am still a bit confused on what > hive is doing in here ? Is there any example I can look at on how to talk > to Spark using Spark SQL JDBC driver alone and not hive ? > > Thanks, > kant > > > > > > On Sat, Aug 12, 2017 at 6:07 AM, Russell Spitzer < > russell.spit...@gmail.com> wrote: > >> You don't have to go through hive. It's just spark sql. The application >> is just a forked hive thrift server. >> >> On Fri, Aug 11, 2017 at 8:53 PM kant kodali wrote: >> >>> @Ryan it looks like if I enable thrift server I need to go through hive. >>> I was talking more about having JDBC connector for Spark SQL itself other >>> words not going through hive. >>> >>> On Fri, Aug 11, 2017 at 6:50 PM, kant kodali wrote: >>> @Ryan Does it work with Spark SQL 2.1.1? On Fri, Aug 11, 2017 at 12:53 AM, Ryan wrote: > the thrift server is a jdbc server, Kanth > > On Fri, Aug 11, 2017 at 2:51 PM, wrote: > >> I also wonder why there isn't a jdbc connector for spark sql? >> >> Sent from my iPhone >> >> On Aug 10, 2017, at 2:45 PM, Jules Damji wrote: >> >> Yes, it's more used in Hive than Spark >> >> Sent from my iPhone >> Pardon the dumb thumb typos :) >> >> On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu < >> vsathishkuma...@gmail.com> wrote: >> >> I think it is for hive dependency. >> On Thu, Aug 10, 2017 at 4:14 PM kant kodali >> wrote: >> >>> Since I see a calcite dependency in Spark I wonder where Calcite is >>> being used? >>> >>> On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu < >>> vsathishkuma...@gmail.com> wrote: >>> Spark SQL doesn't use Calcite On Thu, Aug 10, 2017 at 3:14 PM kant kodali wrote: > Hi All, > > Does Spark SQL uses Calcite? If so, what for? I thought the Spark > SQL has catalyst which would generate its own logical plans, physical > plans > and other optimizations. > > Thanks, > Kant > >>> > >>> >
Re: Does Spark SQL uses Calcite?
Try this link to see how you may connect https://docs.databricks.com/spark/latest/data-sources/sql-databases.html Cheers Jules Sent from my iPhone Pardon the dumb thumb typos :) > On Aug 19, 2017, at 5:27 PM, kant kodali wrote: > > Hi Russell, > > I went through this > https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-sql-thrift-server.html > and I am still a bit confused on what hive is doing in here ? Is there any > example I can look at on how to talk to Spark using Spark SQL JDBC driver > alone and not hive ? > > Thanks, > kant > > > > > >> On Sat, Aug 12, 2017 at 6:07 AM, Russell Spitzer >> wrote: >> You don't have to go through hive. It's just spark sql. The application is >> just a forked hive thrift server. >> >>> On Fri, Aug 11, 2017 at 8:53 PM kant kodali wrote: >>> @Ryan it looks like if I enable thrift server I need to go through hive. I >>> was talking more about having JDBC connector for Spark SQL itself other >>> words not going through hive. >>> On Fri, Aug 11, 2017 at 6:50 PM, kant kodali wrote: @Ryan Does it work with Spark SQL 2.1.1? > On Fri, Aug 11, 2017 at 12:53 AM, Ryan wrote: > the thrift server is a jdbc server, Kanth > >> On Fri, Aug 11, 2017 at 2:51 PM, wrote: >> I also wonder why there isn't a jdbc connector for spark sql? >> >> Sent from my iPhone >> >>> On Aug 10, 2017, at 2:45 PM, Jules Damji wrote: >>> >>> Yes, it's more used in Hive than Spark >>> >>> Sent from my iPhone >>> Pardon the dumb thumb typos :) >>> On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu wrote: I think it is for hive dependency. > On Thu, Aug 10, 2017 at 4:14 PM kant kodali > wrote: > Since I see a calcite dependency in Spark I wonder where Calcite is > being used? > >> On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu >> wrote: >> Spark SQL doesn't use Calcite >> >>> On Thu, Aug 10, 2017 at 3:14 PM kant kodali >>> wrote: >>> Hi All, >>> >>> Does Spark SQL uses Calcite? If so, what for? I thought the Spark >>> SQL has catalyst which would generate its own logical plans, >>> physical plans and other optimizations. >>> >>> Thanks, >>> Kant > > >>> >
Re: Does Spark SQL uses Calcite?
Hi Russell, I went through this https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-sql-thrift-server.html and I am still a bit confused on what hive is doing in here ? Is there any example I can look at on how to talk to Spark using Spark SQL JDBC driver alone and not hive ? Thanks, kant On Sat, Aug 12, 2017 at 6:07 AM, Russell Spitzer wrote: > You don't have to go through hive. It's just spark sql. The application is > just a forked hive thrift server. > > On Fri, Aug 11, 2017 at 8:53 PM kant kodali wrote: > >> @Ryan it looks like if I enable thrift server I need to go through hive. >> I was talking more about having JDBC connector for Spark SQL itself other >> words not going through hive. >> >> On Fri, Aug 11, 2017 at 6:50 PM, kant kodali wrote: >> >>> @Ryan Does it work with Spark SQL 2.1.1? >>> >>> On Fri, Aug 11, 2017 at 12:53 AM, Ryan wrote: >>> the thrift server is a jdbc server, Kanth On Fri, Aug 11, 2017 at 2:51 PM, wrote: > I also wonder why there isn't a jdbc connector for spark sql? > > Sent from my iPhone > > On Aug 10, 2017, at 2:45 PM, Jules Damji wrote: > > Yes, it's more used in Hive than Spark > > Sent from my iPhone > Pardon the dumb thumb typos :) > > On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu < > vsathishkuma...@gmail.com> wrote: > > I think it is for hive dependency. > On Thu, Aug 10, 2017 at 4:14 PM kant kodali > wrote: > >> Since I see a calcite dependency in Spark I wonder where Calcite is >> being used? >> >> On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu < >> vsathishkuma...@gmail.com> wrote: >> >>> Spark SQL doesn't use Calcite >>> >>> On Thu, Aug 10, 2017 at 3:14 PM kant kodali >>> wrote: >>> Hi All, Does Spark SQL uses Calcite? If so, what for? I thought the Spark SQL has catalyst which would generate its own logical plans, physical plans and other optimizations. Thanks, Kant >>> >> >>> >>
Re: Does Spark SQL uses Calcite?
You don't have to go through hive. It's just spark sql. The application is just a forked hive thrift server. On Fri, Aug 11, 2017 at 8:53 PM kant kodali wrote: > @Ryan it looks like if I enable thrift server I need to go through hive. I > was talking more about having JDBC connector for Spark SQL itself other > words not going through hive. > > On Fri, Aug 11, 2017 at 6:50 PM, kant kodali wrote: > >> @Ryan Does it work with Spark SQL 2.1.1? >> >> On Fri, Aug 11, 2017 at 12:53 AM, Ryan wrote: >> >>> the thrift server is a jdbc server, Kanth >>> >>> On Fri, Aug 11, 2017 at 2:51 PM, wrote: >>> I also wonder why there isn't a jdbc connector for spark sql? Sent from my iPhone On Aug 10, 2017, at 2:45 PM, Jules Damji wrote: Yes, it's more used in Hive than Spark Sent from my iPhone Pardon the dumb thumb typos :) On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu < vsathishkuma...@gmail.com> wrote: I think it is for hive dependency. On Thu, Aug 10, 2017 at 4:14 PM kant kodali wrote: > Since I see a calcite dependency in Spark I wonder where Calcite is > being used? > > On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu < > vsathishkuma...@gmail.com> wrote: > >> Spark SQL doesn't use Calcite >> >> On Thu, Aug 10, 2017 at 3:14 PM kant kodali >> wrote: >> >>> Hi All, >>> >>> Does Spark SQL uses Calcite? If so, what for? I thought the Spark >>> SQL has catalyst which would generate its own logical plans, physical >>> plans >>> and other optimizations. >>> >>> Thanks, >>> Kant >>> >> > >>> >> >
Re: Does Spark SQL uses Calcite?
@Ryan it looks like if I enable thrift server I need to go through hive. I was talking more about having JDBC connector for Spark SQL itself other words not going through hive. On Fri, Aug 11, 2017 at 6:50 PM, kant kodali wrote: > @Ryan Does it work with Spark SQL 2.1.1? > > On Fri, Aug 11, 2017 at 12:53 AM, Ryan wrote: > >> the thrift server is a jdbc server, Kanth >> >> On Fri, Aug 11, 2017 at 2:51 PM, wrote: >> >>> I also wonder why there isn't a jdbc connector for spark sql? >>> >>> Sent from my iPhone >>> >>> On Aug 10, 2017, at 2:45 PM, Jules Damji wrote: >>> >>> Yes, it's more used in Hive than Spark >>> >>> Sent from my iPhone >>> Pardon the dumb thumb typos :) >>> >>> On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu < >>> vsathishkuma...@gmail.com> wrote: >>> >>> I think it is for hive dependency. >>> On Thu, Aug 10, 2017 at 4:14 PM kant kodali wrote: >>> Since I see a calcite dependency in Spark I wonder where Calcite is being used? On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu < vsathishkuma...@gmail.com> wrote: > Spark SQL doesn't use Calcite > > On Thu, Aug 10, 2017 at 3:14 PM kant kodali > wrote: > >> Hi All, >> >> Does Spark SQL uses Calcite? If so, what for? I thought the Spark SQL >> has catalyst which would generate its own logical plans, physical plans >> and >> other optimizations. >> >> Thanks, >> Kant >> > >> >
Re: Does Spark SQL uses Calcite?
@Ryan Does it work with Spark SQL 2.1.1? On Fri, Aug 11, 2017 at 12:53 AM, Ryan wrote: > the thrift server is a jdbc server, Kanth > > On Fri, Aug 11, 2017 at 2:51 PM, wrote: > >> I also wonder why there isn't a jdbc connector for spark sql? >> >> Sent from my iPhone >> >> On Aug 10, 2017, at 2:45 PM, Jules Damji wrote: >> >> Yes, it's more used in Hive than Spark >> >> Sent from my iPhone >> Pardon the dumb thumb typos :) >> >> On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu < >> vsathishkuma...@gmail.com> wrote: >> >> I think it is for hive dependency. >> On Thu, Aug 10, 2017 at 4:14 PM kant kodali wrote: >> >>> Since I see a calcite dependency in Spark I wonder where Calcite is >>> being used? >>> >>> On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu < >>> vsathishkuma...@gmail.com> wrote: >>> Spark SQL doesn't use Calcite On Thu, Aug 10, 2017 at 3:14 PM kant kodali wrote: > Hi All, > > Does Spark SQL uses Calcite? If so, what for? I thought the Spark SQL > has catalyst which would generate its own logical plans, physical plans > and > other optimizations. > > Thanks, > Kant > >>> >
Re: Does Spark SQL uses Calcite?
the thrift server is a jdbc server, Kanth On Fri, Aug 11, 2017 at 2:51 PM, wrote: > I also wonder why there isn't a jdbc connector for spark sql? > > Sent from my iPhone > > On Aug 10, 2017, at 2:45 PM, Jules Damji wrote: > > Yes, it's more used in Hive than Spark > > Sent from my iPhone > Pardon the dumb thumb typos :) > > On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu < > vsathishkuma...@gmail.com> wrote: > > I think it is for hive dependency. > On Thu, Aug 10, 2017 at 4:14 PM kant kodali wrote: > >> Since I see a calcite dependency in Spark I wonder where Calcite is being >> used? >> >> On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu < >> vsathishkuma...@gmail.com> wrote: >> >>> Spark SQL doesn't use Calcite >>> >>> On Thu, Aug 10, 2017 at 3:14 PM kant kodali wrote: >>> Hi All, Does Spark SQL uses Calcite? If so, what for? I thought the Spark SQL has catalyst which would generate its own logical plans, physical plans and other optimizations. Thanks, Kant >>> >>
Re: Does Spark SQL uses Calcite?
I also wonder why there isn't a jdbc connector for spark sql? Sent from my iPhone > On Aug 10, 2017, at 2:45 PM, Jules Damji wrote: > > Yes, it's more used in Hive than Spark > > Sent from my iPhone > Pardon the dumb thumb typos :) > >> On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu >> wrote: >> >> I think it is for hive dependency. >>> On Thu, Aug 10, 2017 at 4:14 PM kant kodali wrote: >>> Since I see a calcite dependency in Spark I wonder where Calcite is being >>> used? >>> On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu wrote: Spark SQL doesn't use Calcite > On Thu, Aug 10, 2017 at 3:14 PM kant kodali wrote: > Hi All, > > Does Spark SQL uses Calcite? If so, what for? I thought the Spark SQL has > catalyst which would generate its own logical plans, physical plans and > other optimizations. > > Thanks, > Kant >>>
Re: Does Spark SQL uses Calcite?
Yes, it's more used in Hive than Spark Sent from my iPhone Pardon the dumb thumb typos :) > On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu > wrote: > > I think it is for hive dependency. >> On Thu, Aug 10, 2017 at 4:14 PM kant kodali wrote: >> Since I see a calcite dependency in Spark I wonder where Calcite is being >> used? >> >>> On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu >>> wrote: >>> Spark SQL doesn't use Calcite >>> On Thu, Aug 10, 2017 at 3:14 PM kant kodali wrote: Hi All, Does Spark SQL uses Calcite? If so, what for? I thought the Spark SQL has catalyst which would generate its own logical plans, physical plans and other optimizations. Thanks, Kant >>
Re: Does Spark SQL uses Calcite?
I think it is for hive dependency. On Thu, Aug 10, 2017 at 4:14 PM kant kodali wrote: > Since I see a calcite dependency in Spark I wonder where Calcite is being > used? > > On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu < > vsathishkuma...@gmail.com> wrote: > >> Spark SQL doesn't use Calcite >> >> On Thu, Aug 10, 2017 at 3:14 PM kant kodali wrote: >> >>> Hi All, >>> >>> Does Spark SQL uses Calcite? If so, what for? I thought the Spark SQL >>> has catalyst which would generate its own logical plans, physical plans and >>> other optimizations. >>> >>> Thanks, >>> Kant >>> >> >
Re: Does Spark SQL uses Calcite?
Since I see a calcite dependency in Spark I wonder where Calcite is being used? On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu < vsathishkuma...@gmail.com> wrote: > Spark SQL doesn't use Calcite > > On Thu, Aug 10, 2017 at 3:14 PM kant kodali wrote: > >> Hi All, >> >> Does Spark SQL uses Calcite? If so, what for? I thought the Spark SQL has >> catalyst which would generate its own logical plans, physical plans and >> other optimizations. >> >> Thanks, >> Kant >> >
Re: Does Spark SQL uses Calcite?
Spark SQL doesn't use Calcite On Thu, Aug 10, 2017 at 3:14 PM kant kodali wrote: > Hi All, > > Does Spark SQL uses Calcite? If so, what for? I thought the Spark SQL has > catalyst which would generate its own logical plans, physical plans and > other optimizations. > > Thanks, > Kant >