Re: Does Spark SQL uses Calcite?

[kant kodali]

Hi Jules,

I am looking to connect to Spark via JDBC so I can run Spark SQL queries
via JDBC but not use SPARK SQL to connect to other JDBC sources.

Thanks!

On Sat, Aug 19, 2017 at 5:54 PM, Jules Damji  wrote:

> Try this link to see how you may connect  https://docs.databricks.com/
> spark/latest/data-sources/sql-databases.html
>
> Cheers
> Jules
>
> Sent from my iPhone
> Pardon the dumb thumb typos :)
>
> On Aug 19, 2017, at 5:27 PM, kant kodali  wrote:
>
> Hi Russell,
>
> I went through this https://jaceklaskowski.gitbooks.io/mastering-apache-
> spark/spark-sql-thrift-server.html and I am still a bit confused on what
> hive is doing in here ? Is there any example I can look at on how to talk
> to Spark using Spark SQL JDBC driver alone and not hive ?
>
> Thanks,
> kant
>
>
>
>
>
> On Sat, Aug 12, 2017 at 6:07 AM, Russell Spitzer <
> russell.spit...@gmail.com> wrote:
>
>> You don't have to go through hive. It's just spark sql. The application
>> is just a forked hive thrift server.
>>
>> On Fri, Aug 11, 2017 at 8:53 PM kant kodali  wrote:
>>
>>> @Ryan it looks like if I enable thrift server I need to go through hive.
>>> I was talking more about having JDBC connector for Spark SQL itself other
>>> words not going through hive.
>>>
>>> On Fri, Aug 11, 2017 at 6:50 PM, kant kodali  wrote:
>>>
 @Ryan Does it work with Spark SQL 2.1.1?

 On Fri, Aug 11, 2017 at 12:53 AM, Ryan  wrote:

> the thrift server is a jdbc server, Kanth
>
> On Fri, Aug 11, 2017 at 2:51 PM,  wrote:
>
>> I also wonder why there isn't a jdbc connector for spark sql?
>>
>> Sent from my iPhone
>>
>> On Aug 10, 2017, at 2:45 PM, Jules Damji  wrote:
>>
>> Yes, it's more used in Hive than Spark
>>
>> Sent from my iPhone
>> Pardon the dumb thumb typos :)
>>
>> On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu <
>> vsathishkuma...@gmail.com> wrote:
>>
>> I think it is for hive dependency.
>> On Thu, Aug 10, 2017 at 4:14 PM kant kodali 
>> wrote:
>>
>>> Since I see a calcite dependency in Spark I wonder where Calcite is
>>> being used?
>>>
>>> On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu <
>>> vsathishkuma...@gmail.com> wrote:
>>>
 Spark SQL doesn't use Calcite

 On Thu, Aug 10, 2017 at 3:14 PM kant kodali 
 wrote:

> Hi All,
>
> Does Spark SQL uses Calcite? If so, what for? I thought the Spark
> SQL has catalyst which would generate its own logical plans, physical 
> plans
> and other optimizations.
>
> Thanks,
> Kant
>

>>>
>

>>>
>

Re: Does Spark SQL uses Calcite?

[Jules Damji]

Try this link to see how you may connect  
https://docs.databricks.com/spark/latest/data-sources/sql-databases.html

Cheers
Jules

Sent from my iPhone
Pardon the dumb thumb typos :)

> On Aug 19, 2017, at 5:27 PM, kant kodali  wrote:
> 
> Hi Russell,
> 
> I went through this 
> https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-sql-thrift-server.html
>  and I am still a bit confused on what hive is doing in here ? Is there any 
> example I can look at on how to talk to Spark using Spark SQL JDBC driver 
> alone and not hive ?
> 
> Thanks,
> kant
> 
> 
> 
> 
> 
>> On Sat, Aug 12, 2017 at 6:07 AM, Russell Spitzer  
>> wrote:
>> You don't have to go through hive. It's just spark sql. The application is 
>> just a forked hive thrift server.
>> 
>>> On Fri, Aug 11, 2017 at 8:53 PM kant kodali  wrote:
>>> @Ryan it looks like if I enable thrift server I need to go through hive. I 
>>> was talking more about having JDBC connector for Spark SQL itself other 
>>> words not going through hive.
>>> 
 On Fri, Aug 11, 2017 at 6:50 PM, kant kodali  wrote:
 @Ryan Does it work with Spark SQL 2.1.1?
 
> On Fri, Aug 11, 2017 at 12:53 AM, Ryan  wrote:
> the thrift server is a jdbc server, Kanth
> 
>> On Fri, Aug 11, 2017 at 2:51 PM,  wrote:
>> I also wonder why there isn't a jdbc connector for spark sql?
>> 
>> Sent from my iPhone
>> 
>>> On Aug 10, 2017, at 2:45 PM, Jules Damji  wrote:
>>> 
>>> Yes, it's more used in Hive than Spark 
>>> 
>>> Sent from my iPhone
>>> Pardon the dumb thumb typos :)
>>> 
 On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu 
  wrote:
 
 I think it is for hive dependency.
> On Thu, Aug 10, 2017 at 4:14 PM kant kodali  
> wrote:
> Since I see a calcite dependency in Spark I wonder where Calcite is 
> being used?
> 
>> On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu 
>>  wrote:
>> Spark SQL doesn't use Calcite
>> 
>>> On Thu, Aug 10, 2017 at 3:14 PM kant kodali  
>>> wrote:
>>> Hi All, 
>>> 
>>> Does Spark SQL uses Calcite? If so, what for? I thought the Spark 
>>> SQL has catalyst which would generate its own logical plans, 
>>> physical plans and other optimizations. 
>>> 
>>> Thanks,
>>> Kant
> 
> 
 
>>> 
> 

Re: Does Spark SQL uses Calcite?

[kant kodali]

Hi Russell,

I went through this
https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-sql-thrift-server.html
and I am still a bit confused on what hive is doing in here ? Is there any
example I can look at on how to talk to Spark using Spark SQL JDBC driver
alone and not hive ?

Thanks,
kant





On Sat, Aug 12, 2017 at 6:07 AM, Russell Spitzer 
wrote:

> You don't have to go through hive. It's just spark sql. The application is
> just a forked hive thrift server.
>
> On Fri, Aug 11, 2017 at 8:53 PM kant kodali  wrote:
>
>> @Ryan it looks like if I enable thrift server I need to go through hive.
>> I was talking more about having JDBC connector for Spark SQL itself other
>> words not going through hive.
>>
>> On Fri, Aug 11, 2017 at 6:50 PM, kant kodali  wrote:
>>
>>> @Ryan Does it work with Spark SQL 2.1.1?
>>>
>>> On Fri, Aug 11, 2017 at 12:53 AM, Ryan  wrote:
>>>
 the thrift server is a jdbc server, Kanth

 On Fri, Aug 11, 2017 at 2:51 PM,  wrote:

> I also wonder why there isn't a jdbc connector for spark sql?
>
> Sent from my iPhone
>
> On Aug 10, 2017, at 2:45 PM, Jules Damji  wrote:
>
> Yes, it's more used in Hive than Spark
>
> Sent from my iPhone
> Pardon the dumb thumb typos :)
>
> On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu <
> vsathishkuma...@gmail.com> wrote:
>
> I think it is for hive dependency.
> On Thu, Aug 10, 2017 at 4:14 PM kant kodali 
> wrote:
>
>> Since I see a calcite dependency in Spark I wonder where Calcite is
>> being used?
>>
>> On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu <
>> vsathishkuma...@gmail.com> wrote:
>>
>>> Spark SQL doesn't use Calcite
>>>
>>> On Thu, Aug 10, 2017 at 3:14 PM kant kodali 
>>> wrote:
>>>
 Hi All,

 Does Spark SQL uses Calcite? If so, what for? I thought the Spark
 SQL has catalyst which would generate its own logical plans, physical 
 plans
 and other optimizations.

 Thanks,
 Kant

>>>
>>

>>>
>>

Re: Does Spark SQL uses Calcite?

[Russell Spitzer]

You don't have to go through hive. It's just spark sql. The application is
just a forked hive thrift server.

On Fri, Aug 11, 2017 at 8:53 PM kant kodali  wrote:

> @Ryan it looks like if I enable thrift server I need to go through hive. I
> was talking more about having JDBC connector for Spark SQL itself other
> words not going through hive.
>
> On Fri, Aug 11, 2017 at 6:50 PM, kant kodali  wrote:
>
>> @Ryan Does it work with Spark SQL 2.1.1?
>>
>> On Fri, Aug 11, 2017 at 12:53 AM, Ryan  wrote:
>>
>>> the thrift server is a jdbc server, Kanth
>>>
>>> On Fri, Aug 11, 2017 at 2:51 PM,  wrote:
>>>
 I also wonder why there isn't a jdbc connector for spark sql?

 Sent from my iPhone

 On Aug 10, 2017, at 2:45 PM, Jules Damji  wrote:

 Yes, it's more used in Hive than Spark

 Sent from my iPhone
 Pardon the dumb thumb typos :)

 On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu <
 vsathishkuma...@gmail.com> wrote:

 I think it is for hive dependency.
 On Thu, Aug 10, 2017 at 4:14 PM kant kodali  wrote:

> Since I see a calcite dependency in Spark I wonder where Calcite is
> being used?
>
> On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu <
> vsathishkuma...@gmail.com> wrote:
>
>> Spark SQL doesn't use Calcite
>>
>> On Thu, Aug 10, 2017 at 3:14 PM kant kodali 
>> wrote:
>>
>>> Hi All,
>>>
>>> Does Spark SQL uses Calcite? If so, what for? I thought the Spark
>>> SQL has catalyst which would generate its own logical plans, physical 
>>> plans
>>> and other optimizations.
>>>
>>> Thanks,
>>> Kant
>>>
>>
>
>>>
>>
>

Re: Does Spark SQL uses Calcite?

[kant kodali]

@Ryan it looks like if I enable thrift server I need to go through hive. I
was talking more about having JDBC connector for Spark SQL itself other
words not going through hive.

On Fri, Aug 11, 2017 at 6:50 PM, kant kodali  wrote:

> @Ryan Does it work with Spark SQL 2.1.1?
>
> On Fri, Aug 11, 2017 at 12:53 AM, Ryan  wrote:
>
>> the thrift server is a jdbc server, Kanth
>>
>> On Fri, Aug 11, 2017 at 2:51 PM,  wrote:
>>
>>> I also wonder why there isn't a jdbc connector for spark sql?
>>>
>>> Sent from my iPhone
>>>
>>> On Aug 10, 2017, at 2:45 PM, Jules Damji  wrote:
>>>
>>> Yes, it's more used in Hive than Spark
>>>
>>> Sent from my iPhone
>>> Pardon the dumb thumb typos :)
>>>
>>> On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu <
>>> vsathishkuma...@gmail.com> wrote:
>>>
>>> I think it is for hive dependency.
>>> On Thu, Aug 10, 2017 at 4:14 PM kant kodali  wrote:
>>>
 Since I see a calcite dependency in Spark I wonder where Calcite is
 being used?

 On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu <
 vsathishkuma...@gmail.com> wrote:

> Spark SQL doesn't use Calcite
>
> On Thu, Aug 10, 2017 at 3:14 PM kant kodali 
> wrote:
>
>> Hi All,
>>
>> Does Spark SQL uses Calcite? If so, what for? I thought the Spark SQL
>> has catalyst which would generate its own logical plans, physical plans 
>> and
>> other optimizations.
>>
>> Thanks,
>> Kant
>>
>

>>
>

Re: Does Spark SQL uses Calcite?

[kant kodali]

@Ryan Does it work with Spark SQL 2.1.1?

On Fri, Aug 11, 2017 at 12:53 AM, Ryan  wrote:

> the thrift server is a jdbc server, Kanth
>
> On Fri, Aug 11, 2017 at 2:51 PM,  wrote:
>
>> I also wonder why there isn't a jdbc connector for spark sql?
>>
>> Sent from my iPhone
>>
>> On Aug 10, 2017, at 2:45 PM, Jules Damji  wrote:
>>
>> Yes, it's more used in Hive than Spark
>>
>> Sent from my iPhone
>> Pardon the dumb thumb typos :)
>>
>> On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu <
>> vsathishkuma...@gmail.com> wrote:
>>
>> I think it is for hive dependency.
>> On Thu, Aug 10, 2017 at 4:14 PM kant kodali  wrote:
>>
>>> Since I see a calcite dependency in Spark I wonder where Calcite is
>>> being used?
>>>
>>> On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu <
>>> vsathishkuma...@gmail.com> wrote:
>>>
 Spark SQL doesn't use Calcite

 On Thu, Aug 10, 2017 at 3:14 PM kant kodali  wrote:

> Hi All,
>
> Does Spark SQL uses Calcite? If so, what for? I thought the Spark SQL
> has catalyst which would generate its own logical plans, physical plans 
> and
> other optimizations.
>
> Thanks,
> Kant
>

>>>
>

Re: Does Spark SQL uses Calcite?

[Ryan]

the thrift server is a jdbc server, Kanth

On Fri, Aug 11, 2017 at 2:51 PM,  wrote:

> I also wonder why there isn't a jdbc connector for spark sql?
>
> Sent from my iPhone
>
> On Aug 10, 2017, at 2:45 PM, Jules Damji  wrote:
>
> Yes, it's more used in Hive than Spark
>
> Sent from my iPhone
> Pardon the dumb thumb typos :)
>
> On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu <
> vsathishkuma...@gmail.com> wrote:
>
> I think it is for hive dependency.
> On Thu, Aug 10, 2017 at 4:14 PM kant kodali  wrote:
>
>> Since I see a calcite dependency in Spark I wonder where Calcite is being
>> used?
>>
>> On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu <
>> vsathishkuma...@gmail.com> wrote:
>>
>>> Spark SQL doesn't use Calcite
>>>
>>> On Thu, Aug 10, 2017 at 3:14 PM kant kodali  wrote:
>>>
 Hi All,

 Does Spark SQL uses Calcite? If so, what for? I thought the Spark SQL
 has catalyst which would generate its own logical plans, physical plans and
 other optimizations.

 Thanks,
 Kant

>>>
>>

Re: Does Spark SQL uses Calcite?

[kanth909]

I also wonder why there isn't a jdbc connector for spark sql?

Sent from my iPhone

> On Aug 10, 2017, at 2:45 PM, Jules Damji  wrote:
> 
> Yes, it's more used in Hive than Spark 
> 
> Sent from my iPhone
> Pardon the dumb thumb typos :)
> 
>> On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu 
>>  wrote:
>> 
>> I think it is for hive dependency.
>>> On Thu, Aug 10, 2017 at 4:14 PM kant kodali  wrote:
>>> Since I see a calcite dependency in Spark I wonder where Calcite is being 
>>> used?
>>> 
 On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu 
  wrote:
 Spark SQL doesn't use Calcite
 
> On Thu, Aug 10, 2017 at 3:14 PM kant kodali  wrote:
> Hi All, 
> 
> Does Spark SQL uses Calcite? If so, what for? I thought the Spark SQL has 
> catalyst which would generate its own logical plans, physical plans and 
> other optimizations. 
> 
> Thanks,
> Kant
>>> 

Re: Does Spark SQL uses Calcite?

[Jules Damji]

Yes, it's more used in Hive than Spark 

Sent from my iPhone
Pardon the dumb thumb typos :)

> On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu 
>  wrote:
> 
> I think it is for hive dependency.
>> On Thu, Aug 10, 2017 at 4:14 PM kant kodali  wrote:
>> Since I see a calcite dependency in Spark I wonder where Calcite is being 
>> used?
>> 
>>> On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu 
>>>  wrote:
>>> Spark SQL doesn't use Calcite
>>> 
 On Thu, Aug 10, 2017 at 3:14 PM kant kodali  wrote:
 Hi All, 
 
 Does Spark SQL uses Calcite? If so, what for? I thought the Spark SQL has 
 catalyst which would generate its own logical plans, physical plans and 
 other optimizations. 
 
 Thanks,
 Kant
>> 

Re: Does Spark SQL uses Calcite?

[Sathish Kumaran Vairavelu]

I think it is for hive dependency.
On Thu, Aug 10, 2017 at 4:14 PM kant kodali  wrote:

> Since I see a calcite dependency in Spark I wonder where Calcite is being
> used?
>
> On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu <
> vsathishkuma...@gmail.com> wrote:
>
>> Spark SQL doesn't use Calcite
>>
>> On Thu, Aug 10, 2017 at 3:14 PM kant kodali  wrote:
>>
>>> Hi All,
>>>
>>> Does Spark SQL uses Calcite? If so, what for? I thought the Spark SQL
>>> has catalyst which would generate its own logical plans, physical plans and
>>> other optimizations.
>>>
>>> Thanks,
>>> Kant
>>>
>>
>

Re: Does Spark SQL uses Calcite?

[kant kodali]

Since I see a calcite dependency in Spark I wonder where Calcite is being
used?

On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu <
vsathishkuma...@gmail.com> wrote:

> Spark SQL doesn't use Calcite
>
> On Thu, Aug 10, 2017 at 3:14 PM kant kodali  wrote:
>
>> Hi All,
>>
>> Does Spark SQL uses Calcite? If so, what for? I thought the Spark SQL has
>> catalyst which would generate its own logical plans, physical plans and
>> other optimizations.
>>
>> Thanks,
>> Kant
>>
>

Re: Does Spark SQL uses Calcite?

[Sathish Kumaran Vairavelu]

Spark SQL doesn't use Calcite
On Thu, Aug 10, 2017 at 3:14 PM kant kodali  wrote:

> Hi All,
>
> Does Spark SQL uses Calcite? If so, what for? I thought the Spark SQL has
> catalyst which would generate its own logical plans, physical plans and
> other optimizations.
>
> Thanks,
> Kant
>