Re: Announcing Spark SQL

On Fri, Mar 28, 2014 at 9:53 PM, Rohit Rai wrote: > > Upon discussion with couple of our clients, it seems the reason they would > prefer using hive is that they have already invested a lot in it. Mostly in > UDFs and HiveQL. > 1. Are there any plans to develop the SQL Parser to handdle more compl

Re: Announcing Spark SQL

Thanks Patrick, I was thinking about that... Upon analysis I realized (on date) it would be something similar to the way Hive Context using CustomCatalog stuff. I will review it again, on the lines of implementing SchemaRDD with Cassandra. Thanks for the pointer. Upon discussion with couple of ou

Re: Announcing Spark SQL

Hey Rohit, I think external tables based on Cassandra or other datastores will work out-of-the box if you build Catalyst with Hive support. Michael may have feelings about this but I'd guess the longer term design for having schema support for Cassandra/HBase etc likely wouldn't rely on hive exte

Re: Announcing Spark SQL

2:49 PM, yana wrote: > >> Does Shark not suit your needs? That's what we use at the moment and it's >> been good >> >> >> Sent from my Samsung Galaxy S®4 >> >> >> Original message >> From: andy petrella >&g

Re: Announcing Spark SQL

gt; > > Original message > From: andy petrella > Date:03/27/2014 6:08 AM (GMT-05:00) > To: user@spark.apache.org > Subject: Re: Announcing Spark SQL > > nope (what I said :-P) > > > On Thu, Mar 27, 2014 at 11:05 AM, Pascal Voitot Dev < > pascal.

Re: Announcing Spark SQL

Does Shark not suit your needs? That's what we use at the moment and it's been good Sent from my Samsung Galaxy S®4 Original message From: andy petrella Date:03/27/2014 6:08 AM (GMT-05:00) To: user@spark.apache.org Subject: Re: Announcing Spark SQL nope (w

Re: Announcing Spark SQL

On Thu, Mar 27, 2014 at 11:08 AM, andy petrella wrote: > nope (what I said :-P) > That's also my answer to my own question :D but I didn't understand that in your sentence: "my2c is that this feature is also important to enable ad-hoc queries which is done at runtime." > > > On Thu, Mar 27, 20

Re: Announcing Spark SQL

nope (what I said :-P) On Thu, Mar 27, 2014 at 11:05 AM, Pascal Voitot Dev < pascal.voitot@gmail.com> wrote: > > > > On Thu, Mar 27, 2014 at 10:22 AM, andy petrella > wrote: > >> I just mean queries sent at runtime ^^, like for any RDBMS. >> In our project we have such requirement to have a

Re: Announcing Spark SQL

On Thu, Mar 27, 2014 at 10:22 AM, andy petrella wrote: > I just mean queries sent at runtime ^^, like for any RDBMS. > In our project we have such requirement to have a layer to play with the > data (custom and low level service layer of a lambda arch), and something > like this is interesting. >

Re: Announcing Spark SQL

I just mean queries sent at runtime ^^, like for any RDBMS. In our project we have such requirement to have a layer to play with the data (custom and low level service layer of a lambda arch), and something like this is interesting. On Thu, Mar 27, 2014 at 10:15 AM, Pascal Voitot Dev < pascal.voi

Re: Announcing Spark SQL

Le 27 mars 2014 09:47, "andy petrella" a écrit : > > I hijack the thread, but my2c is that this feature is also important to enable ad-hoc queries which is done at runtime. It doesn't remove interests for such macro for precompiled jobs of course, but it may not be the first use case envisioned wi

Re: Announcing Spark SQL

I hijack the thread, but my2c is that this feature is also important to enable ad-hoc queries which is done at runtime. It doesn't remove interests for such macro for precompiled jobs of course, but it may not be the first use case envisioned with this Spark SQL. Again, only my0.2c (ok I divided b

Re: Announcing Spark SQL

Hi, Quite interesting! Suggestion: why not go even fancier & parse SQL queries at compile-time with a macro ? ;) Pascal On Wed, Mar 26, 2014 at 10:58 PM, Michael Armbrust wrote: > Hey Everyone, > > This already went out to the dev list, but I wanted to put a pointer here > as well to a new fe

Re: Announcing Spark SQL

Great work guys! Have been looking forward to this . . . In the blog it mentions support for reading from Hbase/Avro... What will be the recommended approach for this? Will it be writing custom wrappers for SQLContext like in HiveContext or using Hive's "EXTERNAL TABLE" support? I ask this becaus

Re: Announcing Spark SQL

> > Any plans to make the SQL typesafe using something like Slick ( > http://slick.typesafe.com/) > I would really like to do something like that, and maybe we will in a couple of months. However, in the near term, I think the top priorities are going to be performance and stability. Michael

Re: Announcing Spark SQL

Very nice. Any plans to make the SQL typesafe using something like Slick ( http://slick.typesafe.com/) Thanks ! On Wed, Mar 26, 2014 at 5:58 PM, Michael Armbrust wrote: > Hey Everyone, > > This already went out to the dev list, but I wanted to put a pointer here > as well to a new feature we a

Re: Announcing Spark SQL

+1 Michael, Reynold et al. This is key to some of the things we're doing. -- Christopher T. Nguyen Co-founder & CEO, Adatao linkedin.com/in/ctnguyen On Wed, Mar 26, 2014 at 2:58 PM, Michael Armbrust wrote: > Hey Everyone, > > This already went out to the dev list, but I wan

Re: Announcing Spark SQL

Congrats Michael & co for putting this together — this is probably the neatest piece of technology added to Spark in the past few months, and it will greatly change what users can do as more data sources are added. Matei On Mar 26, 2014, at 3:22 PM, Ognen Duzlevski wrote: > Wow! > Ognen >

Re: Announcing Spark SQL

Wow! Ognen On 3/26/14, 4:58 PM, Michael Armbrust wrote: Hey Everyone, This already went out to the dev list, but I wanted to put a pointer here as well to a new feature we are pretty excited about for Spark 1.0. http://databricks.com/blog/2014/03/26/Spark-SQL-manipulating-structured-data-usi

Re: Announcing Spark SQL

(Long since taken by a W3C RDF project I'm afraid... http://www.w3.org/TR/rdf-sparql-query/ ) On Wed, Mar 26, 2014 at 10:12 PM, Bingham, Skyler wrote: > Fantastic! Although, I think they missed an obvious name choice: SparkQL > (pronounced sparkle) :)

Re: Announcing Spark SQL

Well done guys!! Thanks! 2014-03-26 19:10 GMT-03:00 Nicholas Chammas : > This is so, so COOL. YES. I'm excited about using this once I'm a bit more > comfortable with Spark. > > Nice work, people! > > > On Wed, Mar 26, 2014 at 5:58 PM, Michael Armbrust > wrote: > >> Hey Everyone, >> >> This al

RE: Announcing Spark SQL

Fantastic! Although, I think they missed an obvious name choice: SparkQL (pronounced sparkle) :) Skyler From: Michael Armbrust [mailto:mich...@databricks.com] Sent: Wednesday, March 26, 2014 3:58 PM To: user@spark.apache.org Subject: Announcing Spark SQL Hey Everyone, This already went out to

Re: Announcing Spark SQL

This is so, so COOL. YES. I'm excited about using this once I'm a bit more comfortable with Spark. Nice work, people! On Wed, Mar 26, 2014 at 5:58 PM, Michael Armbrust wrote: > Hey Everyone, > > This already went out to the dev list, but I wanted to put a pointer here > as well to a new feature

Re: Announcing Spark SQL

Thanks for sending this out! Andy Robb Senior Product Manager Phone 1.650.265.7612 ar...@walmartlabs.com On Mar 26, 2014, at 2:58 PM, Michael Armbrust wrote: > Hey Everyone, > > This already went out to the dev list, but I wanted to put a pointer here as > well to a new feature we are pretty