Re: SparkSQL production readiness
OK, good to know data frames are still experimental. Thanks Michael. On Mon, Mar 2, 2015 at 12:37 PM, Michael Armbrust wrote: > We have been using Spark SQL in production for our customers at Databricks > for almost a year now. We also know of some very large production > deployments elsewhere. It is still a young project, but I wouldn't call it > alpha. > > The primary changes to the API are the addition of the DataFrame > interface, which is an expansion of the DSL that was already there. All of > the SQL / HiveQL stuff remains unchanged, as well as the internal execution > engine. DataFrames are still marked experimental, since as you said, we > should let people use them before cementing them. >
Re: SparkSQL production readiness
We have been using Spark SQL in production for our customers at Databricks for almost a year now. We also know of some very large production deployments elsewhere. It is still a young project, but I wouldn't call it alpha. The primary changes to the API are the addition of the DataFrame interface, which is an expansion of the DSL that was already there. All of the SQL / HiveQL stuff remains unchanged, as well as the internal execution engine. DataFrames are still marked experimental, since as you said, we should let people use them before cementing them. On Mon, Mar 2, 2015 at 7:29 AM, Daniel Siegmann wrote: > I thought removing the alpha tag just meant the API was stable? Speaking > of which, aren't there major changes to the API coming in 1.3? Why are you > marking the API as stable before these changes have been widely used? > > > On Sat, Feb 28, 2015 at 5:17 PM, Michael Armbrust > wrote: > >> We are planning to remove the alpha tag in 1.3.0. >> >> On Sat, Feb 28, 2015 at 12:30 AM, Wang, Daoyuan >> wrote: >> >>> Hopefully the alpha tag will be remove in 1.4.0, if the community can >>> review code a little bit faster :P >>> >>> >>> >>> Thanks, >>> >>> Daoyuan >>> >>> >>> >>> *From:* Ashish Mukherjee [mailto:ashish.mukher...@gmail.com] >>> *Sent:* Saturday, February 28, 2015 4:28 PM >>> *To:* user@spark.apache.org >>> *Subject:* SparkSQL production readiness >>> >>> >>> >>> Hi, >>> >>> >>> >>> I am exploring SparkSQL for my purposes of performing large relational >>> operations across a cluster. However, it seems to be in alpha right now. Is >>> there any indication when it would be considered production-level? I don't >>> see any info on the site. >>> >>> >>> >>> Regards, >>> >>> Ashish >>> >> >> >
Re: SparkSQL production readiness
I thought removing the alpha tag just meant the API was stable? Speaking of which, aren't there major changes to the API coming in 1.3? Why are you marking the API as stable before these changes have been widely used? On Sat, Feb 28, 2015 at 5:17 PM, Michael Armbrust wrote: > We are planning to remove the alpha tag in 1.3.0. > > On Sat, Feb 28, 2015 at 12:30 AM, Wang, Daoyuan > wrote: > >> Hopefully the alpha tag will be remove in 1.4.0, if the community can >> review code a little bit faster :P >> >> >> >> Thanks, >> >> Daoyuan >> >> >> >> *From:* Ashish Mukherjee [mailto:ashish.mukher...@gmail.com] >> *Sent:* Saturday, February 28, 2015 4:28 PM >> *To:* user@spark.apache.org >> *Subject:* SparkSQL production readiness >> >> >> >> Hi, >> >> >> >> I am exploring SparkSQL for my purposes of performing large relational >> operations across a cluster. However, it seems to be in alpha right now. Is >> there any indication when it would be considered production-level? I don't >> see any info on the site. >> >> >> >> Regards, >> >> Ashish >> > >
Re: SparkSQL production readiness
We are planning to remove the alpha tag in 1.3.0. On Sat, Feb 28, 2015 at 12:30 AM, Wang, Daoyuan wrote: > Hopefully the alpha tag will be remove in 1.4.0, if the community can > review code a little bit faster :P > > > > Thanks, > > Daoyuan > > > > *From:* Ashish Mukherjee [mailto:ashish.mukher...@gmail.com] > *Sent:* Saturday, February 28, 2015 4:28 PM > *To:* user@spark.apache.org > *Subject:* SparkSQL production readiness > > > > Hi, > > > > I am exploring SparkSQL for my purposes of performing large relational > operations across a cluster. However, it seems to be in alpha right now. Is > there any indication when it would be considered production-level? I don't > see any info on the site. > > > > Regards, > > Ashish >
RE: SparkSQL production readiness
Hopefully the alpha tag will be remove in 1.4.0, if the community can review code a little bit faster :P Thanks, Daoyuan From: Ashish Mukherjee [mailto:ashish.mukher...@gmail.com] Sent: Saturday, February 28, 2015 4:28 PM To: user@spark.apache.org Subject: SparkSQL production readiness Hi, I am exploring SparkSQL for my purposes of performing large relational operations across a cluster. However, it seems to be in alpha right now. Is there any indication when it would be considered production-level? I don't see any info on the site. Regards, Ashish
