RE: Is Spark 1.6 released?
Where can I read more about the dataset api on a user layer? I am failing to get an API doc or understand when to use DataFrame or DataSet, advantages, etc. Thanks, Saif -Original Message- From: Jean-Baptiste Onofré [mailto:j...@nanthrax.net] Sent: Monday, January 04, 2016 2:01 PM To: user@spark.apache.org Subject: Re: Is Spark 1.6 released? It's now OK: Michael published and announced the release. Sorry for the delay. Regards JB On 01/04/2016 10:06 AM, Jung wrote: > Hi > There were Spark 1.6 jars in maven central and github. > I found it 5 days ago. But it doesn't appear on Spark website now. > May I regard Spark 1.6 zip file in github as a stable release? > > Thanks > Jung > -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Is Spark 1.6 released?
Please refer to the following: https://spark.apache.org/docs/latest/sql-programming-guide.html#datasets https://spark.apache.org/docs/latest/sql-programming-guide.html#creating-datasets https://spark.apache.org/docs/latest/sql-programming-guide.html#json-datasets Cheers On Mon, Jan 4, 2016 at 11:59 AM, <saif.a.ell...@wellsfargo.com> wrote: > Where can I read more about the dataset api on a user layer? I am failing > to get an API doc or understand when to use DataFrame or DataSet, > advantages, etc. > > Thanks, > Saif > > -Original Message- > From: Jean-Baptiste Onofré [mailto:j...@nanthrax.net] > Sent: Monday, January 04, 2016 2:01 PM > To: user@spark.apache.org > Subject: Re: Is Spark 1.6 released? > > It's now OK: Michael published and announced the release. > > Sorry for the delay. > > Regards > JB > > On 01/04/2016 10:06 AM, Jung wrote: > > Hi > > There were Spark 1.6 jars in maven central and github. > > I found it 5 days ago. But it doesn't appear on Spark website now. > > May I regard Spark 1.6 zip file in github as a stable release? > > > > Thanks > > Jung > > > > -- > Jean-Baptiste Onofré > jbono...@apache.org > http://blog.nanthrax.net > Talend - http://www.talend.com > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional > commands, e-mail: user-h...@spark.apache.org > > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
Re: Is Spark 1.6 released?
I also wrote about it here: https://databricks.com/blog/2016/01/04/introducing-spark-datasets.html And put together a bunch of examples here: https://docs.cloud.databricks.com/docs/spark/1.6/index.html On Mon, Jan 4, 2016 at 12:02 PM, Annabel Melongo < melongo_anna...@yahoo.com.invalid> wrote: > [1] http://spark.apache.org/releases/spark-release-1-6-0.html > [2] http://spark.apache.org/downloads.html > > > > On Monday, January 4, 2016 2:59 PM, "saif.a.ell...@wellsfargo.com" < > saif.a.ell...@wellsfargo.com> wrote: > > > Where can I read more about the dataset api on a user layer? I am failing > to get an API doc or understand when to use DataFrame or DataSet, > advantages, etc. > > Thanks, > Saif > > -Original Message- > From: Jean-Baptiste Onofré [mailto:j...@nanthrax.net] > Sent: Monday, January 04, 2016 2:01 PM > To: user@spark.apache.org > Subject: Re: Is Spark 1.6 released? > > It's now OK: Michael published and announced the release. > > Sorry for the delay. > > Regards > JB > > On 01/04/2016 10:06 AM, Jung wrote: > > Hi > > There were Spark 1.6 jars in maven central and github. > > I found it 5 days ago. But it doesn't appear on Spark website now. > > May I regard Spark 1.6 zip file in github as a stable release? > > > > Thanks > > Jung > > > > -- > Jean-Baptiste Onofré > jbono...@apache.org > http://blog.nanthrax.net > Talend - http://www.talend.com > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional > commands, e-mail: user-h...@spark.apache.org > > > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > > > >
Re: Is Spark 1.6 released?
[1] http://spark.apache.org/releases/spark-release-1-6-0.html[2] http://spark.apache.org/downloads.html On Monday, January 4, 2016 2:59 PM, "saif.a.ell...@wellsfargo.com" <saif.a.ell...@wellsfargo.com> wrote: Where can I read more about the dataset api on a user layer? I am failing to get an API doc or understand when to use DataFrame or DataSet, advantages, etc. Thanks, Saif -Original Message- From: Jean-Baptiste Onofré [mailto:j...@nanthrax.net] Sent: Monday, January 04, 2016 2:01 PM To: user@spark.apache.org Subject: Re: Is Spark 1.6 released? It's now OK: Michael published and announced the release. Sorry for the delay. Regards JB On 01/04/2016 10:06 AM, Jung wrote: > Hi > There were Spark 1.6 jars in maven central and github. > I found it 5 days ago. But it doesn't appear on Spark website now. > May I regard Spark 1.6 zip file in github as a stable release? > > Thanks > Jung > -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Is Spark 1.6 released?
Hi Jung, yes Spark 1.6.0 has been released December, 28th. The artifacts are on Maven Central: http://repo1.maven.org/maven2/org/apache/spark/ However, the distribution is not available on dist.apache.org: https://dist.apache.org/repos/dist/release/spark/ Let me check with the team to upload the distribution to dist.apache.org. Regards JB On 01/04/2016 10:06 AM, Jung wrote: Hi There were Spark 1.6 jars in maven central and github. I found it 5 days ago. But it doesn't appear on Spark website now. May I regard Spark 1.6 zip file in github as a stable release? Thanks Jung -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Is Spark 1.6 released?
> > bq. In many cases, the current implementation of the Dataset API does not > yet leverage the additional information it has and can be slower than RDDs. > > Are the characteristics of cases above known so that users can decide which > API to use ? > Lots of back to back operations aren't great yet because we serialize deseriaize unnecessarily. For example: https://github.com/databricks/spark-sql-perf/blob/master/src/main/scala/com/databricks/spark/sql/perf/DatasetPerformance.scala#L37 > > For custom encoders, I did a quick search but didn't find the JIRA number. > Can you share the JIRA number ? > This is probably the closest thing: https://issues.apache.org/jira/browse/SPARK-7768
Re: Is Spark 1.6 released?
It's now OK: Michael published and announced the release. Sorry for the delay. Regards JB On 01/04/2016 10:06 AM, Jung wrote: Hi There were Spark 1.6 jars in maven central and github. I found it 5 days ago. But it doesn't appear on Spark website now. May I regard Spark 1.6 zip file in github as a stable release? Thanks Jung -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
