Attempting to attach the release notes again (I think it may have been blocked previously due to not having an extension).
On Sun, Jan 19, 2014 at 8:05 PM, Patrick Wendell <[email protected]> wrote: > I'll add my +1 as well > > On Sun, Jan 19, 2014 at 7:33 PM, Matei Zaharia <[email protected]> > wrote: >> +1 >> >> Re-tested on Mac. >> >> Matei >> >> On Jan 19, 2014, at 7:09 PM, Tathagata Das <[email protected]> >> wrote: >> >>> Starting off. >>> +1 >>> >>> >>> On Sun, Jan 19, 2014 at 2:15 PM, Patrick Wendell <[email protected]> wrote: >>> >>>> Please vote on releasing the following candidate as Apache Spark >>>> (incubating) version 0.9.0. >>>> >>>> A draft of the release notes along with the changes file is attached >>>> to this e-mail. >>>> >>>> The tag to be voted on is v0.9.0-incubating (commit a7760eff): >>>> >>>> https://git-wip-us.apache.org/repos/asf?p=incubator-spark.git;a=commit;h=a7760eff4ea6a474cab68896a88550f63bae8b0d >>>> >>>> The release files, including signatures, digests, etc can be found at: >>>> http://people.apache.org/~pwendell/spark-0.9.0-incubating-rc3/ >>>> >>>> Release artifacts are signed with the following key: >>>> https://people.apache.org/keys/committer/pwendell.asc >>>> >>>> The staging repository for this release can be found at: >>>> https://repository.apache.org/content/repositories/orgapachespark-1004/ >>>> >>>> The documentation corresponding to this release can be found at: >>>> http://people.apache.org/~pwendell/spark-0.9.0-incubating-rc3-docs/ >>>> >>>> Please vote on releasing this package as Apache Spark 0.9.0-incubating! >>>> >>>> The vote is open until Wednesday, January 22, at 22:15 UTC and passes >>>> if a majority of at least 3 +1 PPMC votes are cast. >>>> >>>> [ ] +1 Release this package as Apache Spark 0.9.0-incubating >>>> [ ] -1 Do not release this package because ... >>>> >>>> To learn more about Apache Spark, please see >>>> http://spark.incubator.apache.org/ >>>> >>
Spark 0.9.0 is a major release that adds significant new features. It updates Spark to Scala 2.10, simplifies high availability, and updates numerous components of the project. This release includes a first version of GraphX, a powerful new framework for graph processing that comes with a library of standard algorithms. In addition, Spark Streaming is now out of alpha, and includes significant optimizations and simplified high availability deployment. ### Scala 2.10 Support Spark now runs on Scala 2.10, letting users benefit from the language and library improvements in this version. ### Configuration System The new [SparkConf] class is now the preferred way to configure advanced settings on your SparkContext, though the previous Java system property still works. SparkConf is especially useful in tests to make sure properties donât stay set across tests. ### Spark Streaming Improvements Spark Streaming is no longer alpha, and comes with simplified high availability and several optimizations. * When running on a Spark standalone cluster with the [standalone cluster high availability mode], you can submit a Spark Streaming driver application to the cluster and have it automatically recovered if either the driver or the cluster master crashes. * Windowed operators have been sped up by 30-50%. * Spark Streamingâs input source plugins (e.g. for Twitter, Kafka and Flume) are now separate projects, making it easier to pull in only the dependencies you need. * A new StreamingListener interface has been added for monitoring statistics about the streaming computation. * A few aspects of the API have been improved: * `DStream` and `PairDStream` classes have been moved from `org.apache.spark.streaming` to `org.apache.spark.streaming.dstream` to keep it consistent with `org.apache.spark.rdd.RDD`. * `DStream.foreach` -> `DStream.foreachRDD` to make it explicit that it works for every RDD, not every element * `StreamingContext.awaitTermination()` allows you wait for context shutdown and catch any exception that occurs in the streaming computation. *`StreamingContext.stop()` now allows stopping of StreamingContext without stopping the underlying SparkContext. ### GraphX Alpha GraphX is a new API for graph processing that uses recent advances in graph-parallel computation. It lets you build a graph within a Spark program using the standard Spark operators, then process it with new graph operators that are optimized for distributed computation. It includes basic transformations, a Pregel API for iterative computation, and a standard library of graph loaders and analytics algorithms. By offering these features within the Spark engine, GraphX can significantly speed up processing tasks compared to workflows that use different engines. GraphX features in this release include: * Building graphs from arbitrary Spark RDDs * Basic operations to transform graphs or extract subgraphs * An optimized Pregel API that takes advantage of graph partitioning and indexing * Standard algorithms including PageRank, connected components, strongly connected components, SVD++, and triangle counting * Interactive use from the Spark shell GraphX is still marked as alpha in this first release, but we recommend for new users to use it instead of the more limited Bagel API. ### MLlib Improvements * Sparkâs machine learning library (MLlib) is now available in Python, where it operates on NumPy data (currently requires Python 2.7 and NumPy 1.7) * A new algorithm has been added for Naive Bayes classification * Alternating Least Squares can now be used to predict ratings for multiple items in parallel * MLlibâs documentation was expanded to include more examples in Scala, Java and Python ### Python Changes * Python users can now use MLlib (requires Python 2.7 and NumPy 1.7) * PySpark now shows the call sites of running jobs in the Spark application UI (http://<driver>:4040), making it easy to see which part of your code is running * IPython operation has been improved ### Packaging * Sparkâs scripts have been organized into âbinâ and âsbinâ directories to make it easier to separate admin scripts from user ones and install Spark on standard Linux paths. * Log configuration has been improved so that Spark finds a default log4j.properties file if you donât specify one. ### Core Engine * Sparkâs standalone mode now supports submitting a driver program to run on the cluster instead of on the external machine submitting it. You can access this functionality through the [org.apache.spark.deploy.Client] class. * Large reduce operations now automatically spill data to disk if it does not fit in memory. * Users of standalone mode can now limit how many cores an application will use by default if the application writer didnât configure its size. Previously, such applications took all available cores on the cluster. * `spark-shell` now supports the `-i` option to run a script on startup. * New `histogram` and `countDistinctApprox` operators have been added for working with numerical data. * Methods for reading Hadoop `Writable` objects, such as `SparkContext.sequenceFile` and `hadoopRDD`, now clone the records read by default instead of reusing the same Writable instance, making the resulting RDDs easier to cache; if you only want to use each object once, you can pass an extra flag to disable this cloning * YARN mode now supports distributing extra files with the application, and several bugs have been fixed. ### Compatibility This release is compatible with the previous APIs in stable components, but several language versions and script locations have changed. * Scala programs now need to use Scala 2.10 instead of 2.9. * Scripts such as `spark-shell` and `pyspark` have been moved into the `bin` folder, while administrative scripts to start and stop standalone clusters have been moved into `sbin`. * Spark Streamingâs API has been changed to move external input sources into separate modules, `DStream` and `PairDStream` has been moved to package `org.apache.spark.streaming.dstream` and `DStream.foreach` has been renamed to `foreachRDD`. We expect the current API to be stable now that Spark Streaming is out of alpha. * While the old method of configuring Spark through Java system properties still works, we recommend that users update to the new SparkConf, which is easier to inspect and use. We expect all of the current APIs and script locations in Spark 0.9 to remain stable when we release Spark 1.0. We wanted to make these updates early to give users a chance to switch to the new API.
