FYI, a new release of Spark is out that might be of interest to Mesos users.
Begin forwarded message: > From: Matei Zaharia <[email protected]> > Subject: Spark release 0.6.0 is out! > Date: October 15, 2012 6:19:33 PM PDT > To: [email protected] > > I'm happy to announce that the next major release of Spark, 0.6.0, is now > available. This is the biggest Spark release to date in terms of features > (other than perhaps the first one), as well as the biggest in terms of > contributors, with over a dozen new contributors from Berkeley and outside. > Apart from the visible features, such as a standalone deploy mode and Java > API, it includes a significant rearchitecting of Spark under the hood that > provides up to 2x faster network performance and support for even > lower-latency jobs. > > The major focus points in this release have been accessibility (making Spark > easier to deploy and use) and performance. The full release notes are posted > online at http://www.spark-project.org/release-0.6.0.html, but here are some > highlights: > > * Simpler deployment: Spark now has a pure-Java standalone deploy mode that > lets it run without an external cluster manager, as well as experimental > support for running on YARN (Hadoop NextGen). > > * Java API: exposes all of Spark's features to Java developers in a clean > manner. > > * Expanded documentation: a new documentation site, > http://spark-project.org/docs/0.6.0/, contains significantly expanded docs, > such as a quick start guide, tuning guide, configuration guide, and detailed > Scaladoc help. > > * Engine enhancements: a new, custom communication layer and storage manager > based on Java NIO provide improved performance for network-heavy operations. > > * Debugging enhancements: Spark now prints which line of your code each > operation in its logs corresponds to. > > As mentioned above, this release is also the work of an unprecedentedly large > set of developers. Here are some of the people who contributed to Spark 0.6: > > - Tathagata Das contributed the new communication layer, and parts of the > storage layer. > - Haoyuan Li contributed the new storage manager. > - Denny Britz contributed the YARN deploy mode, key aspects of the standalone > one, and several other features. > - Andy Konwinski contributed the revamped documentation site, Maven > publishing, and several API docs. > - Josh Rosen contributed the Java API, as well as several bug fixes. > - Patrick Wendell contributed the enhanced debugging feature and helped with > testing and documentation. > - Reynold Xin contributed numerous bug and performance fixes. > - Imran Rashid contributed the new Accumulable class. > - Harvey Feng contributed improvements to shuffle operations. > - Shivaram Venkataraman improved Spark's memory estimation and wrote a memory > tuning guide. > - Ravi Pandya contributed Spark run scripts for Windows. > - Mosharaf Chowdhury provided several fixes to broadcast. > - Henry Milner pointed out several bugs in sampling algorithms. > - Ray Racine provided improvements to the EC2 scripts. > - Paul Ruan and Bill Zhao helped with testing. > > We're very proud of this release, and hope that you enjoy it. You can grab > the code at http://www.spark-project.org/release-0.6.0.html. > > Matei
