Hi Mesos users, On January 31st, we’re hosting a first in-person Spark User Meetup, at Klout in San Francisco. Spark is a cluster programming framework built on Mesos that provides fast in-memory computing for iterative and interactive data mining, as well as an easy-to-program interface in the Scala programming language. (Check out www.spark-project.org for details.) This periodic meetup will be a chance to learn Spark from the developers, hear about other peoples’ experiences with Spark, network, and learn about future development plans.
At the first meeting, we’re planning to have a Spark tutorial by Matei Zaharia followed by a presentation from Karthik Thiyagarajan of Quantifind on their experience using Spark to replace Pig for predictive analytics (details below). If you want to follow along, we strongly recommend you sign up for Amazon EC2 and make sure you can launch instances. This way you’ll be able to bring up your own Mesos/Spark cluster on EC2 and play with a Wikipedia dataset in real time! Time: Tuesday January 31, 2012. Pizza and beer start at 6:30, talks start at 7 PM. Location: Klout, 77 Stillman St, San Francisco, CA 94107. RSVP: Please *REGISTER* in advance at http://www.meetup.com/spark-users/ if you'd like to attend. About Quantifind: Quantifind is a small start up into predictive analytics. We are building a platform that automatically identifies relevant signals in both structured and unstructured content, contextualizes them based on what matters most to the customer, and derives insights about past and future events in a way that is directly relevant and actionable. The talk highlights our experience using Spark in various ways ranging from a distributed batch processing framework powering our analytics pipeline to an interactive computing infrastructure serving some of our internal exploratory tools. Over the course of moving our analytics pipeline from Pig to Spark, we realized that Spark's inherent characteristics of low latency and interactivity can be leveraged as an agile way to create new computing services. So we experimented with creating web services on top of Spark which answered queries in real time by performing operations on a cached RDD. Today, Spark acts as a sharded in memory infrastructure for many of the services we use internally helping us explore our data and prototype algorithms in an agile manner. Future Locations: The meetup will rotate among locations in San Francisco, Silicon Valley and Berkeley.
