Tests and Test Infrastructure

2014-09-14 Thread Patrick Wendell
Hey All, Wanted to send a quick update about test infrastructure. With the number of contributors we have and the rate of development, maintaining a well-oiled test infra is really important. Every time a flaky test fails a legitimate pull request, it wastes developer time and effort. 1. Master

Re: Tests and Test Infrastructure

2014-09-14 Thread Josh Rosen
Also, huge thanks to Cheng Lian, who tracked down and fixed the final issue that was causing the Maven master build’s Spark SQL tests to fail! On September 13, 2014 at 11:08:00 PM, Patrick Wendell (pwend...@gmail.com) wrote: Hey All, Wanted to send a quick update about test infrastructure.

Re: Tests and Test Infrastructure

2014-09-14 Thread Nicholas Chammas
I fully support this. A smoothly running test infrastructure helps everybody’s work just flow better. The Jenkins Pull Request Builder is mostly functioning again. However, we are working on a simpler technical pipeline for testing patches, as this plug-in has been a constant source of downtime

Support for Hive buckets

2014-09-14 Thread Cody Koeninger
I noticed that the release notes for 1.1.0 said that spark doesn't support Hive buckets yet. I didn't notice any jira issues related to adding support. Broadly speaking, what would be involved in supporting buckets, especially the bucketmapjoin and sortedmerge optimizations?

Source code for mining big data with Spark

2014-09-14 Thread David Tung
Hi all, I watched am impressed spark demo video by Reynold Xin and Aaron Davidson in youtube ( https://www.youtube.com/watch?v=FjhRkfAuU7I ). Can someone let me know where can I find the source codes for the demo? I can¹t see the source codes from video clearly. Thanks in advance