spark git commit: [SPARK-12349][ML] Make spark.ml PCAModel load backwards compatible

2015-12-21 Thread srowen
Repository: spark Updated Branches: refs/heads/master ce1798b3a -> d0f695089 [SPARK-12349][ML] Make spark.ml PCAModel load backwards compatible Only load explainedVariance in PCAModel if it was written with Spark > 1.6.x jkbradley is this kind of what you had in mind? Author: Sean Owen Clos

spark git commit: [PYSPARK] Pyspark typo & Add missing abstractmethod annotation

2015-12-21 Thread davies
Repository: spark Updated Branches: refs/heads/master d0f695089 -> 1920d72a1 [PYSPARK] Pyspark typo & Add missing abstractmethod annotation No jira is created since this is a trivial change. davies Please help review it Author: Jeff Zhang Closes #10143 from zjffdu/pyspark_typo. Project:

spark git commit: [SPARK-12398] Smart truncation of DataFrame / Dataset toString

2015-12-21 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 1920d72a1 -> 474eb21a3 [SPARK-12398] Smart truncation of DataFrame / Dataset toString When a DataFrame or Dataset has a long schema, we should intelligently truncate to avoid flooding the screen with unreadable information. // Standard out

spark git commit: [SPARK-12321][SQL] JSON format for TreeNode (use reflection)

2015-12-21 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 474eb21a3 -> 7634fe951 [SPARK-12321][SQL] JSON format for TreeNode (use reflection) An alternative solution for https://github.com/apache/spark/pull/10295 , instead of implementing json format for all logical/physical plans and expression

spark git commit: [SPARK-12374][SPARK-12150][SQL] Adding logical/physical operators for Range

2015-12-21 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 7634fe951 -> 4883a5087 [SPARK-12374][SPARK-12150][SQL] Adding logical/physical operators for Range Based on the suggestions from marmbrus , added logical/physical operators for Range for improving the performance. Also added another API f

spark git commit: [SPARK-12392][CORE] Optimize a location order of broadcast blocks by considering preferred local hosts

2015-12-21 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 4883a5087 -> 935f46630 [SPARK-12392][CORE] Optimize a location order of broadcast blocks by considering preferred local hosts When multiple workers exist in a host, we can bypass unnecessary remote access for broadcasts; block managers fe

spark git commit: [SPARK-5882][GRAPHX] Add a test for GraphLoader.edgeListFile

2015-12-21 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 935f46630 -> 1eb90bc9c [SPARK-5882][GRAPHX] Add a test for GraphLoader.edgeListFile Author: Takeshi YAMAMURO Closes #4674 from maropu/AddGraphLoaderSuite. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip

spark git commit: Doc typo: ltrim = trim from left end, not right

2015-12-21 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.6 d6a519ff2 -> c754a0879 Doc typo: ltrim = trim from left end, not right Author: pshearer Closes #10414 from pshearer/patch-1. (cherry picked from commit fc6dbcc7038c2b030ef6a2dc8be5848499ccee1c) Signed-off-by: Andrew Or Project: ht

spark git commit: Doc typo: ltrim = trim from left end, not right

2015-12-21 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.5 eb54c914a -> 4d54ba896 Doc typo: ltrim = trim from left end, not right Author: pshearer Closes #10414 from pshearer/patch-1. (cherry picked from commit fc6dbcc7038c2b030ef6a2dc8be5848499ccee1c) Signed-off-by: Andrew Or Project: ht

spark git commit: Doc typo: ltrim = trim from left end, not right

2015-12-21 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 1eb90bc9c -> fc6dbcc70 Doc typo: ltrim = trim from left end, not right Author: pshearer Closes #10414 from pshearer/patch-1. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark

spark git commit: [SPARK-12339][SPARK-11206][WEBUI] Added a null check that was removed in

2015-12-21 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master fc6dbcc70 -> b0849b8ae [SPARK-12339][SPARK-11206][WEBUI] Added a null check that was removed in Updates made in SPARK-11206 missed an edge case which cause's a NullPointerException when a task is killed. In some cases when a task ends in

spark git commit: [SPARK-2331] SparkContext.emptyRDD should return RDD[T] not EmptyRDD[T]

2015-12-21 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master b0849b8ae -> a820ca19d [SPARK-2331] SparkContext.emptyRDD should return RDD[T] not EmptyRDD[T] Author: Reynold Xin Closes #10394 from rxin/SPARK-2331. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.

spark git commit: [SPARK-12466] Fix harmless NPE in tests

2015-12-21 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master a820ca19d -> d655d37dd [SPARK-12466] Fix harmless NPE in tests ``` [info] ReplayListenerSuite: [info] - Simple replay (58 milliseconds) java.lang.NullPointerException at org.apache.spark.deploy.master.Master$$anonfun$asyncRebuildSp

spark git commit: [SPARK-12466] Fix harmless NPE in tests

2015-12-21 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.6 c754a0879 -> ca3998512 [SPARK-12466] Fix harmless NPE in tests ``` [info] ReplayListenerSuite: [info] - Simple replay (58 milliseconds) java.lang.NullPointerException at org.apache.spark.deploy.master.Master$$anonfun$asyncRebui

spark git commit: [SPARK-12388] change default compression to lz4

2015-12-21 Thread davies
Repository: spark Updated Branches: refs/heads/master d655d37dd -> 29cecd4a4 [SPARK-12388] change default compression to lz4 According the benchmark [1], LZ4-java could be 80% (or 30%) faster than Snappy. After changing the compressor to LZ4, I saw 20% improvement on end-to-end time for a TP

[spark] Git Push Summary

2015-12-21 Thread pwendell
Repository: spark Updated Tags: refs/tags/v1.6.0-rc4 [created] 4062cda30 - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org

[1/2] spark git commit: Preparing Spark release v1.6.0-rc4

2015-12-21 Thread pwendell
Repository: spark Updated Branches: refs/heads/branch-1.6 ca3998512 -> 5b19e7cfd Preparing Spark release v1.6.0-rc4 Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/4062cda3 Tree: http://git-wip-us.apache.org/repos/asf/spar

[2/2] spark git commit: Preparing development version 1.6.0-SNAPSHOT

2015-12-21 Thread pwendell
Preparing development version 1.6.0-SNAPSHOT Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/5b19e7cf Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/5b19e7cf Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/5b1

spark git commit: [SPARK-11807] Remove support for Hadoop < 2.2

2015-12-21 Thread rxin
Repository: spark Updated Branches: refs/heads/master 29cecd4a4 -> 0a38637d0 [SPARK-11807] Remove support for Hadoop < 2.2 i.e. Hadoop 1 and Hadoop 2.0 Author: Reynold Xin Closes #10404 from rxin/SPARK-11807. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wi

spark git commit: [MINOR] Fix typos in JavaStreamingContext

2015-12-21 Thread rxin
Repository: spark Updated Branches: refs/heads/master 0a38637d0 -> 93da8565f [MINOR] Fix typos in JavaStreamingContext Author: Shixiong Zhu Closes #10424 from zsxwing/typo. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commi

spark git commit: [MINOR] Fix typos in JavaStreamingContext

2015-12-21 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-1.6 5b19e7cfd -> 309ef355f [MINOR] Fix typos in JavaStreamingContext Author: Shixiong Zhu Closes #10424 from zsxwing/typo. (cherry picked from commit 93da8565fea42d8ac978df411daced4a9ea3a9c8) Signed-off-by: Reynold Xin Project: http:/

spark git commit: [SPARK-11823][SQL] Fix flaky JDBC cancellation test in HiveThriftBinaryServerSuite

2015-12-21 Thread joshrosen
Repository: spark Updated Branches: refs/heads/master 93da8565f -> 2235cd444 [SPARK-11823][SQL] Fix flaky JDBC cancellation test in HiveThriftBinaryServerSuite This patch fixes a flaky "test jdbc cancel" test in HiveThriftBinaryServerSuite. This test is prone to a race-condition which cause

spark git commit: [SPARK-11823][SQL] Fix flaky JDBC cancellation test in HiveThriftBinaryServerSuite

2015-12-21 Thread joshrosen
Repository: spark Updated Branches: refs/heads/branch-1.6 309ef355f -> 0f905d7df [SPARK-11823][SQL] Fix flaky JDBC cancellation test in HiveThriftBinaryServerSuite This patch fixes a flaky "test jdbc cancel" test in HiveThriftBinaryServerSuite. This test is prone to a race-condition which c

spark git commit: [SPARK-12296][PYSPARK][MLLIB] Feature parity for pyspark mllib standard scaler model

2015-12-21 Thread mlnick
Repository: spark Updated Branches: refs/heads/master 2235cd444 -> 969d5665b [SPARK-12296][PYSPARK][MLLIB] Feature parity for pyspark mllib standard scaler model Some methods are missing, such as ways to access the std, mean, etc. This PR is for feature parity for pyspark.mllib.feature.Stand