spark git commit: [SPARK-15660][CORE] Update RDD `variance/stdev` description and add popVariance/popStdev

2016-06-23 Thread srowen
Repository: spark Updated Branches: refs/heads/master 4374a46bf -> 5eef1e6c6 [SPARK-15660][CORE] Update RDD `variance/stdev` description and add popVariance/popStdev ## What changes were proposed in this pull request? In Spark-11490, `variance/stdev` are redefined as the **sample**

spark git commit: [SPARK-16154][MLLIB] Update spark.ml and spark.mllib package docs

2016-06-23 Thread meng
Repository: spark Updated Branches: refs/heads/branch-2.0 4ad731ed6 -> 567093596 [SPARK-16154][MLLIB] Update spark.ml and spark.mllib package docs ## What changes were proposed in this pull request? Since we decided to switch spark.mllib package into maintenance mode in 2.0, it would be

spark git commit: [SPARK-16154][MLLIB] Update spark.ml and spark.mllib package docs

2016-06-23 Thread meng
Repository: spark Updated Branches: refs/heads/master 5bf2889bf -> 65d1f0f71 [SPARK-16154][MLLIB] Update spark.ml and spark.mllib package docs ## What changes were proposed in this pull request? Since we decided to switch spark.mllib package into maintenance mode in 2.0, it would be nice to

spark git commit: [SPARK-16138] Try to cancel executor requests only if we have at least 1

2016-06-23 Thread tgraves
Repository: spark Updated Branches: refs/heads/master 5eef1e6c6 -> 5bf2889bf [SPARK-16138] Try to cancel executor requests only if we have at least 1 ## What changes were proposed in this pull request? Adding additional check to if statement ## How was this patch tested? I built and deployed

spark git commit: [SPARK-16116][SQL] ConsoleSink should not require checkpointLocation

2016-06-23 Thread zsxwing
Repository: spark Updated Branches: refs/heads/branch-2.0 9f18c8f38 -> 63fd3301c [SPARK-16116][SQL] ConsoleSink should not require checkpointLocation ## What changes were proposed in this pull request? When the user uses `ConsoleSink`, we should use a temp location if `checkpointLocation`

spark git commit: [SPARK-16130][ML] model loading backward compatibility for ml.classfication.LogisticRegression

2016-06-23 Thread meng
Repository: spark Updated Branches: refs/heads/master d85bb10ce -> 60398dabc [SPARK-16130][ML] model loading backward compatibility for ml.classfication.LogisticRegression ## What changes were proposed in this pull request? jira: https://issues.apache.org/jira/browse/SPARK-16130 model

spark git commit: [SPARK-16163] [SQL] Cache the statistics for logical plans

2016-06-23 Thread davies
Repository: spark Updated Branches: refs/heads/branch-2.0 dff3d75db -> 214676d29 [SPARK-16163] [SQL] Cache the statistics for logical plans ## What changes were proposed in this pull request? This calculation of statistics is not trivial anymore, it could be very slow on large query (for

spark git commit: [SPARK-16163] [SQL] Cache the statistics for logical plans

2016-06-23 Thread davies
Repository: spark Updated Branches: refs/heads/master 60398dabc -> 10396d950 [SPARK-16163] [SQL] Cache the statistics for logical plans ## What changes were proposed in this pull request? This calculation of statistics is not trivial anymore, it could be very slow on large query (for

spark git commit: [SPARK-15725][YARN] Ensure ApplicationMaster sleeps for the min interval.

2016-06-23 Thread tgraves
Repository: spark Updated Branches: refs/heads/branch-2.0 214676d29 -> b8818d892 [SPARK-15725][YARN] Ensure ApplicationMaster sleeps for the min interval. ## What changes were proposed in this pull request? Update `ApplicationMaster` to sleep for at least the minimum allocation interval

spark git commit: [SPARK-13723][YARN] Change behavior of --num-executors with dynamic allocation.

2016-06-23 Thread tgraves
Repository: spark Updated Branches: refs/heads/master a410814c8 -> 738f134bf [SPARK-13723][YARN] Change behavior of --num-executors with dynamic allocation. ## What changes were proposed in this pull request? This changes the behavior of --num-executors and spark.executor.instances when

spark git commit: [SPARK-16116][SQL] ConsoleSink should not require checkpointLocation

2016-06-23 Thread zsxwing
Repository: spark Updated Branches: refs/heads/master b5a997667 -> d85bb10ce [SPARK-16116][SQL] ConsoleSink should not require checkpointLocation ## What changes were proposed in this pull request? When the user uses `ConsoleSink`, we should use a temp location if `checkpointLocation` is

spark git commit: [SPARK-16164][SQL] Update `CombineFilters` to try to construct predicates with child predicate first

2016-06-23 Thread meng
Repository: spark Updated Branches: refs/heads/master 738f134bf -> 91b1ef28d [SPARK-16164][SQL] Update `CombineFilters` to try to construct predicates with child predicate first ## What changes were proposed in this pull request? This PR changes `CombineFilters` to compose the final

spark git commit: [SPARK-16164][SQL] Update `CombineFilters` to try to construct predicates with child predicate first

2016-06-23 Thread meng
Repository: spark Updated Branches: refs/heads/branch-2.0 2ce240cfe -> 6cb24de99 [SPARK-16164][SQL] Update `CombineFilters` to try to construct predicates with child predicate first ## What changes were proposed in this pull request? This PR changes `CombineFilters` to compose the final

spark git commit: [SPARK-15443][SQL] Fix 'explain' for streaming Dataset

2016-06-23 Thread yhuai
Repository: spark Updated Branches: refs/heads/branch-2.0 6cb24de99 -> 05677bb5a [SPARK-15443][SQL] Fix 'explain' for streaming Dataset ## What changes were proposed in this pull request? - Fix the `explain` command for streaming Dataset/DataFrame. E.g., ``` == Parsed Logical Plan ==

spark git commit: [SPARK-16088][SPARKR] update setJobGroup, cancelJobGroup, clearJobGroup

2016-06-23 Thread shivaram
Repository: spark Updated Branches: refs/heads/master 65d1f0f71 -> b5a997667 [SPARK-16088][SPARKR] update setJobGroup, cancelJobGroup, clearJobGroup ## What changes were proposed in this pull request? Updated setJobGroup, cancelJobGroup, clearJobGroup to not require sc/SparkContext as

spark git commit: [SPARK-16088][SPARKR] update setJobGroup, cancelJobGroup, clearJobGroup

2016-06-23 Thread shivaram
Repository: spark Updated Branches: refs/heads/branch-2.0 567093596 -> 9f18c8f38 [SPARK-16088][SPARKR] update setJobGroup, cancelJobGroup, clearJobGroup ## What changes were proposed in this pull request? Updated setJobGroup, cancelJobGroup, clearJobGroup to not require sc/SparkContext as

spark git commit: [SPARK-16142][R] group naiveBayes method docs in a single Rd

2016-06-23 Thread meng
Repository: spark Updated Branches: refs/heads/master 14bc5a7f3 -> 4a40d43bb [SPARK-16142][R] group naiveBayes method docs in a single Rd ## What changes were proposed in this pull request? This PR groups `spark.naiveBayes`, `summary(NB)`, `predict(NB)`, and `write.ml(NB)` into a single Rd.

spark git commit: [SPARK-16142][R] group naiveBayes method docs in a single Rd

2016-06-23 Thread meng
Repository: spark Updated Branches: refs/heads/branch-2.0 ea0cf93d3 -> 557eee5b6 [SPARK-16142][R] group naiveBayes method docs in a single Rd ## What changes were proposed in this pull request? This PR groups `spark.naiveBayes`, `summary(NB)`, `predict(NB)`, and `write.ml(NB)` into a single

spark git commit: [SPARK-16133][ML] model loading backward compatibility for ml.feature

2016-06-23 Thread meng
Repository: spark Updated Branches: refs/heads/branch-2.0 557eee5b6 -> 3d8d95644 [SPARK-16133][ML] model loading backward compatibility for ml.feature ## What changes were proposed in this pull request? model loading backward compatibility for ml.feature, ## How was this patch tested?

spark git commit: [SPARK-16165][SQL] Fix the update logic for InMemoryTableScanExec.readBatches

2016-06-23 Thread lian
Repository: spark Updated Branches: refs/heads/branch-2.0 05677bb5a -> e8d1bf60a [SPARK-16165][SQL] Fix the update logic for InMemoryTableScanExec.readBatches ## What changes were proposed in this pull request? Currently, `readBatches` accumulator of `InMemoryTableScanExec` is updated only

spark git commit: [SPARK-16123] Avoid NegativeArraySizeException while reserving additional capacity in VectorizedColumnReader

2016-06-23 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/branch-2.0 e8d1bf60a -> edd5dd24a [SPARK-16123] Avoid NegativeArraySizeException while reserving additional capacity in VectorizedColumnReader ## What changes were proposed in this pull request? This patch fixes an overflow bug in vectorized

spark git commit: [SPARK-16123] Avoid NegativeArraySizeException while reserving additional capacity in VectorizedColumnReader

2016-06-23 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/master 264bc6362 -> cc71d4fa3 [SPARK-16123] Avoid NegativeArraySizeException while reserving additional capacity in VectorizedColumnReader ## What changes were proposed in this pull request? This patch fixes an overflow bug in vectorized

spark git commit: [SPARK-15443][SQL] Fix 'explain' for streaming Dataset

2016-06-23 Thread yhuai
Repository: spark Updated Branches: refs/heads/master 91b1ef28d -> 0e4bdebec [SPARK-15443][SQL] Fix 'explain' for streaming Dataset ## What changes were proposed in this pull request? - Fix the `explain` command for streaming Dataset/DataFrame. E.g., ``` == Parsed Logical Plan ==

spark git commit: [SPARK-16165][SQL] Fix the update logic for InMemoryTableScanExec.readBatches

2016-06-23 Thread lian
Repository: spark Updated Branches: refs/heads/master 0e4bdebec -> 264bc6362 [SPARK-16165][SQL] Fix the update logic for InMemoryTableScanExec.readBatches ## What changes were proposed in this pull request? Currently, `readBatches` accumulator of `InMemoryTableScanExec` is updated only when

spark git commit: [SPARK-16130][ML] model loading backward compatibility for ml.classfication.LogisticRegression

2016-06-23 Thread meng
Repository: spark Updated Branches: refs/heads/branch-2.0 63fd3301c -> dff3d75db [SPARK-16130][ML] model loading backward compatibility for ml.classfication.LogisticRegression ## What changes were proposed in this pull request? jira: https://issues.apache.org/jira/browse/SPARK-16130 model

spark git commit: [SQL][MINOR] ParserUtils.operationNotAllowed should throw exception directly

2016-06-23 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/branch-2.0 edd5dd24a -> a6edec2c5 [SQL][MINOR] ParserUtils.operationNotAllowed should throw exception directly ## What changes were proposed in this pull request? It's weird that `ParserUtils.operationNotAllowed` returns an exception and the

spark git commit: [SQL][MINOR] ParserUtils.operationNotAllowed should throw exception directly

2016-06-23 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/master cc71d4fa3 -> 6a3c6276f [SQL][MINOR] ParserUtils.operationNotAllowed should throw exception directly ## What changes were proposed in this pull request? It's weird that `ParserUtils.operationNotAllowed` returns an exception and the caller

spark git commit: [SPARK-16177][ML] model loading backward compatibility for ml.regression

2016-06-23 Thread meng
Repository: spark Updated Branches: refs/heads/branch-2.0 a6edec2c5 -> ea0cf93d3 [SPARK-16177][ML] model loading backward compatibility for ml.regression ## What changes were proposed in this pull request? jira: https://issues.apache.org/jira/browse/SPARK-16177 model loading backward

spark git commit: [SPARK-16177][ML] model loading backward compatibility for ml.regression

2016-06-23 Thread meng
Repository: spark Updated Branches: refs/heads/master 6a3c6276f -> 14bc5a7f3 [SPARK-16177][ML] model loading backward compatibility for ml.regression ## What changes were proposed in this pull request? jira: https://issues.apache.org/jira/browse/SPARK-16177 model loading backward