spark git commit: Fix missing close-parens for In filter's toString

2017-01-12 Thread rxin
Repository: spark Updated Branches: refs/heads/master 6b34e745b -> b040cef2e Fix missing close-parens for In filter's toString Otherwise the open parentheses isn't closed in query plan descriptions of batch scans. PushedFilters: [In(COL_A, [1,2,4,6,10,16,219,815], IsNotNull(COL_B), ...

spark git commit: Fix missing close-parens for In filter's toString

2017-01-12 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 55d2a1178 -> be527ddc0 Fix missing close-parens for In filter's toString Otherwise the open parentheses isn't closed in query plan descriptions of batch scans. PushedFilters: [In(COL_A, [1,2,4,6,10,16,219,815], IsNotNull(COL_B),

spark git commit: Fix missing close-parens for In filter's toString

2017-01-12 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.1 23944d0d6 -> 0668e061b Fix missing close-parens for In filter's toString Otherwise the open parentheses isn't closed in query plan descriptions of batch scans. PushedFilters: [In(COL_A, [1,2,4,6,10,16,219,815], IsNotNull(COL_B),

spark git commit: [SPARK-19178][SQL] convert string of large numbers to int should return null

2017-01-12 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 7f24a0b6c -> 6b34e745b [SPARK-19178][SQL] convert string of large numbers to int should return null ## What changes were proposed in this pull request? When we convert a string to integral, we will convert that string to `decimal(20, 0)`

spark git commit: [SPARK-19142][SPARKR] spark.kmeans should take seed, initSteps, and tol as parameters

2017-01-12 Thread yliang
Repository: spark Updated Branches: refs/heads/master 3356b8b6a -> 7f24a0b6c [SPARK-19142][SPARKR] spark.kmeans should take seed, initSteps, and tol as parameters ## What changes were proposed in this pull request? spark.kmeans doesn't have interface to set initSteps, seed and tol. As Spark

spark git commit: [SPARK-19092][SQL] Save() API of DataFrameWriter should not scan all the saved files

2017-01-12 Thread wenchen
Repository: spark Updated Branches: refs/heads/master c983267b0 -> 3356b8b6a [SPARK-19092][SQL] Save() API of DataFrameWriter should not scan all the saved files ### What changes were proposed in this pull request? `DataFrameWriter`'s [save()

spark git commit: [SPARK-19110][MLLIB][FOLLOWUP] Add a unit test for testing logPrior and logLikelihood of DistributedLDAModel in MLLIB

2017-01-12 Thread jkbradley
Repository: spark Updated Branches: refs/heads/master 5585ed93b -> c983267b0 [SPARK-19110][MLLIB][FOLLOWUP] Add a unit test for testing logPrior and logLikelihood of DistributedLDAModel in MLLIB ## What changes were proposed in this pull request? #16491 added the fix to mllib and a unit test

spark git commit: [SPARK-17237][SQL] Remove backticks in a pivot result schema

2017-01-12 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 2bc4d4e28 -> 5585ed93b [SPARK-17237][SQL] Remove backticks in a pivot result schema ## What changes were proposed in this pull request? Pivoting adds backticks (e.g. 3_count(\`c\`)) in column names and, in some cases, thes causes analysis

spark git commit: [SPARK-17237][SQL] Remove backticks in a pivot result schema

2017-01-12 Thread lixiao
Repository: spark Updated Branches: refs/heads/branch-2.1 042e32d18 -> 23944d0d6 [SPARK-17237][SQL] Remove backticks in a pivot result schema ## What changes were proposed in this pull request? Pivoting adds backticks (e.g. 3_count(\`c\`)) in column names and, in some cases, thes causes

spark git commit: [SPARK-12757][CORE] lower "block locks were not released" log to info level

2017-01-12 Thread felixcheung
Repository: spark Updated Branches: refs/heads/master c6c37b8af -> 2bc4d4e28 [SPARK-12757][CORE] lower "block locks were not released" log to info level ## What changes were proposed in this pull request? lower "block locks were not released" log to info level, as it is generating a lot of

spark git commit: [SPARK-19055][SQL][PYSPARK] Fix SparkSession initialization when SparkContext is stopped

2017-01-12 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.0 3566e40a4 -> 55d2a1178 [SPARK-19055][SQL][PYSPARK] Fix SparkSession initialization when SparkContext is stopped ## What changes were proposed in this pull request? In SparkSession initialization, we store created the instance of

spark git commit: [SPARK-19055][SQL][PYSPARK] Fix SparkSession initialization when SparkContext is stopped

2017-01-12 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.1 616a78a56 -> 042e32d18 [SPARK-19055][SQL][PYSPARK] Fix SparkSession initialization when SparkContext is stopped ## What changes were proposed in this pull request? In SparkSession initialization, we store created the instance of

spark git commit: [SPARK-19055][SQL][PYSPARK] Fix SparkSession initialization when SparkContext is stopped

2017-01-12 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 871d26664 -> c6c37b8af [SPARK-19055][SQL][PYSPARK] Fix SparkSession initialization when SparkContext is stopped ## What changes were proposed in this pull request? In SparkSession initialization, we store created the instance of

spark git commit: [SPARK-18969][SQL] Support grouping by nondeterministic expressions

2017-01-12 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.0 c94288b57 -> 3566e40a4 [SPARK-18969][SQL] Support grouping by nondeterministic expressions ## What changes were proposed in this pull request? Currently nondeterministic expressions are allowed in `Aggregate`(see the

spark git commit: [SPARK-18969][SQL] Support grouping by nondeterministic expressions

2017-01-12 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.1 9b9867ef5 -> 616a78a56 [SPARK-18969][SQL] Support grouping by nondeterministic expressions ## What changes were proposed in this pull request? Currently nondeterministic expressions are allowed in `Aggregate`(see the

spark git commit: [SPARK-18969][SQL] Support grouping by nondeterministic expressions

2017-01-12 Thread wenchen
Repository: spark Updated Branches: refs/heads/master c71b25481 -> 871d26664 [SPARK-18969][SQL] Support grouping by nondeterministic expressions ## What changes were proposed in this pull request? Currently nondeterministic expressions are allowed in `Aggregate`(see the

spark git commit: [SPARK-18857][SQL] Don't use `Iterator.duplicate` for `incrementalCollect` in Thrift Server

2017-01-12 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.0 ec2fe925c -> c94288b57 [SPARK-18857][SQL] Don't use `Iterator.duplicate` for `incrementalCollect` in Thrift Server ## What changes were proposed in this pull request? To support `FETCH_FIRST`, SPARK-16563 used Scala

spark git commit: [SPARK-18857][SQL] Don't use `Iterator.duplicate` for `incrementalCollect` in Thrift Server

2017-01-12 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.1 0b07634b5 -> 9b9867ef5 [SPARK-18857][SQL] Don't use `Iterator.duplicate` for `incrementalCollect` in Thrift Server ## What changes were proposed in this pull request? To support `FETCH_FIRST`, SPARK-16563 used Scala

spark git commit: [SPARK-19183][SQL] Add deleteWithJob hook to internal commit protocol API

2017-01-12 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 5db35b312 -> c71b25481 [SPARK-19183][SQL] Add deleteWithJob hook to internal commit protocol API ## What changes were proposed in this pull request? Currently in SQL we implement overwrites by calling fs.delete() directly on the original

spark git commit: [SPARK-19164][PYTHON][SQL] Remove unused UserDefinedFunction._broadcast

2017-01-12 Thread rxin
Repository: spark Updated Branches: refs/heads/master 2c586f506 -> 5db35b312 [SPARK-19164][PYTHON][SQL] Remove unused UserDefinedFunction._broadcast ## What changes were proposed in this pull request? Removes `UserDefinedFunction._broadcast` and `UserDefinedFunction.__del__` method. ## How

spark git commit: [SPARK-19158][SPARKR][EXAMPLES] Fix ml.R example fails due to lack of e1071 package.

2017-01-12 Thread yliang
Repository: spark Updated Branches: refs/heads/branch-2.1 82fcc1330 -> 0b07634b5 [SPARK-19158][SPARKR][EXAMPLES] Fix ml.R example fails due to lack of e1071 package. ## What changes were proposed in this pull request? ```ml.R``` example depends on ```e1071``` package, if it's not available

spark git commit: [SPARK-19158][SPARKR][EXAMPLES] Fix ml.R example fails due to lack of e1071 package.

2017-01-12 Thread yliang
Repository: spark Updated Branches: refs/heads/master 24100f162 -> 2c586f506 [SPARK-19158][SPARKR][EXAMPLES] Fix ml.R example fails due to lack of e1071 package. ## What changes were proposed in this pull request? ```ml.R``` example depends on ```e1071``` package, if it's not available in