spark git commit: [SPARK-12470] [SQL] Fix size reduction calculation

Repository: spark Updated Branches: refs/heads/master 6c83d938c -> b504b6a90 [SPARK-12470] [SQL] Fix size reduction calculation also only allocate required buffer size Author: Pete Robbins Closes #10421 from robbinspg/master. Project:

spark git commit: [SPARK-12486] Worker should kill the executors more forcefully if possible.

Repository: spark Updated Branches: refs/heads/master 962aac4db -> 8f659393b [SPARK-12486] Worker should kill the executors more forcefully if possible. This patch updates the ExecutorRunner's terminate path to use the new java 8 API to terminate processes more forcefully if possible. If the

spark git commit: [SPARK-12608][STREAMING] Remove submitJobThreadPool since submitJob doesn't create a separate thread to wait for the job result

Repository: spark Updated Branches: refs/heads/master b504b6a90 -> 43706bf8b [SPARK-12608][STREAMING] Remove submitJobThreadPool since submitJob doesn't create a separate thread to wait for the job result Before #9264, submitJob would create a separate thread to wait for the job result.

spark git commit: [DOC] Adjust coverage for partitionBy()

Repository: spark Updated Branches: refs/heads/master 573ac55d7 -> 40d03960d [DOC] Adjust coverage for partitionBy() This is the related thread: http://search-hadoop.com/m/q3RTtO3ReeJ1iF02=Re+partitioning+json+data+in+spark Michael suggested fixing the doc. Please review. Author: tedyu

spark git commit: [DOC] Adjust coverage for partitionBy()

Repository: spark Updated Branches: refs/heads/branch-1.6 7f37c1e45 -> 1005ee396 [DOC] Adjust coverage for partitionBy() This is the related thread: http://search-hadoop.com/m/q3RTtO3ReeJ1iF02=Re+partitioning+json+data+in+spark Michael suggested fixing the doc. Please review. Author:

spark git commit: [SPARK-12421][SQL] Prevent Internal/External row from exposing state.

Repository: spark Updated Branches: refs/heads/master 40d03960d -> 0171b71e9 [SPARK-12421][SQL] Prevent Internal/External row from exposing state. It is currently possible to change the values of the supposedly immutable ```GenericRow``` and ```GenericInternalRow``` classes. This is caused

spark git commit: [SPARK-12470] [SQL] Fix size reduction calculation

Repository: spark Updated Branches: refs/heads/branch-1.6 cd0203819 -> b5a1f564a [SPARK-12470] [SQL] Fix size reduction calculation also only allocate required buffer size Author: Pete Robbins Closes #10421 from robbinspg/master. (cherry picked from commit

spark git commit: [SPARK-12512][SQL] support column name with dot in withColumn()

Repository: spark Updated Branches: refs/heads/master 43706bf8b -> 573ac55d7 [SPARK-12512][SQL] support column name with dot in withColumn() Author: Xiu Guo Closes #10500 from xguo27/SPARK-12512. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

spark git commit: [SPARK-12579][SQL] Force user-specified JDBC driver to take precedence

Repository: spark Updated Branches: refs/heads/master 8f659393b -> 6c83d938c [SPARK-12579][SQL] Force user-specified JDBC driver to take precedence Spark SQL's JDBC data source allows users to specify an explicit JDBC driver to load (using the `driver` argument), but in the current code it's

spark git commit: [SPARK-12579][SQL] Force user-specified JDBC driver to take precedence

Repository: spark Updated Branches: refs/heads/branch-1.6 b5a1f564a -> 7f37c1e45 [SPARK-12579][SQL] Force user-specified JDBC driver to take precedence Spark SQL's JDBC data source allows users to specify an explicit JDBC driver to load (using the `driver` argument), but in the current code

spark git commit: [SPARK-12486] Worker should kill the executors more forcefully if possible.

Repository: spark Updated Branches: refs/heads/branch-1.6 f7a322382 -> cd0203819 [SPARK-12486] Worker should kill the executors more forcefully if possible. This patch updates the ExecutorRunner's terminate path to use the new java 8 API to terminate processes more forcefully if possible. If

spark git commit: [SPARK-10359][PROJECT-INFRA] Use more random number in dev/test-dependencies.sh; fix version switching

Repository: spark Updated Branches: refs/heads/master 0d165ec20 -> 9fd7a2f02 [SPARK-10359][PROJECT-INFRA] Use more random number in dev/test-dependencies.sh; fix version switching This patch aims to fix another potential source of flakiness in the `dev/test-dependencies.sh` script.

spark git commit: [SPARK-11259][ML] Params.validateParams() should be called automatically

Repository: spark Updated Branches: refs/heads/master 0171b71e9 -> ba5f81859 [SPARK-11259][ML] Params.validateParams() should be called automatically See JIRA: https://issues.apache.org/jira/browse/SPARK-11259 Author: Yanbo Liang Closes #9224 from

spark git commit: [SPARK-9622][ML] DecisionTreeRegressor: provide variance of prediction

Repository: spark Updated Branches: refs/heads/master ba5f81859 -> 93ef9b6a2 [SPARK-9622][ML] DecisionTreeRegressor: provide variance of prediction DecisionTreeRegressor will provide variance of prediction as a Double column. Author: Yanbo Liang Closes #8866 from

spark git commit: [SPARK-12541] [SQL] support cube/rollup as function

Repository: spark Updated Branches: refs/heads/master 93ef9b6a2 -> d084a2de3 [SPARK-12541] [SQL] support cube/rollup as function This PR enable cube/rollup as function, so they can be used as this: ``` select a, b, sum(c) from t group by rollup(a, b) ``` Author: Davies Liu

spark git commit: [SPARK-12589][SQL] Fix UnsafeRowParquetRecordReader to properly set the row length.

Repository: spark Updated Branches: refs/heads/branch-1.6 1005ee396 -> 8ac919809 [SPARK-12589][SQL] Fix UnsafeRowParquetRecordReader to properly set the row length. The reader was previously not setting the row length meaning it was wrong if there were variable length columns. This problem

spark git commit: [SPARK-12509][SQL] Fixed error messages for DataFrame correlation and covariance

Repository: spark Updated Branches: refs/heads/master 34de24abb -> fdfac22d0 [SPARK-12509][SQL] Fixed error messages for DataFrame correlation and covariance Currently, when we call corr or cov on dataframe with invalid input we see these error messages for both corr and cov: -

spark git commit: [SPARK-12589][SQL] Fix UnsafeRowParquetRecordReader to properly set the row length.

Repository: spark Updated Branches: refs/heads/master d084a2de3 -> 34de24abb [SPARK-12589][SQL] Fix UnsafeRowParquetRecordReader to properly set the row length. The reader was previously not setting the row length meaning it was wrong if there were variable length columns. This problem does

spark git commit: [SPARK-12480][SQL] add Hash expression that can calculate hash value for a group of expressions

Repository: spark Updated Branches: refs/heads/master 77ab49b85 -> b1a771231 [SPARK-12480][SQL] add Hash expression that can calculate hash value for a group of expressions just write the arguments into unsafe row and use murmur3 to calculate hash code Author: Wenchen Fan

[1/2] spark git commit: [SPARK-12600][SQL] Remove deprecated methods in Spark SQL

Repository: spark Updated Branches: refs/heads/master fdfac22d0 -> 77ab49b85 http://git-wip-us.apache.org/repos/asf/spark/blob/77ab49b8/sql/core/src/main/scala/org/apache/spark/sql/functions.scala -- diff --git

spark git commit: [SPARK-12600][SQL] follow up: add range check for DecimalType

Repository: spark Updated Branches: refs/heads/master 8896ec9f0 -> b634901bb [SPARK-12600][SQL] follow up: add range check for DecimalType This addresses davies' code review feedback in https://github.com/apache/spark/pull/10559 Author: Reynold Xin Closes #10586 from

[2/2] spark git commit: [SPARK-12600][SQL] Remove deprecated methods in Spark SQL

[SPARK-12600][SQL] Remove deprecated methods in Spark SQL Author: Reynold Xin Closes #10559 from rxin/remove-deprecated-sql. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/77ab49b8 Tree:

spark git commit: [SPARKR][DOC] minor doc update for version in migration guide

Repository: spark Updated Branches: refs/heads/branch-1.6 8ac919809 -> 8950482ee [SPARKR][DOC] minor doc update for version in migration guide checked that the change is in Spark 1.6.0. shivaram Author: felixcheung Closes #10574 from felixcheung/rwritemodedoc.

spark git commit: [SPARK-12513][STREAMING] SocketReceiver hang in Netcat example

Repository: spark Updated Branches: refs/heads/master 9fd7a2f02 -> 962aac4db [SPARK-12513][STREAMING] SocketReceiver hang in Netcat example Explicitly close client side socket connection before restart socket receiver. Author: guoxu1231 Author: Shawn Guo

spark git commit: [SPARK-12568][SQL] Add BINARY to Encoders

Repository: spark Updated Branches: refs/heads/branch-1.6 8950482ee -> d9e4438b5 [SPARK-12568][SQL] Add BINARY to Encoders Author: Michael Armbrust Closes #10516 from marmbrus/datasetCleanup. (cherry picked from commit 53beddc5bf04a35ab73de99158919c2fdd5d4508)

spark git commit: [SPARK-12625][SPARKR][SQL] replace R usage of Spark SQL deprecated API

Repository: spark Updated Branches: refs/heads/master b634901bb -> cc4d5229c [SPARK-12625][SPARKR][SQL] replace R usage of Spark SQL deprecated API rxin davies shivaram Took save mode from my PR #10480, and move everything to writer methods. This is related to PR #10559 - [x] it seems

[2/2] spark git commit: [SPARK-3873][EXAMPLES] Import ordering fixes.

[SPARK-3873][EXAMPLES] Import ordering fixes. Author: Marcelo Vanzin Closes #10575 from vanzin/SPARK-3873-examples. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/7058dc11 Tree:

[1/2] spark git commit: [SPARK-3873][EXAMPLES] Import ordering fixes.

Repository: spark Updated Branches: refs/heads/master cc4d5229c -> 7058dc115 http://git-wip-us.apache.org/repos/asf/spark/blob/7058dc11/examples/src/main/scala/org/apache/spark/examples/mllib/PrefixSpanExample.scala -- diff

spark git commit: [SPARK-12568][SQL] Add BINARY to Encoders

Repository: spark Updated Branches: refs/heads/master 7058dc115 -> 53beddc5b [SPARK-12568][SQL] Add BINARY to Encoders Author: Michael Armbrust Closes #10516 from marmbrus/datasetCleanup. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

svn commit: r11815 - /dev/spark/spark-1.6.0-rc4/ /release/spark/spark-1.6.0/

Author: marmbrus Date: Mon Jan 4 16:20:48 2016 New Revision: 11815 Log: Release Spark 1.6.0 Added: release/spark/spark-1.6.0/ - copied from r11814, dev/spark/spark-1.6.0-rc4/ Removed: dev/spark/spark-1.6.0-rc4/

svn commit: r1722911 [3/3] - in /spark: ./ js/ news/_posts/ releases/_posts/ site/ site/graphx/ site/js/ site/mllib/ site/news/ site/releases/ site/screencasts/ site/sql/ site/streaming/

Added: spark/site/releases/spark-release-1-6-0.html URL: http://svn.apache.org/viewvc/spark/site/releases/spark-release-1-6-0.html?rev=1722911=auto == --- spark/site/releases/spark-release-1-6-0.html (added) +++

svn commit: r1722911 [2/3] - in /spark: ./ js/ news/_posts/ releases/_posts/ site/ site/graphx/ site/js/ site/mllib/ site/news/ site/releases/ site/screencasts/ site/sql/ site/streaming/

Added: spark/site/news/spark-1-6-0-released.html URL: http://svn.apache.org/viewvc/spark/site/news/spark-1-6-0-released.html?rev=1722911=auto == --- spark/site/news/spark-1-6-0-released.html (added) +++

svn commit: r1722932 - in /spark/site/docs: ./ 1.6.0/ 1.6.0/api/ 1.6.0/api/R/ 1.6.0/api/java/ 1.6.0/api/java/lib/ 1.6.0/api/java/org/ 1.6.0/api/java/org/apache/ 1.6.0/api/java/org/apache/spark/ 1.6.0/

Author: marmbrus Date: Mon Jan 4 17:53:21 2016 New Revision: 1722932 URL: http://svn.apache.org/viewvc?rev=1722932=rev Log: Add Spark 1.6.0 docs [This commit notification would consist of 344 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

svn commit: r1722911 [1/3] - in /spark: ./ js/ news/_posts/ releases/_posts/ site/ site/graphx/ site/js/ site/mllib/ site/news/ site/releases/ site/screencasts/ site/sql/ site/streaming/

Author: marmbrus Date: Mon Jan 4 16:22:05 2016 New Revision: 1722911 URL: http://svn.apache.org/viewvc?rev=1722911=rev Log: Update site for Spark 1.6.0 Added: spark/news/_posts/2016-01-04-spark-1-6-0-released.md spark/releases/_posts/2016-01-04-spark-release-1-6-0.md

svn commit: r1722915 - in /spark: _layouts/ site/ site/graphx/ site/mllib/ site/news/ site/releases/ site/screencasts/ site/sql/ site/streaming/

Author: marmbrus Date: Mon Jan 4 16:34:56 2016 New Revision: 1722915 URL: http://svn.apache.org/viewvc?rev=1722915=rev Log: Update site for Spark 1.6.0 Modified: spark/_layouts/global.html spark/site/community.html spark/site/documentation.html spark/site/downloads.html

svn commit: r1722917 - in /spark: releases/_posts/2016-01-04-spark-release-1-6-0.md site/releases/spark-release-1-6-0.html

Author: marmbrus Date: Mon Jan 4 16:42:42 2016 New Revision: 1722917 URL: http://svn.apache.org/viewvc?rev=1722917=rev Log: Remove broken links from Spark 1.6.0 page Modified: spark/releases/_posts/2016-01-04-spark-release-1-6-0.md spark/site/releases/spark-release-1-6-0.html Modified: