git commit: [SPARK-3466] Limit size of results that a driver collects for each action

Repository: spark Updated Branches: refs/heads/master 23f966f47 - 6181577e9 [SPARK-3466] Limit size of results that a driver collects for each action Right now, operations like collect() and take() can crash the driver with an OOM if they bring back too many data. This PR will introduce

git commit: [SPARK-4166][Core][WebUI] Display the executor ID in the Web UI when ExecutorLostFailure happens

Repository: spark Updated Branches: refs/heads/master 6181577e9 - 4e6a7a0b3 [SPARK-4166][Core][WebUI] Display the executor ID in the Web UI when ExecutorLostFailure happens Now when ExecutorLostFailure happens, it only displays `ExecutorLostFailure (executor lost)`. Adding the executor id

git commit: [HOTFIX][SQL] hive test missing some golden files

Repository: spark Updated Branches: refs/heads/master 4e6a7a0b3 - f0a4b630a [HOTFIX][SQL] hive test missing some golden files cc marmbrus Author: wangfei wangf...@huawei.com Closes #3055 from scwf/hotfix and squashes the following commits: d881bd7 [wangfei] miss golden files Project:

git commit: [SPARK-3247][SQL] An API for adding data sources to Spark SQL

Repository: spark Updated Branches: refs/heads/master f0a4b630a - 9c0eb57c7 [SPARK-3247][SQL] An API for adding data sources to Spark SQL This PR introduces a new set of APIs to Spark SQL to allow other developers to add support for reading data from new sources in

git commit: [SPARK-4182][SQL] Fixes ColumnStats classes for boolean, binary and complex data types

Repository: spark Updated Branches: refs/heads/master 9c0eb57c7 - e4b80894b [SPARK-4182][SQL] Fixes ColumnStats classes for boolean, binary and complex data types `NoopColumnStats` was once used for binary, boolean and complex data types. This `ColumnStats` doesn't return properly shaped

git commit: [SQL] Fixes race condition in CliSuite

Repository: spark Updated Branches: refs/heads/master e4b80894b - 495a13203 [SQL] Fixes race condition in CliSuite `CliSuite` has been flaky for a while, this PR tries to improve this situation by fixing a race condition in `CliSuite`. The `captureOutput` function is used to capture both

git commit: [SPARK-3791][SQL] Provides Spark version and Hive version in HiveThriftServer2

Repository: spark Updated Branches: refs/heads/master 495a13203 - c9f840046 [SPARK-3791][SQL] Provides Spark version and Hive version in HiveThriftServer2 This PR overrides the `GetInfo` Hive Thrift API to provide correct version information. Another property `spark.sql.hive.version` is

git commit: [SPARK-4191][SQL]move wrapperFor to HiveInspectors to reuse it

Repository: spark Updated Branches: refs/heads/master c9f840046 - e749f5ded [SPARK-4191][SQL]move wrapperFor to HiveInspectors to reuse it Move wrapperFor in InsertIntoHiveTable to HiveInspectors to reuse them, this method can be reused when writing date with ObjectInspector(such as orc

git commit: [SPARK-4185][SQL] JSON schema inference failed when dealing with type conflicts in arrays

Repository: spark Updated Branches: refs/heads/master e749f5ded - 06232d23f [SPARK-4185][SQL] JSON schema inference failed when dealing with type conflicts in arrays JIRA: https://issues.apache.org/jira/browse/SPARK-4185. This PR also has the fix of #3052. Author: Yin Huai

git commit: [SPARK-2189][SQL] Adds dropTempTable API

Repository: spark Updated Branches: refs/heads/master 06232d23f - 9081b9f9f [SPARK-2189][SQL] Adds dropTempTable API This PR adds an API for unregistering temporary tables. If a temporary table has been cached before, it's unpersisted as well. Author: Cheng Lian lian.cs@gmail.com

git commit: [SPARK-4183] Close transport-related resources between SparkContexts

Repository: spark Updated Branches: refs/heads/master 9081b9f9f - 2ebd1df3f [SPARK-4183] Close transport-related resources between SparkContexts A leak of event loops may be causing test failures. Author: Aaron Davidson aa...@databricks.com Closes #3053 from aarondav/leak and squashes the

git commit: [SPARK-3572] [SQL] Internal API for User-Defined Types

Repository: spark Updated Branches: refs/heads/master 2ebd1df3f - ebd648058 [SPARK-3572] [SQL] Internal API for User-Defined Types This PR adds User-Defined Types (UDTs) to SQL. It is a precursor to using SchemaRDD as a Dataset for the new MLlib API. Currently, the UDT API is private since

git commit: [SPARK-4183] Enable NettyBlockTransferService by default

Repository: spark Updated Branches: refs/heads/master ebd648058 - 1ae51f6dc [SPARK-4183] Enable NettyBlockTransferService by default Note that we're turning this on for at least the first part of the QA period as a trial. We want to enable this (and deprecate the NioBlockTransferService) as

git commit: [SPARK-4109][CORE] Correctly deserialize Task.stageId

Repository: spark Updated Branches: refs/heads/branch-1.0 6d8f1dd15 - 49224fd0f [SPARK-4109][CORE] Correctly deserialize Task.stageId The two subclasses of Task, ShuffleMapTask and ResultTask, do not correctly deserialize stageId. Therefore, the accessing of TaskContext.stageId always

git commit: Close #2971.

Repository: spark Updated Branches: refs/heads/master 1ae51f6dc - d6e4c5917 Close #2971. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d6e4c591 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/d6e4c591 Diff:

git commit: [SPARK-4177][Doc]update build doc since JDBC/CLI support hive 13 now

Repository: spark Updated Branches: refs/heads/master d6e4c5917 - 001acc446 [SPARK-4177][Doc]update build doc since JDBC/CLI support hive 13 now Author: wangfei wangf...@huawei.com Closes #3042 from scwf/patch-9 and squashes the following commits: 3784ed1 [wangfei] remove 'TODO' 1891553

git commit: [SPARK-4163][Core][WebUI] Send the fetch failure message back to Web UI

Repository: spark Updated Branches: refs/heads/master 001acc446 - 76386e1a2 [SPARK-4163][Core][WebUI] Send the fetch failure message back to Web UI This is a PR to send the fetch failure message back to Web UI. Before: