spark git commit: [SPARK-14252] Executors do not try to download remote cached blocks

2016-04-05 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 68be5b9e8 -> 78c1076d0 [SPARK-14252] Executors do not try to download remote cached blocks ## What changes were proposed in this pull request? As mentioned in the ticket this was because one get path in the refactored `BlockManager` did n

spark git commit: [SPARK-14396][SQL] Throw Exceptions for DDLs of Partitioned Views

2016-04-05 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 48467f4eb -> 68be5b9e8 [SPARK-14396][SQL] Throw Exceptions for DDLs of Partitioned Views What changes were proposed in this pull request? Because the concept of partitioning is associated with physical tables, we disable all the supp

spark git commit: [SPARK-14416][CORE] Add thread-safe comments for CoarseGrainedSchedulerBackend's fields

2016-04-05 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master adbfdb878 -> 48467f4eb [SPARK-14416][CORE] Add thread-safe comments for CoarseGrainedSchedulerBackend's fields ## What changes were proposed in this pull request? While I was reviewing #12078, I found most of CoarseGrainedSchedulerBackend

spark git commit: [SPARK-14128][SQL] Alter table DDL followup

2016-04-05 Thread yhuai
Repository: spark Updated Branches: refs/heads/master f6456fa80 -> adbfdb878 [SPARK-14128][SQL] Alter table DDL followup ## What changes were proposed in this pull request? This is just a followup to #12121, which implemented the alter table DDLs using the `SessionCatalog`. Specially, this c

spark git commit: [SPARK-14296][SQL] whole stage codegen support for Dataset.map

2016-04-05 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 8e5c1cbf2 -> f6456fa80 [SPARK-14296][SQL] whole stage codegen support for Dataset.map ## What changes were proposed in this pull request? This PR adds a new operator `MapElements` for `Dataset.map`, it's a 1-1 mapping and is easier to ada

spark git commit: [SPARK-13211][STREAMING] StreamingContext throws NoSuchElementException when created from non-existent checkpoint directory

2016-04-05 Thread zsxwing
Repository: spark Updated Branches: refs/heads/master 7d29c72f6 -> 8e5c1cbf2 [SPARK-13211][STREAMING] StreamingContext throws NoSuchElementException when created from non-existent checkpoint directory ## What changes were proposed in this pull request? Take 2: avoid None.get NoSuchElementExc

spark git commit: [SPARK-14359] Unit tests for java 8 lambda syntax with typed aggregates

2016-04-05 Thread rxin
Repository: spark Updated Branches: refs/heads/master 1146c534d -> 7d29c72f6 [SPARK-14359] Unit tests for java 8 lambda syntax with typed aggregates ## What changes were proposed in this pull request? Adds unit tests for java 8 lambda syntax with typed aggregates as a follow-up to #12168 ##

spark git commit: [SPARK-14353] Dataset Time Window `window` API for R

2016-04-05 Thread davies
Repository: spark Updated Branches: refs/heads/master 48682f6bf -> 1146c534d [SPARK-14353] Dataset Time Window `window` API for R ## What changes were proposed in this pull request? The `window` function was added to Dataset with [this PR](https://github.com/apache/spark/pull/12008). This PR

spark git commit: [HOTFIX] Fix `optional` to `createOptional`.

2016-04-05 Thread rxin
Repository: spark Updated Branches: refs/heads/master d5ee9d5c2 -> 48682f6bf [HOTFIX] Fix `optional` to `createOptional`. ## What changes were proposed in this pull request? This PR fixes the following line. ``` private[spark] val STAGING_DIR = ConfigBuilder("spark.yarn.stagingDir") .

spark git commit: [SPARK-529][SQL] Modify SQLConf to use new config API from core.

2016-04-05 Thread vanzin
Repository: spark Updated Branches: refs/heads/master 7329fe272 -> d5ee9d5c2 [SPARK-529][SQL] Modify SQLConf to use new config API from core. Because SQL keeps track of all known configs, some customization was needed in SQLConf to allow that, since the core API does not have that feature. Te

spark git commit: [SPARK-14411][SQL] Add a note to warn that onQueryProgress is asynchronous

2016-04-05 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 45d8cdee3 -> 7329fe272 [SPARK-14411][SQL] Add a note to warn that onQueryProgress is asynchronous ## What changes were proposed in this pull request? onQueryProgress is asynchronous so the user may see some future status of `ContinuousQue

spark git commit: [SPARK-14129][SPARK-14128][SQL] Alter table DDL commands

2016-04-05 Thread yhuai
Repository: spark Updated Branches: refs/heads/master c59abad05 -> 45d8cdee3 [SPARK-14129][SPARK-14128][SQL] Alter table DDL commands ## What changes were proposed in this pull request? In Spark 2.0, we want to handle the most common `ALTER TABLE` commands ourselves instead of passing the en

spark git commit: [SPARK-14402][SQL] initcap UDF doesn't match Hive/Oracle behavior in lowercasing rest of string

2016-04-05 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 9ee5c2571 -> c59abad05 [SPARK-14402][SQL] initcap UDF doesn't match Hive/Oracle behavior in lowercasing rest of string ## What changes were proposed in this pull request? Current, SparkSQL `initCap` is using `toTitleCase` function. Howeve

spark git commit: [SPARK-14353] Dataset Time Window `window` API for Python, and SQL

2016-04-05 Thread davies
Repository: spark Updated Branches: refs/heads/master 72544d6f2 -> 9ee5c2571 [SPARK-14353] Dataset Time Window `window` API for Python, and SQL ## What changes were proposed in this pull request? The `window` function was added to Dataset with [this PR](https://github.com/apache/spark/pull/1

spark git commit: [SPARK-14243][CORE][BACKPORT-1.6] update task metrics when removing blocks

2016-04-05 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.6 285cb9c66 -> cfe9f02ca [SPARK-14243][CORE][BACKPORT-1.6] update task metrics when removing blocks ## What changes were proposed in this pull request? This patch try to update the `updatedBlockStatuses ` when removing blocks, making s

[1/2] spark git commit: [SPARK-14123][SPARK-14384][SQL] Handle CreateFunction/DropFunction

2016-04-05 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master bc36df127 -> 72544d6f2 http://git-wip-us.apache.org/repos/asf/spark/blob/72544d6f/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala -- diff --git

[2/2] spark git commit: [SPARK-14123][SPARK-14384][SQL] Handle CreateFunction/DropFunction

2016-04-05 Thread andrewor14
[SPARK-14123][SPARK-14384][SQL] Handle CreateFunction/DropFunction ## What changes were proposed in this pull request? This PR implements CreateFunction and DropFunction commands. Besides implementing these two commands, we also change how to manage functions. Here are the main changes. * `Funct

spark git commit: [SPARK-13063][YARN] Make the SPARK YARN STAGING DIR as configurable

2016-04-05 Thread tgraves
Repository: spark Updated Branches: refs/heads/master 463bac001 -> bc36df127 [SPARK-13063][YARN] Make the SPARK YARN STAGING DIR as configurable ## What changes were proposed in this pull request? Made the SPARK YARN STAGING DIR as configurable with the configuration as 'spark.yarn.staging-di

spark git commit: [SPARK-14257][SQL] Allow multiple continuous queries to be started from the same DataFrame

2016-04-05 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master f77f11c67 -> 463bac001 [SPARK-14257][SQL] Allow multiple continuous queries to be started from the same DataFrame ## What changes were proposed in this pull request? Make StreamingRelation store the closure to create the source in Stream

spark git commit: [SPARK-14345][SQL] Decouple deserializer expression resolution from ObjectOperator

2016-04-05 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master e4bd50412 -> f77f11c67 [SPARK-14345][SQL] Decouple deserializer expression resolution from ObjectOperator ## What changes were proposed in this pull request? This PR decouples deserializer expression resolution from `ObjectOperator`, so

spark git commit: [SPARK-14397][WEBUI] and tags are nested in LogPage

2016-04-05 Thread zsxwing
Repository: spark Updated Branches: refs/heads/master d35690158 -> e4bd50412 [SPARK-14397][WEBUI] and tags are nested in LogPage ## What changes were proposed in this pull request? In `LogPage`, the content to be rendered is defined as follows. ``` val content =

spark git commit: [SPARK-14284][ML] KMeansSummary deprecating size; adding clusterSizes

2016-04-05 Thread jkbradley
Repository: spark Updated Branches: refs/heads/master 780717367 -> d35690158 [SPARK-14284][ML] KMeansSummary deprecating size; adding clusterSizes ## What changes were proposed in this pull request? KMeansSummary class : deprecated size and added clusterSizes Author: Shally Sangal Closes #

spark git commit: [SPARK-14349][SQL] Issue Error Messages for Unsupported Operators/DML/DDL in SQL Context.

2016-04-05 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/master 2715bc68b -> 780717367 [SPARK-14349][SQL] Issue Error Messages for Unsupported Operators/DML/DDL in SQL Context. What changes were proposed in this pull request? Currently, the weird error messages are issued if we use Hive Context-o