spark git commit: [SPARK-15107][SQL] Allow varying # iterations by test case in Benchmark

Repository: spark Updated Branches: refs/heads/branch-2.0 fd3accca6 -> 2ff667606 [SPARK-15107][SQL] Allow varying # iterations by test case in Benchmark This patch changes our micro-benchmark util to allow setting different iteration numbers for different test cases. For some of our

spark git commit: [SPARK-15107][SQL] Allow varying # iterations by test case in Benchmark

Repository: spark Updated Branches: refs/heads/master 348c13898 -> 695f0e919 [SPARK-15107][SQL] Allow varying # iterations by test case in Benchmark ## What changes were proposed in this pull request? This patch changes our micro-benchmark util to allow setting different iteration numbers

spark git commit: [SPARK-15095][SQL] remove HiveSessionHook from ThriftServer

Repository: spark Updated Branches: refs/heads/branch-2.0 940b8f60b -> fd3accca6 [SPARK-15095][SQL] remove HiveSessionHook from ThriftServer ## What changes were proposed in this pull request? Remove HiveSessionHook ## How was this patch tested? No tests needed. Author: Davies Liu

spark git commit: [SPARK-15095][SQL] remove HiveSessionHook from ThriftServer

Repository: spark Updated Branches: refs/heads/master 6ba17cd14 -> 348c13898 [SPARK-15095][SQL] remove HiveSessionHook from ThriftServer ## What changes were proposed in this pull request? Remove HiveSessionHook ## How was this patch tested? No tests needed. Author: Davies Liu

spark git commit: [SPARK-14414][SQL] Make DDL exceptions more consistent

Repository: spark Updated Branches: refs/heads/master 9e4928b7e -> 6ba17cd14 [SPARK-14414][SQL] Make DDL exceptions more consistent ## What changes were proposed in this pull request? Just a bunch of small tweaks on DDL exception messages. ## How was this patch tested? `DDLCommandSuite` et

spark git commit: [SPARK-14414][SQL] Make DDL exceptions more consistent

Repository: spark Updated Branches: refs/heads/branch-2.0 95d359abd -> 940b8f60b [SPARK-14414][SQL] Make DDL exceptions more consistent ## What changes were proposed in this pull request? Just a bunch of small tweaks on DDL exception messages. ## How was this patch tested?

spark git commit: [SPARK-15097][SQL] make Dataset.sqlContext a stable identifier for imports

Repository: spark Updated Branches: refs/heads/branch-2.0 5e15615d1 -> 95d359abd [SPARK-15097][SQL] make Dataset.sqlContext a stable identifier for imports ## What changes were proposed in this pull request? Make Dataset.sqlContext a lazy val so that its a stable identifier and can be used

spark git commit: [SPARK-15097][SQL] make Dataset.sqlContext a stable identifier for imports

Repository: spark Updated Branches: refs/heads/master 0903a185c -> 9e4928b7e [SPARK-15097][SQL] make Dataset.sqlContext a stable identifier for imports ## What changes were proposed in this pull request? Make Dataset.sqlContext a lazy val so that its a stable identifier and can be used for

spark git commit: [SPARK-15084][PYTHON][SQL] Use builder pattern to create SparkSession in PySpark.

Repository: spark Updated Branches: refs/heads/branch-2.0 4c7f5a74d -> 5e15615d1 [SPARK-15084][PYTHON][SQL] Use builder pattern to create SparkSession in PySpark. ## What changes were proposed in this pull request? This is a python port of corresponding Scala builder pattern code. `sql.py`

spark git commit: [SPARK-15084][PYTHON][SQL] Use builder pattern to create SparkSession in PySpark.

Repository: spark Updated Branches: refs/heads/master c1839c991 -> 0903a185c [SPARK-15084][PYTHON][SQL] Use builder pattern to create SparkSession in PySpark. ## What changes were proposed in this pull request? This is a python port of corresponding Scala builder pattern code. `sql.py` is

spark git commit: [SPARK-14645][MESOS] Fix python running on cluster mode mesos to have non local uris

Repository: spark Updated Branches: refs/heads/master a8d56f538 -> c1839c991 [SPARK-14645][MESOS] Fix python running on cluster mode mesos to have non local uris ## What changes were proposed in this pull request? Fix SparkSubmit to allow non-local python uris ## How was this patch tested?

spark git commit: [SPARK-14645][MESOS] Fix python running on cluster mode mesos to have non local uris

Repository: spark Updated Branches: refs/heads/branch-2.0 5625b037a -> 4c7f5a74d [SPARK-14645][MESOS] Fix python running on cluster mode mesos to have non local uris ## What changes were proposed in this pull request? Fix SparkSubmit to allow non-local python uris ## How was this patch

spark git commit: [SPARK-14422][SQL] Improve handling of optional configs in SQLConf

Repository: spark Updated Branches: refs/heads/branch-2.0 0d16b7f3a -> 5625b037a [SPARK-14422][SQL] Improve handling of optional configs in SQLConf ## What changes were proposed in this pull request? Create a new API for handling Optional Configs in SQLConf. Right now `getConf` for

spark git commit: [MINOR][DOC] Fixed some python snippets in mllib data types documentation.

Repository: spark Updated Branches: refs/heads/branch-2.0 c212307b9 -> 0d16b7f3a [MINOR][DOC] Fixed some python snippets in mllib data types documentation. ## What changes were proposed in this pull request? Some python snippets is using scala imports and comments. ## How was this patch

spark git commit: [MINOR][DOC] Fixed some python snippets in mllib data types documentation.

Repository: spark Updated Branches: refs/heads/master dbacd9998 -> c4e0fde87 [MINOR][DOC] Fixed some python snippets in mllib data types documentation. ## What changes were proposed in this pull request? Some python snippets is using scala imports and comments. ## How was this patch tested?

spark git commit: [SPARK-15104] Fix spacing in log line

Repository: spark Updated Branches: refs/heads/branch-2.0 51bb0bcc8 -> c212307b9 [SPARK-15104] Fix spacing in log line Otherwise get logs that look like this (note no space before NODE_LOCAL) ``` INFO [2016-05-03 21:18:51,477] org.apache.spark.scheduler.TaskSetManager: Starting task 0.0 in

spark git commit: [SPARK-15104] Fix spacing in log line

Repository: spark Updated Branches: refs/heads/master 028c6a5db -> dbacd9998 [SPARK-15104] Fix spacing in log line Otherwise get logs that look like this (note no space before NODE_LOCAL) ``` INFO [2016-05-03 21:18:51,477] org.apache.spark.scheduler.TaskSetManager: Starting task 0.0 in

spark git commit: [SQL-15102][SQL] remove delegation token support from ThriftServer

Repository: spark Updated Branches: refs/heads/master 71296c041 -> 028c6a5db [SQL-15102][SQL] remove delegation token support from ThriftServer ## What changes were proposed in this pull request? These API is only useful for Hadoop, may not work for Spark SQL. The APIs is kept for source

spark git commit: [SQL-15102][SQL] remove delegation token support from ThriftServer

Repository: spark Updated Branches: refs/heads/branch-2.0 70c5ca220 -> 51bb0bcc8 [SQL-15102][SQL] remove delegation token support from ThriftServer ## What changes were proposed in this pull request? These API is only useful for Hadoop, may not work for Spark SQL. The APIs is kept for

spark git commit: [SPARK-15056][SQL] Parse Unsupported Sampling Syntax and Issue Better Exceptions

Repository: spark Updated Branches: refs/heads/branch-2.0 f21eddb29 -> 70c5ca220 [SPARK-15056][SQL] Parse Unsupported Sampling Syntax and Issue Better Exceptions What changes were proposed in this pull request? Compared with the current Spark parser, there are two extra syntax are

spark git commit: [SPARK-14973][ML] The CrossValidator and TrainValidationSplit miss the seed when saving and loading

Repository: spark Updated Branches: refs/heads/branch-2.0 c50746a4a -> f21eddb29 [SPARK-14973][ML] The CrossValidator and TrainValidationSplit miss the seed when saving and loading ## What changes were proposed in this pull request? https://issues.apache.org/jira/browse/SPARK-14973 Add

spark git commit: [SPARK-14973][ML] The CrossValidator and TrainValidationSplit miss the seed when saving and loading

Repository: spark Updated Branches: refs/heads/master d6c7b2a5c -> 2e2a6211c [SPARK-14973][ML] The CrossValidator and TrainValidationSplit miss the seed when saving and loading ## What changes were proposed in this pull request? https://issues.apache.org/jira/browse/SPARK-14973 Add seed

spark git commit: [SPARK-15095][SQL] drop binary mode in ThriftServer

Repository: spark Updated Branches: refs/heads/branch-2.0 ca9917160 -> c50746a4a [SPARK-15095][SQL] drop binary mode in ThriftServer ## What changes were proposed in this pull request? This PR drop the support for binary mode in ThriftServer, only HTTP mode is supported now, to reduce the

spark git commit: [SPARK-15095][SQL] drop binary mode in ThriftServer

Repository: spark Updated Branches: refs/heads/master 588cac414 -> d6c7b2a5c [SPARK-15095][SQL] drop binary mode in ThriftServer ## What changes were proposed in this pull request? This PR drop the support for binary mode in ThriftServer, only HTTP mode is supported now, to reduce the

spark git commit: [SPARK-15073][SQL] Hide SparkSession constructor from the public

Repository: spark Updated Branches: refs/heads/branch-2.0 b67668bc4 -> ca9917160 [SPARK-15073][SQL] Hide SparkSession constructor from the public ## What changes were proposed in this pull request? Users should use the builder pattern instead. ## How was this patch tested? Jenks. Author:

spark git commit: [SPARK-15073][SQL] Hide SparkSession constructor from the public

Repository: spark Updated Branches: refs/heads/master 83ee92f60 -> 588cac414 [SPARK-15073][SQL] Hide SparkSession constructor from the public ## What changes were proposed in this pull request? Users should use the builder pattern instead. ## How was this patch tested? Jenks. Author:

spark git commit: [SPARK-11316] coalesce doesn't handle UnionRDD with partial locality properly

Repository: spark Updated Branches: refs/heads/master a4aed7171 -> 83ee92f60 [SPARK-11316] coalesce doesn't handle UnionRDD with partial locality properly ## What changes were proposed in this pull request? coalesce doesn't handle UnionRDD with partial locality properly. I had a user who

spark git commit: [SPARK-11316] coalesce doesn't handle UnionRDD with partial locality properly

Repository: spark Updated Branches: refs/heads/branch-2.0 a2369c63f -> b67668bc4 [SPARK-11316] coalesce doesn't handle UnionRDD with partial locality properly ## What changes were proposed in this pull request? coalesce doesn't handle UnionRDD with partial locality properly. I had a user

spark git commit: [SPARK-14521] [SQL] StackOverflowError in Kryo when executing TPC-DS

Repository: spark Updated Branches: refs/heads/master 659f635d3 -> a4aed7171 [SPARK-14521] [SQL] StackOverflowError in Kryo when executing TPC-DS ## What changes were proposed in this pull request? Observed stackOverflowError in Kryo when executing TPC-DS Query27. Spark thrift server

spark git commit: [SPARK-14234][CORE] Executor crashes for TaskRunner thread interruption

Repository: spark Updated Branches: refs/heads/master f5623b460 -> 659f635d3 [SPARK-14234][CORE] Executor crashes for TaskRunner thread interruption ## What changes were proposed in this pull request? Resetting the task interruption status before updating the task status. ## How was this

spark git commit: [SPARK-14234][CORE] Executor crashes for TaskRunner thread interruption

Repository: spark Updated Branches: refs/heads/branch-2.0 50adfcdce -> 4bacdebe9 [SPARK-14234][CORE] Executor crashes for TaskRunner thread interruption ## What changes were proposed in this pull request? Resetting the task interruption status before updating the task status. ## How was this

spark git commit: [SPARK-15059][CORE] Remove fine-grained lock in ChildFirstURLClassLoader to avoid dead lock

Repository: spark Updated Branches: refs/heads/master 84b3a4a87 -> f5623b460 [SPARK-15059][CORE] Remove fine-grained lock in ChildFirstURLClassLoader to avoid dead lock ## What changes were proposed in this pull request? In some cases, fine-grained lock have race condition with class-loader

spark git commit: [SPARK-15059][CORE] Remove fine-grained lock in ChildFirstURLClassLoader to avoid dead lock

Repository: spark Updated Branches: refs/heads/branch-2.0 a08d2c954 -> 50adfcdce [SPARK-15059][CORE] Remove fine-grained lock in ChildFirstURLClassLoader to avoid dead lock ## What changes were proposed in this pull request? In some cases, fine-grained lock have race condition with

spark git commit: [SPARK-9819][STREAMING][DOCUMENTATION] Clarify doc for invReduceFunc in incremental versions of reduceByWindow

Repository: spark Updated Branches: refs/heads/branch-2.0 2b715251d -> 69e030226 [SPARK-9819][STREAMING][DOCUMENTATION] Clarify doc for invReduceFunc in incremental versions of reduceByWindow - that reduceFunc and invReduceFunc should be associative - that the intermediate result in iterated

spark git commit: [SPARK-15082][CORE] Improve unit test coverage for AccumulatorV2

Repository: spark Updated Branches: refs/heads/master 439e36101 -> 84b3a4a87 [SPARK-15082][CORE] Improve unit test coverage for AccumulatorV2 ## What changes were proposed in this pull request? Added tests for ListAccumulator and LegacyAccumulatorWrapper, test for ListAccumulator is one

spark git commit: [SPARK-15082][CORE] Improve unit test coverage for AccumulatorV2

Repository: spark Updated Branches: refs/heads/branch-2.0 69e030226 -> a08d2c954 [SPARK-15082][CORE] Improve unit test coverage for AccumulatorV2 ## What changes were proposed in this pull request? Added tests for ListAccumulator and LegacyAccumulatorWrapper, test for ListAccumulator is one

spark git commit: [SPARK-9819][STREAMING][DOCUMENTATION] Clarify doc for invReduceFunc in incremental versions of reduceByWindow

Repository: spark Updated Branches: refs/heads/master ca813330c -> 439e36101 [SPARK-9819][STREAMING][DOCUMENTATION] Clarify doc for invReduceFunc in incremental versions of reduceByWindow - that reduceFunc and invReduceFunc should be associative - that the intermediate result in iterated

spark git commit: [SPARK-15087][CORE][SQL] Remove AccumulatorV2.localValue and keep only value

Repository: spark Updated Branches: refs/heads/master b545d7521 -> ca813330c [SPARK-15087][CORE][SQL] Remove AccumulatorV2.localValue and keep only value ## What changes were proposed in this pull request? Remove AccumulatorV2.localValue and keep only value ## How was this patch tested?

spark git commit: [SPARK-14860][TESTS] Create a new Waiter in reset to bypass an issue of ScalaTest's Waiter.wait

Repository: spark Updated Branches: refs/heads/master 4ad492c40 -> b545d7521 [SPARK-14860][TESTS] Create a new Waiter in reset to bypass an issue of ScalaTest's Waiter.wait ## What changes were proposed in this pull request? This PR updates `QueryStatusCollector.reset` to create Waiter

spark git commit: [SPARK-14860][TESTS] Create a new Waiter in reset to bypass an issue of ScalaTest's Waiter.wait

Repository: spark Updated Branches: refs/heads/branch-2.0 c5b7e1f70 -> 31e5a2a76 [SPARK-14860][TESTS] Create a new Waiter in reset to bypass an issue of ScalaTest's Waiter.wait ## What changes were proposed in this pull request? This PR updates `QueryStatusCollector.reset` to create Waiter

spark git commit: [SPARK-14716][SQL] Added support for partitioning in FileStreamSink

Repository: spark Updated Branches: refs/heads/branch-2.0 45bc65519 -> c5b7e1f70 [SPARK-14716][SQL] Added support for partitioning in FileStreamSink # What changes were proposed in this pull request? Support partitioning in the file stream sink. This is implemented using a new, but simpler

spark git commit: [SPARK-14884][SQL][STREAMING][WEBUI] Fix call site for continuous queries

Repository: spark Updated Branches: refs/heads/branch-2.0 17996e7d0 -> 45bc65519 [SPARK-14884][SQL][STREAMING][WEBUI] Fix call site for continuous queries ## What changes were proposed in this pull request? Since we've been processing continuous queries in separate threads, the call sites

spark git commit: [SPARK-14884][SQL][STREAMING][WEBUI] Fix call site for continuous queries

Repository: spark Updated Branches: refs/heads/master 5503e453b -> 5bd9a2f69 [SPARK-14884][SQL][STREAMING][WEBUI] Fix call site for continuous queries ## What changes were proposed in this pull request? Since we've been processing continuous queries in separate threads, the call sites are

spark git commit: [SPARK-15088] [SQL] Remove SparkSqlSerializer

Repository: spark Updated Branches: refs/heads/master 8b6491fc0 -> 5503e453b [SPARK-15088] [SQL] Remove SparkSqlSerializer ## What changes were proposed in this pull request? This patch removes SparkSqlSerializer. I believe this is now dead code. ## How was this patch tested? Removed a test

spark git commit: [SPARK-15088] [SQL] Remove SparkSqlSerializer

Repository: spark Updated Branches: refs/heads/branch-2.0 a373c39a9 -> 17996e7d0 [SPARK-15088] [SQL] Remove SparkSqlSerializer ## What changes were proposed in this pull request? This patch removes SparkSqlSerializer. I believe this is now dead code. ## How was this patch tested? Removed a

spark git commit: [SPARK-15091][SPARKR] Fix warnings and a failure in SparkR test cases with testthat version 1.0.1

Repository: spark Updated Branches: refs/heads/branch-2.0 932e1b5b2 -> a373c39a9 [SPARK-15091][SPARKR] Fix warnings and a failure in SparkR test cases with testthat version 1.0.1 ## What changes were proposed in this pull request? Fix warnings and a failure in SparkR test cases with testthat

spark git commit: [SPARK-15091][SPARKR] Fix warnings and a failure in SparkR test cases with testthat version 1.0.1

Repository: spark Updated Branches: refs/heads/master d26f7cb01 -> 8b6491fc0 [SPARK-15091][SPARKR] Fix warnings and a failure in SparkR test cases with testthat version 1.0.1 ## What changes were proposed in this pull request? Fix warnings and a failure in SparkR test cases with testthat

spark git commit: [SPARK-14971][ML][PYSPARK] PySpark ML Params setter code clean up

Repository: spark Updated Branches: refs/heads/branch-2.0 f03bf7eac -> 932e1b5b2 [SPARK-14971][ML][PYSPARK] PySpark ML Params setter code clean up ## What changes were proposed in this pull request? PySpark ML Params setter code clean up. For examples, ```setInputCol``` can be simplified from

spark git commit: [SPARK-14971][ML][PYSPARK] PySpark ML Params setter code clean up

Repository: spark Updated Branches: refs/heads/master 46965cd01 -> d26f7cb01 [SPARK-14971][ML][PYSPARK] PySpark ML Params setter code clean up ## What changes were proposed in this pull request? PySpark ML Params setter code clean up. For examples, ```setInputCol``` can be simplified from ```

spark git commit: [SPARK-15057][GRAPHX] Remove stale TODO comment for making `enum` in GraphGenerators

Repository: spark Updated Branches: refs/heads/master 57ac7c182 -> 46965cd01 [SPARK-15057][GRAPHX] Remove stale TODO comment for making `enum` in GraphGenerators ## What changes were proposed in this pull request? This PR removes a stale TODO comment in `GraphGenerators.scala` ## How was

spark git commit: [SPARK-14897][CORE] Upgrade Jetty to latest version of 8

Repository: spark Updated Branches: refs/heads/branch-1.6 10d513feb -> 2db19a3af [SPARK-14897][CORE] Upgrade Jetty to latest version of 8 Update Jetty 8.1 to the latest 2016/02 release, from a 2013/10 release, for security and bug fixes. This does not resolve the JIRA necessarily, as it's

spark git commit: [SPARK-14897][CORE] Upgrade Jetty to latest version of 8

Repository: spark Updated Branches: refs/heads/branch-2.0 b802979ad -> f03bf7eac [SPARK-14897][CORE] Upgrade Jetty to latest version of 8 ## What changes were proposed in this pull request? Update Jetty 8.1 to the latest 2016/02 release, from a 2013/10 release, for security and bug fixes.

spark git commit: [SPARK-14897][CORE] Upgrade Jetty to latest version of 8

Repository: spark Updated Branches: refs/heads/master d557a5e01 -> 57ac7c182 [SPARK-14897][CORE] Upgrade Jetty to latest version of 8 ## What changes were proposed in this pull request? Update Jetty 8.1 to the latest 2016/02 release, from a 2013/10 release, for security and bug fixes. This

spark git commit: [SPARK-15081] Move AccumulatorV2 and subclasses into util package

Repository: spark Updated Branches: refs/heads/master a74445707 -> d557a5e01 [SPARK-15081] Move AccumulatorV2 and subclasses into util package ## What changes were proposed in this pull request? This patch moves AccumulatorV2 and subclasses into util package. ## How was this patch tested?

spark git commit: [SPARK-15081] Move AccumulatorV2 and subclasses into util package

Repository: spark Updated Branches: refs/heads/branch-2.0 38f6e66af -> b802979ad [SPARK-15081] Move AccumulatorV2 and subclasses into util package ## What changes were proposed in this pull request? This patch moves AccumulatorV2 and subclasses into util package. ## How was this patch

spark git commit: [SPARK-15053][BUILD] Fix Java Lint errors on Hive-Thriftserver module

Repository: spark Updated Branches: refs/heads/branch-2.0 07a02e8bb -> 38f6e66af [SPARK-15053][BUILD] Fix Java Lint errors on Hive-Thriftserver module ## What changes were proposed in this pull request? This issue fixes or hides 181 Java linter errors introduced by SPARK-14987 which copied

spark git commit: [SPARK-15053][BUILD] Fix Java Lint errors on Hive-Thriftserver module

Repository: spark Updated Branches: refs/heads/master dfd9723dd -> a74445707 [SPARK-15053][BUILD] Fix Java Lint errors on Hive-Thriftserver module ## What changes were proposed in this pull request? This issue fixes or hides 181 Java linter errors introduced by SPARK-14987 which copied hive

spark git commit: [MINOR][DOCS] Fix type Information in Quick Start and Programming Guide

Repository: spark Updated Branches: refs/heads/master f10ae4b1e -> dfd9723dd [MINOR][DOCS] Fix type Information in Quick Start and Programming Guide Author: Sandeep Singh Closes #12841 from techaddict/improve_docs_1. Project:

spark git commit: [MINOR][DOCS] Fix type Information in Quick Start and Programming Guide

Repository: spark Updated Branches: refs/heads/branch-2.0 27efd92e3 -> 07a02e8bb [MINOR][DOCS] Fix type Information in Quick Start and Programming Guide Author: Sandeep Singh Closes #12841 from techaddict/improve_docs_1. (cherry picked from commit

spark git commit: [SPARK-6717][ML] Clear shuffle files after checkpointing in ALS

Repository: spark Updated Branches: refs/heads/branch-2.0 52308103e -> 27efd92e3 [SPARK-6717][ML] Clear shuffle files after checkpointing in ALS ## What changes were proposed in this pull request? When ALS is run with a checkpoint interval, during the checkpoint materialize the current

spark git commit: [SPARK-6717][ML] Clear shuffle files after checkpointing in ALS

Repository: spark Updated Branches: refs/heads/master d8f528ceb -> f10ae4b1e [SPARK-6717][ML] Clear shuffle files after checkpointing in ALS ## What changes were proposed in this pull request? When ALS is run with a checkpoint interval, during the checkpoint materialize the current state