spark git commit: [SPARK-14426][SQL] Merge PerserUtils and ParseUtils

Repository: spark Updated Branches: refs/heads/master 90ca18448 -> 10494feae [SPARK-14426][SQL] Merge PerserUtils and ParseUtils ## What changes were proposed in this pull request? We have ParserUtils and ParseUtils which are both utility collections for use during the parsing process.

spark git commit: [SPARK-14468] Always enable OutputCommitCoordinator

ugh `spark.hadoop.outputCommitCoordination.enabled`, but they really shouldn't... ## How was this patch tested? `OutputCommitCoordinator*Suite` Author: Andrew Or <and...@databricks.com> Closes #12244 from andrewor14/always-occ. (cherry picked from commit 3e29e372ff518827bae9dcd26087946

spark git commit: [DOCS][MINOR] Remove sentence about Mesos not supporting cluster mode.

Repository: spark Updated Branches: refs/heads/branch-1.6 dca0d9a48 -> 8a94a59f9 [DOCS][MINOR] Remove sentence about Mesos not supporting cluster mode. Docs change to remove the sentence about Mesos not supporting cluster mode. It was not. Author: Michael Gummelt

spark git commit: [SPARK-14243][CORE][BACKPORT-1.6] update task metrics when removing blocks

Repository: spark Updated Branches: refs/heads/branch-1.6 285cb9c66 -> cfe9f02ca [SPARK-14243][CORE][BACKPORT-1.6] update task metrics when removing blocks ## What changes were proposed in this pull request? This patch try to update the `updatedBlockStatuses ` when removing blocks, making

[2/2] spark git commit: [SPARK-14123][SPARK-14384][SQL] Handle CreateFunction/DropFunction

[SPARK-14123][SPARK-14384][SQL] Handle CreateFunction/DropFunction ## What changes were proposed in this pull request? This PR implements CreateFunction and DropFunction commands. Besides implementing these two commands, we also change how to manage functions. Here are the main changes. *

[1/2] spark git commit: [SPARK-14123][SPARK-14384][SQL] Handle CreateFunction/DropFunction

Repository: spark Updated Branches: refs/heads/master bc36df127 -> 72544d6f2 http://git-wip-us.apache.org/repos/asf/spark/blob/72544d6f/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala -- diff

spark git commit: [SPARK-14391][LAUNCHER] Increase test timeouts.

Repository: spark Updated Branches: refs/heads/master 5a4b11a90 -> de4792605 [SPARK-14391][LAUNCHER] Increase test timeouts. Most of the time tests should still pass really quickly; it's just when machines are overloaded that the tests may take a little time, but that's still preferable over

spark git commit: [SPARK-12133][STREAMING] Streaming dynamic allocation

Repository: spark Updated Branches: refs/heads/master de4792605 -> 9af5423ec [SPARK-12133][STREAMING] Streaming dynamic allocation ## What changes were proposed in this pull request? Added a new Executor Allocation Manager for the Streaming scheduler for doing Streaming Dynamic Allocation.

spark git commit: [SPARK-14444][BUILD] Add a new scalastyle `NoScalaDoc` to prevent ScalaDoc-style multiline comments

Repository: spark Updated Branches: refs/heads/master 457e58bef -> d717ae1fd [SPARK-1][BUILD] Add a new scalastyle `NoScalaDoc` to prevent ScalaDoc-style multiline comments ## What changes were proposed in this pull request? According to the [Spark Code Style

spark git commit: [SPARK-14424][BUILD][DOCS] Update the build docs to switch from assembly to package and add a no…

Repository: spark Updated Branches: refs/heads/master 9af5423ec -> 457e58bef [SPARK-14424][BUILD][DOCS] Update the build docs to switch from assembly to package and add a no… ## What changes were proposed in this pull request? Change our build docs & shell scripts to that developers are

spark git commit: [SPARK-13112][CORE] Make sure RegisterExecutorResponse arrive before LaunchTask

Repository: spark Updated Branches: refs/heads/master c4bb02abf -> f1def573f [SPARK-13112][CORE] Make sure RegisterExecutorResponse arrive before LaunchTask ## What changes were proposed in this pull request? Send `RegisterExecutorResponse` using `executorRef` in order to make sure

spark git commit: [SPARK-14252] Executors do not try to download remote cached blocks

Repository: spark Updated Branches: refs/heads/master 68be5b9e8 -> 78c1076d0 [SPARK-14252] Executors do not try to download remote cached blocks ## What changes were proposed in this pull request? As mentioned in the ticket this was because one get path in the refactored `BlockManager` did

spark git commit: [SPARK-14396][SQL] Throw Exceptions for DDLs of Partitioned Views

Repository: spark Updated Branches: refs/heads/master 48467f4eb -> 68be5b9e8 [SPARK-14396][SQL] Throw Exceptions for DDLs of Partitioned Views What changes were proposed in this pull request? Because the concept of partitioning is associated with physical tables, we disable all the

spark git commit: [SPARK-14455][STREAMING] Fix NPE in allocatedExecutors when calling in receiver-less scenario

Repository: spark Updated Branches: refs/heads/master 3fb09afd5 -> 2c95e4e96 [SPARK-14455][STREAMING] Fix NPE in allocatedExecutors when calling in receiver-less scenario ## What changes were proposed in this pull request? When calling `ReceiverTracker#allocatedExecutors` in receiver-less

spark git commit: [SPARK-14357][CORE] Properly handle the root cause being a commit denied exception

Repository: spark Updated Branches: refs/heads/branch-1.5 4f9d1f8c7 -> 1e61ff4ca [SPARK-14357][CORE] Properly handle the root cause being a commit denied exception ## What changes were proposed in this pull request? When deciding whether a CommitDeniedException caused a task to fail,

spark git commit: [SPARK-14506][SQL] HiveClientImpl's toHiveTable misses a table property for external tables

Repository: spark Updated Branches: refs/heads/master aea30a1a9 -> 3fb09afd5 [SPARK-14506][SQL] HiveClientImpl's toHiveTable misses a table property for external tables ## What changes were proposed in this pull request? For an external table's metadata (in Hive's representation), its table

spark git commit: [SPARK-14416][CORE] Add thread-safe comments for CoarseGrainedSchedulerBackend's fields

Repository: spark Updated Branches: refs/heads/master adbfdb878 -> 48467f4eb [SPARK-14416][CORE] Add thread-safe comments for CoarseGrainedSchedulerBackend's fields ## What changes were proposed in this pull request? While I was reviewing #12078, I found most of

spark git commit: [SPARK-14468] Always enable OutputCommitCoordinator

ugh `spark.hadoop.outputCommitCoordination.enabled`, but they really shouldn't... ## How was this patch tested? `OutputCommitCoordinator*Suite` Author: Andrew Or <and...@databricks.com> Closes #12244 from andrewor14/always-occ. (cherry picked from commit 3e29e372ff518827bae9dcd26087946

spark git commit: [SPARK-14468] Always enable OutputCommitCoordinator

ugh `spark.hadoop.outputCommitCoordination.enabled`, but they really shouldn't... ## How was this patch tested? `OutputCommitCoordinator*Suite` Author: Andrew Or <and...@databricks.com> Closes #12244 from andrewor14/always-occ. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Com

[2/3] spark git commit: [SPARK-14014][SQL] Integrate session catalog (attempt #2)

http://git-wip-us.apache.org/repos/asf/spark/blob/20ddf5fd/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala -- diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala

[1/3] spark git commit: [SPARK-14014][SQL] Integrate session catalog (attempt #2)

Repository: spark Updated Branches: refs/heads/master 1c70b7650 -> 20ddf5fdd http://git-wip-us.apache.org/repos/asf/spark/blob/20ddf5fd/sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcQuerySuite.scala -- diff --git

[3/3] spark git commit: [SPARK-14014][SQL] Integrate session catalog (attempt #2)

` and `HiveContextSuite`. Author: Andrew Or <and...@databricks.com> Closes #11938 from andrewor14/session-catalog-again. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/20ddf5fd Tree: http://git-wip-us.apache.org/repos/asf/spar

[1/2] spark git commit: Revert "[SPARK-14014][SQL] Replace existing catalog with SessionCatalog"

Repository: spark Updated Branches: refs/heads/master cf823bead -> c44d140ca http://git-wip-us.apache.org/repos/asf/spark/blob/c44d140c/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala -- diff --git

[2/2] spark git commit: Revert "[SPARK-14014][SQL] Replace existing catalog with SessionCatalog"

Revert "[SPARK-14014][SQL] Replace existing catalog with SessionCatalog" This reverts commit 5dfc01976bb0d72489620b4f32cc12d620bb6260. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c44d140c Tree:

spark git commit: [SPARK-13689][SQL] Move helper things in CatalystQl to new utils object

rya's changes in #11048. It prefaces the bigger fix for SPARK-13139 to make the diff of that patch smaller. ## How was this patch tested? No change in functionality, so just Jenkins. Author: Andrew Or <and...@databricks.com> Closes #11529 from andrewor14/parser-utils. Project: http://gi

spark git commit: [SPARK-13659] Refactor BlockStore put*() APIs to remove returnValues

Repository: spark Updated Branches: refs/heads/master 017cdf2be -> e52e597db [SPARK-13659] Refactor BlockStore put*() APIs to remove returnValues In preparation for larger refactoring, this patch removes the confusing `returnValues` option from the BlockStore put() APIs: returning the value

[2/2] spark git commit: [SPARK-12817] Add BlockManager.getOrElseUpdate and remove CacheManager

andrewor14 and nongli for review. Note that this changes the locking semantics of a couple of internal BlockManager methods (`doPut()` and `lockNewBlockForWriting`), so please pay attention to the Scaladoc changes and new test cases for those methods. Author: Josh Rosen <joshro...@databricks.com>

[1/3] spark git commit: [SPARK-13633][SQL] Move things into catalyst.parser package

Repository: spark Updated Branches: refs/heads/master 83302c3bf -> b7d414742 http://git-wip-us.apache.org/repos/asf/spark/blob/b7d41474/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DataTypeParserSuite.scala

[2/3] spark git commit: [SPARK-13633][SQL] Move things into catalyst.parser package

http://git-wip-us.apache.org/repos/asf/spark/blob/b7d41474/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/CatalystQl.scala -- diff --git

spark git commit: [SPARK-10570][CORE] Add version info to json api

Repository: spark Updated Branches: refs/heads/master 15c0b0006 -> d26c42982 [SPARK-10570][CORE] Add version info to json api Add a new api endpoint `/api/v1/version` to retrieve various version info. This PR only adds support for finding the current spark version, however other version

spark git commit: [SPARK-14232][WEBUI] Fix event timeline display issue when an executor is removed with a multiple line reason.

Repository: spark Updated Branches: refs/heads/branch-1.6 c2ce247ea -> 84ad2544f [SPARK-14232][WEBUI] Fix event timeline display issue when an executor is removed with a multiple line reason. ## What changes were proposed in this pull request? The event timeline doesn't show on job page if

spark git commit: [SPARK-14232][WEBUI] Fix event timeline display issue when an executor is removed with a multiple line reason.

Repository: spark Updated Branches: refs/heads/master d2a819a63 -> 15c0b0006 [SPARK-14232][WEBUI] Fix event timeline display issue when an executor is removed with a multiple line reason. ## What changes were proposed in this pull request? The event timeline doesn't show on job page if an

spark git commit: [SPARK-13845][CORE][BACKPORT-1.6] Using onBlockUpdated to replace onTaskEnd avioding driver OOM

Repository: spark Updated Branches: refs/heads/branch-1.6 a7579444d -> c2ce247ea [SPARK-13845][CORE][BACKPORT-1.6] Using onBlockUpdated to replace onTaskEnd avioding driver OOM ## What changes were proposed in this pull request? We have a streaming job using `FlumePollInputStream` always

spark git commit: [SPARK-13796] Redirect error message to logWarning

Repository: spark Updated Branches: refs/heads/master 358692932 -> ac1b8b302 [SPARK-13796] Redirect error message to logWarning ## What changes were proposed in this pull request? Redirect error message to logWarning ## How was this patch tested? Unit tests, manual tests JoshRosen

spark git commit: [SPARK-11327][MESOS] Dispatcher does not respect all args from the Submit request

Repository: spark Updated Branches: refs/heads/master 0abee534f -> 10508f36a [SPARK-11327][MESOS] Dispatcher does not respect all args from the Submit request Supersedes https://github.com/apache/spark/pull/9752 Author: Jo Voordeckers Author: Iulian Dragos

spark git commit: [SPARK-14304][SQL][TESTS] Fix tests that don't create temp files in the `java.io.tmpdir` folder

Repository: spark Updated Branches: refs/heads/master 3cfbeb70b -> e78540282 [SPARK-14304][SQL][TESTS] Fix tests that don't create temp files in the `java.io.tmpdir` folder ## What changes were proposed in this pull request? If I press `CTRL-C` when running these tests, the temp files will

spark git commit: [SPARK-14069][SQL] Improve SparkStatusTracker to also track executor information

Repository: spark Updated Branches: refs/heads/master 4d93b653f -> 0abee534f [SPARK-14069][SQL] Improve SparkStatusTracker to also track executor information ## What changes were proposed in this pull request? Track executor information like host and port, cache size, running tasks. TODO:

spark git commit: [SPARK-14243][CORE] update task metrics when removing blocks

Repository: spark Updated Branches: refs/heads/master 446c45bd8 -> 8a333d2da [SPARK-14243][CORE] update task metrics when removing blocks ## What changes were proposed in this pull request? This PR try to use `incUpdatedBlockStatuses ` to update the `updatedBlockStatuses ` when removing

spark git commit: [SPARK-14182][SQL] Parse DDL Command: Alter View

Repository: spark Updated Branches: refs/heads/master ac1b8b302 -> 446c45bd8 [SPARK-14182][SQL] Parse DDL Command: Alter View This PR is to provide native parsing support for DDL commands: `Alter View`. Since its AST trees are highly similar to `Alter Table`. Thus, both implementation are

spark git commit: [Docs] Update monitoring.md to accurately describe the history server

Repository: spark Updated Branches: refs/heads/master 8a333d2da -> 4d93b653f [Docs] Update monitoring.md to accurately describe the history server It looks like the docs were recently updated to reflect the History Server's support for incomplete applications, but they still had wording that

spark git commit: [SPARK-14364][SPARK] HeartbeatReceiver object should be private

Repository: spark Updated Branches: refs/heads/master 5743c6476 -> 27dad6f65 [SPARK-14364][SPARK] HeartbeatReceiver object should be private ## What changes were proposed in this pull request? It's a mistake that HeartbeatReceiver object was made public in Spark 1.x. ## How was this patch

spark git commit: [SPARK-11327][MESOS] Backport dispatcher does not respect all args f…

Repository: spark Updated Branches: refs/heads/branch-1.6 f12f11e57 -> 91530b09e [SPARK-11327][MESOS] Backport dispatcher does not respect all args f… Backport for https://github.com/apache/spark/pull/10370 andrewor14 Author: Jo Voordeckers <jo.voordeck...@gmail.com> Closes #1

spark git commit: [SPARK-14358] Change SparkListener from a trait to an abstract class

Repository: spark Updated Branches: refs/heads/master 27dad6f65 -> 714390470 [SPARK-14358] Change SparkListener from a trait to an abstract class ## What changes were proposed in this pull request? Scala traits are difficult to maintain binary compatibility on, and as a result we had to

spark git commit: [SPARK-14124][SQL] Implement Database-related DDL Commands

nds. In the Database-related DDL commands, we will issue an error exception for `ALTER (DATABASE|SCHEMA) database_name SET OWNER [USER|ROLE] user_or_role`. cc yhuai andrewor14 rxin Could you review the changes? Is it in the right direction? Thanks! How was this patch tested? Added a few t

spark git commit: [SPARK-14721][SQL] Remove HiveContext (part 2)

ext in the REPL (fixed by #12589) ## How was this patch tested? No change in functionality. Author: Andrew Or <and...@databricks.com> Closes #12585 from andrewor14/delete-hive-context. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/

spark git commit: [MINOR] Comment whitespace changes in #12553

Repository: spark Updated Branches: refs/heads/master a2e8d4fdd -> ef6be7bed [MINOR] Comment whitespace changes in #12553 Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/ef6be7be Tree:

spark git commit: [SPARK-14988][PYTHON] SparkSession API follow-ups

rew Or <and...@databricks.com> Closes #12784 from andrewor14/python-followup. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d33e3d57 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/d33e3d57 Diff: http://git-wip-us.apache.org/repos/

spark git commit: [SPARK-15019][SQL] Propagate all Spark Confs to HiveConf created in HiveClientImpl

Repository: spark Updated Branches: refs/heads/master dcfaeadea -> b33d6b728 [SPARK-15019][SQL] Propagate all Spark Confs to HiveConf created in HiveClientImpl ## What changes were proposed in this pull request? This PR makes two changes: 1. We will propagate Spark Confs to HiveConf created

spark git commit: [MINOR] Follow-up to #12625

rew Or <and...@databricks.com> Closes #12686 from andrewor14/visibility. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/2a3d39f4 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/2a3d39f4 Diff: http://git-wip-us.a

spark git commit: [SPARK-15417][SQL][PYTHON] PySpark shell always uses in-memory catalog

rew Or <and...@databricks.com> Closes #13203 from andrewor14/fix-pyspark-shell. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c32b1b16 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c32b1b16 Diff: http://git-wip-us.apache.

spark git commit: [SPARK-15417][SQL][PYTHON] PySpark shell always uses in-memory catalog

rew Or <and...@databricks.com> Closes #13203 from andrewor14/fix-pyspark-shell. (cherry picked from commit c32b1b162e7e5ecc5c823f79ba9f23cbd1407dbf) Signed-off-by: Andrew Or <and...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apach

spark git commit: [SPARK-15421][SQL] Validate DDL property values

row exceptions instead. ## How was this patch tested? `DDLCommandSuite` Author: Andrew Or <and...@databricks.com> Closes #13205 from andrewor14/ddl-prop-values. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/2573

spark git commit: [SPARK-15421][SQL] Validate DDL property values

uld throw exceptions instead. ## How was this patch tested? `DDLCommandSuite` Author: Andrew Or <and...@databricks.com> Closes #13205 from andrewor14/ddl-prop-values. (cherry picked from commit 257375019266ab9e3c320e33026318cc31f58ada) Signed-off-by: Andrew Or <and...@databricks.com>

spark git commit: [HOTFIX] Add back intended change from SPARK-15392

Repository: spark Updated Branches: refs/heads/branch-2.0 2ef645724 -> 612866473 [HOTFIX] Add back intended change from SPARK-15392 This was accidentally reverted in f8d0177. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

spark git commit: [HOTFIX] Test compilation error from 52b967f

Repository: spark Updated Branches: refs/heads/branch-2.0 2126fb0c2 -> 1fc0f95eb [HOTFIX] Test compilation error from 52b967f Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/1fc0f95e Tree:

spark git commit: [SPARK-15279][SQL] Catch conflicting SerDe when creating table

tch: - `ROW FORMAT DELIMITED` is only compatible with `TEXTFILE` - `ROW FORMAT SERDE` is only compatible with `TEXTFILE`, `RCFILE` and `SEQUENCEFILE` ## How was this patch tested? New tests in `DDLCommandSuite`. Author: Andrew Or <and...@databricks.com> Closes #13068 from andrewor14/r

spark git commit: [SPARK-15279][SQL] Catch conflicting SerDe when creating table

tch: - `ROW FORMAT DELIMITED` is only compatible with `TEXTFILE` - `ROW FORMAT SERDE` is only compatible with `TEXTFILE`, `RCFILE` and `SEQUENCEFILE` ## How was this patch tested? New tests in `DDLCommandSuite`. Author: Andrew Or <and...@databricks.com> Closes #13068 from andrewor14/r

spark git commit: Revert "[SPARK-15285][SQL] Generated SpecificSafeProjection.apply method grows beyond 64 KB"

Repository: spark Updated Branches: refs/heads/branch-2.0 d0bcec157 -> 1890f5fdf Revert "[SPARK-15285][SQL] Generated SpecificSafeProjection.apply method grows beyond 64 KB" This reverts commit d0bcec157d2bd2ed4eff848f831841bef4745904. Project:

spark git commit: Revert "[SPARK-15285][SQL] Generated SpecificSafeProjection.apply method grows beyond 64 KB"

Repository: spark Updated Branches: refs/heads/master fa244e5a9 -> de726b0d5 Revert "[SPARK-15285][SQL] Generated SpecificSafeProjection.apply method grows beyond 64 KB" This reverts commit fa244e5a90690d6a31be50f2aa203ae1a2e9a1cf. Project: http://git-wip-us.apache.org/repos/asf/spark/repo

spark git commit: [SPARK-15311][SQL] Disallow DML on Regular Tables when Using In-Memory Catalog

Repository: spark Updated Branches: refs/heads/master 01659bc50 -> 5afd927a4 [SPARK-15311][SQL] Disallow DML on Regular Tables when Using In-Memory Catalog What changes were proposed in this pull request? So far, when using In-Memory Catalog, we allow DDL operations for the tables.

spark git commit: [SPARK-15311][SQL] Disallow DML on Regular Tables when Using In-Memory Catalog

Repository: spark Updated Branches: refs/heads/branch-2.0 3def56120 -> 220b9a08e [SPARK-15311][SQL] Disallow DML on Regular Tables when Using In-Memory Catalog What changes were proposed in this pull request? So far, when using In-Memory Catalog, we allow DDL operations for the tables.

spark git commit: [SPARK-15464][ML][MLLIB][SQL][TESTS] Replace SQLContext and SparkContext with SparkSession using builder pattern in python test code

Repository: spark Updated Branches: refs/heads/master 5afd927a4 -> a15ca5533 [SPARK-15464][ML][MLLIB][SQL][TESTS] Replace SQLContext and SparkContext with SparkSession using builder pattern in python test code ## What changes were proposed in this pull request? Replace SQLContext and

spark git commit: [SPARK-15464][ML][MLLIB][SQL][TESTS] Replace SQLContext and SparkContext with SparkSession using builder pattern in python test code

Repository: spark Updated Branches: refs/heads/branch-2.0 220b9a08e -> f3162b96d [SPARK-15464][ML][MLLIB][SQL][TESTS] Replace SQLContext and SparkContext with SparkSession using builder pattern in python test code ## What changes were proposed in this pull request? Replace SQLContext and

spark git commit: [SPARK-15397][SQL] fix string udf locate as hive

Repository: spark Updated Branches: refs/heads/master de726b0d5 -> d642b2735 [SPARK-15397][SQL] fix string udf locate as hive ## What changes were proposed in this pull request? in hive, `locate("aa", "aaa", 0)` would yield 0, `locate("aa", "aaa", 1)` would yield 1 and `locate("aa", "aaa",

spark git commit: [SPARK-15397][SQL] fix string udf locate as hive

Repository: spark Updated Branches: refs/heads/branch-2.0 1890f5fdf -> 6adbc0613 [SPARK-15397][SQL] fix string udf locate as hive ## What changes were proposed in this pull request? in hive, `locate("aa", "aaa", 0)` would yield 0, `locate("aa", "aaa", 1)` would yield 1 and `locate("aa",

spark git commit: [SPARK-15388][SQL] Fix spark sql CREATE FUNCTION with hive 1.2.1

Repository: spark Updated Branches: refs/heads/master a313a5ae7 -> 784cc07d1 [SPARK-15388][SQL] Fix spark sql CREATE FUNCTION with hive 1.2.1 ## What changes were proposed in this pull request? spark.sql("CREATE FUNCTION myfunc AS 'com.haizhi.bdp.udf.UDFGetGeoCode'") throws

spark git commit: [SPARK-15388][SQL] Fix spark sql CREATE FUNCTION with hive 1.2.1

Repository: spark Updated Branches: refs/heads/branch-2.0 988d4dbf4 -> 1bb0aa4b0 [SPARK-15388][SQL] Fix spark sql CREATE FUNCTION with hive 1.2.1 ## What changes were proposed in this pull request? spark.sql("CREATE FUNCTION myfunc AS 'com.haizhi.bdp.udf.UDFGetGeoCode'") throws

spark git commit: [MINOR][CORE][TEST] Update obsolete `takeSample` test case.

Repository: spark Updated Branches: refs/heads/branch-2.0 1bb0aa4b0 -> 2574abea0 [MINOR][CORE][TEST] Update obsolete `takeSample` test case. ## What changes were proposed in this pull request? This PR fixes some obsolete comments and assertion in `takeSample` testcase of `RDDSuite.scala`.

spark git commit: [MINOR][CORE][TEST] Update obsolete `takeSample` test case.

Repository: spark Updated Branches: refs/heads/master 784cc07d1 -> be99a99fe [MINOR][CORE][TEST] Update obsolete `takeSample` test case. ## What changes were proposed in this pull request? This PR fixes some obsolete comments and assertion in `takeSample` testcase of `RDDSuite.scala`. ##

spark git commit: [SPARK-15031][EXAMPLE] Use SparkSession in examples

LIB` is not recommended to use now, so examples in `MLLIB` are ignored in this PR. `StreamingContext` can not be directly obtained from `SparkSession`, so example in `Streaming` are ignored too. cc andrewor14 ## How was this patch tested? manual tests with spark-submit Author: Zheng RuiFeng <

spark git commit: [SPARK-15031][EXAMPLE] Use SparkSession in examples

LIB` is not recommended to use now, so examples in `MLLIB` are ignored in this PR. `StreamingContext` can not be directly obtained from `SparkSession`, so example in `Streaming` are ignored too. cc andrewor14 ## How was this patch tested? manual tests with spark-submit Author: Zheng RuiFeng <

spark git commit: [SPARK-15456][PYSPARK] Fixed PySpark shell context initialization when HiveConf not present

Repository: spark Updated Branches: refs/heads/branch-2.0 684167862 -> c7e013f18 [SPARK-15456][PYSPARK] Fixed PySpark shell context initialization when HiveConf not present ## What changes were proposed in this pull request? When PySpark shell cannot find HiveConf, it will fallback to

spark git commit: [SPARK-15456][PYSPARK] Fixed PySpark shell context initialization when HiveConf not present

Repository: spark Updated Branches: refs/heads/master 127bf1bb0 -> 021c19702 [SPARK-15456][PYSPARK] Fixed PySpark shell context initialization when HiveConf not present ## What changes were proposed in this pull request? When PySpark shell cannot find HiveConf, it will fallback to create a

spark git commit: [SPARK-15387][SQL] SessionCatalog in SimpleAnalyzer does not need to make database directory.

Repository: spark Updated Branches: refs/heads/master ad182086c -> faafd1e9d [SPARK-15387][SQL] SessionCatalog in SimpleAnalyzer does not need to make database directory. ## What changes were proposed in this pull request? After #12871 is fixed, we are forced to make `/user/hive/warehouse`

spark git commit: [SPARK-14603][SQL][FOLLOWUP] Verification of Metadata Operations by Session Catalog

Repository: spark Updated Branches: refs/heads/master 9308bf119 -> ef7a5e0bc [SPARK-14603][SQL][FOLLOWUP] Verification of Metadata Operations by Session Catalog What changes were proposed in this pull request? This follow-up PR is to address the remaining comments in

spark git commit: [SPARK-15322][SQL][FOLLOW-UP] Update deprecated accumulator usage into accumulatorV2

Repository: spark Updated Branches: refs/heads/branch-2.0 9c817d027 -> 554e0f30a [SPARK-15322][SQL][FOLLOW-UP] Update deprecated accumulator usage into accumulatorV2 ## What changes were proposed in this pull request? This PR corrects another case that uses deprecated

spark git commit: [SPARK-14603][SQL][FOLLOWUP] Verification of Metadata Operations by Session Catalog

Repository: spark Updated Branches: refs/heads/branch-2.0 2604eadcf -> 496f6d0fc [SPARK-14603][SQL][FOLLOWUP] Verification of Metadata Operations by Session Catalog What changes were proposed in this pull request? This follow-up PR is to address the remaining comments in

spark git commit: [SPARK-15300] Fix writer lock conflict when remove a block

Repository: spark Updated Branches: refs/heads/master ef7a5e0bc -> ad182086c [SPARK-15300] Fix writer lock conflict when remove a block ## What changes were proposed in this pull request? A writer lock could be acquired when 1) create a new block 2) remove a block 3) evict a block to disk.

spark git commit: [SPARK-15387][SQL] SessionCatalog in SimpleAnalyzer does not need to make database directory.

Repository: spark Updated Branches: refs/heads/branch-2.0 96a473a11 -> 9c817d027 [SPARK-15387][SQL] SessionCatalog in SimpleAnalyzer does not need to make database directory. ## What changes were proposed in this pull request? After #12871 is fixed, we are forced to make

spark git commit: [SPARK-14346][SQL] Lists unsupported Hive features in SHOW CREATE TABLE output

Repository: spark Updated Branches: refs/heads/branch-2.0 97fd9a09c -> 4f8639f9d [SPARK-14346][SQL] Lists unsupported Hive features in SHOW CREATE TABLE output ## What changes were proposed in this pull request? This PR is a follow-up of #13079. It replaces `hasUnsupportedFeatures: Boolean`

spark git commit: [SPARK-14346][SQL] Lists unsupported Hive features in SHOW CREATE TABLE output

Repository: spark Updated Branches: refs/heads/master e71cd96bf -> 6ac1c3a04 [SPARK-14346][SQL] Lists unsupported Hive features in SHOW CREATE TABLE output ## What changes were proposed in this pull request? This PR is a follow-up of #13079. It replaces `hasUnsupportedFeatures: Boolean` in

spark git commit: [SPARK-15317][CORE] Don't store accumulators for every task in listeners

Repository: spark Updated Branches: refs/heads/master 6ac1c3a04 -> 4e3cb7a5d [SPARK-15317][CORE] Don't store accumulators for every task in listeners ## What changes were proposed in this pull request? In general, the Web UI doesn't need to store the Accumulator/AccumulableInfo for every

spark git commit: [SPARK-15392][SQL] fix default value of size estimation of logical plan

Repository: spark Updated Branches: refs/heads/master 4e3cb7a5d -> 5ccecc078 [SPARK-15392][SQL] fix default value of size estimation of logical plan ## What changes were proposed in this pull request? We use autoBroadcastJoinThreshold + 1L as the default value of size estimation, that is

spark git commit: [SPARK-15317][CORE] Don't store accumulators for every task in listeners

Repository: spark Updated Branches: refs/heads/branch-2.0 4f8639f9d -> 62e5158f1 [SPARK-15317][CORE] Don't store accumulators for every task in listeners ## What changes were proposed in this pull request? In general, the Web UI doesn't need to store the Accumulator/AccumulableInfo for

spark git commit: [SPARK-15392][SQL] fix default value of size estimation of logical plan

Repository: spark Updated Branches: refs/heads/branch-2.0 62e5158f1 -> d1b5df83d [SPARK-15392][SQL] fix default value of size estimation of logical plan ## What changes were proposed in this pull request? We use autoBroadcastJoinThreshold + 1L as the default value of size estimation, that

spark git commit: [SPARK-15093][SQL] create/delete/rename directory for InMemoryCatalog operations if needed

Repository: spark Updated Branches: refs/heads/branch-2.0 fb73663db -> 5cdb7bea5 [SPARK-15093][SQL] create/delete/rename directory for InMemoryCatalog operations if needed ## What changes were proposed in this pull request? following operations have file system operation now: 1. CREATE

spark git commit: [SPARK-15093][SQL] create/delete/rename directory for InMemoryCatalog operations if needed

Repository: spark Updated Branches: refs/heads/master ee3b17156 -> beb16ec55 [SPARK-15093][SQL] create/delete/rename directory for InMemoryCatalog operations if needed ## What changes were proposed in this pull request? following operations have file system operation now: 1. CREATE

spark git commit: [SPARK-15199][SQL] Disallow Dropping Build-in Functions

Repository: spark Updated Branches: refs/heads/master beb16ec55 -> b1e01fd51 [SPARK-15199][SQL] Disallow Dropping Build-in Functions What changes were proposed in this pull request? As Hive and the major RDBMS behave, the built-in functions are not allowed to drop. In the current

spark git commit: [SPARK-15199][SQL] Disallow Dropping Build-in Functions

Repository: spark Updated Branches: refs/heads/branch-2.0 5cdb7bea5 -> 29bc8d2ec [SPARK-15199][SQL] Disallow Dropping Build-in Functions What changes were proposed in this pull request? As Hive and the major RDBMS behave, the built-in functions are not allowed to drop. In the current

spark git commit: [MINOR][DOCS] Remove remaining sqlContext in documentation at examples

Repository: spark Updated Branches: refs/heads/master 671b382a8 -> 2992a215c [MINOR][DOCS] Remove remaining sqlContext in documentation at examples This PR removes `sqlContext` in examples. Actual usage was all replaced in https://github.com/apache/spark/pull/12809 but there are some in

spark git commit: [MINOR][DOCS] Remove remaining sqlContext in documentation at examples

Repository: spark Updated Branches: refs/heads/branch-2.0 de6afc887 -> 6371197c6 [MINOR][DOCS] Remove remaining sqlContext in documentation at examples This PR removes `sqlContext` in examples. Actual usage was all replaced in https://github.com/apache/spark/pull/12809 but there are some in

spark git commit: [SPARK-15223][DOCS] fix wrongly named config reference

Repository: spark Updated Branches: refs/heads/master 2992a215c -> 65b4ab281 [SPARK-15223][DOCS] fix wrongly named config reference ## What changes were proposed in this pull request? The configuration setting `spark.executor.logs.rolling.size.maxBytes` was changed to

spark git commit: [SPARK-15223][DOCS] fix wrongly named config reference

Repository: spark Updated Branches: refs/heads/branch-2.0 6371197c6 -> 1b4e99ff1 [SPARK-15223][DOCS] fix wrongly named config reference ## What changes were proposed in this pull request? The configuration setting `spark.executor.logs.rolling.size.maxBytes` was changed to

spark git commit: [SPARK-15225][SQL] Replace SQLContext with SparkSession in Encoder documentation

Repository: spark Updated Branches: refs/heads/branch-2.0 1b4e99ff1 -> 8f0ed2891 [SPARK-15225][SQL] Replace SQLContext with SparkSession in Encoder documentation `Encoder`'s doc mentions `sqlContext.implicits._`. We should use `sparkSession.implicits._` instead now. Only doc update.

spark git commit: [SPARK-15225][SQL] Replace SQLContext with SparkSession in Encoder documentation

Repository: spark Updated Branches: refs/heads/master 65b4ab281 -> e083db2e9 [SPARK-15225][SQL] Replace SQLContext with SparkSession in Encoder documentation `Encoder`'s doc mentions `sqlContext.implicits._`. We should use `sparkSession.implicits._` instead now. Only doc update. Author:

spark git commit: [SPARK-15067][YARN] YARN executors are launched with fixed perm gen size

Repository: spark Updated Branches: refs/heads/branch-2.0 8f0ed2891 -> 3c6f686f9 [SPARK-15067][YARN] YARN executors are launched with fixed perm gen size ## What changes were proposed in this pull request? Look for MaxPermSize arguments anywhere in an arg, to account for quoted args. See

spark git commit: [SPARK-15067][YARN] YARN executors are launched with fixed perm gen size

Repository: spark Updated Branches: refs/heads/master e083db2e9 -> 6747171eb [SPARK-15067][YARN] YARN executors are launched with fixed perm gen size ## What changes were proposed in this pull request? Look for MaxPermSize arguments anywhere in an arg, to account for quoted args. See JIRA

spark git commit: [MINOR][SQL] Enhance the exception message if checkpointLocation is not set

Repository: spark Updated Branches: refs/heads/master 6747171eb -> ee6a8d7ea [MINOR][SQL] Enhance the exception message if checkpointLocation is not set Enhance the exception message when `checkpointLocation` is not set, previously the message is: ``` java.util.NoSuchElementException:

spark git commit: [MINOR][SQL] Enhance the exception message if checkpointLocation is not set

Repository: spark Updated Branches: refs/heads/branch-2.0 3c6f686f9 -> 1d5615857 [MINOR][SQL] Enhance the exception message if checkpointLocation is not set Enhance the exception message when `checkpointLocation` is not set, previously the message is: ``` java.util.NoSuchElementException:

spark git commit: [SAPRK-15220][UI] add hyperlink to running application and completed application

Repository: spark Updated Branches: refs/heads/branch-2.0 1d5615857 -> c6d23b660 [SAPRK-15220][UI] add hyperlink to running application and completed application ## What changes were proposed in this pull request? Add hyperlink to "running application" and "completed application", so user can

<    8   9   10   11   12   13   14   15   >