spark git commit: [SPARK-15552][SQL] Remove unnecessary private[sql] methods in SparkSession

Repository: spark Updated Branches: refs/heads/master 594a1bf20 -> 0f61d6efb [SPARK-15552][SQL] Remove unnecessary private[sql] methods in SparkSession ## What changes were proposed in this pull request? SparkSession has a list of unnecessary private[sql] methods. These methods cause some

spark git commit: [SPARK-15538][SPARK-15539][SQL] Truncate table fixes round 2

ION`) (2) Disallow truncating external tables or views ## How was this patch tested? `DDLSuite` Author: Andrew Or <and...@databricks.com> Closes #13315 from andrewor14/truncate-table. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/

spark git commit: [SPARK-15538][SPARK-15539][SQL] Truncate table fixes round 2

out `PARTITION`) (2) Disallow truncating external tables or views ## How was this patch tested? `DDLSuite` Author: Andrew Or <and...@databricks.com> Closes #13315 from andrewor14/truncate-table. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/

spark git commit: [SPARK-15646][SQL] When spark.sql.hive.convertCTAS is true, the conversion rule needs to respect TEXTFILE/SEQUENCEFILE format and the user-defined location

Repository: spark Updated Branches: refs/heads/master c8fb776d4 -> 6dddb70c3 [SPARK-15646][SQL] When spark.sql.hive.convertCTAS is true, the conversion rule needs to respect TEXTFILE/SEQUENCEFILE format and the user-defined location ## What changes were proposed in this pull request? When

spark git commit: [SPARK-15646][SQL] When spark.sql.hive.convertCTAS is true, the conversion rule needs to respect TEXTFILE/SEQUENCEFILE format and the user-defined location

Repository: spark Updated Branches: refs/heads/branch-2.0 35195f6ce -> 5a835b99f [SPARK-15646][SQL] When spark.sql.hive.convertCTAS is true, the conversion rule needs to respect TEXTFILE/SEQUENCEFILE format and the user-defined location ## What changes were proposed in this pull request?

spark git commit: [MINOR][SQL][DOCS] Fix docs of Dataset.scala and SQLImplicits.scala.

Repository: spark Updated Branches: refs/heads/master dad5a6881 -> 196a0d827 [MINOR][SQL][DOCS] Fix docs of Dataset.scala and SQLImplicits.scala. This PR fixes a sample code, a description, and indentations in docs. Manual. Author: Dongjoon Hyun Closes #13420 from

spark git commit: [SPARK-15998][SQL] Verification of SQLConf HIVE_METASTORE_PARTITION_PRUNING

Repository: spark Updated Branches: refs/heads/master 7a89f2adb -> 796429d71 [SPARK-15998][SQL] Verification of SQLConf HIVE_METASTORE_PARTITION_PRUNING What changes were proposed in this pull request? `HIVE_METASTORE_PARTITION_PRUNING` is a public `SQLConf`. When `true`, some

spark git commit: [MINOR][DOCS][SQL] Fix some comments about types(TypeCoercion, Partition) and exceptions.

Repository: spark Updated Branches: refs/heads/master 796429d71 -> 2d27eb1e7 [MINOR][DOCS][SQL] Fix some comments about types(TypeCoercion,Partition) and exceptions. ## What changes were proposed in this pull request? This PR contains a few changes on code comments. - `HiveTypeCoercion` is

spark git commit: [SPARK-15998][SQL] Verification of SQLConf HIVE_METASTORE_PARTITION_PRUNING

Repository: spark Updated Branches: refs/heads/branch-2.0 1230516d9 -> 7d8cddfb4 [SPARK-15998][SQL] Verification of SQLConf HIVE_METASTORE_PARTITION_PRUNING What changes were proposed in this pull request? `HIVE_METASTORE_PARTITION_PRUNING` is a public `SQLConf`. When `true`, some

spark git commit: [SPARK-15978][SQL] improve 'show tables' command related codes

Repository: spark Updated Branches: refs/heads/branch-2.0 095ddb4c9 -> d9dd46edd [SPARK-15978][SQL] improve 'show tables' command related codes ## What changes were proposed in this pull request? I've found some minor issues in "show tables" command: 1. In the `SessionCatalog.scala`,

spark git commit: [SPARK-15978][SQL] improve 'show tables' command related codes

Repository: spark Updated Branches: refs/heads/master 457126e42 -> bbad4cb48 [SPARK-15978][SQL] improve 'show tables' command related codes ## What changes were proposed in this pull request? I've found some minor issues in "show tables" command: 1. In the `SessionCatalog.scala`,

spark git commit: [SPARK-15868][WEB UI] Executors table in Executors tab should sort Executor IDs in numerical order

Repository: spark Updated Branches: refs/heads/master 2d27eb1e7 -> e849285df [SPARK-15868][WEB UI] Executors table in Executors tab should sort Executor IDs in numerical order ## What changes were proposed in this pull request? Currently the Executors table sorts by id using a string sort

spark git commit: [SPARK-15868][WEB UI] Executors table in Executors tab should sort Executor IDs in numerical order

Repository: spark Updated Branches: refs/heads/branch-2.0 fb0fab63c -> 27e274c3e [SPARK-15868][WEB UI] Executors table in Executors tab should sort Executor IDs in numerical order ## What changes were proposed in this pull request? Currently the Executors table sorts by id using a string

spark git commit: [MINOR][DOCS][SQL] Fix some comments about types(TypeCoercion, Partition) and exceptions.

Repository: spark Updated Branches: refs/heads/branch-2.0 7d8cddfb4 -> fb0fab63c [MINOR][DOCS][SQL] Fix some comments about types(TypeCoercion,Partition) and exceptions. ## What changes were proposed in this pull request? This PR contains a few changes on code comments. - `HiveTypeCoercion`

spark git commit: [SQL] Minor HashAggregateExec string output fixes

Repository: spark Updated Branches: refs/heads/master acef843f6 -> 7a89f2adb [SQL] Minor HashAggregateExec string output fixes ## What changes were proposed in this pull request? This PR fixes some minor `.toString` format issues for `HashAggregateExec`. Before: ```

spark git commit: [SQL] Minor HashAggregateExec string output fixes

Repository: spark Updated Branches: refs/heads/branch-2.0 938988757 -> 1230516d9 [SQL] Minor HashAggregateExec string output fixes ## What changes were proposed in this pull request? This PR fixes some minor `.toString` format issues for `HashAggregateExec`. Before: ```

spark git commit: [SPARK-15975] Fix improper Popen retcode code handling in dev/run-tests

Repository: spark Updated Branches: refs/heads/branch-2.0 d9dd46edd -> 938988757 [SPARK-15975] Fix improper Popen retcode code handling in dev/run-tests In the `dev/run-tests.py` script we check a `Popen.retcode` for success using `retcode > 0`, but this is subtlety wrong because Popen's

spark git commit: [SPARK-15975] Fix improper Popen retcode code handling in dev/run-tests

Repository: spark Updated Branches: refs/heads/branch-1.5 6043fa8df -> 1891e04a6 [SPARK-15975] Fix improper Popen retcode code handling in dev/run-tests In the `dev/run-tests.py` script we check a `Popen.retcode` for success using `retcode > 0`, but this is subtlety wrong because Popen's

spark git commit: [SPARK-15975] Fix improper Popen retcode code handling in dev/run-tests

Repository: spark Updated Branches: refs/heads/branch-1.6 cffc0800b -> 0a8ada506 [SPARK-15975] Fix improper Popen retcode code handling in dev/run-tests In the `dev/run-tests.py` script we check a `Popen.retcode` for success using `retcode > 0`, but this is subtlety wrong because Popen's

spark git commit: [SPARK-15975] Fix improper Popen retcode code handling in dev/run-tests

Repository: spark Updated Branches: refs/heads/master bbad4cb48 -> acef843f6 [SPARK-15975] Fix improper Popen retcode code handling in dev/run-tests In the `dev/run-tests.py` script we check a `Popen.retcode` for success using `retcode > 0`, but this is subtlety wrong because Popen's return

spark git commit: [SPARK-15749][SQL] make the error message more meaningful

Repository: spark Updated Branches: refs/heads/branch-2.0 27e274c3e -> 2280ad8a3 [SPARK-15749][SQL] make the error message more meaningful ## What changes were proposed in this pull request? For table test1 (C1 varchar (10), C2 varchar (10)), when I insert a row using ```

spark git commit: [SPARK-15966][DOC] Add closing tag to fix rendering issue for Spark monitoring

Repository: spark Updated Branches: refs/heads/master 9040d83bc -> f1bf0d2f3 [SPARK-15966][DOC] Add closing tag to fix rendering issue for Spark monitoring ## What changes were proposed in this pull request? Adds the missing closing tag for spark.ui.view.acls.groups ## How was this patch

spark git commit: [SPARK-12174] Speed up BlockManagerSuite getRemoteBytes() test

Repository: spark Updated Branches: refs/heads/master bcc7373f6 -> 25782981c [SPARK-12174] Speed up BlockManagerSuite getRemoteBytes() test This patch significantly speeds up the BlockManagerSuite's "SPARK-9591: getRemoteBytes from another location when Exception throw" test, reducing the

spark git commit: [SPARK-10985][CORE] Avoid passing evicted blocks throughout BlockManager

Repository: spark Updated Branches: refs/heads/master 302bb569f -> b8cb548a4 [SPARK-10985][CORE] Avoid passing evicted blocks throughout BlockManager This patch refactors portions of the BlockManager and CacheManager in order to avoid having to pass `evictedBlocks` lists throughout the code.

spark git commit: [SPARK-5095][MESOS] Support launching multiple mesos executors in coarse grained mesos mode.

ved with andrewor14 and pwendell to implement the Mesos scheduler's support of `spark.executor.cores` to be consistent with YARN and Standalone. This PR implements that resolution. This PR implements two high-level features. These two features are co-dependent, so they're implemented both h

spark git commit: [SPARK-13052] waitingApps metric doesn't show the number of apps currently in the WAITING state

Repository: spark Updated Branches: refs/heads/master 7a4b37f02 -> 6dbfc4077 [SPARK-13052] waitingApps metric doesn't show the number of apps currently in the WAITING state Author: Raafat Akkad Closes #10959 from RaafatAkkad/master. Project:

spark git commit: [SPARK-13002][MESOS] Send initial request of executors for dyn allocation

Repository: spark Updated Branches: refs/heads/master 66e1383de -> 0bb5b7338 [SPARK-13002][MESOS] Send initial request of executors for dyn allocation Fix for [SPARK-13002](https://issues.apache.org/jira/browse/SPARK-13002) about the initial number of executors when running with dynamic

spark git commit: [SPARK-13214][DOCS] update dynamicAllocation documentation

Repository: spark Updated Branches: refs/heads/master 1ed354a53 -> 66e1383de [SPARK-13214][DOCS] update dynamicAllocation documentation Author: Bill Chambers Closes #11094 from anabranch/dynamic-docs. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

spark git commit: [SPARK-13214][DOCS] update dynamicAllocation documentation

Repository: spark Updated Branches: refs/heads/branch-1.6 a907c7c64 -> 3ca5dc307 [SPARK-13214][DOCS] update dynamicAllocation documentation Author: Bill Chambers Closes #11094 from anabranch/dynamic-docs. (cherry picked from commit

spark git commit: MAINTENANCE: Automated closing of pull requests.

ted by yhuai) Closes #9288 (requested by andrewor14) Closes #9321 (requested by andrewor14) Closes #9935 (requested by JoshRosen) Closes #10442 (requested by andrewor14) Closes #10585 (requested by srowen) Closes #10785 (requested by srowen) Closes #10832 (requested by andrewor14) Closes #10941 (reques

spark git commit: [SPARK-12330][MESOS] Fix mesos coarse mode cleanup

Repository: spark Updated Branches: refs/heads/master dee801adb -> 2eaeafe8a [SPARK-12330][MESOS] Fix mesos coarse mode cleanup In the current implementation the mesos coarse scheduler does not wait for the mesos tasks to complete before ending the driver. This causes a race where the task

spark git commit: [SPARK-13164][CORE] Replace deprecated synchronized buffer in core

Repository: spark Updated Branches: refs/heads/master 2eaeafe8a -> 62a7c2838 [SPARK-13164][CORE] Replace deprecated synchronized buffer in core Building with scala 2.11 results in the warning trait SynchronizedBuffer in package mutable is deprecated: Synchronization via traits is deprecated

spark git commit: [SPARK-13162] Standalone mode does not respect initial executors

ied `spark.dynamicAllocation.initialExecutors`, the config would not take effect. This is similar to #11047 but for standalone mode. Author: Andrew Or <and...@databricks.com> Closes #11054 from andrewor14/standalone-da-initial. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/

spark git commit: [SPARK-13053][TEST] Unignore tests in InternalAccumulatorSuite

rew Or <and...@databricks.com> Closes #10969 from andrewor14/unignore-accum-test. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/15205da8 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/15205da8 Diff: http://git-wip-us.apache.org/

spark git commit: [HOTFIX] Fix style violation caused by c756bda

Repository: spark Updated Branches: refs/heads/master 8e2f29630 -> 7a4b37f02 [HOTFIX] Fix style violation caused by c756bda Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/7a4b37f0 Tree:

spark git commit: [SPARK-12265][MESOS] Spark calls System.exit inside driver instead of throwing exception

Repository: spark Updated Branches: refs/heads/master 51b03b71f -> a41b68b95 [SPARK-12265][MESOS] Spark calls System.exit inside driver instead of throwing exception This takes over #10729 and makes sure that `spark-shell` fails with a proper error message. There is a slight behavioral

spark git commit: [SPARK-6847][CORE][STREAMING] Fix stack overflow issue when updateStateByKey is followed by a checkpointed dstream

Repository: spark Updated Branches: refs/heads/master c1da4d421 -> 6075573a9 [SPARK-6847][CORE][STREAMING] Fix stack overflow issue when updateStateByKey is followed by a checkpointed dstream Add a local property to indicate if checkpointing all RDDs that are marked with the checkpoint

spark git commit: [SPARK-12979][MESOS] Don’t resolve paths on the local file system in Mesos scheduler

Repository: spark Updated Branches: refs/heads/master a41b68b95 -> c9b89a0a0 [SPARK-12979][MESOS] Don’t resolve paths on the local file system in Mesos scheduler The driver filesystem is likely different from where the executors will run, so resolving paths (and symlinks, etc.) will lead

[8/9] spark git commit: [SPARK-12790][CORE] Remove HistoryServer old multiple files format

http://git-wip-us.apache.org/repos/asf/spark/blob/0df3cfb8/core/src/test/resources/spark-events/local-1422981759269 -- diff --git a/core/src/test/resources/spark-events/local-1422981759269

[7/9] spark git commit: [SPARK-12790][CORE] Remove HistoryServer old multiple files format

http://git-wip-us.apache.org/repos/asf/spark/blob/0df3cfb8/core/src/test/resources/spark-events/local-1422981759269/EVENT_LOG_1 -- diff --git a/core/src/test/resources/spark-events/local-1422981759269/EVENT_LOG_1

[4/9] spark git commit: [SPARK-12790][CORE] Remove HistoryServer old multiple files format

http://git-wip-us.apache.org/repos/asf/spark/blob/0df3cfb8/core/src/test/resources/spark-events/local-1425081759269 -- diff --git a/core/src/test/resources/spark-events/local-1425081759269

[6/9] spark git commit: [SPARK-12790][CORE] Remove HistoryServer old multiple files format

http://git-wip-us.apache.org/repos/asf/spark/blob/0df3cfb8/core/src/test/resources/spark-events/local-1422981780767 -- diff --git a/core/src/test/resources/spark-events/local-1422981780767

[5/9] spark git commit: [SPARK-12790][CORE] Remove HistoryServer old multiple files format

http://git-wip-us.apache.org/repos/asf/spark/blob/0df3cfb8/core/src/test/resources/spark-events/local-1422981780767/EVENT_LOG_1 -- diff --git a/core/src/test/resources/spark-events/local-1422981780767/EVENT_LOG_1

[2/9] spark git commit: [SPARK-12790][CORE] Remove HistoryServer old multiple files format

http://git-wip-us.apache.org/repos/asf/spark/blob/0df3cfb8/core/src/test/resources/spark-events/local-1426533911241 -- diff --git a/core/src/test/resources/spark-events/local-1426533911241

[9/9] spark git commit: [SPARK-12790][CORE] Remove HistoryServer old multiple files format

[SPARK-12790][CORE] Remove HistoryServer old multiple files format Removed isLegacyLogDirectory code path and updated tests andrewor14 Author: felixcheung <felixcheun...@hotmail.com> Closes #10860 from felixcheung/historyserverformat. Project: http://git-wip-us.apache.org/repos/asf/spar

spark git commit: [SPARK-12637][CORE] Print stage info of finished stages properly

Repository: spark Updated Branches: refs/heads/master be7a2fc07 -> 715a19d56 [SPARK-12637][CORE] Print stage info of finished stages properly Improve printing of StageInfo in onStageCompleted See also https://github.com/apache/spark/pull/10585 Author: Sean Owen Closes

[3/9] spark git commit: [SPARK-12790][CORE] Remove HistoryServer old multiple files format

http://git-wip-us.apache.org/repos/asf/spark/blob/0df3cfb8/core/src/test/resources/spark-events/local-1425081759269/EVENT_LOG_1 -- diff --git a/core/src/test/resources/spark-events/local-1425081759269/EVENT_LOG_1

[1/9] spark git commit: [SPARK-12790][CORE] Remove HistoryServer old multiple files format

Repository: spark Updated Branches: refs/heads/master 715a19d56 -> 0df3cfb8a http://git-wip-us.apache.org/repos/asf/spark/blob/0df3cfb8/core/src/test/resources/spark-events/local-1426633911242 -- diff --git

[1/2] spark git commit: [SPARK-13088] Fix DAG viz in latest version of chrome

gre-d3.min.js @@ -1,4 +1,5 @@ -/* This is a custom version of dagre-d3 on top of v0.4.3. The full list of commits can be found at http://github.com/andrewor14/dagre-d3/ */!function(e){if("object"==typeof exports&&"undefined"!=typeof module)module.exports=e();else if(&qu

spark git commit: [SPARK-13071] Coalescing HadoopRDD overwrites existing input metrics

ks.com> Closes #10973 from andrewor14/fix-input-metrics-coalesce. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/12252d1d Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/12252d1d Diff: http://git-wip-us.apache.org/repos/

spark git commit: [SPARK-13096][TEST] Fix flaky verifyPeakExecutionMemorySet

sed, i.e. until the listener queue is empty. https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.7/79/testReport/junit/org.apache.spark.util.collection/ExternalAppendOnlyMapSuite/spilling/ Author: Andrew Or <and...@databricks.com> Closes #10990 from andrewor14/accum-s

spark git commit: [SPARK-13152][CORE] Fix task metrics deprecation warning

now incBytesRead and incRecordsRead are marked as deprecated and for internal use only. We should make private[spark] versions which are not deprecated and switch to those internally so as to not clutter up the warning messages when building. cc andrewor14 who did the initial deprecation Author: Hol

[1/2] spark git commit: [SPARK-13088] Fix DAG viz in latest version of chrome

gre-d3.min.js @@ -1,4 +1,5 @@ -/* This is a custom version of dagre-d3 on top of v0.4.3. The full list of commits can be found at http://github.com/andrewor14/dagre-d3/ */!function(e){if("object"==typeof exports&&"undefined"!=typeof module)module.exports=e();else if(&qu

[2/2] spark git commit: [SPARK-13088] Fix DAG viz in latest version of chrome

[SPARK-13088] Fix DAG viz in latest version of chrome Apparently chrome removed `SVGElement.prototype.getTransformToElement`, which is used by our JS library dagre-d3 when creating edges. The real diff can be found here: https://github.com/andrewor14/dagre-d3/commit

[1/2] spark git commit: [SPARK-13088] Fix DAG viz in latest version of chrome

-d3.min.js @@ -1,4 +1,5 @@ -/* This is a custom version of dagre-d3 on top of v0.4.3. The full list of commits can be found at http://github.com/andrewor14/dagre-d3/ */!function(e){if("object"==typeof exports&&"undefined"!=typeof module)module.exports=e();else if(&qu

[2/2] spark git commit: [SPARK-13088] Fix DAG viz in latest version of chrome

[SPARK-13088] Fix DAG viz in latest version of chrome Apparently chrome removed `SVGElement.prototype.getTransformToElement`, which is used by our JS library dagre-d3 when creating edges. The real diff can be found here: https://github.com/andrewor14/dagre-d3/commit

[1/2] spark git commit: [SPARK-13088] Fix DAG viz in latest version of chrome

gre-d3.min.js @@ -1,4 +1,5 @@ -/* This is a custom version of dagre-d3 on top of v0.4.3. The full list of commits can be found at http://github.com/andrewor14/dagre-d3/ */!function(e){if("object"==typeof exports&&"undefined"!=typeof module)module.exports=e();else if(&qu

[2/2] spark git commit: [SPARK-13088] Fix DAG viz in latest version of chrome

[SPARK-13088] Fix DAG viz in latest version of chrome Apparently chrome removed `SVGElement.prototype.getTransformToElement`, which is used by our JS library dagre-d3 when creating edges. The real diff can be found here: https://github.com/andrewor14/dagre-d3/commit

[2/2] spark git commit: [SPARK-13088] Fix DAG viz in latest version of chrome

[SPARK-13088] Fix DAG viz in latest version of chrome Apparently chrome removed `SVGElement.prototype.getTransformToElement`, which is used by our JS library dagre-d3 when creating edges. The real diff can be found here: https://github.com/andrewor14/dagre-d3/commit

spark git commit: [SPARK-12463][SPARK-12464][SPARK-12465][SPARK-10647][MESOS] Fix zookeeper dir with mesos conf and add docs.

Repository: spark Updated Branches: refs/heads/master 711ce048a -> 51b03b71f [SPARK-12463][SPARK-12464][SPARK-12465][SPARK-10647][MESOS] Fix zookeeper dir with mesos conf and add docs. Fix zookeeper dir configuration used in cluster mode, and also add documentation around these settings.

spark git commit: [SPARK-13122] Fix race condition in MemoryStore.unrollSafely()

Repository: spark Updated Branches: refs/heads/master 21112e8a1 -> ff71261b6 [SPARK-13122] Fix race condition in MemoryStore.unrollSafely() https://issues.apache.org/jira/browse/SPARK-13122 A race condition can occur in MemoryStore's unrollSafely() method if two threads that return the same

spark git commit: [SPARK-13122] Fix race condition in MemoryStore.unrollSafely()

Repository: spark Updated Branches: refs/heads/branch-1.6 e81333be0 -> 2f8abb4af [SPARK-13122] Fix race condition in MemoryStore.unrollSafely() https://issues.apache.org/jira/browse/SPARK-13122 A race condition can occur in MemoryStore's unrollSafely() method if two threads that return the

spark git commit: [SPARK-10749][MESOS] Support multiple roles with mesos cluster mode.

Repository: spark Updated Branches: refs/heads/master 40e6d40fe -> 00461bb91 [SPARK-10749][MESOS] Support multiple roles with mesos cluster mode. Currently the Mesos cluster dispatcher is not using offers from multiple roles correctly, as it simply aggregates all the offers resource values

spark git commit: [SPARK-13522][CORE] Fix the exit log place for heartbeat

Repository: spark Updated Branches: refs/heads/master 17a253cbf -> 644dbb641 [SPARK-13522][CORE] Fix the exit log place for heartbeat ## What changes were proposed in this pull request? Just fixed the log place introduced by #11401 ## How was this patch tested? unit tests. Author:

spark git commit: [SPARK-13522][CORE] Executor should kill itself when it's unable to heartbeat to driver more than N times

Repository: spark Updated Branches: refs/heads/master bc65f60ef -> 17a253cbf [SPARK-13522][CORE] Executor should kill itself when it's unable to heartbeat to driver more than N times ## What changes were proposed in this pull request? Sometimes, network disconnection event won't be

spark git commit: [SPARK-13519][CORE] Driver should tell Executor to stop itself when cleaning executor's state

Repository: spark Updated Branches: refs/heads/master 1e5fcdf96 -> ad615291f [SPARK-13519][CORE] Driver should tell Executor to stop itself when cleaning executor's state ## What changes were proposed in this pull request? When the driver removes an executor's state, the connection between

spark git commit: [SPARK-13298][CORE][UI] Escape "label" to avoid DAG being broken by some special character

Repository: spark Updated Branches: refs/heads/master 33ef3aa7e -> a11b39951 [SPARK-13298][CORE][UI] Escape "label" to avoid DAG being broken by some special character ## What changes were proposed in this pull request? When there are some special characters (e.g., `"`, `\`) in `label`, DAG

spark git commit: [SPARK-13298][CORE][UI] Escape "label" to avoid DAG being broken by some special character

Repository: spark Updated Branches: refs/heads/branch-1.6 699644c69 -> 85e6a2205 [SPARK-13298][CORE][UI] Escape "label" to avoid DAG being broken by some special character ## What changes were proposed in this pull request? When there are some special characters (e.g., `"`, `\`) in `label`,

[2/2] spark git commit: [SPARK-12757] Add block-level read/write locks to BlockManager

[SPARK-12757] Add block-level read/write locks to BlockManager ## Motivation As a pre-requisite to off-heap caching of blocks, we need a mechanism to prevent pages / blocks from being evicted while they are being read. With on-heap objects, evicting a block while it is being read merely leads

[1/2] spark git commit: [SPARK-12757] Add block-level read/write locks to BlockManager

Repository: spark Updated Branches: refs/heads/master 712995757 -> 633d63a48 http://git-wip-us.apache.org/repos/asf/spark/blob/633d63a4/core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala -- diff --git

spark git commit: [SPARK-13501] Remove use of Guava Stopwatch

Repository: spark Updated Branches: refs/heads/master 7a6ee8a8f -> f2cfafdfe [SPARK-13501] Remove use of Guava Stopwatch Our nightly doc snapshot builds are failing due to some issue involving the Guava Stopwatch constructor: ``` [error]

spark git commit: [SPARK-13387][MESOS] Add support for SPARK_DAEMON_JAVA_OPTS with MesosClusterDispatcher.

Repository: spark Updated Branches: refs/heads/master f2cfafdfe -> 712995757 [SPARK-13387][MESOS] Add support for SPARK_DAEMON_JAVA_OPTS with MesosClusterDispatcher. ## What changes were proposed in this pull request? Add support for SPARK_DAEMON_JAVA_OPTS with MesosClusterDispatcher. ##

spark git commit: [SPARK-12009][YARN] Avoid to re-allocating yarn container while driver want to stop all Executors

Repository: spark Updated Branches: refs/heads/master dc6c5ea4c -> 7a6ee8a8f [SPARK-12009][YARN] Avoid to re-allocating yarn container while driver want to stop all Executors Author: hushan Closes #9992 from suyanNone/tricky. Project:

spark git commit: [SPARK-12887] Do not expose var's in TaskMetrics

nse in the case of TaskMetrics because these are just aggregated metrics that we want to collect throughout the task, so it doesn't matter who's incrementing them. Parent PR: #10717 Author: Andrew Or <and...@databricks.com> Author: Josh Rosen <joshro...@databricks.com> Aut

spark git commit: [SPARK-5095] Fix style in mesos coarse grained scheduler code

Repository: spark Updated Branches: refs/heads/master 42d656814 -> 38bc6018e [SPARK-5095] Fix style in mesos coarse grained scheduler code andrewor14 This addressed your style comments from #10993 Author: Michael Gummelt <mgumm...@mesosphere.io> Closes #11187 from mgummelt/fix_me

spark git commit: [SPARK-5095] remove flaky test

Repository: spark Updated Branches: refs/heads/master 38bc6018e -> 62b1c07e7 [SPARK-5095] remove flaky test Overrode the start() method, which was previously starting a thread causing a race condition. I believe this should fix the flaky test. Author: Michael Gummelt

spark git commit: [SPARK-13371][CORE][STRING] TaskSetManager.dequeueSpeculativeTask compares Option and String directly.

Repository: spark Updated Branches: refs/heads/master 892b2dd6d -> 78562535f [SPARK-13371][CORE][STRING] TaskSetManager.dequeueSpeculativeTask compares Option and String directly. ## What changes were proposed in this pull request? Fix some comparisons between unequal types that cause IJ

spark git commit: [SPARK-13371][CORE][STRING] TaskSetManager.dequeueSpeculativeTask compares Option and String directly.

Repository: spark Updated Branches: refs/heads/branch-1.6 66106a660 -> 16f35c4c6 [SPARK-13371][CORE][STRING] TaskSetManager.dequeueSpeculativeTask compares Option and String directly. ## What changes were proposed in this pull request? Fix some comparisons between unequal types that cause

[2/2] spark git commit: [SPARK-13696] Remove BlockStore class & simplify interfaces of mem. & disk stores

[SPARK-13696] Remove BlockStore class & simplify interfaces of mem. & disk stores Today, both the MemoryStore and DiskStore implement a common `BlockStore` API, but I feel that this API is inappropriate because it abstracts away important distinctions between the behavior of these two stores.

[1/2] spark git commit: [SPARK-13696] Remove BlockStore class & simplify interfaces of mem. & disk stores

Repository: spark Updated Branches: refs/heads/master 3d2b6f56e -> 81d48532d http://git-wip-us.apache.org/repos/asf/spark/blob/81d48532/core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala -- diff --git

spark git commit: [SPARK-13604][CORE] Sync worker's state after registering with master

Repository: spark Updated Branches: refs/heads/master 020ff8cd3 -> 27fe6bacc [SPARK-13604][CORE] Sync worker's state after registering with master ## What changes were proposed in this pull request? Here lists all cases that Master cannot talk with Worker for a while and then network is

spark git commit: [SPARK-13833] Guard against race condition when re-caching disk blocks in memory

Repository: spark Updated Branches: refs/heads/master 9a1680c2c -> 9a87afd7d [SPARK-13833] Guard against race condition when re-caching disk blocks in memory When reading data from the DiskStore and attempting to cache it back into the memory store, we should guard against race conditions

spark git commit: [SPARK-12583][MESOS] Mesos shuffle service: Don't delete shuffle files before application has stopped

Repository: spark Updated Branches: refs/heads/master 07cb323e7 -> 310981d49 [SPARK-12583][MESOS] Mesos shuffle service: Don't delete shuffle files before application has stopped ## Problem description: Mesos shuffle service is completely unusable since Spark 1.6.0 . The problem seems to

spark git commit: [MINOR][COMMON] Fix copy-paste oversight in variable naming

Repository: spark Updated Branches: refs/heads/master 9f13f0fc1 -> e06493cb7 [MINOR][COMMON] Fix copy-paste oversight in variable naming ## What changes were proposed in this pull request? JavaUtils.java has methods to convert time and byte strings for internal use, this change renames a

spark git commit: [MINOR][COMMON] Fix copy-paste oversight in variable naming

Repository: spark Updated Branches: refs/heads/branch-1.5 9ecd33e96 -> 7a24d94a8 [MINOR][COMMON] Fix copy-paste oversight in variable naming ## What changes were proposed in this pull request? JavaUtils.java has methods to convert time and byte strings for internal use, this change renames

spark git commit: [MINOR][DOCS] Added Missing back slashes

Repository: spark Updated Branches: refs/heads/master 310981d49 -> 9f13f0fc1 [MINOR][DOCS] Added Missing back slashes ## What changes were proposed in this pull request? When studying spark many users just copy examples on the documentation and paste on their terminals and because of that

spark git commit: [MINOR][DOCS] Added Missing back slashes

Repository: spark Updated Branches: refs/heads/branch-1.6 3519ce9f1 -> 13bd12264 [MINOR][DOCS] Added Missing back slashes ## What changes were proposed in this pull request? When studying spark many users just copy examples on the documentation and paste on their terminals and because of

spark git commit: [SPARK-13328][CORE] Poor read performance for broadcast variables with dynamic resource allocation

Repository: spark Updated Branches: refs/heads/master eb650a81f -> ff776b2fc [SPARK-13328][CORE] Poor read performance for broadcast variables with dynamic resource allocation When dynamic resource allocation is enabled fetching broadcast variables from removed executors were causing job

spark git commit: [SPARK-13778][CORE] Set the executor state for a worker when removing it

Repository: spark Updated Branches: refs/heads/master 37fcda3e6 -> 40e067675 [SPARK-13778][CORE] Set the executor state for a worker when removing it ## What changes were proposed in this pull request? When a worker is lost, the executors on this worker are also lost. But Master's

spark git commit: [MINOR] Fix typo in 'hypot' docstring

Repository: spark Updated Branches: refs/heads/master 238447db5 -> 5f7dbdba6 [MINOR] Fix typo in 'hypot' docstring Minor typo: docstring for pyspark.sql.functions: hypot has extra characters N/A Author: Tristan Reid Closes #11616 from tristanreid/master. Project:

spark git commit: [SPARK-13775] History page sorted by completed time desc by default.

Repository: spark Updated Branches: refs/heads/master 40e067675 -> 238447db5 [SPARK-13775] History page sorted by completed time desc by default. ## What changes were proposed in this pull request? Originally the page is sorted by AppID by default. After tests with users' feedback, we think

spark git commit: [SPARK-13492][MESOS] Configurable Mesos framework webui URL.

Repository: spark Updated Branches: refs/heads/master 5f7dbdba6 -> a4a0addcc [SPARK-13492][MESOS] Configurable Mesos framework webui URL. ## What changes were proposed in this pull request? Previously the Mesos framework webui URL was being derived only from the Spark UI address leaving no

spark git commit: [SPARK-13845][CORE] Using onBlockUpdated to replace onTaskEnd avioding driver OOM

Repository: spark Updated Branches: refs/heads/master a916d2a45 -> ad9e3d50f [SPARK-13845][CORE] Using onBlockUpdated to replace onTaskEnd avioding driver OOM ## What changes were proposed in this pull request? We have a streaming job using `FlumePollInputStream` always driver OOM after

spark git commit: [SPARK-14086][SQL] Add DDL commands to ANTLR4 parser

ove them in follow-up PRs. How was this patch tested? The existing DDLCommandSuite. cc rxin andrewor14 yhuai Author: Herman van Hovell <hvanhov...@questtec.nl> Closes #12011 from hvanhovell/SPARK-14086. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http:

spark git commit: [SPARK-13447][YARN][CORE] Clean the stale states for AM failure and restart situation

ean-up work only for dynamic allocation enabled scenario. Here we should also clean the states in `CoarseGrainedSchedulerBackend` for dynamic allocation disabled scenario. Please review, CC andrewor14 lianhuiwang , thanks a lot. ## How was this patch tested? Run the unit test locally, a

spark git commit: [SPARK-13923][SPARK-14014][SQL] Session catalog follow-ups

ull list of changes in this patch, just trace the commits. ## How was this patch tested? `SessionCatalogSuite` and `CatalogTestCases` Author: Andrew Or <and...@databricks.com> Closes #12006 from andrewor14/session-catalog-followup. Project: http://git-wip-us.apache.org/repos/asf/s

spark git commit: [SPARK-14180][CORE] Fix a deadlock in CoarseGrainedExecutorBackend Shutdown

Repository: spark Updated Branches: refs/heads/master 328c71161 -> 34c0638ee [SPARK-14180][CORE] Fix a deadlock in CoarseGrainedExecutorBackend Shutdown ## What changes were proposed in this pull request? Call `executor.stop` in a new thread to eliminate deadlock. ## How was this patch

spark git commit: [SPARK-14119][SPARK-14120][SPARK-14122][SQL] Throw exception on unsupported DDL commands

his patch tested? `HiveQuerySuite` Author: Andrew Or <and...@databricks.com> Closes #11948 from andrewor14/ddl-role-management. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a916d2a4 Tree: http://git-wip-us.apache.or

spark git commit: [SPARK-14383][SQL] missing "|" in the g4 file

Repository: spark Updated Branches: refs/heads/master 5abd02c02 -> 3c8d88216 [SPARK-14383][SQL] missing "|" in the g4 file ## What changes were proposed in this pull request? A very trivial one. It missed "|" between DISTRIBUTE and UNSET. ## How was this patch tested? I do not think it is

spark git commit: [SPARK-14429][SQL] Improve LIKE pattern in "SHOW TABLES / FUNCTIONS LIKE " DDL

some changes to the utility method to make sure we will get the same result as Hive does. A new method was created in StringUtil and test cases were added. andrewor14 Author: bomeng <bm...@us.ibm.com> Closes #12206 from bomeng/SPARK-14429. Project: http://git-wip-us.apache.org/repos/asf/spar

<    7   8   9   10   11   12   13   14   15   >