spark git commit: [SPARK-18715][ML] Fix AIC calculations in Binomial GLM

2016-12-13 Thread srowen
ate. srowen sethah yanboliang HyukjinKwon zhengruifeng ## What changes were proposed in this pull request? I suggest changing the current aic calculation for the Binomial family from ``` -2.0 * predictions.map { case (y: Double, mu: Double, weight: Double) => weight * dist.Binomial

spark git commit: [SPARK-18471][MLLIB] In LBFGS, avoid sending huge vectors of 0

2016-12-13 Thread srowen
Repository: spark Updated Branches: refs/heads/master e57e3938c -> 9e8a9d7c6 [SPARK-18471][MLLIB] In LBFGS, avoid sending huge vectors of 0 ## What changes were proposed in this pull request? CostFun used to send a dense vector of zeroes as a closure in a treeAggregate call. To avoid that, we

spark git commit: [SPARK-18816][WEB UI] Executors Logs column only ran visibility check on initial table load

2016-12-13 Thread srowen
Repository: spark Updated Branches: refs/heads/master 9e8a9d7c6 -> aebf44e50 [SPARK-18816][WEB UI] Executors Logs column only ran visibility check on initial table load ## What changes were proposed in this pull request? When I added a visibility check for the logs column on the executors pa

spark git commit: [SPARK-18816][WEB UI] Executors Logs column only ran visibility check on initial table load

2016-12-13 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.1 d5c4a5d06 -> 292a37f24 [SPARK-18816][WEB UI] Executors Logs column only ran visibility check on initial table load ## What changes were proposed in this pull request? When I added a visibility check for the logs column on the executor

spark git commit: [SPARK-18842][TESTS][LAUNCHER] De-duplicate paths in classpaths in commands for local-cluster mode to work around the path length limitation on Windows

2016-12-14 Thread srowen
Repository: spark Updated Branches: refs/heads/master ba4aab9b8 -> c6b8eb71a [SPARK-18842][TESTS][LAUNCHER] De-duplicate paths in classpaths in commands for local-cluster mode to work around the path length limitation on Windows ## What changes were proposed in this pull request? Currently,

spark git commit: [SPARK-18830][TESTS] Fix tests in PipedRDDSuite to pass on Windows

2016-12-14 Thread srowen
Repository: spark Updated Branches: refs/heads/master c6b8eb71a -> 169b9d73e [SPARK-18830][TESTS] Fix tests in PipedRDDSuite to pass on Windows ## What changes were proposed in this pull request? This PR proposes to fix the tests failed on Windows as below: ``` [info] - pipe with empty parti

spark git commit: [SPARK-18742][CORE] Clarify that user-defined BroadcastFactory is not supported

2016-12-16 Thread srowen
Repository: spark Updated Branches: refs/heads/master d7f3058e1 -> 53ab8fb34 [SPARK-18742][CORE] Clarify that user-defined BroadcastFactory is not supported ## What changes were proposed in this pull request? After SPARK-12588 Remove HTTPBroadcast [1], the one and only implementation of Broad

spark git commit: [SPARK-18723][DOC] Expanded programming guide information on wholeTex…

2016-12-16 Thread srowen
Repository: spark Updated Branches: refs/heads/master dc2a4d4ad -> 836c95b10 [SPARK-18723][DOC] Expanded programming guide information on wholeTex… ## What changes were proposed in this pull request? Add additional information to wholeTextFiles in the Programming Guide. Also explain partit

spark git commit: [SPARK-18708][CORE] Improvement/improve docs in spark context file

2016-12-16 Thread srowen
Repository: spark Updated Branches: refs/heads/master 836c95b10 -> f7a574a6c [SPARK-18708][CORE] Improvement/improve docs in spark context file ## What changes were proposed in this pull request? SparkContext.scala was created a long time ago and contains several types of Scaladocs/Javadocs

spark git commit: [MINOR][BUILD] Fix lint-check failures and javadoc8 break

2016-12-16 Thread srowen
Repository: spark Updated Branches: refs/heads/master f7a574a6c -> ed84cd068 [MINOR][BUILD] Fix lint-check failures and javadoc8 break ## What changes were proposed in this pull request? This PR proposes to fix lint-check failures and javadoc8 break. Few errors were introduced as below: **l

spark git commit: [SPARK-18485][CORE] Underlying integer overflow when create ChunkedByteBufferOutputStream in MemoryStore

2016-12-17 Thread srowen
Repository: spark Updated Branches: refs/heads/master 2bc1c9515 -> 6d2379b3b [SPARK-18485][CORE] Underlying integer overflow when create ChunkedByteBufferOutputStream in MemoryStore ## What changes were proposed in this pull request? There is an underlying integer overflow when create Chunk

spark git commit: [SPARK-18918][DOC] Missing in Configuration page

2016-12-18 Thread srowen
Repository: spark Updated Branches: refs/heads/master 38fd163d0 -> c0c9e1d27 [SPARK-18918][DOC] Missing in Configuration page ### What changes were proposed in this pull request? The configuration page looks messy now, as shown in the nightly build: https://people.apache.org/~pwendell/spark-n

spark git commit: [SPARK-18918][DOC] Missing in Configuration page

2016-12-18 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.1 001f49b7c -> 4b8a643f9 [SPARK-18918][DOC] Missing in Configuration page ### What changes were proposed in this pull request? The configuration page looks messy now, as shown in the nightly build: https://people.apache.org/~pwendell/spa

spark git commit: [SPARK-18827][CORE] Fix cannot read broadcast on disk

2016-12-18 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.1 4b8a643f9 -> a5da8db85 [SPARK-18827][CORE] Fix cannot read broadcast on disk ## What changes were proposed in this pull request? `NoSuchElementException` will throw since https://github.com/apache/spark/pull/15056 if a broadcast cannot

spark git commit: [SPARK-18827][CORE] Fix cannot read broadcast on disk

2016-12-18 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.0 1935bf446 -> b41668349 [SPARK-18827][CORE] Fix cannot read broadcast on disk ## What changes were proposed in this pull request? `NoSuchElementException` will throw since https://github.com/apache/spark/pull/15056 if a broadcast cannot

spark git commit: [SPARK-18827][CORE] Fix cannot read broadcast on disk

2016-12-18 Thread srowen
Repository: spark Updated Branches: refs/heads/master c0c9e1d27 -> 1e5c51f33 [SPARK-18827][CORE] Fix cannot read broadcast on disk ## What changes were proposed in this pull request? `NoSuchElementException` will throw since https://github.com/apache/spark/pull/15056 if a broadcast cannot cac

spark git commit: [SPARK-18356][ML] KMeans should cache RDD before training

2016-12-19 Thread srowen
Repository: spark Updated Branches: refs/heads/master 1e5c51f33 -> 7db09abb0 [SPARK-18356][ML] KMeans should cache RDD before training ## What changes were proposed in this pull request? According to request of Mr. Joseph Bradley , I did this update of my PR https://github.com/apache/spark/p

spark git commit: [SPARK-18923][DOC][BUILD] Support skipping R/Python API docs

2016-12-21 Thread srowen
Repository: spark Updated Branches: refs/heads/master 24c0c9412 -> ba4468bb2 [SPARK-18923][DOC][BUILD] Support skipping R/Python API docs ## What changes were proposed in this pull request? We can build Python API docs by `cd ./python/docs && make html for Python` and R API docs by `cd ./R &

spark git commit: [SPARK-18953][CORE][WEB UI] Do now show the link to a dead worker on the master page

2016-12-22 Thread srowen
Repository: spark Updated Branches: refs/heads/master 2e861df96 -> f489339c7 [SPARK-18953][CORE][WEB UI] Do now show the link to a dead worker on the master page ## What changes were proposed in this pull request? For a dead worker, we will not be able to see its worker page anyway. This PR

spark git commit: [SPARK-18922][TESTS] Fix more resource-closing-related and path-related test failures in identified ones on Windows

2016-12-22 Thread srowen
Repository: spark Updated Branches: refs/heads/master f489339c7 -> 4186aba63 [SPARK-18922][TESTS] Fix more resource-closing-related and path-related test failures in identified ones on Windows ## What changes were proposed in this pull request? There are several tests failing due to resource

spark git commit: [SPARK-18963] o.a.s.unsafe.types.UTF8StringSuite.writeToOutputStreamIntArray test

2016-12-23 Thread srowen
Repository: spark Updated Branches: refs/heads/master 17579bda3 -> 1311448ea [SPARK-18963] o.a.s.unsafe.types.UTF8StringSuite.writeToOutputStreamIntArray test fails on big endian. Only change byte order on little endian ## What changes were proposed in this pull request? Fix test to only ch

spark-website git commit: Fix typo in contributing.md: 'h elping' -> 'helping'

2016-12-23 Thread srowen
Repository: spark-website Updated Branches: refs/heads/asf-site 066dfa274 -> ecf94f284 Fix typo in contributing.md: 'h elping' -> 'helping' Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-us.apache.org/repos/asf/spark-website/commit/ecf94f28 Tree: htt

spark git commit: [SPARK-18800][SQL] Correct the assert in UnsafeKVExternalSorter which ensures array size

2016-12-24 Thread srowen
Repository: spark Updated Branches: refs/heads/master 3cff81615 -> 07fcbea51 [SPARK-18800][SQL] Correct the assert in UnsafeKVExternalSorter which ensures array size ## What changes were proposed in this pull request? `UnsafeKVExternalSorter` uses `UnsafeInMemorySorter` to sort the records o

spark git commit: [SPARK-18837][WEBUI] Very long stage descriptions do not wrap in the UI

2016-12-24 Thread srowen
Repository: spark Updated Branches: refs/heads/master 07fcbea51 -> f2ceb2abe [SPARK-18837][WEBUI] Very long stage descriptions do not wrap in the UI ## What changes were proposed in this pull request? This issue was reported by wangyum. In the AllJobsPage, JobPage and StagePage, the descript

spark git commit: [SPARK-18837][WEBUI] Very long stage descriptions do not wrap in the UI

2016-12-24 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.1 5bafdc45d -> ca25b1e51 [SPARK-18837][WEBUI] Very long stage descriptions do not wrap in the UI ## What changes were proposed in this pull request? This issue was reported by wangyum. In the AllJobsPage, JobPage and StagePage, the desc

spark git commit: [SPARK-18842][TESTS] De-duplicate paths in classpaths in processes for local-cluster mode in ReplSuite to work around the length limitation on Windows

2016-12-27 Thread srowen
Repository: spark Updated Branches: refs/heads/master 2404d8e54 -> d8e14db84 [SPARK-18842][TESTS] De-duplicate paths in classpaths in processes for local-cluster mode in ReplSuite to work around the length limitation on Windows ## What changes were proposed in this pull request? `ReplSuite`s

spark git commit: [SPARK-19006][DOCS] mention spark.kryoserializer.buffer.max must be less than 2048m in doc

2016-12-27 Thread srowen
Repository: spark Updated Branches: refs/heads/master d8e14db84 -> 28ab0ec49 [SPARK-19006][DOCS] mention spark.kryoserializer.buffer.max must be less than 2048m in doc ## What changes were proposed in this pull request? On configuration doc page:https://spark.apache.org/docs/latest/configur

spark git commit: [SPARK-19010][CORE] Include Kryo exception in case of overflow

2016-12-28 Thread srowen
Repository: spark Updated Branches: refs/heads/master 9cff67f34 -> 67fb33e7e [SPARK-19010][CORE] Include Kryo exception in case of overflow ## What changes were proposed in this pull request? This is to workaround an implicit result of #4947 which suppressed the original Kryo exception if the

spark git commit: [SPARK-18960][SQL][SS] Avoid double reading file which is being copied.

2016-12-28 Thread srowen
Repository: spark Updated Branches: refs/heads/master 67fb33e7e -> 76e9bd748 [SPARK-18960][SQL][SS] Avoid double reading file which is being copied. ## What changes were proposed in this pull request? In HDFS, when we copy a file into target directory, there will a temporary `._COPY_` file f

spark git commit: [MINOR][DOC] Fix doc of ForeachWriter to use writeStream

2016-12-28 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.1 ca25b1e51 -> ac7107fe7 [MINOR][DOC] Fix doc of ForeachWriter to use writeStream ## What changes were proposed in this pull request? Fix the document of `ForeachWriter` to use `writeStream` instead of `write` for a streaming dataset.

spark git commit: [MINOR][DOC] Fix doc of ForeachWriter to use writeStream

2016-12-28 Thread srowen
Repository: spark Updated Branches: refs/heads/master 76e9bd748 -> 2a5f52a71 [MINOR][DOC] Fix doc of ForeachWriter to use writeStream ## What changes were proposed in this pull request? Fix the document of `ForeachWriter` to use `writeStream` instead of `write` for a streaming dataset. ## H

spark git commit: [SPARK-18993][BUILD] Unable to build/compile Spark in IntelliJ due to missing Scala deps in spark-tags

2016-12-28 Thread srowen
ses from spark-tags because its Scala annotations need them. ## How was this patch tested? Existing tests Author: Sean Owen Closes #16418 from srowen/SPARK-18993. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d7bce3bd T

spark git commit: [SPARK-18993][BUILD] Unable to build/compile Spark in IntelliJ due to missing Scala deps in spark-tags

2016-12-28 Thread srowen
ary classes from spark-tags because its Scala annotations need them. ## How was this patch tested? Existing tests Author: Sean Owen Closes #16418 from srowen/SPARK-18993. (cherry picked from commit d7bce3bd31ec193274718042dc017706989d7563) Signed-off-by: Sean Owen Project: http://git-

spark git commit: [SPARK-18993][BUILD] Unable to build/compile Spark in IntelliJ due to missing Scala deps in spark-tags

2016-12-28 Thread srowen
ary classes from spark-tags because its Scala annotations need them. ## How was this patch tested? Existing tests Author: Sean Owen Closes #16418 from srowen/SPARK-18993. (cherry picked from commit d7bce3bd31ec193274718042dc017706989d7563) Signed-off-by: Sean Owen Project: http://git-

spark git commit: [SPARK-19003][DOCS] Add Java example in Spark Streaming Guide, section Design Patterns for using foreachRDD

2016-12-29 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.1 80d583bd0 -> 47ab4afed [SPARK-19003][DOCS] Add Java example in Spark Streaming Guide, section Design Patterns for using foreachRDD ## What changes were proposed in this pull request? Added missing Java example under section "Design Pa

spark git commit: [SPARK-19003][DOCS] Add Java example in Spark Streaming Guide, section Design Patterns for using foreachRDD

2016-12-29 Thread srowen
Repository: spark Updated Branches: refs/heads/master 87bc4112c -> dba81e1dc [SPARK-19003][DOCS] Add Java example in Spark Streaming Guide, section Design Patterns for using foreachRDD ## What changes were proposed in this pull request? Added missing Java example under section "Design Patter

spark git commit: [SPARK-18808][ML][MLLIB] ml.KMeansModel.transform is very inefficient

2016-12-30 Thread srowen
oid this form of overhead on this and other code paths. ## How was this patch tested? Existing tests. Author: Sean Owen Closes #16328 from srowen/SPARK-18808. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/56d3a7eb Tree: h

spark git commit: [SPARK-18922][TESTS] Fix more path-related test failures on Windows

2016-12-30 Thread srowen
Repository: spark Updated Branches: refs/heads/master 56d3a7eb8 -> 852782b83 [SPARK-18922][TESTS] Fix more path-related test failures on Windows ## What changes were proposed in this pull request? This PR proposes to fix the test failures due to different format of paths on Windows. Failed

spark git commit: [BUILD] Close stale PRs

2017-01-02 Thread srowen
Repository: spark Updated Branches: refs/heads/master 239404737 -> ba4881268 [BUILD] Close stale PRs Closes #12968 Closes #16215 Closes #16212 Closes #16086 Closes #15713 Closes #16413 Closes #16396 Author: Sean Owen Closes #16447 from srowen/CloseStalePRs. Project: http://git-

spark git commit: [SPARK-19041][SS] Fix code snippet compilation issues in Structured Streaming Programming Guide

2017-01-02 Thread srowen
Repository: spark Updated Branches: refs/heads/master ba4881268 -> 808b84e2d [SPARK-19041][SS] Fix code snippet compilation issues in Structured Streaming Programming Guide ## What changes were proposed in this pull request? Currently some code snippets in the programming guide just do not c

spark git commit: [MINOR][DOC] Minor doc change for YARN credential providers

2017-01-02 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.1 3483defeb -> 63857c8d3 [MINOR][DOC] Minor doc change for YARN credential providers ## What changes were proposed in this pull request? The configuration `spark.yarn.security.tokens.{service}.enabled` is deprecated. Now we should use `

spark git commit: [MINOR][DOC] Minor doc change for YARN credential providers

2017-01-02 Thread srowen
Repository: spark Updated Branches: refs/heads/master 808b84e2d -> 0ac2f1e71 [MINOR][DOC] Minor doc change for YARN credential providers ## What changes were proposed in this pull request? The configuration `spark.yarn.security.tokens.{service}.enabled` is deprecated. Now we should use `spar

spark git commit: [SPARK-19022][TESTS] Fix tests dependent on OS due to different newline characters

2017-01-02 Thread srowen
Repository: spark Updated Branches: refs/heads/master 0ac2f1e71 -> f1330b1d9 [SPARK-19022][TESTS] Fix tests dependent on OS due to different newline characters ## What changes were proposed in this pull request? There are two tests failing on Windows due to the different newlines. ``` - St

spark git commit: [SPARK-19002][BUILD][PYTHON] Check pep8 against all Python scripts

2017-01-02 Thread srowen
Repository: spark Updated Branches: refs/heads/master f1330b1d9 -> 46b212602 [SPARK-19002][BUILD][PYTHON] Check pep8 against all Python scripts ## What changes were proposed in this pull request? This PR proposes to check pep8 against all other Python scripts and fix the errors as below: ``

spark git commit: [SPARK-18379][SQL] Make the parallelism of parallelPartitionDiscovery configurable.

2017-01-02 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.1 63857c8d3 -> 517f39833 [SPARK-18379][SQL] Make the parallelism of parallelPartitionDiscovery configurable. ## What changes were proposed in this pull request? The largest parallelism in PartitioningAwareFileIndex #listLeafFilesInPara

spark git commit: [MINOR] Add missing sc.stop() to end of examples

2017-01-03 Thread srowen
Repository: spark Updated Branches: refs/heads/master ae83c2112 -> e5c307c50 [MINOR] Add missing sc.stop() to end of examples ## What changes were proposed in this pull request? Add `finally` clause for `sc.stop()` in the `test("register and deregister Spark listener from SparkContext")`. #

spark git commit: [SPARK-19073] LauncherState should be only set to SUBMITTED after the application is submitted

2017-01-04 Thread srowen
Repository: spark Updated Branches: refs/heads/master cbd11d235 -> fe1c895e1 [SPARK-19073] LauncherState should be only set to SUBMITTED after the application is submitted ## What changes were proposed in this pull request? LauncherState should be only set to SUBMITTED after the application i

spark git commit: [SPARK-19054][ML] Eliminate extra pass in NB

2017-01-04 Thread srowen
Repository: spark Updated Branches: refs/heads/master 101556d0f -> 7a8250581 [SPARK-19054][ML] Eliminate extra pass in NB ## What changes were proposed in this pull request? eliminate unnecessary extra pass in NB's train ## How was this patch tested? existing tests Author: Zheng RuiFeng Cl

spark git commit: [MINOR][DOCS] Remove consecutive duplicated words/typo in Spark Repo

2017-01-04 Thread srowen
Repository: spark Updated Branches: refs/heads/master 7a8250581 -> a1e40b1f5 [MINOR][DOCS] Remove consecutive duplicated words/typo in Spark Repo ## What changes were proposed in this pull request? There are many locations in the Spark repo where the same word occurs consecutively. Sometimes

spark git commit: [SPARK-13748][PYSPARK][DOC] Add the description for explictly setting None for a named argument for a Row

2017-01-07 Thread srowen
Repository: spark Updated Branches: refs/heads/master d60f6f62d -> 68ea290b3 [SPARK-13748][PYSPARK][DOC] Add the description for explictly setting None for a named argument for a Row ## What changes were proposed in this pull request? It seems allowed to not set a key and value for a dict to

spark git commit: [MINOR] Bump R version to 2.2.0.

2017-01-07 Thread srowen
Repository: spark Updated Branches: refs/heads/master 68ea290b3 -> cdda3372a [MINOR] Bump R version to 2.2.0. ## What changes were proposed in this pull request? #16126 bumps master branch version to 2.2.0-SNAPSHOT, but it seems R version was omitted. ## How was this patch tested? N/A Autho

spark git commit: [SPARK-19106][DOCS] Styling for the configuration docs is broken

2017-01-07 Thread srowen
n't rendering, being recognized correctly. Removed extra p tags and pulled level 4 titles up to level 3, since level 3 had been skipped. This improves the TOC. ## How was this patch tested? Doc build, manual check. Author: Sean Owen Closes #16490 from srowen/SPARK-19106. Project: http:/

spark git commit: [SPARK-19106][DOCS] Styling for the configuration docs is broken

2017-01-07 Thread srowen
extra p tags and pulled level 4 titles up to level 3, since level 3 had been skipped. This improves the TOC. Doc build, manual check. Author: Sean Owen Closes #16490 from srowen/SPARK-19106. (cherry picked from commit 54138f6e89abfc17101b4f2812715784a2b98331) Signed-off-by: Sean Owen Proje

spark git commit: [SPARK-19026] SPARK_LOCAL_DIRS(multiple directories on different disks) cannot be deleted

2017-01-08 Thread srowen
Repository: spark Updated Branches: refs/heads/master 6b6b555a1 -> cd1d00ada [SPARK-19026] SPARK_LOCAL_DIRS(multiple directories on different disks) cannot be deleted JIRA Issue: https://issues.apache.org/jira/browse/SPARK-19026 SPARK_LOCAL_DIRS (Standalone) can be a comma-separated list of

spark git commit: [SPARK-17645][MLLIB][ML][FOLLOW-UP] document minor change

2017-01-10 Thread srowen
Repository: spark Updated Branches: refs/heads/master acfc5f354 -> 32286ba68 [SPARK-17645][MLLIB][ML][FOLLOW-UP] document minor change ## What changes were proposed in this pull request? Add FDR test case in ml/feature/ChiSqSelectorSuite. Improve some comments in the code. This is a follow-up

spark git commit: [SPARK-18922][SQL][CORE][STREAMING][TESTS] Fix all identified tests failed due to path and resource-not-closed problems on Windows

2017-01-10 Thread srowen
Repository: spark Updated Branches: refs/heads/master 32286ba68 -> 4e27578fa [SPARK-18922][SQL][CORE][STREAMING][TESTS] Fix all identified tests failed due to path and resource-not-closed problems on Windows ## What changes were proposed in this pull request? This PR proposes to fix all the

spark git commit: [SPARK-19117][TESTS] Skip the tests using script transformation on Windows

2017-01-10 Thread srowen
Repository: spark Updated Branches: refs/heads/master 4e27578fa -> 2cfd41ac0 [SPARK-19117][TESTS] Skip the tests using script transformation on Windows ## What changes were proposed in this pull request? This PR proposes to skip the tests for script transformation failed on Windows due to fi

spark git commit: [SPARK-18857][SQL] Don't use `Iterator.duplicate` for `incrementalCollect` in Thrift Server

2017-01-10 Thread srowen
Repository: spark Updated Branches: refs/heads/master 2cfd41ac0 -> a2c6adcc5 [SPARK-18857][SQL] Don't use `Iterator.duplicate` for `incrementalCollect` in Thrift Server ## What changes were proposed in this pull request? To support `FETCH_FIRST`, SPARK-16563 used Scala `Iterator.duplicate`.

spark git commit: [SPARK-19113][SS][TESTS] Set UncaughtExceptionHandler in onQueryStarted to ensure catching fatal errors during query initialization

2017-01-10 Thread srowen
Repository: spark Updated Branches: refs/heads/master a2c6adcc5 -> 3ef183a94 [SPARK-19113][SS][TESTS] Set UncaughtExceptionHandler in onQueryStarted to ensure catching fatal errors during query initialization ## What changes were proposed in this pull request? StreamTest sets `UncaughtExcept

spark-website git commit: First Java example does not work with recent Spark version (see https://issues.apache.org/jira/browse/SPARK-19156)

2017-01-11 Thread srowen
Repository: spark-website Updated Branches: refs/heads/asf-site 46a7a8027 -> e95223137 First Java example does not work with recent Spark version (see https://issues.apache.org/jira/browse/SPARK-19156) Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-

spark git commit: [SPARK-20085][MESOS] Configurable mesos labels for executors

2017-04-06 Thread srowen
Repository: spark Updated Branches: refs/heads/master e156b5dd3 -> c8fc1f3ba [SPARK-20085][MESOS] Configurable mesos labels for executors ## What changes were proposed in this pull request? Add spark.mesos.task.labels configuration option to add mesos key:value labels to the executor. "k1:

spark git commit: [SPARK-20064][PYSPARK] Bump the PySpark verison number to 2.2

2017-04-06 Thread srowen
Repository: spark Updated Branches: refs/heads/master c8fc1f3ba -> d009fb369 [SPARK-20064][PYSPARK] Bump the PySpark verison number to 2.2 ## What changes were proposed in this pull request? PySpark version in version.py was lagging behind Versioning is in line with PEP 440: https://www.pytho

spark git commit: [SPARK-20218][DOC][APP-ID] applications//stages' in REST API, add description.

2017-04-07 Thread srowen
Repository: spark Updated Branches: refs/heads/master 1a52a6237 -> 9e0893b53 [SPARK-20218][DOC][APP-ID] applications//stages' in REST API,add description. ## What changes were proposed in this pull request? 1. '/applications/[app-id]/stages' in rest api.status should add description '?status

spark git commit: [SPARK-20218][DOC][APP-ID] applications//stages' in REST API, add description.

2017-04-07 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.1 fb81a412e -> 77911201c [SPARK-20218][DOC][APP-ID] applications//stages' in REST API,add description. ## What changes were proposed in this pull request? 1. '/applications/[app-id]/stages' in rest api.status should add description '?st

spark git commit: [MINOR] Issue: Change "slice" vs "partition" in exception messages (and code?)

2017-04-08 Thread srowen
Repository: spark Updated Branches: refs/heads/master e1afc4dcc -> 34fc48fb5 [MINOR] Issue: Change "slice" vs "partition" in exception messages (and code?) ## What changes were proposed in this pull request? Came across the term "slice" when running some spark scala code. Consequently, a Goo

spark git commit: [SPARK-19991][CORE][YARN] FileSegmentManagedBuffer performance improvement

2017-04-09 Thread srowen
ack to default. This apparently causes non-trivial overhead in at least one path, and can easily be avoided. See https://github.com/apache/spark/pull/17329 ## How was this patch tested? Existing tests Author: Sean Owen Closes #17567 from srowen/SPARK-19991. Project: http://git-

spark git commit: [SPARK-20260][MLLIB] String interpolation required for error message

2017-04-09 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.1 658b35885 -> 43a7fcad5 [SPARK-20260][MLLIB] String interpolation required for error message ## What changes were proposed in this pull request? This error message doesn't get properly formatted because of a missing `s`. Currently the

spark git commit: [SPARK-20260][MLLIB] String interpolation required for error message

2017-04-09 Thread srowen
Repository: spark Updated Branches: refs/heads/master 1f0de3c1c -> 261eaf514 [SPARK-20260][MLLIB] String interpolation required for error message ## What changes were proposed in this pull request? This error message doesn't get properly formatted because of a missing `s`. Currently the erro

spark-website git commit: Add R code style guide

2017-04-10 Thread srowen
Repository: spark-website Updated Branches: refs/heads/asf-site 8b27c470c -> d39c4ecac Add R code style guide Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-us.apache.org/repos/asf/spark-website/commit/d39c4eca Tree: http://git-wip-us.apache.org/repo

[3/3] spark git commit: [SPARK-20156][CORE][SQL][STREAMING][MLLIB] Java String toLowerCase "Turkish locale bug" causes Spark problems

2017-04-10 Thread srowen
riation in behavior (aka the "Turkish locale problem"). The change looks large but it is just adding `Locale.ROOT` (the locale with no country or language specified) to every call to these methods. ## How was this patch tested? Existing tests. Author: Sean Owen Closes #17527 from srowen/SP

[1/3] spark git commit: [SPARK-20156][CORE][SQL][STREAMING][MLLIB] Java String toLowerCase "Turkish locale bug" causes Spark problems

2017-04-10 Thread srowen
Repository: spark Updated Branches: refs/heads/master fd711ea13 -> a26e3ed5e http://git-wip-us.apache.org/repos/asf/spark/blob/a26e3ed5/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala ---

[2/3] spark git commit: [SPARK-20156][CORE][SQL][STREAMING][MLLIB] Java String toLowerCase "Turkish locale bug" causes Spark problems

2017-04-10 Thread srowen
http://git-wip-us.apache.org/repos/asf/spark/blob/a26e3ed5/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala -- diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expre

spark git commit: Document Master URL format in high availability set up

2017-04-11 Thread srowen
Repository: spark Updated Branches: refs/heads/master 0d2b79642 -> d11ef3d77 Document Master URL format in high availability set up ## What changes were proposed in this pull request? Add documentation for adding master url in multi host, port format for standalone cluster with high availabi

spark git commit: [MINOR][DOCS] Fix spacings in Structured Streaming Programming Guide

2017-04-12 Thread srowen
Repository: spark Updated Branches: refs/heads/master b14bfc3f8 -> b93843824 [MINOR][DOCS] Fix spacings in Structured Streaming Programming Guide ## What changes were proposed in this pull request? 1. Omitted space between the sentences: `... on static data.The Spark SQL engine will ...` ->

spark git commit: [MINOR][DOCS] Fix spacings in Structured Streaming Programming Guide

2017-04-12 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.1 46e212d2f -> b2970d971 [MINOR][DOCS] Fix spacings in Structured Streaming Programming Guide ## What changes were proposed in this pull request? 1. Omitted space between the sentences: `... on static data.The Spark SQL engine will ...`

spark git commit: [MINOR][DOCS] JSON APIs related documentation fixes

2017-04-12 Thread srowen
Repository: spark Updated Branches: refs/heads/master b93843824 -> bca4259f1 [MINOR][DOCS] JSON APIs related documentation fixes ## What changes were proposed in this pull request? This PR proposes corrections related to JSON APIs as below: - Rendering links in Python documentation - Replaci

spark git commit: [SPARK-20298][SPARKR][MINOR] fixed spelling mistake "charactor"

2017-04-12 Thread srowen
Repository: spark Updated Branches: refs/heads/master bca4259f1 -> 044f7ecbf [SPARK-20298][SPARKR][MINOR] fixed spelling mistake "charactor" ## What changes were proposed in this pull request? Fixed spelling of "charactor" ## How was this patch tested? Spelling change only Author: Brendan

spark git commit: [SPARK-20296][TRIVIAL][DOCS] Count distinct error message for streaming

2017-04-12 Thread srowen
Repository: spark Updated Branches: refs/heads/master ffc57b011 -> 2e1fd46e1 [SPARK-20296][TRIVIAL][DOCS] Count distinct error message for streaming ## What changes were proposed in this pull request? Update count distinct error message for streaming datasets/dataframes to match current behav

spark git commit: [SPARK-20296][TRIVIAL][DOCS] Count distinct error message for streaming

2017-04-12 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.1 b2970d971 -> dbb6d1b44 [SPARK-20296][TRIVIAL][DOCS] Count distinct error message for streaming ## What changes were proposed in this pull request? Update count distinct error message for streaming datasets/dataframes to match current b

spark git commit: [SPARK-18692][BUILD][DOCS] Test Java 8 unidoc build on Jenkins

2017-04-12 Thread srowen
Repository: spark Updated Branches: refs/heads/master 2e1fd46e1 -> ceaf77ae4 [SPARK-18692][BUILD][DOCS] Test Java 8 unidoc build on Jenkins ## What changes were proposed in this pull request? This PR proposes to run Spark unidoc to test Javadoc 8 build as Javadoc 8 is easily re-breakable. T

spark git commit: [SPARK-20189][DSTREAM] Fix spark kinesis testcases to remove deprecated createStream and use Builders

2017-04-13 Thread srowen
Repository: spark Updated Branches: refs/heads/master c5f1cc370 -> ec68d8f8c [SPARK-20189][DSTREAM] Fix spark kinesis testcases to remove deprecated createStream and use Builders ## What changes were proposed in this pull request? The spark-kinesis testcases use the KinesisUtils.createStream

spark git commit: [SPARK-20265][MLLIB] Improve Prefix'span pre-processing efficiency

2017-04-13 Thread srowen
Repository: spark Updated Branches: refs/heads/master ec68d8f8c -> 095d1cb3a [SPARK-20265][MLLIB] Improve Prefix'span pre-processing efficiency ## What changes were proposed in this pull request? Improve PrefixSpan pre-processing efficency by preventing sequences of zero in the cleaned datab

spark git commit: [SPARK-20284][CORE] Make {Des, S}erializationStream extend Closeable

2017-04-13 Thread srowen
Repository: spark Updated Branches: refs/heads/master 095d1cb3a -> a4293c284 [SPARK-20284][CORE] Make {Des,S}erializationStream extend Closeable ## What changes were proposed in this pull request? This PR allows to use `SerializationStream` and `DeserializationStream` in try-with-resources.

[2/2] spark-website git commit: regenerate html pointing to 2.11

2017-04-14 Thread srowen
regenerate html pointing to 2.11 Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-us.apache.org/repos/asf/spark-website/commit/fe9e3a88 Tree: http://git-wip-us.apache.org/repos/asf/spark-website/tree/fe9e3a88 Diff: http://git-wip-us.apache.org/repos/asf/sp

[1/2] spark-website git commit: add intellij information

2017-04-14 Thread srowen
Repository: spark-website Updated Branches: refs/heads/add_more_intellij_instructions [created] fe9e3a88c add intellij information Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-us.apache.org/repos/asf/spark-website/commit/cbe2a9b8 Tree: http://git-w

[2/2] spark-website git commit: regenerate html pointing to 2.11

2017-04-14 Thread srowen
regenerate html pointing to 2.11 Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-us.apache.org/repos/asf/spark-website/commit/fe9e3a88 Tree: http://git-wip-us.apache.org/repos/asf/spark-website/tree/fe9e3a88 Diff: http://git-wip-us.apache.org/repos/asf/sp

[1/2] spark-website git commit: add intellij information

2017-04-14 Thread srowen
Repository: spark-website Updated Branches: refs/heads/asf-site d39c4ecac -> fe9e3a88c add intellij information Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-us.apache.org/repos/asf/spark-website/commit/cbe2a9b8 Tree: http://git-wip-us.apache.org/re

[spark-website] Git Push Summary

2017-04-14 Thread srowen
Repository: spark-website Updated Branches: refs/heads/add_more_intellij_instructions [deleted] fe9e3a88c - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.

spark git commit: [SPARK-20316][SQL] Val and Var should strictly follow the Scala syntax

2017-04-15 Thread srowen
Repository: spark Updated Branches: refs/heads/master fb036c441 -> 98b41ecbc [SPARK-20316][SQL] Val and Var should strictly follow the Scala syntax ## What changes were proposed in this pull request? val and var should strictly follow the Scala syntax ## How was this patch tested? manual te

spark git commit: [SPARK-19740][MESOS] Add support in Spark to pass arbitrary parameters into docker when running on mesos with docker containerizer

2017-04-16 Thread srowen
Repository: spark Updated Branches: refs/heads/master e090f3c0c -> a888fed30 [SPARK-19740][MESOS] Add support in Spark to pass arbitrary parameters into docker when running on mesos with docker containerizer ## What changes were proposed in this pull request? Allow passing in arbitrary param

spark git commit: [SPARK-20343][BUILD] Add avro dependency in core POM to resolve build failure in SBT Hadoop 2.6 master on Jenkins

2017-04-16 Thread srowen
Repository: spark Updated Branches: refs/heads/master a888fed30 -> ad935f526 [SPARK-20343][BUILD] Add avro dependency in core POM to resolve build failure in SBT Hadoop 2.6 master on Jenkins ## What changes were proposed in this pull request? This PR proposes to add ``` org.a

[1/3] spark-website git commit: Alternate commit of PR 44 -- add new Spark Summits

2017-04-18 Thread srowen
Repository: spark-website Updated Branches: refs/heads/asf-site fe9e3a88c -> cc799c06f http://git-wip-us.apache.org/repos/asf/spark-website/blob/cc799c06/site/releases/spark-release-1-6-3.html -- diff --git a/site/releases/spar

[3/3] spark-website git commit: Alternate commit of PR 44 -- add new Spark Summits

2017-04-18 Thread srowen
Alternate commit of PR 44 -- add new Spark Summits Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-us.apache.org/repos/asf/spark-website/commit/cc799c06 Tree: http://git-wip-us.apache.org/repos/asf/spark-website/tree/cc799c06 Diff: http://git-wip-us.apach

[2/3] spark-website git commit: Alternate commit of PR 44 -- add new Spark Summits

2017-04-18 Thread srowen
http://git-wip-us.apache.org/repos/asf/spark-website/blob/cc799c06/site/news/spark-and-shark-in-the-news.html -- diff --git a/site/news/spark-and-shark-in-the-news.html b/site/news/spark-and-shark-in-the-news.html index 714fe85..9

spark-website git commit: Oops, forgot to add new generated HTML post

2017-04-18 Thread srowen
Repository: spark-website Updated Branches: refs/heads/asf-site cc799c06f -> aa1c66e42 Oops, forgot to add new generated HTML post Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-us.apache.org/repos/asf/spark-website/commit/aa1c66e4 Tree: http://git-w

spark git commit: [SPARK-20344][SCHEDULER] Duplicate call in FairSchedulableBuilder.addTaskSetManager

2017-04-18 Thread srowen
Repository: spark Updated Branches: refs/heads/master b0a1e93e9 -> 07fd94e0d [SPARK-20344][SCHEDULER] Duplicate call in FairSchedulableBuilder.addTaskSetManager ## What changes were proposed in this pull request? Eliminate the duplicate call to `Pool.getSchedulableByName()` in `FairSchedula

spark git commit: [SPARK-20343][BUILD] Force Avro 1.7.7 in sbt build to resolve build failure in SBT Hadoop 2.6 master on Jenkins

2017-04-18 Thread srowen
Repository: spark Updated Branches: refs/heads/master 07fd94e0d -> d4f10cbbe [SPARK-20343][BUILD] Force Avro 1.7.7 in sbt build to resolve build failure in SBT Hadoop 2.6 master on Jenkins ## What changes were proposed in this pull request? This PR proposes to force Avro's version to 1.7.7 i

spark git commit: [SPARK-20343][BUILD] Avoid Unidoc build only if Hadoop 2.6 is explicitly set in SBT build

2017-04-19 Thread srowen
Repository: spark Updated Branches: refs/heads/master 773754b6c -> 35378766a [SPARK-20343][BUILD] Avoid Unidoc build only if Hadoop 2.6 is explicitly set in SBT build ## What changes were proposed in this pull request? This PR proposes two things as below: - Avoid Unidoc build only if Hadoo

spark git commit: [SPARK-20343][BUILD] Avoid Unidoc build only if Hadoop 2.6 is explicitly set in SBT build

2017-04-19 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.2 a87e21dd2 -> 8baa970bc [SPARK-20343][BUILD] Avoid Unidoc build only if Hadoop 2.6 is explicitly set in SBT build ## What changes were proposed in this pull request? This PR proposes two things as below: - Avoid Unidoc build only if H

spark git commit: [SPARK-20036][DOC] Note incompatible dependencies on org.apache.kafka artifacts

2017-04-19 Thread srowen
Repository: spark Updated Branches: refs/heads/master 35378766a -> 71a8e9df1 [SPARK-20036][DOC] Note incompatible dependencies on org.apache.kafka artifacts ## What changes were proposed in this pull request? Note that you shouldn't manually add dependencies on org.apache.kafka artifacts ##

<    2   3   4   5   6   7   8   9   10   11   >