[GitHub] spark issue #21007: [SPARK-23942][PYTHON][SQL] Makes collect in PySpark as a...

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21007 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89290/ Test FAILed. ---

[GitHub] spark issue #21007: [SPARK-23942][PYTHON][SQL] Makes collect in PySpark as a...

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21007 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21007: [SPARK-23942][PYTHON][SQL] Makes collect in PySpark as a...

2018-04-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21007 **[Test build #89290 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89290/testReport)** for PR 21007 at commit [`7c1b3c6`](https://github.com/apache/spark/commit/7

[GitHub] spark issue #20858: [SPARK-23736][SQL] Extending the concat function to supp...

2018-04-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20858 **[Test build #89299 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89299/testReport)** for PR 20858 at commit [`57b250c`](https://github.com/apache/spark/commit/57

[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...

2018-04-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21056 **[Test build #89298 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89298/testReport)** for PR 21056 at commit [`6aa36d4`](https://github.com/apache/spark/commit/6a

[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...

2018-04-12 Thread MaxGekk
Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/21056 jenkins, retest this, please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

2018-04-12 Thread madanadit
Github user madanadit commented on the issue: https://github.com/apache/spark/pull/21032 @foxish @liyinan926 Thanks for your comments. To make the configuration more general, here is an initial proposal. Context: - volumeMount requires `[name, mountPath, readWriteMode]`

[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21056 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21056 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89289/ Test FAILed. ---

[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...

2018-04-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21056 **[Test build #89289 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89289/testReport)** for PR 21056 at commit [`6aa36d4`](https://github.com/apache/spark/commit/6

[GitHub] spark issue #21058: [SPARK-23971] Should not leak Spark sessions across test...

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21058 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21058: [SPARK-23971] Should not leak Spark sessions across test...

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21058 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2287/ Tes

[GitHub] spark issue #20983: [SPARK-23747][Structured Streaming] Add EpochCoordinator...

2018-04-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20983 **[Test build #89297 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89297/testReport)** for PR 20983 at commit [`89a1e11`](https://github.com/apache/spark/commit/89

[GitHub] spark issue #20983: [SPARK-23747][Structured Streaming] Add EpochCoordinator...

2018-04-12 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/20983 jenkins test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark issue #20983: [SPARK-23747][Structured Streaming] Add EpochCoordinator...

2018-04-12 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/20983 jenkins ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@

[GitHub] spark issue #21058: [SPARK-23971] Should not leak Spark sessions across test...

2018-04-12 Thread ericl
Github user ericl commented on the issue: https://github.com/apache/spark/pull/21058 This is a followup to https://github.com/apache/spark/pull/20971 @gatorsmile --- - To unsubscribe, e-mail: reviews-unsubscr...@sp

[GitHub] spark issue #21058: [SPARK-23971] Should not leak Spark sessions across test...

2018-04-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21058 **[Test build #89296 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89296/testReport)** for PR 21058 at commit [`92afcc2`](https://github.com/apache/spark/commit/92

[GitHub] spark pull request #21058: [SPARK-23971] Should not leak Spark sessions acro...

2018-04-12 Thread ericl
GitHub user ericl opened a pull request: https://github.com/apache/spark/pull/21058 [SPARK-23971] Should not leak Spark sessions across test suites ## What changes were proposed in this pull request? Many suites currently leak Spark sessions (sometimes with stopped SparkCon

[GitHub] spark pull request #21033: [SPARK-19320][MESOS][WIP]allow specifying a hard ...

2018-04-12 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/21033#discussion_r181199842 --- Diff: resource-managers/mesos/src/test/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackendSuite.scala --- @@ -165,18 +165

[GitHub] spark issue #20923: [SPARK-23807][BUILD][WIP] Add Hadoop 3 profile with rele...

2018-04-12 Thread steveloughran
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/20923 The jetty problem has been dealt with; because of the shading declaration of jetty-util as provided (it isn't needed in spark any more), it wasn't getting into dist/jars even for those depende

[GitHub] spark pull request #21032: [SPARK-23529][K8s] Support mounting hostPath volu...

2018-04-12 Thread madanadit
Github user madanadit commented on a diff in the pull request: https://github.com/apache/spark/pull/21032#discussion_r181197979 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala --- @@ -117,6 +117,13 @@ private[spark] object Config

[GitHub] spark issue #20923: [SPARK-23807][BUILD][WIP] Add Hadoop 3 profile with rele...

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20923 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #21045: [WIP][SPARK-23931][SQL] Adds zip function to spar...

2018-04-12 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r181196604 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -87,6 +87,62 @@ case class MapKeys(ch

[GitHub] spark issue #20923: [SPARK-23807][BUILD][WIP] Add Hadoop 3 profile with rele...

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20923 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2286/ Tes

[GitHub] spark issue #20923: [SPARK-23807][BUILD][WIP] Add Hadoop 3 profile with rele...

2018-04-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20923 **[Test build #89295 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89295/testReport)** for PR 20923 at commit [`f6b9dc8`](https://github.com/apache/spark/commit/f6

[GitHub] spark issue #21054: [SPARK-23907][SQL] Add regr_* functions

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21054 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21054: [SPARK-23907][SQL] Add regr_* functions

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21054 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89278/ Test PASSed. ---

[GitHub] spark issue #21054: [SPARK-23907][SQL] Add regr_* functions

2018-04-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21054 **[Test build #89278 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89278/testReport)** for PR 21054 at commit [`90c266e`](https://github.com/apache/spark/commit/9

[GitHub] spark pull request #20838: [SPARK-23698] Resolve undefined names in Python 3

2018-04-12 Thread cclauss
Github user cclauss commented on a diff in the pull request: https://github.com/apache/spark/pull/20838#discussion_r181192109 --- Diff: python/pyspark/cloudpickle.py --- @@ -802,9 +802,8 @@ def save_not_implemented(self, obj): self.save_reduce(_gen_not_implemented, ())

[GitHub] spark issue #20838: [SPARK-23698] Resolve undefined names in Python 3

2018-04-12 Thread cclauss
Github user cclauss commented on the issue: https://github.com/apache/spark/pull/20838 Other issues / suggestions? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: r

[GitHub] spark issue #21024: [SPARK-23917][SQL] Add array_max function

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21024 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89271/ Test FAILed. ---

[GitHub] spark issue #21024: [SPARK-23917][SQL] Add array_max function

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21024 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21024: [SPARK-23917][SQL] Add array_max function

2018-04-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21024 **[Test build #89271 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89271/testReport)** for PR 21024 at commit [`e739a0a`](https://github.com/apache/spark/commit/e

[GitHub] spark pull request #21045: [WIP][SPARK-23931][SQL] Adds zip function to spar...

2018-04-12 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r181189694 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -87,6 +87,62 @@ case class MapKeys(ch

[GitHub] spark pull request #21045: [WIP][SPARK-23931][SQL] Adds zip function to spar...

2018-04-12 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r181189546 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -87,6 +87,62 @@ case class MapKeys(ch

[GitHub] spark issue #20828: [SPARK-23687][SS] Add a memory source for continuous pro...

2018-04-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20828 **[Test build #89294 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89294/testReport)** for PR 20828 at commit [`7b2a964`](https://github.com/apache/spark/commit/7b

[GitHub] spark issue #20828: [SPARK-23687][SS] Add a memory source for continuous pro...

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20828 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89293/ Test FAILed. ---

[GitHub] spark issue #20828: [SPARK-23687][SS] Add a memory source for continuous pro...

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20828 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #20828: [SPARK-23687][SS] Add a memory source for continu...

2018-04-12 Thread jose-torres
Github user jose-torres commented on a diff in the pull request: https://github.com/apache/spark/pull/20828#discussion_r181187372 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/ContinuousMemoryStream.scala --- @@ -0,0 +1,211 @@ +/* + * L

[GitHub] spark pull request #20828: [SPARK-23687][SS] Add a memory source for continu...

2018-04-12 Thread jose-torres
Github user jose-torres commented on a diff in the pull request: https://github.com/apache/spark/pull/20828#discussion_r181186281 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/streaming/continuous/ContinuousSuite.scala --- @@ -53,32 +54,24 @@ class ContinuousSuiteBase ext

[GitHub] spark issue #21050: [SPARK-23912][SQL]add array_distinct

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21050 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89276/ Test PASSed. ---

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

2018-04-12 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21032 Yes, that is what I was indicating as well. We should have a design for all different volume types. Extending this PR as the design evolves is a good way to go about that. On Thu, Apr

[GitHub] spark issue #21050: [SPARK-23912][SQL]add array_distinct

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21050 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21050: [SPARK-23912][SQL]add array_distinct

2018-04-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21050 **[Test build #89276 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89276/testReport)** for PR 21050 at commit [`3bfb23e`](https://github.com/apache/spark/commit/3

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

2018-04-12 Thread liyinan926
Github user liyinan926 commented on the issue: https://github.com/apache/spark/pull/21032 I think we should have a more general solution so it works for other types of volumes. Particularly, we should have a design discussion on the format of values of such a property so it works for

[GitHub] spark pull request #20828: [SPARK-23687][SS] Add a memory source for continu...

2018-04-12 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20828#discussion_r181181014 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/ContinuousMemoryStream.scala --- @@ -0,0 +1,211 @@ +/* + * Licensed

[GitHub] spark pull request #20828: [SPARK-23687][SS] Add a memory source for continu...

2018-04-12 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20828#discussion_r181178566 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/streaming/continuous/ContinuousSuite.scala --- @@ -53,32 +54,24 @@ class ContinuousSuiteBase extends St

[GitHub] spark pull request #20828: [SPARK-23687][SS] Add a memory source for continu...

2018-04-12 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20828#discussion_r181180114 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/streaming/continuous/ContinuousSuite.scala --- @@ -53,32 +54,24 @@ class ContinuousSuiteBase extends St

[GitHub] spark pull request #20828: [SPARK-23687][SS] Add a memory source for continu...

2018-04-12 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20828#discussion_r181180748 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/ContinuousMemoryStream.scala --- @@ -0,0 +1,211 @@ +/* + * Licensed

[GitHub] spark pull request #20828: [SPARK-23687][SS] Add a memory source for continu...

2018-04-12 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20828#discussion_r181180867 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/ContinuousMemoryStream.scala --- @@ -0,0 +1,211 @@ +/* + * Licensed

[GitHub] spark pull request #20828: [SPARK-23687][SS] Add a memory source for continu...

2018-04-12 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20828#discussion_r181180235 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/ContinuousMemoryStream.scala --- @@ -0,0 +1,211 @@ +/* + * Licensed

[GitHub] spark pull request #20828: [SPARK-23687][SS] Add a memory source for continu...

2018-04-12 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20828#discussion_r181177756 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/memory.scala --- @@ -43,20 +45,51 @@ object MemoryStream { protected val cur

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

2018-04-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21032 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2240/ ---

[GitHub] spark issue #21004: [SPARK-23896][SQL]Improve PartitioningAwareFileIndex

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21004 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89277/ Test FAILed. ---

[GitHub] spark issue #21004: [SPARK-23896][SQL]Improve PartitioningAwareFileIndex

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21004 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21004: [SPARK-23896][SQL]Improve PartitioningAwareFileIndex

2018-04-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21004 **[Test build #89277 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89277/testReport)** for PR 21004 at commit [`60d5b6b`](https://github.com/apache/spark/commit/6

[GitHub] spark issue #21007: [SPARK-23942][PYTHON][SQL] Makes collect in PySpark as a...

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21007 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2285/ Tes

[GitHub] spark pull request #21032: [SPARK-23529][K8s] Support mounting hostPath volu...

2018-04-12 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/21032#discussion_r181176057 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala --- @@ -117,6 +117,13 @@ private[spark] object Config ex

[GitHub] spark issue #21007: [SPARK-23942][PYTHON][SQL] Makes collect in PySpark as a...

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21007 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

2018-04-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21032 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2240/ --- --

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21032 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2284/ Tes

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21032 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

2018-04-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21032 **[Test build #89291 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89291/testReport)** for PR 21032 at commit [`c72cbc1`](https://github.com/apache/spark/commit/c

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21032 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89291/ Test FAILed. ---

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21032 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #21032: [SPARK-23529][K8s] Support mounting hostPath volu...

2018-04-12 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/21032#discussion_r181172885 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala --- @@ -117,6 +117,13 @@ private[spark] object Config ex

[GitHub] spark issue #20828: [SPARK-23687][SS] Add a memory source for continuous pro...

2018-04-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20828 **[Test build #89292 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89292/testReport)** for PR 20828 at commit [`6e9a2c1`](https://github.com/apache/spark/commit/6e

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

2018-04-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21032 **[Test build #89291 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89291/testReport)** for PR 21032 at commit [`c72cbc1`](https://github.com/apache/spark/commit/c7

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

2018-04-12 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21032 Jenkins, ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h.

[GitHub] spark issue #21007: [SPARK-23942][PYTHON][SQL] Makes collect in PySpark as a...

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21007 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21007: [SPARK-23942][PYTHON][SQL] Makes collect in PySpark as a...

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21007 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2283/ Tes

[GitHub] spark issue #20910: [SPARK-22839] [K8s] Refactor to unify driver and executo...

2018-04-12 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/20910 @mccheah I think it's fine as is now. We can take care of moving abstractions between the submission client and the driver in a future PR if necessary. Just the scala style issue needs taking care of

[GitHub] spark issue #21007: [SPARK-23942][PYTHON][SQL] Makes collect in PySpark as a...

2018-04-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21007 **[Test build #89290 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89290/testReport)** for PR 21007 at commit [`7c1b3c6`](https://github.com/apache/spark/commit/7c

[GitHub] spark issue #21057: 2 Improvements to Pyspark docs

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21057 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21007: [SPARK-23942][PYTHON][SQL] Makes collect in PySpark as a...

2018-04-12 Thread BryanCutler
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/21007 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: review

[GitHub] spark issue #21025: [SPARK-23918][SQL] Add array_min function

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21025 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89270/ Test PASSed. ---

[GitHub] spark issue #21025: [SPARK-23918][SQL] Add array_min function

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21025 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21025: [SPARK-23918][SQL] Add array_min function

2018-04-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21025 **[Test build #89270 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89270/testReport)** for PR 21025 at commit [`fc51766`](https://github.com/apache/spark/commit/f

[GitHub] spark issue #21057: 2 Improvements to Pyspark docs

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21057 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21057: 2 Improvements to Pyspark docs

2018-04-12 Thread aviv-ebates
GitHub user aviv-ebates opened a pull request: https://github.com/apache/spark/pull/21057 2 Improvements to Pyspark docs Each of these 2 items has cost me a few hours of debugging. Hopefully, this will stop others from having to debug the same thing. 1. Describe the expecte

[GitHub] spark issue #21007: [SPARK-23942][PYTHON][SQL] Makes collect in PySpark as a...

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21007 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21007: [SPARK-23942][PYTHON][SQL] Makes collect in PySpark as a...

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21007 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89279/ Test FAILed. ---

[GitHub] spark issue #21007: [SPARK-23942][PYTHON][SQL] Makes collect in PySpark as a...

2018-04-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21007 **[Test build #89279 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89279/testReport)** for PR 21007 at commit [`7c1b3c6`](https://github.com/apache/spark/commit/7

[GitHub] spark issue #21004: [SPARK-23896][SQL]Improve PartitioningAwareFileIndex

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21004 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2282/ Tes

[GitHub] spark issue #21004: [SPARK-23896][SQL]Improve PartitioningAwareFileIndex

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21004 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...

2018-04-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21056 **[Test build #89289 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89289/testReport)** for PR 21056 at commit [`6aa36d4`](https://github.com/apache/spark/commit/6a

[GitHub] spark issue #21048: [SPARK-23966][SS] Refactoring all checkpoint file writin...

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21048 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21048: [SPARK-23966][SS] Refactoring all checkpoint file writin...

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21048 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2281/ Tes

[GitHub] spark issue #21004: [SPARK-23896][SQL]Improve PartitioningAwareFileIndex

2018-04-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21004 **[Test build #89288 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89288/testReport)** for PR 21004 at commit [`60d5b6b`](https://github.com/apache/spark/commit/60

[GitHub] spark issue #21048: [SPARK-23966][SS] Refactoring all checkpoint file writin...

2018-04-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21048 **[Test build #89287 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89287/testReport)** for PR 21048 at commit [`12177b1`](https://github.com/apache/spark/commit/12

[GitHub] spark issue #21004: [SPARK-23896][SQL]Improve PartitioningAwareFileIndex

2018-04-12 Thread gengliangwang
Github user gengliangwang commented on the issue: https://github.com/apache/spark/pull/21004 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: rev

[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21056 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89285/ Test FAILed. ---

[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...

2018-04-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21056 **[Test build #89285 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89285/testReport)** for PR 21056 at commit [`a31b2e2`](https://github.com/apache/spark/commit/a

[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21056 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...

2018-04-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21056 **[Test build #89285 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89285/testReport)** for PR 21056 at commit [`a31b2e2`](https://github.com/apache/spark/commit/a3

[GitHub] spark issue #20858: [SPARK-23736][SQL] Extending the concat function to supp...

2018-04-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20858 **[Test build #89286 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89286/testReport)** for PR 20858 at commit [`6bb33e6`](https://github.com/apache/spark/commit/6b

[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...

2018-04-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21056 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21056: [SPARK-23849][SQL] Tests for samplingRatio of jso...

2018-04-12 Thread MaxGekk
GitHub user MaxGekk opened a pull request: https://github.com/apache/spark/pull/21056 [SPARK-23849][SQL] Tests for samplingRatio of json datasource ## What changes were proposed in this pull request? Added the `samplingRatio` option to the `json()` method of PySpark DataFra

[GitHub] spark pull request #20959: [SPARK-23846][SQL] The samplingRatio option for C...

2018-04-12 Thread MaxGekk
Github user MaxGekk commented on a diff in the pull request: https://github.com/apache/spark/pull/20959#discussion_r181165340 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala --- @@ -528,6 +529,7 @@ class DataFrameReader private[sql](sparkSession: Spark

[GitHub] spark pull request #20828: [SPARK-23687][SS] Add a memory source for continu...

2018-04-12 Thread jose-torres
Github user jose-torres commented on a diff in the pull request: https://github.com/apache/spark/pull/20828#discussion_r181163750 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousMemoryStream.scala --- @@ -0,0 +1,220 @@ +/* +

<    1   2   3   4   5   6   7   >