[GitHub] spark pull request #21516: [SPARK-24501][MESOS] Add Dispatcher and Driver me...

2018-07-05 Thread nickbp
Github user nickbp commented on a diff in the pull request: https://github.com/apache/spark/pull/21516#discussion_r200470719 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerSource.scala --- @@ -0,0 +1,250 @@

[GitHub] spark pull request #21516: [SPARK-24501][MESOS] Add Dispatcher and Driver me...

2018-07-05 Thread nickbp
Github user nickbp commented on a diff in the pull request: https://github.com/apache/spark/pull/21516#discussion_r200472180 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala --- @@ -796,6 +811,38 @@

[GitHub] spark pull request #21516: [SPARK-24501][MESOS] Add Dispatcher and Driver me...

2018-07-05 Thread nickbp
Github user nickbp commented on a diff in the pull request: https://github.com/apache/spark/pull/21516#discussion_r200469780 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterSchedulerSource.scala --- @@ -17,25 +17,170 @@

[GitHub] spark pull request #21516: [SPARK-24501][MESOS] Add Dispatcher and Driver me...

2018-07-05 Thread nickbp
Github user nickbp commented on a diff in the pull request: https://github.com/apache/spark/pull/21516#discussion_r200470681 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerSource.scala --- @@ -0,0 +1,250 @@

[GitHub] spark pull request #21516: [SPARK-24501][MESOS] Add Dispatcher and Driver me...

2018-07-05 Thread nickbp
Github user nickbp commented on a diff in the pull request: https://github.com/apache/spark/pull/21516#discussion_r200470645 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerSource.scala --- @@ -0,0 +1,250 @@

[GitHub] spark pull request #21516: [SPARK-24501][MESOS] Add Dispatcher and Driver me...

2018-07-05 Thread nickbp
Github user nickbp commented on a diff in the pull request: https://github.com/apache/spark/pull/21516#discussion_r200469746 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterSchedulerSource.scala --- @@ -17,25 +17,170 @@

[GitHub] spark pull request #21516: [SPARK-24501][MESOS] Add Dispatcher and Driver me...

2018-07-05 Thread nickbp
Github user nickbp commented on a diff in the pull request: https://github.com/apache/spark/pull/21516#discussion_r200470228 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterSchedulerSource.scala --- @@ -17,25 +17,170 @@

[GitHub] spark issue #21673: [SPARK-24697][SS] Fix the reported start offsets in stre...

2018-07-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21673 **[Test build #92652 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92652/testReport)** for PR 21673 at commit [`aeadd5a`](https://github.com/apache/spark/commit/a

[GitHub] spark issue #21673: [SPARK-24697][SS] Fix the reported start offsets in stre...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21673 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92652/ Test PASSed. ---

[GitHub] spark issue #21673: [SPARK-24697][SS] Fix the reported start offsets in stre...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21673 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21672: [SPARK-24694][K8S] Pass all app args to integration test...

2018-07-05 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/21672 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@

[GitHub] spark pull request #21672: [SPARK-24694][K8S] Pass all app args to integrati...

2018-07-05 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21672 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21719: [SPARK-24747] Make Instrumentation class more fle...

2018-07-05 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/21719#discussion_r200498732 --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/Instrumentation.scala --- @@ -148,12 +170,25 @@ private[spark] class Instrumentation[E <: Estimator[

[GitHub] spark pull request #21719: [SPARK-24747] Make Instrumentation class more fle...

2018-07-05 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/21719#discussion_r200499315 --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/Instrumentation.scala --- @@ -169,22 +204,33 @@ private[spark] object Instrumentation { val

[GitHub] spark pull request #21719: [SPARK-24747] Make Instrumentation class more fle...

2018-07-05 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/21719#discussion_r200497861 --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/Instrumentation.scala --- @@ -35,32 +36,47 @@ import org.apache.spark.util.Utils /** * A s

[GitHub] spark pull request #21719: [SPARK-24747] Make Instrumentation class more fle...

2018-07-05 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/21719#discussion_r200495412 --- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala --- @@ -490,7 +490,7 @@ class LogisticRegression @Since("1.2.0")

[GitHub] spark pull request #21719: [SPARK-24747] Make Instrumentation class more fle...

2018-07-05 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/21719#discussion_r200499164 --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/Instrumentation.scala --- @@ -169,22 +204,33 @@ private[spark] object Instrumentation { val

[GitHub] spark pull request #21719: [SPARK-24747] Make Instrumentation class more fle...

2018-07-05 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/21719#discussion_r200497209 --- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala --- @@ -500,7 +500,7 @@ class LogisticRegression @Since("1.2.0")

[GitHub] spark pull request #21719: [SPARK-24747] Make Instrumentation class more fle...

2018-07-05 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/21719#discussion_r200498918 --- Diff: mllib/src/main/scala/org/apache/spark/ml/util/Instrumentation.scala --- @@ -148,12 +170,25 @@ private[spark] class Instrumentation[E <: Estimator[

[GitHub] spark issue #21608: [SPARK-24626] [SQL] Improve location size calculation in...

2018-07-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21608 **[Test build #92655 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92655/testReport)** for PR 21608 at commit [`1e1db66`](https://github.com/apache/spark/commit/1e

[GitHub] spark pull request #21720: [SPARK-24163][SPARK-24164][SQL] Support column li...

2018-07-05 Thread maryannxue
GitHub user maryannxue opened a pull request: https://github.com/apache/spark/pull/21720 [SPARK-24163][SPARK-24164][SQL] Support column list as the pivot column in Pivot ## What changes were proposed in this pull request? 1. Bug fixing in Analyzer for Pivot node. 2. Ext

[GitHub] spark issue #21720: [SPARK-24163][SPARK-24164][SQL] Support column list as t...

2018-07-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21720 **[Test build #92656 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92656/testReport)** for PR 21720 at commit [`942a30d`](https://github.com/apache/spark/commit/94

[GitHub] spark issue #21720: [SPARK-24163][SPARK-24164][SQL] Support column list as t...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21720 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/710/

[GitHub] spark issue #21720: [SPARK-24163][SPARK-24164][SQL] Support column list as t...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21720 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21662: [SPARK-24662][SQL][SS] Support limit in structured strea...

2018-07-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21662 **[Test build #92654 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92654/testReport)** for PR 21662 at commit [`d12fb1f`](https://github.com/apache/spark/commit/d

[GitHub] spark issue #21662: [SPARK-24662][SQL][SS] Support limit in structured strea...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21662 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21662: [SPARK-24662][SQL][SS] Support limit in structured strea...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21662 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92654/ Test PASSed. ---

[GitHub] spark issue #21676: [SPARK-24699][SS][WIP] Watermark / Append mode should wo...

2018-07-05 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/21676 I think the right solution is to record the updateat watermark in the commit log, so that the updated watermark can be read back from the commit log next time the stream is started. ---

[GitHub] spark pull request #21620: [SPARK-24636][SQL] Type coercion of arrays for ar...

2018-07-05 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21620#discussion_r200514583 --- Diff: sql/core/src/test/resources/sql-tests/results/typeCoercion/native/arrayJoin.sql.out --- @@ -0,0 +1,90 @@ +-- Automatically generated by SQ

[GitHub] spark issue #21673: [SPARK-24697][SS] Fix the reported start offsets in stre...

2018-07-05 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/21673 Thanks @arunmahadevan for making this PR. However, I dont like the solution of adding another field as a workaround thus making the control flow harder to reason about. I think the fundamental problem

[GitHub] spark pull request #21652: [SPARK-24551][K8S] Add integration tests for secr...

2018-07-05 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/21652#discussion_r200514722 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/BasicTestsSuite.scala --- @@ -0,0 +1,110

[GitHub] spark pull request #21652: [SPARK-24551][K8S] Add integration tests for secr...

2018-07-05 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/21652#discussion_r200514767 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/BasicTestsSuite.scala --- @@ -0,0 +1,110

[GitHub] spark pull request #21652: [SPARK-24551][K8S] Add integration tests for secr...

2018-07-05 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/21652#discussion_r200514990 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala --- @@ -31,19 +31,

[GitHub] spark pull request #21652: [SPARK-24551][K8S] Add integration tests for secr...

2018-07-05 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/21652#discussion_r200515051 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/SecretsTestsSuite.scala --- @@ -0,0 +1,1

[GitHub] spark issue #21541: [SPARK-20168][Streaming Kinesis] Setting the timestamp d...

2018-07-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21541 **[Test build #4209 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4209/testReport)** for PR 21541 at commit [`133ba87`](https://github.com/apache/spark/commit/1

[GitHub] spark issue #21541: [SPARK-20168][Streaming Kinesis] Setting the timestamp d...

2018-07-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21541 **[Test build #4209 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4209/testReport)** for PR 21541 at commit [`133ba87`](https://github.com/apache/spark/commit/

[GitHub] spark pull request #21720: [SPARK-24163][SPARK-24164][SQL] Support column li...

2018-07-05 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21720#discussion_r200518399 --- Diff: sql/core/src/test/resources/sql-tests/inputs/pivot.sql --- @@ -88,12 +93,12 @@ PIVOT ( ); -- pivot with aliases and projection

[GitHub] spark issue #21684: [SPARK-24470][Core] RestSubmissionClient to be robust ag...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21684 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21684: [SPARK-24470][Core] RestSubmissionClient to be robust ag...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21684 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/711/

[GitHub] spark issue #21684: [SPARK-24470][Core] RestSubmissionClient to be robust ag...

2018-07-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21684 **[Test build #92657 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92657/testReport)** for PR 21684 at commit [`e043901`](https://github.com/apache/spark/commit/e0

[GitHub] spark pull request #21556: [SPARK-24549][SQL] Support Decimal type push down...

2018-07-05 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/21556#discussion_r200526464 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala --- @@ -82,6 +120,30 @@ private[parquet] class Pa

[GitHub] spark issue #21673: [SPARK-24697][SS] Fix the reported start offsets in stre...

2018-07-05 Thread arunmahadevan
Github user arunmahadevan commented on the issue: https://github.com/apache/spark/pull/21673 @tdas , thanks for your comments. Yes theres problem with the current abstraction, and I didn't consider refactoring it since there have been multiple changes to this class without changing th

[GitHub] spark issue #21720: [SPARK-24163][SPARK-24164][SQL] Support column list as t...

2018-07-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21720 **[Test build #92656 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92656/testReport)** for PR 21720 at commit [`942a30d`](https://github.com/apache/spark/commit/9

[GitHub] spark issue #21720: [SPARK-24163][SPARK-24164][SQL] Support column list as t...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21720 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21720: [SPARK-24163][SPARK-24164][SQL] Support column list as t...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21720 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92656/ Test FAILed. ---

[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21710 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21710 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/712/

[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan

2018-07-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21710 **[Test build #92658 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92658/testReport)** for PR 21710 at commit [`d2e85f5`](https://github.com/apache/spark/commit/d2

[GitHub] spark issue #21608: [SPARK-24626] [SQL] Improve location size calculation in...

2018-07-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21608 **[Test build #92655 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92655/testReport)** for PR 21608 at commit [`1e1db66`](https://github.com/apache/spark/commit/1

[GitHub] spark issue #21608: [SPARK-24626] [SQL] Improve location size calculation in...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21608 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21608: [SPARK-24626] [SQL] Improve location size calculation in...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21608 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92655/ Test PASSed. ---

[GitHub] spark issue #21700: [SPARK-24717][SS] Split out max retain version of state ...

2018-07-05 Thread HeartSaVioR
Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/21700 I would like to add numbers to pursuade how much this patch is helpful for end users of Apache Spark. I crafted and published a project which implements some stateful use cases with IoT

[GitHub] spark pull request #21721: [SPARK-24748][SS] Support for reporting custom me...

2018-07-05 Thread arunmahadevan
GitHub user arunmahadevan opened a pull request: https://github.com/apache/spark/pull/21721 [SPARK-24748][SS] Support for reporting custom metrics via StreamingQuery Progress ## What changes were proposed in this pull request? Currently the Structured Streaming sources and

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

2018-07-05 Thread arunmahadevan
Github user arunmahadevan commented on the issue: https://github.com/apache/spark/pull/21721 @tdas @jose-torres @HeartSaVioR --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional command

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21721 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21721 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

2018-07-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21721 **[Test build #92659 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92659/testReport)** for PR 21721 at commit [`b7b2c3b`](https://github.com/apache/spark/commit/b7

[GitHub] spark issue #21718: [SPARK-24744][STRUCTRURED STREAMING] Set the SparkSessio...

2018-07-05 Thread bjkonglu
Github user bjkonglu commented on the issue: https://github.com/apache/spark/pull/21718 Thx @HeartSaVioR .I am looking forward to your solution. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #21701: [SPARK-24730][SS] Add policy to choose max as global wat...

2018-07-05 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/21701 @zsxwing @brkyvz --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@s

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

2018-07-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21721 **[Test build #92659 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92659/testReport)** for PR 21721 at commit [`b7b2c3b`](https://github.com/apache/spark/commit/b

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21721 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21721 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92659/ Test FAILed. ---

[GitHub] spark pull request #21658: [SPARK-24678][Spark-Streaming] Give priority in u...

2018-07-05 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/21658#discussion_r200535022 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -1569,7 +1569,7 @@ private[spark] object BlockManager { va

[GitHub] spark issue #21666: [SPARK-24535][SPARKR] fix tests on java check error

2018-07-05 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/21666 Any update on this issue @felixcheung , this blocks 2.3.2 release, just want to make sure if it is still in-progress. --- - To

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

2018-07-05 Thread bjkonglu
Github user bjkonglu commented on the issue: https://github.com/apache/spark/pull/21721 I think this proposal is necessary. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan

2018-07-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21710 **[Test build #92658 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92658/testReport)** for PR 21710 at commit [`d2e85f5`](https://github.com/apache/spark/commit/d

[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21710 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92658/ Test PASSed. ---

[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21710 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #21658: [SPARK-24678][Spark-Streaming] Give priority in u...

2018-07-05 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/21658#discussion_r200536325 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -1569,7 +1569,7 @@ private[spark] object BlockManager { va

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

2018-07-05 Thread bjkonglu
Github user bjkonglu commented on the issue: https://github.com/apache/spark/pull/21721 I think that SS'metrics should be report to Spark UI. Then user can lean application operations! --- - To unsubscribe, e-mail:

[GitHub] spark pull request #21620: [SPARK-24636][SQL] Type coercion of arrays for ar...

2018-07-05 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21620#discussion_r200537707 --- Diff: sql/core/src/test/resources/sql-tests/results/typeCoercion/native/arrayJoin.sql.out --- @@ -0,0 +1,90 @@ +-- Automatically generated by SQLQue

[GitHub] spark pull request #21721: [SPARK-24748][SS] Support for reporting custom me...

2018-07-05 Thread jose-torres
Github user jose-torres commented on a diff in the pull request: https://github.com/apache/spark/pull/21721#discussion_r200537276 --- Diff: sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/SupportsCustomMetrics.java --- @@ -0,0 +1,30 @@ +/* --- End diff --

[GitHub] spark pull request #21721: [SPARK-24748][SS] Support for reporting custom me...

2018-07-05 Thread jose-torres
Github user jose-torres commented on a diff in the pull request: https://github.com/apache/spark/pull/21721#discussion_r200537454 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/progress.scala --- @@ -178,12 +180,18 @@ class SourceProgress protected[sql](

[GitHub] spark pull request #21721: [SPARK-24748][SS] Support for reporting custom me...

2018-07-05 Thread jose-torres
Github user jose-torres commented on a diff in the pull request: https://github.com/apache/spark/pull/21721#discussion_r200538008 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/JsonUtils.scala --- @@ -95,4 +95,25 @@ private object JsonUtils {

[GitHub] spark pull request #21721: [SPARK-24748][SS] Support for reporting custom me...

2018-07-05 Thread jose-torres
Github user jose-torres commented on a diff in the pull request: https://github.com/apache/spark/pull/21721#discussion_r200537533 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchReader.scala --- @@ -379,3 +384,16 @@ private[kafka010] c

[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat

2018-07-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21073 **[Test build #92660 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92660/testReport)** for PR 21073 at commit [`3c0da03`](https://github.com/apache/spark/commit/3c

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

2018-07-05 Thread yuanboliu
Github user yuanboliu commented on the issue: https://github.com/apache/spark/pull/21690 @koeninger Thanks for your details. Sorry quite busy this week. I will delete the last pause, test the patch on my own cluster this weekend and give feedback asap. --- ---

[GitHub] spark issue #21713: [SPARK-24737][SQL] Type coercion between StructTypes.

2018-07-05 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21713 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-

[GitHub] spark issue #21693: [SPARK-24673][SQL] scala sql function from_utc_timestamp...

2018-07-05 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21693 Will go ahead on this weekends. If you guys find some times please go ahead till then. --- - To unsubscribe, e-mail: reviews

[GitHub] spark issue #21677: [SPARK-24692][TESTS] Improvement FilterPushdownBenchmark

2018-07-05 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21677 I am merging this to show up other benchmark results in @wangyum's PRs. --- - To unsubscribe, e-mail: reviews-unsubscr...@spa

[GitHub] spark issue #21677: [SPARK-24692][TESTS] Improvement FilterPushdownBenchmark

2018-07-05 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21677 Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark pull request #21713: [SPARK-24737][SQL] Type coercion between StructTy...

2018-07-05 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21713 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21677: [SPARK-24692][TESTS] Improvement FilterPushdownBe...

2018-07-05 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21677 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21023: [SPARK-23949] makes && supports the function of predicat...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21023 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21611: [SPARK-24569][SQL] Aggregator with output type Option sh...

2018-07-05 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21611 Based on the comment, this null check is for preventing encoding a Product type to a null top-level row. For `Option[Int]`, it is encoded to an int column in a top-level row. An example look

[GitHub] spark issue #21700: [SPARK-24717][SS] Split out max retain version of state ...

2018-07-05 Thread tedyu
Github user tedyu commented on the issue: https://github.com/apache/spark/pull/21700 Please publish the above results to the thread where you requested review from committers. --- - To unsubscribe, e-mail: reviews-u

[GitHub] spark issue #21611: [SPARK-24569][SQL] Aggregator with output type Option sh...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21611 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/713/

[GitHub] spark issue #21611: [SPARK-24569][SQL] Aggregator with output type Option sh...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21611 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21611: [SPARK-24569][SQL] Aggregator with output type Option sh...

2018-07-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21611 **[Test build #92661 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92661/testReport)** for PR 21611 at commit [`e67594a`](https://github.com/apache/spark/commit/e6

[GitHub] spark issue #19222: [SPARK-10399][SPARK-23879][CORE][SQL] Introduce multiple...

2018-07-05 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/19222 ping @rednaxelafx --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@

[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat

2018-07-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21073 **[Test build #92662 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92662/testReport)** for PR 21073 at commit [`03328a4`](https://github.com/apache/spark/commit/03

[GitHub] spark pull request #21061: [SPARK-23914][SQL] Add array_union function

2018-07-05 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21061#discussion_r200541020 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -3261,3 +3261,339 @@ case class ArrayDisti

[GitHub] spark pull request #21061: [SPARK-23914][SQL] Add array_union function

2018-07-05 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21061#discussion_r200540913 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -3261,3 +3261,339 @@ case class ArrayDisti

[GitHub] spark pull request #21061: [SPARK-23914][SQL] Add array_union function

2018-07-05 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21061#discussion_r200540949 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -3261,3 +3261,339 @@ case class ArrayDisti

[GitHub] spark pull request #21061: [SPARK-23914][SQL] Add array_union function

2018-07-05 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21061#discussion_r200543679 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -3261,3 +3261,339 @@ case class ArrayDisti

[GitHub] spark pull request #21061: [SPARK-23914][SQL] Add array_union function

2018-07-05 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21061#discussion_r200541138 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -3261,3 +3261,339 @@ case class ArrayDisti

[GitHub] spark pull request #21061: [SPARK-23914][SQL] Add array_union function

2018-07-05 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21061#discussion_r200541043 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -3261,3 +3261,339 @@ case class ArrayDisti

[GitHub] spark pull request #21608: [SPARK-24626] [SQL] Improve location size calcula...

2018-07-05 Thread Achuth17
Github user Achuth17 commented on a diff in the pull request: https://github.com/apache/spark/pull/21608#discussion_r200544127 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala --- @@ -47,15 +48,26 @@ object CommandUtils extends Logging {

[GitHub] spark issue #21684: [SPARK-24470][Core] RestSubmissionClient to be robust ag...

2018-07-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21684 **[Test build #92657 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92657/testReport)** for PR 21684 at commit [`e043901`](https://github.com/apache/spark/commit/e

[GitHub] spark issue #21684: [SPARK-24470][Core] RestSubmissionClient to be robust ag...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21684 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

<    1   2   3   4   >