[GitHub] spark pull request #21622: [SPARK-24637][SS] Add metrics regarding state and...

2018-07-05 Thread HeartSaVioR
Github user HeartSaVioR commented on a diff in the pull request: https://github.com/apache/spark/pull/21622#discussion_r200554917 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MetricsReporter.scala --- @@ -39,6 +42,23 @@ class MetricsReporter(

[GitHub] spark issue #21693: [SPARK-24673][SQL] scala sql function from_utc_timestamp...

2018-07-05 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/21693 @gatorsmile btw, on a personal note, I got a new (& first) baby in my family hours ago ;)) --- - To unsubscribe, e-mail:

[GitHub] spark issue #21260: [SPARK-23529][K8s] Support mounting volumes

2018-07-05 Thread liyinan926
Github user liyinan926 commented on the issue: https://github.com/apache/spark/pull/21260 @felixcheung @mccheah Can you take a look and merge this? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #21684: [SPARK-24470][Core] RestSubmissionClient to be robust ag...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21684 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21684: [SPARK-24470][Core] RestSubmissionClient to be robust ag...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21684 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/716/

[GitHub] spark issue #21684: [SPARK-24470][Core] RestSubmissionClient to be robust ag...

2018-07-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21684 **[Test build #92665 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92665/testReport)** for PR 21684 at commit

[GitHub] spark issue #21693: [SPARK-24673][SQL] scala sql function from_utc_timestamp...

2018-07-05 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21693 Thanks, @maropu. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21700: [SPARK-24717][SS] Split out max retain version of state ...

2018-07-05 Thread HeartSaVioR
Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/21700 @tedyu Thanks for the suggestion. Published the result to the mail thread.

[GitHub] spark issue #21684: [SPARK-24470][Core] RestSubmissionClient to be robust ag...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21684 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21684: [SPARK-24470][Core] RestSubmissionClient to be robust ag...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21684 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/715/

[GitHub] spark issue #21693: [SPARK-24673][SQL] scala sql function from_utc_timestamp...

2018-07-05 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/21693 ok, I get time now, so I'll take it ;) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21684: [SPARK-24470][Core] RestSubmissionClient to be robust ag...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21684 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21684: [SPARK-24470][Core] RestSubmissionClient to be robust ag...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21684 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92664/ Test FAILed. ---

[GitHub] spark issue #21684: [SPARK-24470][Core] RestSubmissionClient to be robust ag...

2018-07-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21684 **[Test build #92664 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92664/testReport)** for PR 21684 at commit

[GitHub] spark issue #21684: [SPARK-24470][Core] RestSubmissionClient to be robust ag...

2018-07-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21684 **[Test build #92664 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92664/testReport)** for PR 21684 at commit

[GitHub] spark issue #21684: [SPARK-24470][Core] RestSubmissionClient to be robust ag...

2018-07-05 Thread rekhajoshm
Github user rekhajoshm commented on the issue: https://github.com/apache/spark/pull/21684 ``` Discovery starting. Discovery completed in 5 seconds, 946 milliseconds. Run starting. Expected test count is: 16 StandaloneRestSubmitSuite: - construct submit request -

[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21073 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92662/ Test FAILed. ---

[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21073 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat

2018-07-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21073 **[Test build #92662 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92662/testReport)** for PR 21073 at commit

[GitHub] spark issue #21709: [SPARK-5152][CORE] Read metrics config file from Hadoop ...

2018-07-05 Thread jzhuge
Github user jzhuge commented on the issue: https://github.com/apache/spark/pull/21709 Not quite there with SparkConf. We already uses option `--properties-file` for a different purpose and spark-submit does not allow multiple occurrences of `--properties-file`. How about the

[GitHub] spark issue #21722: Spark-24742: Fix NullPointerexception in Field Metadata

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21722 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21722: Spark-24742: Fix NullPointerexception in Field Metadata

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21722 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21722: Spark-24742: Fix NullPointerexception in Field Metadata

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21722 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21722: Spark-24742: Fix NullPointerexception in Field Me...

2018-07-05 Thread kupferk
GitHub user kupferk opened a pull request: https://github.com/apache/spark/pull/21722 Spark-24742: Fix NullPointerexception in Field Metadata ## What changes were proposed in this pull request? This pull request provides a fix for SPARK-24742: SQL Field MetaData was

[GitHub] spark issue #21684: [SPARK-24470][Core] RestSubmissionClient to be robust ag...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21684 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21684: [SPARK-24470][Core] RestSubmissionClient to be robust ag...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21684 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/714/

[GitHub] spark issue #21684: [SPARK-24470][Core] RestSubmissionClient to be robust ag...

2018-07-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21684 **[Test build #92663 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92663/testReport)** for PR 21684 at commit

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

2018-07-05 Thread HeartSaVioR
Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/21721 Though I haven't take a look yet, I would like to see this feature (mentioned from https://github.com/apache/spark/pull/21622#issuecomment-399677099) and happy to see this being implemented!

[GitHub] spark issue #21684: [SPARK-24470][Core] RestSubmissionClient to be robust ag...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21684 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21684: [SPARK-24470][Core] RestSubmissionClient to be robust ag...

2018-07-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21684 **[Test build #92657 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92657/testReport)** for PR 21684 at commit

[GitHub] spark pull request #21608: [SPARK-24626] [SQL] Improve location size calcula...

2018-07-05 Thread Achuth17
Github user Achuth17 commented on a diff in the pull request: https://github.com/apache/spark/pull/21608#discussion_r200544127 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala --- @@ -47,15 +48,26 @@ object CommandUtils extends Logging {

[GitHub] spark pull request #21061: [SPARK-23914][SQL] Add array_union function

2018-07-05 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21061#discussion_r200541043 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -3261,3 +3261,339 @@ case class

[GitHub] spark pull request #21061: [SPARK-23914][SQL] Add array_union function

2018-07-05 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21061#discussion_r200543679 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -3261,3 +3261,339 @@ case class

[GitHub] spark pull request #21061: [SPARK-23914][SQL] Add array_union function

2018-07-05 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21061#discussion_r200541138 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -3261,3 +3261,339 @@ case class

[GitHub] spark pull request #21061: [SPARK-23914][SQL] Add array_union function

2018-07-05 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21061#discussion_r200540949 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -3261,3 +3261,339 @@ case class

[GitHub] spark pull request #21061: [SPARK-23914][SQL] Add array_union function

2018-07-05 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21061#discussion_r200540913 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -3261,3 +3261,339 @@ case class

[GitHub] spark pull request #21061: [SPARK-23914][SQL] Add array_union function

2018-07-05 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21061#discussion_r200541020 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -3261,3 +3261,339 @@ case class

[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat

2018-07-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21073 **[Test build #92662 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92662/testReport)** for PR 21073 at commit

[GitHub] spark issue #19222: [SPARK-10399][SPARK-23879][CORE][SQL] Introduce multiple...

2018-07-05 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/19222 ping @rednaxelafx --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21611: [SPARK-24569][SQL] Aggregator with output type Option sh...

2018-07-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21611 **[Test build #92661 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92661/testReport)** for PR 21611 at commit

[GitHub] spark issue #21611: [SPARK-24569][SQL] Aggregator with output type Option sh...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21611 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21611: [SPARK-24569][SQL] Aggregator with output type Option sh...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21611 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/713/

[GitHub] spark issue #21700: [SPARK-24717][SS] Split out max retain version of state ...

2018-07-05 Thread tedyu
Github user tedyu commented on the issue: https://github.com/apache/spark/pull/21700 Please publish the above results to the thread where you requested review from committers. --- - To unsubscribe, e-mail:

[GitHub] spark issue #21611: [SPARK-24569][SQL] Aggregator with output type Option sh...

2018-07-05 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21611 Based on the comment, this null check is for preventing encoding a Product type to a null top-level row. For `Option[Int]`, it is encoded to an int column in a top-level row. An example

[GitHub] spark issue #21023: [SPARK-23949] makes && supports the function of predicat...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21023 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21677: [SPARK-24692][TESTS] Improvement FilterPushdownBe...

2018-07-05 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21677 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21713: [SPARK-24737][SQL] Type coercion between StructTy...

2018-07-05 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21713 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21677: [SPARK-24692][TESTS] Improvement FilterPushdownBenchmark

2018-07-05 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21677 Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21677: [SPARK-24692][TESTS] Improvement FilterPushdownBenchmark

2018-07-05 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21677 I am merging this to show up other benchmark results in @wangyum's PRs. --- - To unsubscribe, e-mail:

[GitHub] spark issue #21693: [SPARK-24673][SQL] scala sql function from_utc_timestamp...

2018-07-05 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21693 Will go ahead on this weekends. If you guys find some times please go ahead till then. --- - To unsubscribe, e-mail:

[GitHub] spark issue #21713: [SPARK-24737][SQL] Type coercion between StructTypes.

2018-07-05 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21713 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

2018-07-05 Thread yuanboliu
Github user yuanboliu commented on the issue: https://github.com/apache/spark/pull/21690 @koeninger Thanks for your details. Sorry quite busy this week. I will delete the last pause, test the patch on my own cluster this weekend and give feedback asap. ---

[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat

2018-07-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21073 **[Test build #92660 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92660/testReport)** for PR 21073 at commit

[GitHub] spark pull request #21721: [SPARK-24748][SS] Support for reporting custom me...

2018-07-05 Thread jose-torres
Github user jose-torres commented on a diff in the pull request: https://github.com/apache/spark/pull/21721#discussion_r200538008 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/JsonUtils.scala --- @@ -95,4 +95,25 @@ private object JsonUtils {

[GitHub] spark pull request #21721: [SPARK-24748][SS] Support for reporting custom me...

2018-07-05 Thread jose-torres
Github user jose-torres commented on a diff in the pull request: https://github.com/apache/spark/pull/21721#discussion_r200537533 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchReader.scala --- @@ -379,3 +384,16 @@ private[kafka010]

[GitHub] spark pull request #21721: [SPARK-24748][SS] Support for reporting custom me...

2018-07-05 Thread jose-torres
Github user jose-torres commented on a diff in the pull request: https://github.com/apache/spark/pull/21721#discussion_r200537276 --- Diff: sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/SupportsCustomMetrics.java --- @@ -0,0 +1,30 @@ +/* --- End diff --

[GitHub] spark pull request #21721: [SPARK-24748][SS] Support for reporting custom me...

2018-07-05 Thread jose-torres
Github user jose-torres commented on a diff in the pull request: https://github.com/apache/spark/pull/21721#discussion_r200537454 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/progress.scala --- @@ -178,12 +180,18 @@ class SourceProgress protected[sql](

[GitHub] spark pull request #21620: [SPARK-24636][SQL] Type coercion of arrays for ar...

2018-07-05 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21620#discussion_r200537707 --- Diff: sql/core/src/test/resources/sql-tests/results/typeCoercion/native/arrayJoin.sql.out --- @@ -0,0 +1,90 @@ +-- Automatically generated by

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

2018-07-05 Thread bjkonglu
Github user bjkonglu commented on the issue: https://github.com/apache/spark/pull/21721 I think that SS'metrics should be report to Spark UI. Then user can lean application operations! --- - To unsubscribe, e-mail:

[GitHub] spark pull request #21658: [SPARK-24678][Spark-Streaming] Give priority in u...

2018-07-05 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/21658#discussion_r200536325 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -1569,7 +1569,7 @@ private[spark] object BlockManager {

[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21710 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21710 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92658/ Test PASSed. ---

[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan

2018-07-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21710 **[Test build #92658 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92658/testReport)** for PR 21710 at commit

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

2018-07-05 Thread bjkonglu
Github user bjkonglu commented on the issue: https://github.com/apache/spark/pull/21721 I think this proposal is necessary. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21666: [SPARK-24535][SPARKR] fix tests on java check error

2018-07-05 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/21666 Any update on this issue @felixcheung , this blocks 2.3.2 release, just want to make sure if it is still in-progress. --- -

[GitHub] spark pull request #21658: [SPARK-24678][Spark-Streaming] Give priority in u...

2018-07-05 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/21658#discussion_r200535022 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -1569,7 +1569,7 @@ private[spark] object BlockManager {

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21721 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92659/ Test FAILed. ---

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21721 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

2018-07-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21721 **[Test build #92659 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92659/testReport)** for PR 21721 at commit

[GitHub] spark issue #21701: [SPARK-24730][SS] Add policy to choose max as global wat...

2018-07-05 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/21701 @zsxwing @brkyvz --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21718: [SPARK-24744][STRUCTRURED STREAMING] Set the SparkSessio...

2018-07-05 Thread bjkonglu
Github user bjkonglu commented on the issue: https://github.com/apache/spark/pull/21718 Thx @HeartSaVioR .I am looking forward to your solution. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

2018-07-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21721 **[Test build #92659 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92659/testReport)** for PR 21721 at commit

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21721 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21721 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...

2018-07-05 Thread arunmahadevan
Github user arunmahadevan commented on the issue: https://github.com/apache/spark/pull/21721 @tdas @jose-torres @HeartSaVioR --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21721: [SPARK-24748][SS] Support for reporting custom me...

2018-07-05 Thread arunmahadevan
GitHub user arunmahadevan opened a pull request: https://github.com/apache/spark/pull/21721 [SPARK-24748][SS] Support for reporting custom metrics via StreamingQuery Progress ## What changes were proposed in this pull request? Currently the Structured Streaming sources and

[GitHub] spark issue #21700: [SPARK-24717][SS] Split out max retain version of state ...

2018-07-05 Thread HeartSaVioR
Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/21700 I would like to add numbers to pursuade how much this patch is helpful for end users of Apache Spark. I crafted and published a project which implements some stateful use cases with

[GitHub] spark issue #21608: [SPARK-24626] [SQL] Improve location size calculation in...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21608 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92655/ Test PASSed. ---

[GitHub] spark issue #21608: [SPARK-24626] [SQL] Improve location size calculation in...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21608 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21608: [SPARK-24626] [SQL] Improve location size calculation in...

2018-07-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21608 **[Test build #92655 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92655/testReport)** for PR 21608 at commit

[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan

2018-07-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21710 **[Test build #92658 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92658/testReport)** for PR 21710 at commit

[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21710 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/712/

[GitHub] spark issue #21710: [SPARK-24207][R]add R API for PrefixSpan

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21710 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21720: [SPARK-24163][SPARK-24164][SQL] Support column list as t...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21720 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21720: [SPARK-24163][SPARK-24164][SQL] Support column list as t...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21720 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92656/ Test FAILed. ---

[GitHub] spark issue #21720: [SPARK-24163][SPARK-24164][SQL] Support column list as t...

2018-07-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21720 **[Test build #92656 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92656/testReport)** for PR 21720 at commit

[GitHub] spark issue #21673: [SPARK-24697][SS] Fix the reported start offsets in stre...

2018-07-05 Thread arunmahadevan
Github user arunmahadevan commented on the issue: https://github.com/apache/spark/pull/21673 @tdas , thanks for your comments. Yes theres problem with the current abstraction, and I didn't consider refactoring it since there have been multiple changes to this class without changing

[GitHub] spark pull request #21556: [SPARK-24549][SQL] Support Decimal type push down...

2018-07-05 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/21556#discussion_r200526464 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala --- @@ -82,6 +120,30 @@ private[parquet] class

[GitHub] spark issue #21684: [SPARK-24470][Core] RestSubmissionClient to be robust ag...

2018-07-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21684 **[Test build #92657 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92657/testReport)** for PR 21684 at commit

[GitHub] spark issue #21684: [SPARK-24470][Core] RestSubmissionClient to be robust ag...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21684 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/711/

[GitHub] spark issue #21684: [SPARK-24470][Core] RestSubmissionClient to be robust ag...

2018-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21684 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21720: [SPARK-24163][SPARK-24164][SQL] Support column li...

2018-07-05 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21720#discussion_r200518399 --- Diff: sql/core/src/test/resources/sql-tests/inputs/pivot.sql --- @@ -88,12 +93,12 @@ PIVOT ( ); -- pivot with aliases and projection

[GitHub] spark issue #21541: [SPARK-20168][Streaming Kinesis] Setting the timestamp d...

2018-07-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21541 **[Test build #4209 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4209/testReport)** for PR 21541 at commit

[GitHub] spark issue #21541: [SPARK-20168][Streaming Kinesis] Setting the timestamp d...

2018-07-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21541 **[Test build #4209 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4209/testReport)** for PR 21541 at commit

[GitHub] spark pull request #21652: [SPARK-24551][K8S] Add integration tests for secr...

2018-07-05 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/21652#discussion_r200515051 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/SecretsTestsSuite.scala --- @@ -0,0

[GitHub] spark pull request #21652: [SPARK-24551][K8S] Add integration tests for secr...

2018-07-05 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/21652#discussion_r200514990 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala --- @@ -31,19

[GitHub] spark pull request #21652: [SPARK-24551][K8S] Add integration tests for secr...

2018-07-05 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/21652#discussion_r200514722 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/BasicTestsSuite.scala --- @@ -0,0

[GitHub] spark pull request #21652: [SPARK-24551][K8S] Add integration tests for secr...

2018-07-05 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/21652#discussion_r200514767 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/BasicTestsSuite.scala --- @@ -0,0

[GitHub] spark issue #21673: [SPARK-24697][SS] Fix the reported start offsets in stre...

2018-07-05 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/21673 Thanks @arunmahadevan for making this PR. However, I dont like the solution of adding another field as a workaround thus making the control flow harder to reason about. I think the fundamental problem

[GitHub] spark pull request #21620: [SPARK-24636][SQL] Type coercion of arrays for ar...

2018-07-05 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21620#discussion_r200514583 --- Diff: sql/core/src/test/resources/sql-tests/results/typeCoercion/native/arrayJoin.sql.out --- @@ -0,0 +1,90 @@ +-- Automatically generated by

  1   2   3   4   >