[GitHub] [spark] SparkQA commented on issue #26668: [SPARK-30031][BUILD][SQL] Remove `hive-2.3` profile from `sql/hive/pom.xml`
SparkQA commented on issue #26668: [SPARK-30031][BUILD][SQL] Remove `hive-2.3` profile from `sql/hive/pom.xml` URL: https://github.com/apache/spark/pull/26668#issuecomment-558322672 **[Test build #114428 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114428/testReport)** for PR 26668 at commit [`d807933`](https://github.com/apache/spark/commit/d807933f9a110dba0785a99926847507989e0b55). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26658: [SPARK-30015][BUILD] Move hive-storage-api dependency from `hive-2.3` to `sql/core`
AmplabJenkins removed a comment on issue #26658: [SPARK-30015][BUILD] Move hive-storage-api dependency from `hive-2.3` to `sql/core` URL: https://github.com/apache/spark/pull/26658#issuecomment-558320539 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26658: [SPARK-30015][BUILD] Move hive-storage-api dependency from `hive-2.3` to `sql/core`
AmplabJenkins removed a comment on issue #26658: [SPARK-30015][BUILD] Move hive-storage-api dependency from `hive-2.3` to `sql/core` URL: https://github.com/apache/spark/pull/26658#issuecomment-558320551 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114416/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26658: [SPARK-30015][BUILD] Move hive-storage-api dependency from `hive-2.3` to `sql/core`
AmplabJenkins commented on issue #26658: [SPARK-30015][BUILD] Move hive-storage-api dependency from `hive-2.3` to `sql/core` URL: https://github.com/apache/spark/pull/26658#issuecomment-558320551 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114416/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26658: [SPARK-30015][BUILD] Move hive-storage-api dependency from `hive-2.3` to `sql/core`
AmplabJenkins commented on issue #26658: [SPARK-30015][BUILD] Move hive-storage-api dependency from `hive-2.3` to `sql/core` URL: https://github.com/apache/spark/pull/26658#issuecomment-558320539 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun opened a new pull request #26668: [SPARK-30031][BUILD][SQL] Remove `hive-2.3` profile from `sql/hive/pom.xml`
dongjoon-hyun opened a new pull request #26668: [SPARK-30031][BUILD][SQL] Remove `hive-2.3` profile from `sql/hive/pom.xml` URL: https://github.com/apache/spark/pull/26668 … ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce any user-facing change? ### How was this patch tested? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26658: [SPARK-30015][BUILD] Move hive-storage-api dependency from `hive-2.3` to `sql/core`
SparkQA removed a comment on issue #26658: [SPARK-30015][BUILD] Move hive-storage-api dependency from `hive-2.3` to `sql/core` URL: https://github.com/apache/spark/pull/26658#issuecomment-558260901 **[Test build #114416 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114416/testReport)** for PR 26658 at commit [`afb1af1`](https://github.com/apache/spark/commit/afb1af157a0c2b811270ed9e8ceb1f06bb952170). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26658: [SPARK-30015][BUILD] Move hive-storage-api dependency from `hive-2.3` to `sql/core`
SparkQA commented on issue #26658: [SPARK-30015][BUILD] Move hive-storage-api dependency from `hive-2.3` to `sql/core` URL: https://github.com/apache/spark/pull/26658#issuecomment-558319876 **[Test build #114416 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114416/testReport)** for PR 26658 at commit [`afb1af1`](https://github.com/apache/spark/commit/afb1af157a0c2b811270ed9e8ceb1f06bb952170). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LucaCanali commented on issue #24132: [SPARK-27189][CORE] Add Executor metrics and memory usage instrumentation to the metrics system
LucaCanali commented on issue #24132: [SPARK-27189][CORE] Add Executor metrics and memory usage instrumentation to the metrics system URL: https://github.com/apache/spark/pull/24132#issuecomment-558318439 I think it should be OK to have ExecutorMetrics on by default. It is the case for most metrics at present. BTW, the JVM-related memory metrics in ExecutorMetrics can also be used as a partial (but in most cases good enough) replacement for the optional `org.apache.spark.metrics.source.JvmSource`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #26666: [SPARK-30030][INFRA] Use RegexChecker instead of TokenChecker to check `org.apache.commons.lang.`
dongjoon-hyun closed pull request #2: [SPARK-30030][INFRA] Use RegexChecker instead of TokenChecker to check `org.apache.commons.lang.` URL: https://github.com/apache/spark/pull/2 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26649: [SPARK-29862][SQL] CREATE (OR REPLACE) ... VIEW should look up catalog/table like v2 commands
AmplabJenkins removed a comment on issue #26649: [SPARK-29862][SQL] CREATE (OR REPLACE) ... VIEW should look up catalog/table like v2 commands URL: https://github.com/apache/spark/pull/26649#issuecomment-558317751 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19263/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26649: [SPARK-29862][SQL] CREATE (OR REPLACE) ... VIEW should look up catalog/table like v2 commands
AmplabJenkins removed a comment on issue #26649: [SPARK-29862][SQL] CREATE (OR REPLACE) ... VIEW should look up catalog/table like v2 commands URL: https://github.com/apache/spark/pull/26649#issuecomment-558317742 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26666: [SPARK-30030][INFRA] Use RegexChecker instead of TokenChecker to check `regex`
dongjoon-hyun commented on issue #2: [SPARK-30030][INFRA] Use RegexChecker instead of TokenChecker to check `regex` URL: https://github.com/apache/spark/pull/2#issuecomment-558317675 Since the GitHub Action Linter passed with the new rule and the changed code, I'll merge this PR. Thank you, @srowen and @databricks-david-lewis . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26649: [SPARK-29862][SQL] CREATE (OR REPLACE) ... VIEW should look up catalog/table like v2 commands
AmplabJenkins commented on issue #26649: [SPARK-29862][SQL] CREATE (OR REPLACE) ... VIEW should look up catalog/table like v2 commands URL: https://github.com/apache/spark/pull/26649#issuecomment-558317742 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26649: [SPARK-29862][SQL] CREATE (OR REPLACE) ... VIEW should look up catalog/table like v2 commands
AmplabJenkins commented on issue #26649: [SPARK-29862][SQL] CREATE (OR REPLACE) ... VIEW should look up catalog/table like v2 commands URL: https://github.com/apache/spark/pull/26649#issuecomment-558317751 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19263/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26649: [SPARK-29862][SQL] CREATE (OR REPLACE) ... VIEW should look up catalog/table like v2 commands
SparkQA commented on issue #26649: [SPARK-29862][SQL] CREATE (OR REPLACE) ... VIEW should look up catalog/table like v2 commands URL: https://github.com/apache/spark/pull/26649#issuecomment-558317104 **[Test build #114427 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114427/testReport)** for PR 26649 at commit [`e878207`](https://github.com/apache/spark/commit/e878207877ef699ebd99aaa1cd1b490118f9ee90). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #26649: [SPARK-29862][SQL] CREATE (OR REPLACE) ... VIEW should look up catalog/table like v2 commands
huaxingao commented on a change in pull request #26649: [SPARK-29862][SQL]
CREATE (OR REPLACE) ... VIEW should look up catalog/table like v2 commands
URL: https://github.com/apache/spark/pull/26649#discussion_r350391937
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala
##
@@ -1669,6 +1669,105 @@ class DDLParserSuite extends AnalysisTest {
comparePlans(parsed, expected)
}
+ test("create view -- basic") {
+val v1 = "CREATE VIEW view1 AS SELECT * FROM tab1"
+val parsed1 = parsePlan(v1)
+
+val expected1 = CreateViewStatement(
+ Seq("view1"),
+ Seq.empty[(String, Option[String])],
+ None,
+ Map.empty[String, String],
+ Some("SELECT * FROM tab1"),
+ parsePlan("SELECT * FROM tab1"),
+ false,
+ false,
+ PersistedView)
+comparePlans(parsed1, expected1)
+
+val v2 = "CREATE VIEW a.b.c AS SELECT * FROM tab1"
+val parsed2 = parsePlan(v2)
+
+val expected2 = CreateViewStatement(
+ Seq("a", "b", "c"),
+ Seq.empty[(String, Option[String])],
+ None,
+ Map.empty[String, String],
+ Some("SELECT * FROM tab1"),
+ parsePlan("SELECT * FROM tab1"),
+ false,
+ false,
+ PersistedView)
+comparePlans(parsed2, expected2)
+ }
+
+ test("create view - full") {
+val v1 =
+ """
+|CREATE OR REPLACE VIEW view1
+|(col1, col3 COMMENT 'hello')
+|TBLPROPERTIES('prop1Key'="prop1Val")
+|COMMENT 'BLABLA'
+|AS SELECT * FROM tab1
+ """.stripMargin
+val parsed1 = parsePlan(v1)
+val expected1 = CreateViewStatement(
+ Seq("view1"),
+ Seq("col1" -> None, "col3" -> Some("hello")),
+ Some("BLABLA"),
+ Map("prop1Key" -> "prop1Val"),
+ Some("SELECT * FROM tab1"),
+ parsePlan("SELECT * FROM tab1"),
+ false,
+ true,
+ PersistedView)
+comparePlans(parsed1, expected1)
+
+val v2 =
+ """
+|CREATE OR REPLACE VIEW a.b.c
+|(col1, col3 COMMENT 'hello')
+|TBLPROPERTIES('prop1Key'="prop1Val")
+|COMMENT 'BLABLA'
+|AS SELECT * FROM tab1
+ """.stripMargin
+val parsed2 = parsePlan(v2)
+val expected2 = CreateViewStatement(
+ Seq("a", "b", "c"),
+ Seq("col1" -> None, "col3" -> Some("hello")),
+ Some("BLABLA"),
+ Map("prop1Key" -> "prop1Val"),
+ Some("SELECT * FROM tab1"),
+ parsePlan("SELECT * FROM tab1"),
+ false,
+ true,
+ PersistedView)
Review comment:
Sure. Updated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26666: [SPARK-30030][INFRA] Use RegexChecker instead of TokenChecker to check `regex`
dongjoon-hyun commented on a change in pull request #2: [SPARK-30030][INFRA] Use RegexChecker instead of TokenChecker to check `regex` URL: https://github.com/apache/spark/pull/2#discussion_r350388967 ## File path: scalastyle-config.xml ## @@ -258,7 +258,7 @@ This file is divided into 3 sections: scala.collection.JavaConverters._ and use .asScala / .asJava methods - + Review comment: @srowen and @databricks-david-lewis . This was the root cause. I'm not sure if there is any change inside `TokenChecker`, but `RegexChecker` is better at detecting `org.apache.commons.lang.` since it's not a token. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26666: [SPARK-30030][INFRA] Use RegexChecker instead of TokenChecker to check `regex`
AmplabJenkins removed a comment on issue #2: [SPARK-30030][INFRA] Use RegexChecker instead of TokenChecker to check `regex` URL: https://github.com/apache/spark/pull/2#issuecomment-558314357 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19262/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26666: [SPARK-30030][INFRA] Use RegexChecker instead of TokenChecker to check `regex`
AmplabJenkins removed a comment on issue #2: [SPARK-30030][INFRA] Use RegexChecker instead of TokenChecker to check `regex` URL: https://github.com/apache/spark/pull/2#issuecomment-558314354 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26666: [SPARK-30030][INFRA] Use RegexChecker instead of TokenChecker to check `regex`
AmplabJenkins commented on issue #2: [SPARK-30030][INFRA] Use RegexChecker instead of TokenChecker to check `regex` URL: https://github.com/apache/spark/pull/2#issuecomment-558314357 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19262/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26666: [SPARK-30030][INFRA] Use RegexChecker instead of TokenChecker to check `regex`
AmplabJenkins commented on issue #2: [SPARK-30030][INFRA] Use RegexChecker instead of TokenChecker to check `regex` URL: https://github.com/apache/spark/pull/2#issuecomment-558314354 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26666: [SPARK-30030][INFRA] Use RegexChecker instead of TokenChecker to check `regex`
dongjoon-hyun commented on a change in pull request #2: [SPARK-30030][INFRA] Use RegexChecker instead of TokenChecker to check `regex` URL: https://github.com/apache/spark/pull/2#discussion_r350388967 ## File path: scalastyle-config.xml ## @@ -258,7 +258,7 @@ This file is divided into 3 sections: scala.collection.JavaConverters._ and use .asScala / .asJava methods - + Review comment: @srowen and @databricks-david-lewis . This was the root cause. I'm not sure if there is any change inside `TokenChecker`, but `RegexChecker` is better for `org.apache.commons.lang.` since it's not a token. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26666: [SPARK-30030][INFRA] Use RegexChecker instead of TokenChecker to check `regex`
SparkQA commented on issue #2: [SPARK-30030][INFRA] Use RegexChecker instead of TokenChecker to check `regex` URL: https://github.com/apache/spark/pull/2#issuecomment-558313912 **[Test build #114425 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114425/testReport)** for PR 2 at commit [`f3b84d8`](https://github.com/apache/spark/commit/f3b84d806c9a57a79fbc616fce49bacfcf0ff246). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26082: [SPARK-29431][WebUI] Improve Web UI / Sql tab visualization with cached dataframes.
SparkQA commented on issue #26082: [SPARK-29431][WebUI] Improve Web UI / Sql tab visualization with cached dataframes. URL: https://github.com/apache/spark/pull/26082#issuecomment-558313910 **[Test build #114426 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114426/testReport)** for PR 26082 at commit [`09b4ed3`](https://github.com/apache/spark/commit/09b4ed3480f128f3a35690ea34295734065f292e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26651: [SPARK-30008][SQL] The dataType of collect_list/collect_set aggs should be ArrayType(_, false)
AmplabJenkins removed a comment on issue #26651: [SPARK-30008][SQL] The dataType of collect_list/collect_set aggs should be ArrayType(_, false) URL: https://github.com/apache/spark/pull/26651#issuecomment-558313160 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114411/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26651: [SPARK-30008][SQL] The dataType of collect_list/collect_set aggs should be ArrayType(_, false)
AmplabJenkins removed a comment on issue #26651: [SPARK-30008][SQL] The dataType of collect_list/collect_set aggs should be ArrayType(_, false) URL: https://github.com/apache/spark/pull/26651#issuecomment-558313150 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26651: [SPARK-30008][SQL] The dataType of collect_list/collect_set aggs should be ArrayType(_, false)
AmplabJenkins commented on issue #26651: [SPARK-30008][SQL] The dataType of collect_list/collect_set aggs should be ArrayType(_, false) URL: https://github.com/apache/spark/pull/26651#issuecomment-558313160 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114411/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26651: [SPARK-30008][SQL] The dataType of collect_list/collect_set aggs should be ArrayType(_, false)
AmplabJenkins commented on issue #26651: [SPARK-30008][SQL] The dataType of collect_list/collect_set aggs should be ArrayType(_, false) URL: https://github.com/apache/spark/pull/26651#issuecomment-558313150 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26651: [SPARK-30008][SQL] The dataType of collect_list/collect_set aggs should be ArrayType(_, false)
SparkQA removed a comment on issue #26651: [SPARK-30008][SQL] The dataType of collect_list/collect_set aggs should be ArrayType(_, false) URL: https://github.com/apache/spark/pull/26651#issuecomment-558208448 **[Test build #114411 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114411/testReport)** for PR 26651 at commit [`ff9d524`](https://github.com/apache/spark/commit/ff9d524aea2171ba4a515b50a9d7e52c90d3d090). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26651: [SPARK-30008][SQL] The dataType of collect_list/collect_set aggs should be ArrayType(_, false)
SparkQA commented on issue #26651: [SPARK-30008][SQL] The dataType of collect_list/collect_set aggs should be ArrayType(_, false) URL: https://github.com/apache/spark/pull/26651#issuecomment-558312435 **[Test build #114411 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114411/testReport)** for PR 26651 at commit [`ff9d524`](https://github.com/apache/spark/commit/ff9d524aea2171ba4a515b50a9d7e52c90d3d090). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26082: [SPARK-29431][WebUI] Improve Web UI / Sql tab visualization with cached dataframes.
AmplabJenkins removed a comment on issue #26082: [SPARK-29431][WebUI] Improve Web UI / Sql tab visualization with cached dataframes. URL: https://github.com/apache/spark/pull/26082#issuecomment-558311307 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26633: [SPARK-29994][CORE] Add WILDCARD task location
AmplabJenkins commented on issue #26633: [SPARK-29994][CORE] Add WILDCARD task location URL: https://github.com/apache/spark/pull/26633#issuecomment-558311338 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26633: [SPARK-29994][CORE] Add WILDCARD task location
AmplabJenkins removed a comment on issue #26633: [SPARK-29994][CORE] Add WILDCARD task location URL: https://github.com/apache/spark/pull/26633#issuecomment-558311347 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114415/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26082: [SPARK-29431][WebUI] Improve Web UI / Sql tab visualization with cached dataframes.
AmplabJenkins removed a comment on issue #26082: [SPARK-29431][WebUI] Improve Web UI / Sql tab visualization with cached dataframes. URL: https://github.com/apache/spark/pull/26082#issuecomment-558311320 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19261/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26082: [SPARK-29431][WebUI] Improve Web UI / Sql tab visualization with cached dataframes.
AmplabJenkins commented on issue #26082: [SPARK-29431][WebUI] Improve Web UI / Sql tab visualization with cached dataframes. URL: https://github.com/apache/spark/pull/26082#issuecomment-558311320 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19261/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26633: [SPARK-29994][CORE] Add WILDCARD task location
AmplabJenkins commented on issue #26633: [SPARK-29994][CORE] Add WILDCARD task location URL: https://github.com/apache/spark/pull/26633#issuecomment-558311347 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114415/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26082: [SPARK-29431][WebUI] Improve Web UI / Sql tab visualization with cached dataframes.
AmplabJenkins commented on issue #26082: [SPARK-29431][WebUI] Improve Web UI / Sql tab visualization with cached dataframes. URL: https://github.com/apache/spark/pull/26082#issuecomment-558311307 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] planga82 commented on issue #26082: [SPARK-29431][WebUI] Improve Web UI / Sql tab visualization with cached dataframes.
planga82 commented on issue #26082: [SPARK-29431][WebUI] Improve Web UI / Sql tab visualization with cached dataframes. URL: https://github.com/apache/spark/pull/26082#issuecomment-558311529 @dongjoon-hyun What is your opinion about this changes? Thank you This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26633: [SPARK-29994][CORE] Add WILDCARD task location
AmplabJenkins removed a comment on issue #26633: [SPARK-29994][CORE] Add WILDCARD task location URL: https://github.com/apache/spark/pull/26633#issuecomment-558311338 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26633: [SPARK-29994][CORE] Add WILDCARD task location
SparkQA removed a comment on issue #26633: [SPARK-29994][CORE] Add WILDCARD task location URL: https://github.com/apache/spark/pull/26633#issuecomment-558244366 **[Test build #114415 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114415/testReport)** for PR 26633 at commit [`72a946c`](https://github.com/apache/spark/commit/72a946cb5649a08a8bfc8de03924fd95349347e1). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26667: [SPARK-29922][SQL] SHOW FUNCTIONS should do multi-catalog resolution
AmplabJenkins removed a comment on issue #26667: [SPARK-29922][SQL] SHOW FUNCTIONS should do multi-catalog resolution URL: https://github.com/apache/spark/pull/26667#issuecomment-558307932 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26667: [SPARK-29922][SQL] SHOW FUNCTIONS should do multi-catalog resolution
AmplabJenkins commented on issue #26667: [SPARK-29922][SQL] SHOW FUNCTIONS should do multi-catalog resolution URL: https://github.com/apache/spark/pull/26667#issuecomment-558310499 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26633: [SPARK-29994][CORE] Add WILDCARD task location
SparkQA commented on issue #26633: [SPARK-29994][CORE] Add WILDCARD task location URL: https://github.com/apache/spark/pull/26633#issuecomment-558310516 **[Test build #114415 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114415/testReport)** for PR 26633 at commit [`72a946c`](https://github.com/apache/spark/commit/72a946cb5649a08a8bfc8de03924fd95349347e1). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26082: [SPARK-29431][WebUI] Improve Web UI / Sql tab visualization with cached dataframes.
SparkQA commented on issue #26082: [SPARK-29431][WebUI] Improve Web UI / Sql tab visualization with cached dataframes. URL: https://github.com/apache/spark/pull/26082#issuecomment-558310694 **[Test build #114424 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114424/testReport)** for PR 26082 at commit [`c030902`](https://github.com/apache/spark/commit/c03090224b5fd1c9c77d6fd9321376a187e0cad5). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #26649: [SPARK-29862][SQL] CREATE (OR REPLACE) ... VIEW should look up catalog/table like v2 commands
viirya commented on a change in pull request #26649: [SPARK-29862][SQL] CREATE
(OR REPLACE) ... VIEW should look up catalog/table like v2 commands
URL: https://github.com/apache/spark/pull/26649#discussion_r350384426
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala
##
@@ -1669,6 +1669,105 @@ class DDLParserSuite extends AnalysisTest {
comparePlans(parsed, expected)
}
+ test("create view -- basic") {
+val v1 = "CREATE VIEW view1 AS SELECT * FROM tab1"
+val parsed1 = parsePlan(v1)
+
+val expected1 = CreateViewStatement(
+ Seq("view1"),
+ Seq.empty[(String, Option[String])],
+ None,
+ Map.empty[String, String],
+ Some("SELECT * FROM tab1"),
+ parsePlan("SELECT * FROM tab1"),
+ false,
+ false,
+ PersistedView)
+comparePlans(parsed1, expected1)
+
+val v2 = "CREATE VIEW a.b.c AS SELECT * FROM tab1"
+val parsed2 = parsePlan(v2)
+
+val expected2 = CreateViewStatement(
+ Seq("a", "b", "c"),
+ Seq.empty[(String, Option[String])],
+ None,
+ Map.empty[String, String],
+ Some("SELECT * FROM tab1"),
+ parsePlan("SELECT * FROM tab1"),
+ false,
+ false,
+ PersistedView)
+comparePlans(parsed2, expected2)
+ }
+
+ test("create view - full") {
+val v1 =
+ """
+|CREATE OR REPLACE VIEW view1
+|(col1, col3 COMMENT 'hello')
+|TBLPROPERTIES('prop1Key'="prop1Val")
+|COMMENT 'BLABLA'
+|AS SELECT * FROM tab1
+ """.stripMargin
+val parsed1 = parsePlan(v1)
+val expected1 = CreateViewStatement(
+ Seq("view1"),
+ Seq("col1" -> None, "col3" -> Some("hello")),
+ Some("BLABLA"),
+ Map("prop1Key" -> "prop1Val"),
+ Some("SELECT * FROM tab1"),
+ parsePlan("SELECT * FROM tab1"),
+ false,
+ true,
+ PersistedView)
+comparePlans(parsed1, expected1)
+
+val v2 =
+ """
+|CREATE OR REPLACE VIEW a.b.c
+|(col1, col3 COMMENT 'hello')
+|TBLPROPERTIES('prop1Key'="prop1Val")
+|COMMENT 'BLABLA'
+|AS SELECT * FROM tab1
+ """.stripMargin
+val parsed2 = parsePlan(v2)
+val expected2 = CreateViewStatement(
+ Seq("a", "b", "c"),
+ Seq("col1" -> None, "col3" -> Some("hello")),
+ Some("BLABLA"),
+ Map("prop1Key" -> "prop1Val"),
+ Some("SELECT * FROM tab1"),
+ parsePlan("SELECT * FROM tab1"),
+ false,
+ true,
+ PersistedView)
Review comment:
these tests are all for PersistedView, can we have few for other view types?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26667: [SPARK-29922][SQL] SHOW FUNCTIONS should do multi-catalog resolution
AmplabJenkins commented on issue #26667: [SPARK-29922][SQL] SHOW FUNCTIONS should do multi-catalog resolution URL: https://github.com/apache/spark/pull/26667#issuecomment-558307932 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] planga82 commented on issue #26667: [SPARK-29922][SQL] SHOW FUNCTIONS should do multi-catalog resolution
planga82 commented on issue #26667: [SPARK-29922][SQL] SHOW FUNCTIONS should do multi-catalog resolution URL: https://github.com/apache/spark/pull/26667#issuecomment-558307837 cc: @cloud-fan @rdblue @viirya @imback82 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] planga82 opened a new pull request #26667: [SPARK-29922][SQL] SHOW FUNCTIONS should do multi-catalog resolution
planga82 opened a new pull request #26667: [SPARK-29922][SQL] SHOW FUNCTIONS should do multi-catalog resolution URL: https://github.com/apache/spark/pull/26667 ### What changes were proposed in this pull request? Add ShowFunctionsStatement and make SHOW FUNCTIONS go through the same catalog/table resolution framework of v2 commands. We don’t have this methods in the catalog to implement an V2 command * catalog.listFunctions ### Why are the changes needed? It's important to make all the commands have the same table resolution behavior, to avoid confusing `SHOW FUNCTIONS LIKE namespace.function` ### Does this PR introduce any user-facing change? Yes. When running SHOW FUNCTIONS LIKE namespace.function Spark fails the command if the current catalog is set to a v2 catalog. ### How was this patch tested? Unit tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26666: [SPARK-29081][CORE][FOLLOWUP] Use commons-lang3 instead of commons-lang2
dongjoon-hyun commented on issue #2: [SPARK-29081][CORE][FOLLOWUP] Use commons-lang3 instead of commons-lang2 URL: https://github.com/apache/spark/pull/2#issuecomment-558306997 Thank you for review, @databricks-david-lewis . Yes. I'm also suspecting that and checking now (if we have more instances like that). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] databricks-david-lewis commented on issue #26666: [SPARK-29081][CORE][FOLLOWUP] Use commons-lang3 instead of commons-lang2
databricks-david-lewis commented on issue #2: [SPARK-29081][CORE][FOLLOWUP] Use commons-lang3 instead of commons-lang2 URL: https://github.com/apache/spark/pull/2#issuecomment-558305738 LGTM, sorry for letting it sneak back in! Maybe benchmark code is excluded from that check? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26664: [SPARK-9853][Doc][Follow-up] Update the description for all the SQLConf affected by spark.sql.adaptive.enabled
SparkQA commented on issue #26664: [SPARK-9853][Doc][Follow-up] Update the description for all the SQLConf affected by spark.sql.adaptive.enabled URL: https://github.com/apache/spark/pull/26664#issuecomment-558304425 **[Test build #114423 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114423/testReport)** for PR 26664 at commit [`81c536d`](https://github.com/apache/spark/commit/81c536d7aa91572006168d1887dd102e7df18ff1). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26666: [SPARK-29081][CORE][FOLLOWUP] Use commons-lang3 instead of commons-lang2
dongjoon-hyun commented on issue #2: [SPARK-29081][CORE][FOLLOWUP] Use commons-lang3 instead of commons-lang2 URL: https://github.com/apache/spark/pull/2#issuecomment-558303060 Oh, interesting. We have a rule for that. Let me dig the reason how this escape from the checker. ``` org\.apache\.commons\.lang\. Use Commons Lang 3 classes (package org.apache.commons.lang3.*) instead of Commons Lang 2 (package org.apache.commons.lang.*) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26664: [SPARK-9853][Doc][Follow-up] Update the description for all the SQLConf affected by spark.sql.adaptive.enabled
AmplabJenkins removed a comment on issue #26664: [SPARK-9853][Doc][Follow-up] Update the description for all the SQLConf affected by spark.sql.adaptive.enabled URL: https://github.com/apache/spark/pull/26664#issuecomment-558301849 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26655: [SPARK-29293][BUILD] Update scalafmt plugin to 1.0.2 for compatibility with Scala 2.13
AmplabJenkins removed a comment on issue #26655: [SPARK-29293][BUILD] Update scalafmt plugin to 1.0.2 for compatibility with Scala 2.13 URL: https://github.com/apache/spark/pull/26655#issuecomment-558301853 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26664: [SPARK-9853][Doc][Follow-up] Update the description for all the SQLConf affected by spark.sql.adaptive.enabled
AmplabJenkins removed a comment on issue #26664: [SPARK-9853][Doc][Follow-up] Update the description for all the SQLConf affected by spark.sql.adaptive.enabled URL: https://github.com/apache/spark/pull/26664#issuecomment-558301858 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19259/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26655: [SPARK-29293][BUILD] Update scalafmt plugin to 1.0.2 for compatibility with Scala 2.13
AmplabJenkins removed a comment on issue #26655: [SPARK-29293][BUILD] Update scalafmt plugin to 1.0.2 for compatibility with Scala 2.13 URL: https://github.com/apache/spark/pull/26655#issuecomment-558301863 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19260/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #26664: [SPARK-9853][Doc][Follow-up] Update the description for all the SQLConf affected by spark.sql.adaptive.enabled
xuanyuanking commented on a change in pull request #26664:
[SPARK-9853][Doc][Follow-up] Update the description for all the SQLConf
affected by spark.sql.adaptive.enabled
URL: https://github.com/apache/spark/pull/26664#discussion_r350375087
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -355,29 +355,33 @@ object SQLConf {
val SHUFFLE_TARGET_POSTSHUFFLE_INPUT_SIZE =
buildConf("spark.sql.adaptive.shuffle.targetPostShuffleInputSize")
- .doc("The target post-shuffle input size in bytes of a task.")
+ .doc("The target post-shuffle input size in bytes of a task. This
configuration only has " +
+s"an effect when '${ADAPTIVE_EXECUTION_ENABLED.key}' is enabled.")
Review comment:
Thanks, fix the doc of SHUFFLE_MIN_NUM_POSTSHUFFLE_PARTITIONS and
SHUFFLE_TARGET_POSTSHUFFLE_INPUT_SIZE.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26664: [SPARK-9853][Doc][Follow-up] Update the description for all the SQLConf affected by spark.sql.adaptive.enabled
AmplabJenkins commented on issue #26664: [SPARK-9853][Doc][Follow-up] Update the description for all the SQLConf affected by spark.sql.adaptive.enabled URL: https://github.com/apache/spark/pull/26664#issuecomment-558301858 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19259/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26655: [SPARK-29293][BUILD] Update scalafmt plugin to 1.0.2 for compatibility with Scala 2.13
AmplabJenkins commented on issue #26655: [SPARK-29293][BUILD] Update scalafmt plugin to 1.0.2 for compatibility with Scala 2.13 URL: https://github.com/apache/spark/pull/26655#issuecomment-558301853 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26655: [SPARK-29293][BUILD] Update scalafmt plugin to 1.0.2 for compatibility with Scala 2.13
AmplabJenkins commented on issue #26655: [SPARK-29293][BUILD] Update scalafmt plugin to 1.0.2 for compatibility with Scala 2.13 URL: https://github.com/apache/spark/pull/26655#issuecomment-558301863 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19260/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26664: [SPARK-9853][Doc][Follow-up] Update the description for all the SQLConf affected by spark.sql.adaptive.enabled
AmplabJenkins commented on issue #26664: [SPARK-9853][Doc][Follow-up] Update the description for all the SQLConf affected by spark.sql.adaptive.enabled URL: https://github.com/apache/spark/pull/26664#issuecomment-558301849 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shahidki31 commented on issue #26616: [SPARK-25392][Webui]Prevent error page when accessing pools page from history server
shahidki31 commented on issue #26616: [SPARK-25392][Webui]Prevent error page when accessing pools page from history server URL: https://github.com/apache/spark/pull/26616#issuecomment-558301458 cc @vanzin @srowen Could you please review this patch. seems a minor change to fix avoiding the error page. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26655: [SPARK-29293][BUILD] Update scalafmt plugin to 1.0.2 for compatibility with Scala 2.13
SparkQA commented on issue #26655: [SPARK-29293][BUILD] Update scalafmt plugin to 1.0.2 for compatibility with Scala 2.13 URL: https://github.com/apache/spark/pull/26655#issuecomment-558301344 **[Test build #114422 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114422/testReport)** for PR 26655 at commit [`9142562`](https://github.com/apache/spark/commit/914256246611b234a8ab66d290615e7d9d56469d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26666: [SPARK-29081][CORE][FOLLOWUP] Use commons-lang3 instead of commons-lang2
dongjoon-hyun commented on issue #2: [SPARK-29081][CORE][FOLLOWUP] Use commons-lang3 instead of commons-lang2 URL: https://github.com/apache/spark/pull/2#issuecomment-558301021 Thanks you, @srowen . Yes. I'll try to clean up the pom and dependency like this at this time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on issue #26655: [SPARK-29293][BUILD] Update scalafmt plugin to 1.0.2 for compatibility with Scala 2.13
srowen commented on issue #26655: [SPARK-29293][BUILD] Update scalafmt plugin to 1.0.2 for compatibility with Scala 2.13 URL: https://github.com/apache/spark/pull/26655#issuecomment-558300497 I updated to just: - Move scalafmt under the 2.12 profile - Move it to pluginManagement for good measure (still works) - Add the extra skip configs that version 1.x wants, before I forget - Manually tested that it does reformat, and only files in the current changelist This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26658: [SPARK-30015][BUILD] Move hive-storage-api dependency from `hive-2.3` to `sql/core`
dongjoon-hyun commented on issue #26658: [SPARK-30015][BUILD] Move hive-storage-api dependency from `hive-2.3` to `sql/core` URL: https://github.com/apache/spark/pull/26658#issuecomment-558300522 Thanks for confirming. Yes. Right. `hadoop-3.2` only works with `Hive 2.3`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host
tgravescs commented on a change in pull request #25299: [SPARK-27651][Core]
Avoid the network when shuffle blocks are fetched from the same host
URL: https://github.com/apache/spark/pull/25299#discussion_r350372735
##
File path:
core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
##
@@ -441,6 +540,10 @@ final class ShuffleBlockFetcherIterator(
// Get Local Blocks
fetchLocalBlocks()
logDebug(s"Got local blocks in ${Utils.getUsedTimeNs(startTimeNs)}")
+
+if (hostLocalBlocks.nonEmpty) {
+ blockManager.hostLocalDirManager.foreach(fetchHostLocalBlocks)
Review comment:
might be nice to add a logDebug similar to above about how long it took
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] tgravescs commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host
tgravescs commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host URL: https://github.com/apache/spark/pull/25299#issuecomment-558299984 Definitely agree, sounds like gain and been wanting this for a while so thanks @attilapiros for working on it. And you're right I wasn't thinking about this not even opening the file so should be very fast. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26378: [SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI
AmplabJenkins removed a comment on issue #26378: [SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI URL: https://github.com/apache/spark/pull/26378#issuecomment-558298885 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19258/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26378: [SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI
AmplabJenkins removed a comment on issue #26378: [SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI URL: https://github.com/apache/spark/pull/26378#issuecomment-558298881 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26378: [SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI
AmplabJenkins commented on issue #26378: [SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI URL: https://github.com/apache/spark/pull/26378#issuecomment-558298885 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19258/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26378: [SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI
AmplabJenkins commented on issue #26378: [SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI URL: https://github.com/apache/spark/pull/26378#issuecomment-558298881 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on issue #26524: [SPARK-29898][SQL] Support Avro Custom Logical Types
gengliangwang commented on issue #26524: [SPARK-29898][SQL] Support Avro Custom Logical Types URL: https://github.com/apache/spark/pull/26524#issuecomment-558298514 > With this extension you can add your custom logical types and map it to the properly SQL type. For example; read a string ISO date from Avro (marked as iso-date logical type) and use as DateType in Spark. Sorry I don't get it. Could you provide a reasonable scenario? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26378: [SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI
SparkQA commented on issue #26378: [SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI URL: https://github.com/apache/spark/pull/26378#issuecomment-558298329 **[Test build #114421 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114421/testReport)** for PR 26378 at commit [`607dded`](https://github.com/apache/spark/commit/607ddedf9599addc598426694e9eafab296514e3). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] liancheng commented on issue #26437: [SPARK-29800][SQL] Plan non-correlated Exists 's subquery in PlanSubqueries
liancheng commented on issue #26437: [SPARK-29800][SQL] Plan non-correlated Exists 's subquery in PlanSubqueries URL: https://github.com/apache/spark/pull/26437#issuecomment-558296560 @AngersZh, I think there's probably some thing wrong with your latest master merge... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26378: [SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI
AmplabJenkins removed a comment on issue #26378: [SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI URL: https://github.com/apache/spark/pull/26378#issuecomment-558295959 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114420/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26378: [SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI
AmplabJenkins removed a comment on issue #26378: [SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI URL: https://github.com/apache/spark/pull/26378#issuecomment-558295950 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26378: [SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI
SparkQA removed a comment on issue #26378: [SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI URL: https://github.com/apache/spark/pull/26378#issuecomment-558295163 **[Test build #114420 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114420/testReport)** for PR 26378 at commit [`4ab4e92`](https://github.com/apache/spark/commit/4ab4e92e7dbe05b7a6f49c6eee1a6744c795e7a5). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26378: [SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI
AmplabJenkins commented on issue #26378: [SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI URL: https://github.com/apache/spark/pull/26378#issuecomment-558295950 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26378: [SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI
SparkQA commented on issue #26378: [SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI URL: https://github.com/apache/spark/pull/26378#issuecomment-558295940 **[Test build #114420 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114420/testReport)** for PR 26378 at commit [`4ab4e92`](https://github.com/apache/spark/commit/4ab4e92e7dbe05b7a6f49c6eee1a6744c795e7a5). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26378: [SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI
AmplabJenkins commented on issue #26378: [SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI URL: https://github.com/apache/spark/pull/26378#issuecomment-558295959 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114420/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26378: [SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI
AmplabJenkins removed a comment on issue #26378: [SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI URL: https://github.com/apache/spark/pull/26378#issuecomment-558295781 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19257/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26378: [SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI
AmplabJenkins removed a comment on issue #26378: [SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI URL: https://github.com/apache/spark/pull/26378#issuecomment-558295771 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26378: [SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI
AmplabJenkins commented on issue #26378: [SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI URL: https://github.com/apache/spark/pull/26378#issuecomment-558295771 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26378: [SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI
AmplabJenkins commented on issue #26378: [SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI URL: https://github.com/apache/spark/pull/26378#issuecomment-558295781 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19257/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26378: [SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI
SparkQA commented on issue #26378: [SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI URL: https://github.com/apache/spark/pull/26378#issuecomment-558295163 **[Test build #114420 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114420/testReport)** for PR 26378 at commit [`4ab4e92`](https://github.com/apache/spark/commit/4ab4e92e7dbe05b7a6f49c6eee1a6744c795e7a5). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shahidki31 commented on a change in pull request #26378: [SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI
shahidki31 commented on a change in pull request #26378:
[SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer
WebUI
URL: https://github.com/apache/spark/pull/26378#discussion_r350366443
##
File path:
sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/HiveThriftServer2Listener.scala
##
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hive.thriftserver.ui
+
+import java.util.concurrent.ConcurrentHashMap
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.hive.service.server.HiveServer2
+
+import org.apache.spark.{SparkConf, SparkContext}
+import org.apache.spark.scheduler._
+import org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.ExecutionState
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.status.{ElementTrackingStore, KVUtils, LiveEntity}
+
+/**
+ * An inner sparkListener called in sc.stop to clean up the HiveThriftServer2
+ */
+private[thriftserver] class HiveThriftServer2Listener(
+ kvstore: ElementTrackingStore,
+ sparkConf: SparkConf,
+ server: Option[HiveServer2],
+ live: Boolean = true) extends SparkListener {
+
+ private val sessionList = new ConcurrentHashMap[String, LiveSessionData]()
+ private val executionList = new ConcurrentHashMap[String,
LiveExecutionData]()
+
+ private val (retainedStatements: Int, retainedSessions: Int) = {
+(sparkConf.get(SQLConf.THRIFTSERVER_UI_STATEMENT_LIMIT),
+ sparkConf.get(SQLConf.THRIFTSERVER_UI_SESSION_LIMIT))
+ }
+
+ // Returns true if this listener has no live data. Exposed for tests only.
+ private[thriftserver] def noLiveData(): Boolean = {
+sessionList.isEmpty && executionList.isEmpty
+ }
+
+ kvstore.addTrigger(classOf[SessionInfo], retainedSessions) { count =>
+cleanupSession(count)
+ }
+
+ kvstore.addTrigger(classOf[ExecutionInfo], retainedStatements) { count =>
+cleanupExecutions(count)
+ }
+
+ kvstore.onFlush {
+if (!live) {
+ val now = System.nanoTime()
+ flush(update(_, now))
+}
+ }
+
+ override def onApplicationEnd(applicationEnd: SparkListenerApplicationEnd):
Unit = {
+if (live) {
+ server.foreach(_.stop())
+}
+ }
+
+ override def onJobStart(jobStart: SparkListenerJobStart): Unit = {
+val properties = jobStart.properties
+if (properties != null) {
+ val groupId = properties.getProperty(SparkContext.SPARK_JOB_GROUP_ID)
+ if (groupId != null) {
+updateJobDetails(jobStart.jobId.toString, groupId)
+}
+ }
+}
+
+ /**
+ * This method is to handle out of order events. ie. if Job event come after
execution end event.
+ * @param jobId
+ * @param groupId
+ */
+ private def updateJobDetails(jobId: String, groupId: String): Unit = {
+val execList = executionList.values().asScala.filter(_.groupId ==
groupId).toSeq
+if (execList.nonEmpty) {
+ execList.foreach { exec =>
+exec.jobId += jobId.toString
+updateLiveStore(exec)
+ }
+} else {
+ // Here will come only if JobStart event comes after Execution End event.
+ val storeExecInfo =
kvstore.view(classOf[ExecutionInfo]).asScala.filter(_.groupId == groupId)
+ storeExecInfo.foreach { exec =>
+val liveExec = getOrCreateExecution(exec.execId, exec.statement,
exec.sessionId,
+ exec.startTimestamp, exec.userName)
+liveExec.jobId += jobId.toString
+updateLiveStore(liveExec, true)
+executionList.remove(liveExec.execId)
+ }
+}
+ }
+
+ override def onOtherEvent(event: SparkListenerEvent): Unit = {
+event match {
+ case e: SparkListenerSessionCreated => onSessionCreated(e)
+ case e: SparkListenerSessionClosed => onSessionClosed(e)
+ case e: SparkListenerOperationStart => onOperationStart(e)
+ case e: SparkListenerOperationParsed => onOperationParsed(e)
+ case e: SparkListenerOperationCanceled => onOperationCanceled(e)
+ case e: SparkListenerOperationError => onOperationError(e)
+ case e: SparkListenerOperationFinish => onOperationFinished(e)
+ case e: SparkListenerOperationClosed =>
[GitHub] [spark] shahidki31 commented on a change in pull request #26378: [SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI
shahidki31 commented on a change in pull request #26378:
[SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer
WebUI
URL: https://github.com/apache/spark/pull/26378#discussion_r350366189
##
File path:
sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/HiveThriftServer2Listener.scala
##
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hive.thriftserver.ui
+
+import java.util.concurrent.ConcurrentHashMap
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.hive.service.server.HiveServer2
+
+import org.apache.spark.{SparkConf, SparkContext}
+import org.apache.spark.scheduler._
+import org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.ExecutionState
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.status.{ElementTrackingStore, KVUtils, LiveEntity}
+
+/**
+ * An inner sparkListener called in sc.stop to clean up the HiveThriftServer2
+ */
+private[thriftserver] class HiveThriftServer2Listener(
+ kvstore: ElementTrackingStore,
Review comment:
Right, ideally style check should catch this. Let me check my IJ settings
again.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shahidki31 commented on a change in pull request #26378: [SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI
shahidki31 commented on a change in pull request #26378:
[SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer
WebUI
URL: https://github.com/apache/spark/pull/26378#discussion_r350365849
##
File path:
sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/HiveThriftServer2EventManager.scala
##
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.hive.thriftserver.ui
+
+import org.apache.spark.SparkContext
+import org.apache.spark.scheduler.SparkListenerEvent
+
+/**
+ * This class manages events generated by the thriftserver application. It
converts the
+ * operation and session events to listener events and post it into the live
listener bus.
+ */
+private[thriftserver] class HiveThriftServer2EventManager(sc: SparkContext) {
+
+ def postLiveListenerBus(event: SparkListenerEvent): Unit = {
+ sc.listenerBus.post(event)
+ }
+
+ def onSessionCreated(ip: String, sessionId: String, userName: String =
"UNKNOWN"): Unit = {
+postLiveListenerBus(SparkListenerSessionCreated(ip, sessionId,
+ userName, System.currentTimeMillis()))
+ }
+
+ def onSessionClosed(sessionId: String): Unit = {
+postLiveListenerBus(SparkListenerSessionClosed(sessionId,
System.currentTimeMillis()))
+ }
+
+ def onStatementStart(
+ id: String,
+ sessionId: String,
+ statement: String,
+ groupId: String,
+ userName: String = "UNKNOWN"): Unit = {
+postLiveListenerBus(SparkListenerOperationStart(id, sessionId, statement,
groupId,
+ System.currentTimeMillis(), userName))
+ }
+
+ def onStatementParsed(id: String, executionPlan: String): Unit = {
+postLiveListenerBus(SparkListenerOperationParsed(id, executionPlan))
+ }
+
+ def onStatementCanceled(id: String): Unit = {
+postLiveListenerBus(SparkListenerOperationCanceled(id,
System.currentTimeMillis()))
+ }
+
+ def onStatementError(id: String, errorMsg: String, errorTrace: String): Unit
= {
+postLiveListenerBus(SparkListenerOperationError(id, errorMsg, errorTrace,
+ System.currentTimeMillis()))
+ }
+
+ def onStatementFinish(id: String): Unit = {
+postLiveListenerBus(SparkListenerOperationFinish(id,
System.currentTimeMillis()))
+
+ }
+
+ def onOperationClosed(id: String): Unit = {
+postLiveListenerBus(SparkListenerOperationClosed(id,
System.currentTimeMillis()))
+ }
+}
+
+private[thriftserver] case class SparkListenerSessionCreated(
Review comment:
Yes, updated
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shahidki31 commented on a change in pull request #26378: [SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI
shahidki31 commented on a change in pull request #26378:
[SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer
WebUI
URL: https://github.com/apache/spark/pull/26378#discussion_r350365752
##
File path:
sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/HiveThriftServer2EventManager.scala
##
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.hive.thriftserver.ui
+
+import org.apache.spark.SparkContext
+import org.apache.spark.scheduler.SparkListenerEvent
+
+/**
+ * This class manages events generated by the thriftserver application. It
converts the
+ * operation and session events to listener events and post it into the live
listener bus.
+ */
+private[thriftserver] class HiveThriftServer2EventManager(sc: SparkContext) {
+
+ def postLiveListenerBus(event: SparkListenerEvent): Unit = {
+ sc.listenerBus.post(event)
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shahidki31 commented on a change in pull request #26378: [SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI
shahidki31 commented on a change in pull request #26378:
[SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer
WebUI
URL: https://github.com/apache/spark/pull/26378#discussion_r350365671
##
File path:
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##
@@ -352,17 +352,36 @@ private[history] class FsHistoryProvider(conf:
SparkConf, clock: Clock)
val ui = SparkUI.create(None, new HistoryAppStatusStore(conf, kvstore),
conf, secManager,
app.info.name, HistoryServer.getAttemptURI(appId,
attempt.info.attemptId),
attempt.info.startTime.getTime(), attempt.info.appSparkVersion)
-loadPlugins().foreach(_.setupUI(ui))
-
+setupPluginUI(ui)
val loadedUI = LoadedAppUI(ui)
-
synchronized {
activeUIs((appId, attemptId)) = loadedUI
}
Some(loadedUI)
}
+ private def setupPluginUI(ui: SparkUI): Unit = {
+val plugins = loadPlugins().toSeq
+
+var sqlTab: Option[AppHistoryServerPlugin] = None
+var jdbcTab: Option[AppHistoryServerPlugin] = None
+val otherTabs: Seq[AppHistoryServerPlugin] = Seq()
+
+plugins.foreach{ plugin =>
+ if (plugin.toString.contains("SQL")) {
+sqlTab = Some(plugin)
+ } else if (plugin.toString.contains("ThriftServer")) {
Review comment:
Thanks. Updated
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26666: [SPARK-29081][CORE][FOLLOWUP] Use commons-lang3 instead of commons-lang2
dongjoon-hyun commented on issue #2: [SPARK-29081][CORE][FOLLOWUP] Use commons-lang3 instead of commons-lang2 URL: https://github.com/apache/spark/pull/2#issuecomment-558292404 cc @databricks-david-lewis and @srowen This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26658: [SPARK-30015][BUILD] Move hive-storage-api dependency from `hive-2.3` to `sql/core`
dongjoon-hyun commented on issue #26658: [SPARK-30015][BUILD] Move hive-storage-api dependency from `hive-2.3` to `sql/core` URL: https://github.com/apache/spark/pull/26658#issuecomment-558289134 @tgravescs . After merging this, I'd like to change the following. ``` sql/hive/pom.xml: hive-2.3 ``` BTW, what you want when you use `-Phive`. `-Phive-1.2` or `-Phive-2.3`? > so what are the rest of the jiras to fix the build? I'm wondering your decision on `Hive 1.2.1` or `Hive 2.3.6`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26662: [SPARK-30026][SQL] Whitespaces can be identified as delimiters in interval string
AmplabJenkins removed a comment on issue #26662: [SPARK-30026][SQL] Whitespaces can be identified as delimiters in interval string URL: https://github.com/apache/spark/pull/26662#issuecomment-558288203 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26662: [SPARK-30026][SQL] Whitespaces can be identified as delimiters in interval string
AmplabJenkins removed a comment on issue #26662: [SPARK-30026][SQL] Whitespaces can be identified as delimiters in interval string URL: https://github.com/apache/spark/pull/26662#issuecomment-558288223 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114410/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26662: [SPARK-30026][SQL] Whitespaces can be identified as delimiters in interval string
AmplabJenkins commented on issue #26662: [SPARK-30026][SQL] Whitespaces can be identified as delimiters in interval string URL: https://github.com/apache/spark/pull/26662#issuecomment-558288203 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26662: [SPARK-30026][SQL] Whitespaces can be identified as delimiters in interval string
AmplabJenkins commented on issue #26662: [SPARK-30026][SQL] Whitespaces can be identified as delimiters in interval string URL: https://github.com/apache/spark/pull/26662#issuecomment-558288223 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114410/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26662: [SPARK-30026][SQL] Whitespaces can be identified as delimiters in interval string
SparkQA removed a comment on issue #26662: [SPARK-30026][SQL] Whitespaces can be identified as delimiters in interval string URL: https://github.com/apache/spark/pull/26662#issuecomment-558190748 **[Test build #114410 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114410/testReport)** for PR 26662 at commit [`7af388b`](https://github.com/apache/spark/commit/7af388ba56115d86bd3bc5845bbf3680c7a98107). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26662: [SPARK-30026][SQL] Whitespaces can be identified as delimiters in interval string
SparkQA commented on issue #26662: [SPARK-30026][SQL] Whitespaces can be identified as delimiters in interval string URL: https://github.com/apache/spark/pull/26662#issuecomment-558287437 **[Test build #114410 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114410/testReport)** for PR 26662 at commit [`7af388b`](https://github.com/apache/spark/commit/7af388ba56115d86bd3bc5845bbf3680c7a98107). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
