[GitHub] [spark] AmplabJenkins removed a comment on pull request #28909: [SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
AmplabJenkins removed a comment on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648607669 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28909: [SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
AmplabJenkins commented on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648607669 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #28781: [SPARK-31953][SS] Add Spark Structured Streaming History Server Support
gengliangwang commented on pull request #28781: URL: https://github.com/apache/spark/pull/28781#issuecomment-648607182 @uncleGen Thanks for the work! I think the major issue of this PR is that there are no related test cases. How about upload event log of streaming workload and create test cases like HistoryServerSuite This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
Ngone51 commented on a change in pull request #28880:
URL: https://github.com/apache/spark/pull/28880#discussion_r444658287
##
File path:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##
@@ -211,9 +211,11 @@ private[spark] class ApplicationMaster(
final def run(): Int = {
try {
val attemptID = if (isClusterMode) {
-// Set the web ui port to be ephemeral for yarn so we don't conflict
with
-// other spark processes running on the same box
-System.setProperty(UI_PORT.key, "0")
+// Set the web ui port to be ephemeral for yarn if not set explicitly
+// so we don't conflict with other spark processes running on the same
box
+if (System.getProperty(UI_PORT.key) != null) {
+ System.setProperty(UI_PORT.key, "0")
+}
Review comment:
I see. This is what we expected, right?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28900: [SPARK-32056][SQL] Coalesce partitions for repartition by expressions when AQE is enabled
AmplabJenkins removed a comment on pull request #28900: URL: https://github.com/apache/spark/pull/28900#issuecomment-648602795 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28900: [SPARK-32056][SQL] Coalesce partitions for repartition by expressions when AQE is enabled
AmplabJenkins commented on pull request #28900: URL: https://github.com/apache/spark/pull/28900#issuecomment-648602795 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #28900: [SPARK-32056][SQL] Coalesce partitions for repartition by expressions when AQE is enabled
viirya commented on a change in pull request #28900:
URL: https://github.com/apache/spark/pull/28900#discussion_r444656437
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
##
@@ -1026,13 +1026,57 @@ class AdaptiveQueryExecSuite
Seq(true, false).foreach { enableAQE =>
withSQLConf(
SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> enableAQE.toString,
+SQLConf.COALESCE_PARTITIONS_ENABLED.key -> "true",
SQLConf.SHUFFLE_PARTITIONS.key -> "6",
SQLConf.COALESCE_PARTITIONS_INITIAL_PARTITION_NUM.key -> "7") {
-val partitionsNum =
spark.range(10).repartition($"id").rdd.collectPartitions().length
+val df1 = spark.range(10).repartition($"id")
+val df2 = spark.range(10).repartition(10, $"id")
+val df3 = spark.range(10).repartition(10)
+
+val partitionsNum1 = df1.rdd.collectPartitions().length
+if (enableAQE) {
+ assert(partitionsNum1 < 6)
+
+ val plan = df1.queryExecution.executedPlan
+ assert(plan.isInstanceOf[AdaptiveSparkPlanExec])
+ val shuffle =
plan.asInstanceOf[AdaptiveSparkPlanExec].executedPlan.collect {
+case s: ShuffleExchangeExec => s
+ }
+ assert(shuffle.size == 1)
+ assert(shuffle(0).outputPartitioning.numPartitions == 7)
+} else {
+ assert(partitionsNum1 === 6)
+}
+
+assert(df2.rdd.collectPartitions().length == 10)
+assert(df3.rdd.collectPartitions().length == 10)
+ }
+}
+ }
+
+ test("SPARK-32056 coalesce partitions for repartition by expressions when
AQE is enabled") {
Review comment:
Separated to two tests.
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
##
@@ -1026,13 +1026,57 @@ class AdaptiveQueryExecSuite
Seq(true, false).foreach { enableAQE =>
withSQLConf(
SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> enableAQE.toString,
+SQLConf.COALESCE_PARTITIONS_ENABLED.key -> "true",
SQLConf.SHUFFLE_PARTITIONS.key -> "6",
SQLConf.COALESCE_PARTITIONS_INITIAL_PARTITION_NUM.key -> "7") {
-val partitionsNum =
spark.range(10).repartition($"id").rdd.collectPartitions().length
+val df1 = spark.range(10).repartition($"id")
+val df2 = spark.range(10).repartition(10, $"id")
+val df3 = spark.range(10).repartition(10)
Review comment:
Added it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28900: [SPARK-32056][SQL] Coalesce partitions for repartition by expressions when AQE is enabled
SparkQA commented on pull request #28900: URL: https://github.com/apache/spark/pull/28900#issuecomment-648602410 **[Test build #124461 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124461/testReport)** for PR 28900 at commit [`7ceaebc`](https://github.com/apache/spark/commit/7ceaebcdff8c149a76106d789d0270205592ca68). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
AmplabJenkins commented on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648600562 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
AmplabJenkins removed a comment on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648600562 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
SparkQA commented on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648600152 **[Test build #124460 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124460/testReport)** for PR 28909 at commit [`741b7a5`](https://github.com/apache/spark/commit/741b7a527ca95fdff4e48d69bb3b960348d3dbab). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ulysses-you edited a comment on pull request #28899: [SPARK-32062][SQL] Reset listenerRegistered in SparkSession
ulysses-you edited a comment on pull request #28899: URL: https://github.com/apache/spark/pull/28899#issuecomment-648591904 @cloud-fan thanks for merging ! BTW do we need a backport for 3.0.1 ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #28899: [SPARK-32062][SQL] Reset listenerRegistered in SparkSession
ulysses-you commented on pull request #28899: URL: https://github.com/apache/spark/pull/28899#issuecomment-648591904 @cloud-fan thanks for merging ! BTW do we need to a backport for 3.0.1 ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28804: [SPARK-31973][SQL] Add ability to disable Sort,Spill in Partial aggregation
AmplabJenkins removed a comment on pull request #28804: URL: https://github.com/apache/spark/pull/28804#issuecomment-648590893 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124450/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28804: [SPARK-31973][SQL] Add ability to disable Sort,Spill in Partial aggregation
AmplabJenkins removed a comment on pull request #28804: URL: https://github.com/apache/spark/pull/28804#issuecomment-648590889 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #28900: [SPARK-32056][SQL] Coalesce partitions for repartition by expressions when AQE is enabled
ulysses-you commented on pull request #28900: URL: https://github.com/apache/spark/pull/28900#issuecomment-648591126 > Seems currently the COALESCE hint doesn't allow default partition number usage. I'm not sure the reason about it. I mean the repartition, such as this sql `select /*+ repartition(col) */ * from test`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28804: [SPARK-31973][SQL] Add ability to disable Sort,Spill in Partial aggregation
AmplabJenkins commented on pull request #28804: URL: https://github.com/apache/spark/pull/28804#issuecomment-648590889 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28804: [SPARK-31973][SQL] Add ability to disable Sort,Spill in Partial aggregation
SparkQA commented on pull request #28804: URL: https://github.com/apache/spark/pull/28804#issuecomment-648590750 **[Test build #124450 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124450/testReport)** for PR 28804 at commit [`99c1d22`](https://github.com/apache/spark/commit/99c1d2226d170f789dfa534ffc658f7fc430c38d). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28804: [SPARK-31973][SQL] Add ability to disable Sort,Spill in Partial aggregation
SparkQA removed a comment on pull request #28804: URL: https://github.com/apache/spark/pull/28804#issuecomment-648552423 **[Test build #124450 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124450/testReport)** for PR 28804 at commit [`99c1d22`](https://github.com/apache/spark/commit/99c1d2226d170f789dfa534ffc658f7fc430c38d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28900: [SPARK-32056][SQL] Coalesce partitions for repartition by expressions when AQE is enabled
cloud-fan commented on a change in pull request #28900:
URL: https://github.com/apache/spark/pull/28900#discussion_r444647502
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
##
@@ -1026,13 +1026,57 @@ class AdaptiveQueryExecSuite
Seq(true, false).foreach { enableAQE =>
withSQLConf(
SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> enableAQE.toString,
+SQLConf.COALESCE_PARTITIONS_ENABLED.key -> "true",
SQLConf.SHUFFLE_PARTITIONS.key -> "6",
SQLConf.COALESCE_PARTITIONS_INITIAL_PARTITION_NUM.key -> "7") {
-val partitionsNum =
spark.range(10).repartition($"id").rdd.collectPartitions().length
+val df1 = spark.range(10).repartition($"id")
+val df2 = spark.range(10).repartition(10, $"id")
+val df3 = spark.range(10).repartition(10)
+
+val partitionsNum1 = df1.rdd.collectPartitions().length
+if (enableAQE) {
+ assert(partitionsNum1 < 6)
+
+ val plan = df1.queryExecution.executedPlan
+ assert(plan.isInstanceOf[AdaptiveSparkPlanExec])
+ val shuffle =
plan.asInstanceOf[AdaptiveSparkPlanExec].executedPlan.collect {
+case s: ShuffleExchangeExec => s
+ }
+ assert(shuffle.size == 1)
+ assert(shuffle(0).outputPartitioning.numPartitions == 7)
+} else {
+ assert(partitionsNum1 === 6)
+}
+
+assert(df2.rdd.collectPartitions().length == 10)
+assert(df3.rdd.collectPartitions().length == 10)
+ }
+}
+ }
+
+ test("SPARK-32056 coalesce partitions for repartition by expressions when
AQE is enabled") {
Review comment:
Can we have a clear separation of the tests? e.g. one for `repartition`
and one for `repartitionByRange`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28900: [SPARK-32056][SQL] Coalesce partitions for repartition by expressions when AQE is enabled
cloud-fan commented on a change in pull request #28900:
URL: https://github.com/apache/spark/pull/28900#discussion_r444646225
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
##
@@ -1026,13 +1026,57 @@ class AdaptiveQueryExecSuite
Seq(true, false).foreach { enableAQE =>
withSQLConf(
SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> enableAQE.toString,
+SQLConf.COALESCE_PARTITIONS_ENABLED.key -> "true",
SQLConf.SHUFFLE_PARTITIONS.key -> "6",
SQLConf.COALESCE_PARTITIONS_INITIAL_PARTITION_NUM.key -> "7") {
-val partitionsNum =
spark.range(10).repartition($"id").rdd.collectPartitions().length
+val df1 = spark.range(10).repartition($"id")
+val df2 = spark.range(10).repartition(10, $"id")
+val df3 = spark.range(10).repartition(10)
Review comment:
`repartitionByRange` also takes `numPartitions`. Can we test it as well
and check it doesn't coalesce?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
AmplabJenkins removed a comment on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648586209 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124459/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
SparkQA removed a comment on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648585829 **[Test build #124459 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124459/testReport)** for PR 28909 at commit [`1d7cfaf`](https://github.com/apache/spark/commit/1d7cfaf30f6bacea34352a04ff4f153e4e0f7ae5). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
AmplabJenkins removed a comment on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648586202 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #28905: [SPARK-32071][SQL] Benchmark `make_interval`
MaxGekk commented on pull request #28905: URL: https://github.com/apache/spark/pull/28905#issuecomment-648586569 @cloud-fan I will re-generate results tomorrow. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
SparkQA commented on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648586197 **[Test build #124459 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124459/testReport)** for PR 28909 at commit [`1d7cfaf`](https://github.com/apache/spark/commit/1d7cfaf30f6bacea34352a04ff4f153e4e0f7ae5). * This patch **fails build dependency tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
AmplabJenkins commented on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648586202 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
SparkQA commented on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648585829 **[Test build #124459 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124459/testReport)** for PR 28909 at commit [`1d7cfaf`](https://github.com/apache/spark/commit/1d7cfaf30f6bacea34352a04ff4f153e4e0f7ae5). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #28900: [SPARK-32056][SQL] Coalesce partitions for repartition by expressions when AQE is enabled
viirya commented on a change in pull request #28900:
URL: https://github.com/apache/spark/pull/28900#discussion_r444643475
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
##
@@ -1026,13 +1026,55 @@ class AdaptiveQueryExecSuite
Seq(true, false).foreach { enableAQE =>
withSQLConf(
SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> enableAQE.toString,
+SQLConf.COALESCE_PARTITIONS_ENABLED.key -> "true",
SQLConf.SHUFFLE_PARTITIONS.key -> "6",
SQLConf.COALESCE_PARTITIONS_INITIAL_PARTITION_NUM.key -> "7") {
-val partitionsNum =
spark.range(10).repartition($"id").rdd.collectPartitions().length
+val df1 = spark.range(10).repartition($"id")
+val df2 = spark.range(10).repartition(10, $"id")
+val df3 = spark.range(10).repartition(10)
+
+val partitionsNum1 = df1.rdd.collectPartitions().length
+if (enableAQE) {
+ assert(partitionsNum1 < 6)
+
+ val plan = df1.queryExecution.executedPlan
+ assert(plan.isInstanceOf[AdaptiveSparkPlanExec])
+ val shuffle =
plan.asInstanceOf[AdaptiveSparkPlanExec].executedPlan.collect {
+case s: ShuffleExchangeExec => s
+ }
+ assert(shuffle.size == 1)
+ assert(shuffle(0).outputPartitioning.numPartitions == 7)
+} else {
+ assert(partitionsNum1 === 6)
+}
+
+val partitionsNum2 = df2.rdd.collectPartitions().length
+assert(partitionsNum2 == 10)
+
+val partitionsNum3 = df3.rdd.collectPartitions().length
+assert(partitionsNum3 == 10)
+ }
+}
+ }
+
+ test("SPARK-32056 coalesce partitions for repartition by expressions when
AQE is enabled") {
+Seq(true, false).foreach { enableAQE =>
+ withSQLConf(
+SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> enableAQE.toString,
+SQLConf.COALESCE_PARTITIONS_ENABLED.key -> "true",
+SQLConf.COALESCE_PARTITIONS_INITIAL_PARTITION_NUM.key -> "50",
+SQLConf.SHUFFLE_PARTITIONS.key -> "10") {
+val partitionsNum1 = (1 to 10).toDF.repartition($"value")
Review comment:
added.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
AmplabJenkins removed a comment on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648584008 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
AmplabJenkins commented on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648584008 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #28899: [SPARK-32062][SQL] Reset listenerRegistered in SparkSession
cloud-fan commented on pull request #28899: URL: https://github.com/apache/spark/pull/28899#issuecomment-648582994 thanks, merging to master! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #28899: [SPARK-32062][SQL] Reset listenerRegistered in SparkSession
cloud-fan closed pull request #28899: URL: https://github.com/apache/spark/pull/28899 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #28833: [SPARK-20680][SQL] Make null type in Spark sql to be compatible with Hive void datatype
cloud-fan commented on pull request #28833: URL: https://github.com/apache/spark/pull/28833#issuecomment-648582548 Can we have another PR to forbid creating tables with void type? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan edited a comment on pull request #28833: [SPARK-20680][SQL] Make null type in Spark sql to be compatible with Hive void datatype
cloud-fan edited a comment on pull request #28833: URL: https://github.com/apache/spark/pull/28833#issuecomment-648582548 Can we have another PR to forbid creating tables with void type, via an analyzer rule? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
AmplabJenkins removed a comment on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648581937 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124458/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
AmplabJenkins commented on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648581934 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
AmplabJenkins removed a comment on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648581934 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on pull request #28900: [SPARK-32056][SQL] Coalesce partitions for repartition by expressions when AQE is enabled
viirya commented on pull request #28900: URL: https://github.com/apache/spark/pull/28900#issuecomment-648582061 > Can we add the feature in `ResolveCoalesceHints` ? Hint can call repartition with default shuffle number. Do you mean like `SELECT /*+ COALESCE() */ ...` ? When no partition number is not specified, let it be default partition number and AQE can coalesce it if enabled? Seems currently the `COALESCE` hint doesn't allow default partition number usage. I'm not sure the reason about it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
SparkQA removed a comment on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648581651 **[Test build #124458 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124458/testReport)** for PR 28909 at commit [`1a6c3b0`](https://github.com/apache/spark/commit/1a6c3b000c26300dfd58b7e7af84ff0f4b5311c1). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
SparkQA commented on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648581923 **[Test build #124458 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124458/testReport)** for PR 28909 at commit [`1a6c3b0`](https://github.com/apache/spark/commit/1a6c3b000c26300dfd58b7e7af84ff0f4b5311c1). * This patch **fails build dependency tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
SparkQA commented on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648581651 **[Test build #124458 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124458/testReport)** for PR 28909 at commit [`1a6c3b0`](https://github.com/apache/spark/commit/1a6c3b000c26300dfd58b7e7af84ff0f4b5311c1). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #28905: [SPARK-32071][SQL] Benchmark `make_interval`
cloud-fan commented on pull request #28905: URL: https://github.com/apache/spark/pull/28905#issuecomment-648581509 do we need to regenerate the result as https://github.com/apache/spark/pull/28906 is merged? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #28906: [SPARK-32072][CORE][TESTS] Fix table formatting with benchmark results
cloud-fan commented on pull request #28906: URL: https://github.com/apache/spark/pull/28906#issuecomment-648581305 thanks, merging to master! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #28906: [SPARK-32072][CORE][TESTS] Fix table formatting with benchmark results
cloud-fan closed pull request #28906: URL: https://github.com/apache/spark/pull/28906 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28906: [SPARK-32072][CORE][TESTS] Fix table formatting with benchmark results
cloud-fan commented on a change in pull request #28906: URL: https://github.com/apache/spark/pull/28906#discussion_r444640751 ## File path: sql/core/benchmarks/MakeDateTimeBenchmark-results.txt ## @@ -2,21 +2,21 @@ OpenJDK 64-Bit Server VM 1.8.0_252-8u252-b09-1~18.04-b09 on Linux 4.15.0-1063-aw Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz make_date(): Best Time(ms) Avg Time(ms) Stdev(ms)Rate(M/s) Per Row(ns) Relative -prepare make_date()2951 3182 355 33.9 29.5 1.0X -make_date(2019, 9, 16) 2325 2415 101 43.0 23.2 1.3X -make_date(*, *, *) 4556 4573 17 21.9 45.6 0.6X +prepare make_date()3309 3429 110 30.2 33.1 1.0X Review comment: I think so, as the head is written by the benchmark framework automatically. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #28910: [SPARK-32075][DOCS]Fix a few issues in parameters table
HyukjinKwon closed pull request #28910: URL: https://github.com/apache/spark/pull/28910 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28910: [SPARK-32075][DOCS]Fix a few issues in parameters table
HyukjinKwon commented on pull request #28910: URL: https://github.com/apache/spark/pull/28910#issuecomment-648580552 Merged to master and branch-3.0. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
AmplabJenkins removed a comment on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648580130 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
AmplabJenkins commented on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648580130 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28903: [SPARK-19939] [ML] Add support for association rules in ML
AmplabJenkins removed a comment on pull request #28903: URL: https://github.com/apache/spark/pull/28903#issuecomment-648579531 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28903: [SPARK-19939] [ML] Add support for association rules in ML
SparkQA removed a comment on pull request #28903: URL: https://github.com/apache/spark/pull/28903#issuecomment-648556577 **[Test build #124452 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124452/testReport)** for PR 28903 at commit [`c80ec10`](https://github.com/apache/spark/commit/c80ec101079b038457d6ffb39001275c9878826b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28903: [SPARK-19939] [ML] Add support for association rules in ML
AmplabJenkins commented on pull request #28903: URL: https://github.com/apache/spark/pull/28903#issuecomment-648579531 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28903: [SPARK-19939] [ML] Add support for association rules in ML
SparkQA commented on pull request #28903: URL: https://github.com/apache/spark/pull/28903#issuecomment-648579225 **[Test build #124452 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124452/testReport)** for PR 28903 at commit [`c80ec10`](https://github.com/apache/spark/commit/c80ec101079b038457d6ffb39001275c9878826b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
AmplabJenkins removed a comment on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648577676 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124457/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
AmplabJenkins commented on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648577907 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
SparkQA removed a comment on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648577580 **[Test build #124457 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124457/testReport)** for PR 28909 at commit [`22f2481`](https://github.com/apache/spark/commit/22f24819f8595c06fb558861337208af01fb453c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
AmplabJenkins removed a comment on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648577672 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
SparkQA commented on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648577580 **[Test build #124457 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124457/testReport)** for PR 28909 at commit [`22f2481`](https://github.com/apache/spark/commit/22f24819f8595c06fb558861337208af01fb453c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
AmplabJenkins commented on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648577672 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
SparkQA commented on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648577661 **[Test build #124457 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124457/testReport)** for PR 28909 at commit [`22f2481`](https://github.com/apache/spark/commit/22f24819f8595c06fb558861337208af01fb453c). * This patch **fails some tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28911: [SPARK-32077][CORE] Support host-local shuffle data reading when external shuffle service is disabled
AmplabJenkins removed a comment on pull request #28911: URL: https://github.com/apache/spark/pull/28911#issuecomment-648574962 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124454/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28911: [SPARK-32077][CORE] Support host-local shuffle data reading when external shuffle service is disabled
AmplabJenkins commented on pull request #28911: URL: https://github.com/apache/spark/pull/28911#issuecomment-648574961 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28911: [SPARK-32077][CORE] Support host-local shuffle data reading when external shuffle service is disabled
SparkQA removed a comment on pull request #28911: URL: https://github.com/apache/spark/pull/28911#issuecomment-648571518 **[Test build #124454 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124454/testReport)** for PR 28911 at commit [`0d62ccb`](https://github.com/apache/spark/commit/0d62ccb24e326c2ace1d7b185e0bf18591a7099e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28911: [SPARK-32077][CORE] Support host-local shuffle data reading when external shuffle service is disabled
AmplabJenkins removed a comment on pull request #28911: URL: https://github.com/apache/spark/pull/28911#issuecomment-648574961 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28911: [SPARK-32077][CORE] Support host-local shuffle data reading when external shuffle service is disabled
SparkQA commented on pull request #28911: URL: https://github.com/apache/spark/pull/28911#issuecomment-648574824 **[Test build #124454 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124454/testReport)** for PR 28911 at commit [`0d62ccb`](https://github.com/apache/spark/commit/0d62ccb24e326c2ace1d7b185e0bf18591a7099e). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
AmplabJenkins removed a comment on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648573904 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
AmplabJenkins commented on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648573899 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
SparkQA commented on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648573577 **[Test build #124456 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124456/testReport)** for PR 28909 at commit [`5b1929a`](https://github.com/apache/spark/commit/5b1929ac2d24f9bb6f58df0a545d3691cd3f4e48). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
AmplabJenkins removed a comment on pull request #28647: URL: https://github.com/apache/spark/pull/28647#issuecomment-648572028 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124441/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
AmplabJenkins removed a comment on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648572273 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
AmplabJenkins commented on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648572273 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
AmplabJenkins removed a comment on pull request #28647: URL: https://github.com/apache/spark/pull/28647#issuecomment-648572023 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
AmplabJenkins commented on pull request #28647: URL: https://github.com/apache/spark/pull/28647#issuecomment-648572023 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
SparkQA removed a comment on pull request #28647: URL: https://github.com/apache/spark/pull/28647#issuecomment-648521944 **[Test build #124441 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124441/testReport)** for PR 28647 at commit [`78ff34f`](https://github.com/apache/spark/commit/78ff34f57a8756717a30bbe279fecfee26abe296). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28911: [SPARK-32077][CORE] Support host-local shuffle data reading when external shuffle service is disabled
AmplabJenkins commented on pull request #28911: URL: https://github.com/apache/spark/pull/28911#issuecomment-648571878 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
AmplabJenkins removed a comment on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648571517 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124455/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28911: [SPARK-32077][CORE] Support host-local shuffle data reading when external shuffle service is disabled
AmplabJenkins removed a comment on pull request #28911: URL: https://github.com/apache/spark/pull/28911#issuecomment-648571878 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
SparkQA removed a comment on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648571488 **[Test build #124455 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124455/testReport)** for PR 28909 at commit [`fa4729b`](https://github.com/apache/spark/commit/fa4729ba7dc3db0abd57d0e1e24bdea3560746dc). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
AmplabJenkins removed a comment on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648571513 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] rajatahujaatinmobi commented on a change in pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
rajatahujaatinmobi commented on a change in pull request #28880:
URL: https://github.com/apache/spark/pull/28880#discussion_r444633578
##
File path:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##
@@ -211,9 +211,11 @@ private[spark] class ApplicationMaster(
final def run(): Int = {
try {
val attemptID = if (isClusterMode) {
-// Set the web ui port to be ephemeral for yarn so we don't conflict
with
-// other spark processes running on the same box
-System.setProperty(UI_PORT.key, "0")
+// Set the web ui port to be ephemeral for yarn if not set explicitly
+// so we don't conflict with other spark processes running on the same
box
+if (System.getProperty(UI_PORT.key) != null) {
+ System.setProperty(UI_PORT.key, "0")
+}
Review comment:
I blocked ports from **18000 to 18050 s**o now spark.ui.port can not get
port b/w that range. But in Spark properties, I mentioned **spark.ui.port as
18018 and spark.port.maxRetries=60** so now spark service will start trying
from 18018 port until **18018 + 60** in an increment way but ports are blocked
until 18050 so spark.ui.port will pick up **18051** port.
##
File path:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##
@@ -211,9 +211,11 @@ private[spark] class ApplicationMaster(
final def run(): Int = {
try {
val attemptID = if (isClusterMode) {
-// Set the web ui port to be ephemeral for yarn so we don't conflict
with
-// other spark processes running on the same box
-System.setProperty(UI_PORT.key, "0")
+// Set the web ui port to be ephemeral for yarn if not set explicitly
+// so we don't conflict with other spark processes running on the same
box
+if (System.getProperty(UI_PORT.key) != null) {
+ System.setProperty(UI_PORT.key, "0")
+}
Review comment:
I blocked ports from **18000 to 18050 s**o now spark.ui.port can not get
port b/w that range. But in Spark properties, I mentioned **spark.ui.port as
18018 and spark.port.maxRetries=60** so now spark service will start trying
from 18018 port until **18018 + 60** in an increment way but ports are blocked
until 18050 so spark.ui.port will pick up **18051** port. @Ngone51
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
SparkQA commented on pull request #28647: URL: https://github.com/apache/spark/pull/28647#issuecomment-648571620 **[Test build #124441 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124441/testReport)** for PR 28647 at commit [`78ff34f`](https://github.com/apache/spark/commit/78ff34f57a8756717a30bbe279fecfee26abe296). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
SparkQA commented on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648571488 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
AmplabJenkins commented on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648571513 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28911: [SPARK-32077][CORE] Support host-local shuffle data reading when external shuffle service is disabled
SparkQA commented on pull request #28911: URL: https://github.com/apache/spark/pull/28911#issuecomment-648571518 **[Test build #124454 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124454/testReport)** for PR 28911 at commit [`0d62ccb`](https://github.com/apache/spark/commit/0d62ccb24e326c2ace1d7b185e0bf18591a7099e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] rajatahujaatinmobi commented on a change in pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly
rajatahujaatinmobi commented on a change in pull request #28880:
URL: https://github.com/apache/spark/pull/28880#discussion_r444552932
##
File path:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##
@@ -211,9 +211,11 @@ private[spark] class ApplicationMaster(
final def run(): Int = {
try {
val attemptID = if (isClusterMode) {
-// Set the web ui port to be ephemeral for yarn so we don't conflict
with
-// other spark processes running on the same box
-System.setProperty(UI_PORT.key, "0")
+// Set the web ui port to be ephemeral for yarn if not set explicitly
+// so we don't conflict with other spark processes running on the same
box
+if (System.getProperty(UI_PORT.key) != null) {
+ System.setProperty(UI_PORT.key, "0")
+}
Review comment:
I blocked a set of Port range from 18000 to 18050 and configure web or
to start at 18008 but it eventually Started at 18051
**Properties Set: spark.ui.port=18018 spark.port.maxRetries=60**
Successful Case:
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18018. Attempting port 18019.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18019. Attempting port 18020.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18020. Attempting port 18021.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18021. Attempting port 18022.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18022. Attempting port 18023.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18023. Attempting port 18024.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18024. Attempting port 18025.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18025. Attempting port 18026.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18026. Attempting port 18027.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18027. Attempting port 18028.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18028. Attempting port 18029.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18029. Attempting port 18030.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18030. Attempting port 18031.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18031. Attempting port 18032.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18032. Attempting port 18033.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18033. Attempting port 18034.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18034. Attempting port 18035.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18035. Attempting port 18036.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18036. Attempting port 18037.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18037. Attempting port 18038.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18038. Attempting port 18039.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18039. Attempting port 18040.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18040. Attempting port 18041.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18041. Attempting port 18042.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18042. Attempting port 18043.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18043. Attempting port 18044.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18044. Attempting port 18045.
20/06/23 22:18:44 WARN util.Utils: Service 'SparkUI' could not bind on port
18045. Attempting port 18046.
20/06/23 22:18:44 WARN util.Utils: Service 'SparkUI' could not bind on port
18046. Attempting port 18047.
20/06/23 22:18:44 WARN util.Utils: Service 'SparkUI' could not bind on port
18047. Attempting port 18048.
20/06/23 22:18:44 WARN util.Utils: Service 'SparkUI' could not bind on port
18048. Attempting port 18049.
20/06/23 22:18:44 WARN util.Utils: Service 'SparkUI' could not bind on port
18049. Attempting port 18050.
20/06/23 22:18:44 WARN util.Utils: Service 'SparkUI' could not bind on port
18050. Attempting port 18051.
20/06/23 22:18:44 INFO server.AbstractConnector: Started
ServerConnector@6f330eb9{HTTP/1.1,[http/1.1]}{0.0.0.0:18051}
20/06/23 22:18:44 INFO util.Utils: Successfully started service 'SparkUI' on
port 18051.
And Job succeeded**
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
AmplabJenkins removed a comment on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648570005 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124453/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
AmplabJenkins removed a comment on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648570002 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
AmplabJenkins commented on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648569867 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #28840: [SPARK-31999][SQL] Add REFRESH FUNCTION command
ulysses-you commented on pull request #28840: URL: https://github.com/apache/spark/pull/28840#issuecomment-648570103 So now we should make `ResolvedFunc` like `ResolvedView` that only hold a `Identifier`. After we have function api in v2 catalog, `ResolvedFunc` should hold the catalog info and function info. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
SparkQA removed a comment on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648569562 **[Test build #124453 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124453/testReport)** for PR 28909 at commit [`4b0a539`](https://github.com/apache/spark/commit/4b0a53998532db2768bd6e899621599a779b5eff). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
AmplabJenkins removed a comment on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648569867 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
SparkQA commented on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648569982 **[Test build #124453 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124453/testReport)** for PR 28909 at commit [`4b0a539`](https://github.com/apache/spark/commit/4b0a53998532db2768bd6e899621599a779b5eff). * This patch **fails some tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
AmplabJenkins commented on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648570002 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #28911: [SPARK-32077][CORE] Support host-local shuffle data reading when external shuffle service is disabled
Ngone51 commented on pull request #28911: URL: https://github.com/apache/spark/pull/28911#issuecomment-648569632 Jenkins, retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28909: [WIP][SPARK-32074][BUILD][R] Update AppVeyor R version to 4.0.2
SparkQA commented on pull request #28909: URL: https://github.com/apache/spark/pull/28909#issuecomment-648569562 **[Test build #124453 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124453/testReport)** for PR 28909 at commit [`4b0a539`](https://github.com/apache/spark/commit/4b0a53998532db2768bd6e899621599a779b5eff). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #28911: [SPARK-32077][CORE] Support host-local shuffle data reading when external shuffle service is disabled
Ngone51 commented on pull request #28911: URL: https://github.com/apache/spark/pull/28911#issuecomment-648567102 @tgravescs updated the description, thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #28840: [SPARK-31999][SQL] Add REFRESH FUNCTION command
ulysses-you commented on a change in pull request #28840:
URL: https://github.com/apache/spark/pull/28840#discussion_r444628868
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala
##
@@ -236,6 +236,59 @@ case class ShowFunctionsCommand(
}
}
+
+/**
+ * A command for users to refresh the persistent function.
+ * The syntax of using this command in SQL is:
+ * {{{
+ *REFRESH FUNCTION functionName
+ * }}}
+ */
+case class RefreshFunctionCommand(
+databaseName: Option[String],
+functionName: String)
+ extends RunnableCommand {
+
+ override def run(sparkSession: SparkSession): Seq[Row] = {
+val catalog = sparkSession.sessionState.catalog
+if
(FunctionRegistry.builtin.functionExists(FunctionIdentifier(functionName))) {
+ throw new AnalysisException(s"Cannot refresh builtin function
$functionName")
+}
+if (catalog.isTemporaryFunction(FunctionIdentifier(functionName,
databaseName))) {
+ throw new AnalysisException(s"Cannot refresh temporary function
$functionName")
+}
+
+// we only refresh the permanent function.
+// there are 4 cases:
+// 1. registry exists externalCatalog exists
+// 2. registry exists externalCatalog not exists
+// 3. registry not exists externalCatalog exists
+// 4. registry not exists externalCatalog not exists
+val identifier = FunctionIdentifier(
+ functionName, Some(databaseName.getOrElse(catalog.getCurrentDatabase)))
+val isRegisteredFunction = catalog.isRegisteredFunction(identifier)
+val isPersistentFunction = catalog.isPersistentFunction(identifier)
+if (isRegisteredFunction && isPersistentFunction) {
+ // re-register function
+ catalog.unregisterFunction(identifier)
+ val func = catalog.getFunctionMetadata(identifier)
+ catalog.registerFunction(func, true)
+} else if (isRegisteredFunction && !isPersistentFunction) {
+ // unregister function and throw NoSuchFunctionException
+ catalog.unregisterFunction(identifier)
+ throw new NoSuchFunctionException(identifier.database.get, functionName)
+} else if (!isRegisteredFunction && isPersistentFunction) {
+ // register function
+ val func = catalog.getFunctionMetadata(identifier)
+ catalog.registerFunction(func, true)
Review comment:
register function is light. I think it's ok to cache the function right
away instead lazy.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #28840: [SPARK-31999][SQL] Add REFRESH FUNCTION command
ulysses-you commented on a change in pull request #28840:
URL: https://github.com/apache/spark/pull/28840#discussion_r444628279
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala
##
@@ -236,6 +236,59 @@ case class ShowFunctionsCommand(
}
}
+
+/**
+ * A command for users to refresh the persistent function.
+ * The syntax of using this command in SQL is:
+ * {{{
+ *REFRESH FUNCTION functionName
+ * }}}
+ */
+case class RefreshFunctionCommand(
+databaseName: Option[String],
+functionName: String)
+ extends RunnableCommand {
+
+ override def run(sparkSession: SparkSession): Seq[Row] = {
+val catalog = sparkSession.sessionState.catalog
+if
(FunctionRegistry.builtin.functionExists(FunctionIdentifier(functionName))) {
+ throw new AnalysisException(s"Cannot refresh builtin function
$functionName")
+}
+if (catalog.isTemporaryFunction(FunctionIdentifier(functionName,
databaseName))) {
+ throw new AnalysisException(s"Cannot refresh temporary function
$functionName")
+}
+
+// we only refresh the permanent function.
+// there are 4 cases:
+// 1. registry exists externalCatalog exists
+// 2. registry exists externalCatalog not exists
+// 3. registry not exists externalCatalog exists
+// 4. registry not exists externalCatalog not exists
+val identifier = FunctionIdentifier(
+ functionName, Some(databaseName.getOrElse(catalog.getCurrentDatabase)))
+val isRegisteredFunction = catalog.isRegisteredFunction(identifier)
+val isPersistentFunction = catalog.isPersistentFunction(identifier)
+if (isRegisteredFunction && isPersistentFunction) {
+ // re-register function
+ catalog.unregisterFunction(identifier)
+ val func = catalog.getFunctionMetadata(identifier)
+ catalog.registerFunction(func, true)
+} else if (isRegisteredFunction && !isPersistentFunction) {
+ // unregister function and throw NoSuchFunctionException
+ catalog.unregisterFunction(identifier)
+ throw new NoSuchFunctionException(identifier.database.get, functionName)
Review comment:
unregister is for clean the cache since the function is not exists. This
is the core behavior for `refresh function`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #28900: [SPARK-32056][SQL] Coalesce partitions for repartition by expressions when AQE is enabled
ulysses-you commented on pull request #28900: URL: https://github.com/apache/spark/pull/28900#issuecomment-648564896 Can we add the feature in `ResolveCoalesceHints` ? Hint can call repartition with default shuffle number. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
