[GitHub] [spark] dongjoon-hyun commented on issue #26233: [SPARK-29533][SQL][TESTS][FOLLOWUP] Regenerate the result on EC2
dongjoon-hyun commented on issue #26233: [SPARK-29533][SQL][TESTS][FOLLOWUP] Regenerate the result on EC2 URL: https://github.com/apache/spark/pull/26233#issuecomment-545591973 @MaxGekk , @HyukjinKwon , @cloud-fan This is preparation for https://github.com/apache/spark/pull/26190 . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26232: [SPARK-29093][PYTHON][ML] Remove automatically generated param setters in _shared_params_code_gen.py
AmplabJenkins removed a comment on issue #26232: [SPARK-29093][PYTHON][ML] Remove automatically generated param setters in _shared_params_code_gen.py URL: https://github.com/apache/spark/pull/26232#issuecomment-545590513 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112559/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun opened a new pull request #26233: [SPARK-29533][SQL][TESTS][FOLLOWUP] Regenerate the result on EC2
dongjoon-hyun opened a new pull request #26233: [SPARK-29533][SQL][TESTS][FOLLOWUP] Regenerate the result on EC2 URL: https://github.com/apache/spark/pull/26233 ### What changes were proposed in this pull request? This regenerates the result on EC2. ### Why are the changes needed? This will be used for the other PR reviews. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? N/A. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26232: [SPARK-29093][PYTHON][ML] Remove automatically generated param setters in _shared_params_code_gen.py
SparkQA removed a comment on issue #26232: [SPARK-29093][PYTHON][ML] Remove automatically generated param setters in _shared_params_code_gen.py URL: https://github.com/apache/spark/pull/26232#issuecomment-545586632 **[Test build #112559 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112559/testReport)** for PR 26232 at commit [`a67163e`](https://github.com/apache/spark/commit/a67163e285876651dbb7eeb1d0ee66aa30dbce31). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26232: [SPARK-29093][PYTHON][ML] Remove automatically generated param setters in _shared_params_code_gen.py
AmplabJenkins removed a comment on issue #26232: [SPARK-29093][PYTHON][ML] Remove automatically generated param setters in _shared_params_code_gen.py URL: https://github.com/apache/spark/pull/26232#issuecomment-545590503 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26190: [SPARK-29532][SQL] simplify interval string parsing
AmplabJenkins removed a comment on issue #26190: [SPARK-29532][SQL] simplify interval string parsing URL: https://github.com/apache/spark/pull/26190#issuecomment-545590627 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #26170: [SPARK-29397][core] Extend plugin interface to include the driver.
tgravescs commented on a change in pull request #26170: [SPARK-29397][core]
Extend plugin interface to include the driver.
URL: https://github.com/apache/spark/pull/26170#discussion_r338226990
##
File path:
core/src/main/scala/org/apache/spark/internal/plugin/PluginContainer.scala
##
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.internal.plugin
+
+import scala.collection.JavaConverters._
+import scala.util.{Either, Left, Right}
+
+import org.apache.spark.{SparkContext, SparkEnv}
+import org.apache.spark.api.plugin._
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config._
+import org.apache.spark.util.Utils
+
+sealed abstract class PluginContainer {
+
+ def shutdown(): Unit
+
+}
+
+private class DriverPluginContainer(sc: SparkContext, plugins:
Seq[SparkPlugin])
+ extends PluginContainer with Logging {
+
+ private val driverPlugins: Seq[(String, DriverPlugin)] = plugins.flatMap { p
=>
+val driverPlugin = p.driverPlugin()
+if (driverPlugin != null) {
+ val name = p.getClass().getName()
+ val ctx = new PluginContextImpl(name, sc.env.rpcEnv,
sc.env.metricsSystem, sc.conf,
+sc.env.executorId)
+
+ val extraConf = driverPlugin.init(sc, ctx)
+ if (extraConf != null) {
+extraConf.asScala.foreach { case (k, v) =>
+ sc.conf.set(s"${PluginContainer.EXTRA_CONF_PREFIX}$name.$k", v)
+}
+ }
+ ctx.registerMetrics()
+ logInfo(s"Initialized driver component for plugin $name.")
+ Some(p.getClass().getName() -> driverPlugin)
+} else {
+ None
+}
+ }
+
+ if (driverPlugins.nonEmpty) {
+sc.env.rpcEnv.setupEndpoint(classOf[PluginEndpoint].getName(),
+ new PluginEndpoint(driverPlugins.toMap, sc.env.rpcEnv))
+ }
+
+ override def shutdown(): Unit = {
+driverPlugins.foreach { case (name, plugin) =>
+ try {
+logDebug(s"Stopping plugin $name.")
+plugin.shutdown()
+ } catch {
+case t: Throwable =>
+ logInfo(s"Exception while shutting down plugin $name.", t)
+ }
+}
+ }
+
+}
+
+private class ExecutorPluginContainer(env: SparkEnv, plugins: Seq[SparkPlugin])
+ extends PluginContainer with Logging {
+
+ private val executorPlugins: Seq[(String, ExecutorPlugin)] = {
+val allExtraConf =
env.conf.getAllWithPrefix(PluginContainer.EXTRA_CONF_PREFIX)
+
+plugins.flatMap { p =>
+ val executorPlugin = p.executorPlugin()
+ if (executorPlugin != null) {
+val name = p.getClass().getName()
+val prefix = name + "."
+val extraConf = allExtraConf
+ .filter { case (k, v) => k.startsWith(prefix) }
+ .map { case (k, v) => k.substring(prefix.length()) -> v }
+ .toMap
+ .asJava
+val ctx = new PluginContextImpl(name, env.rpcEnv, env.metricsSystem,
env.conf,
+ env.executorId)
+executorPlugin.init(ctx, extraConf)
+ctx.registerMetrics()
+
+logInfo(s"Initialized executor component for plugin $name.")
+Some(p.getClass().getName() -> executorPlugin)
+ } else {
+None
+ }
+}
+ }
+
+ override def shutdown(): Unit = {
+executorPlugins.foreach { case (name, plugin) =>
+ try {
+logDebug(s"Stopping plugin $name.")
+plugin.shutdown()
+ } catch {
+case t: Throwable =>
+ logInfo(s"Exception while shutting down plugin $name.", t)
+ }
+}
+ }
+}
+
+object PluginContainer {
+
+ val EXTRA_CONF_PREFIX = "spark.plugins.__internal_conf__."
Review comment:
sorry I was thinking just spark.plugins.internal.conf. We have the
internal() option to config builder so figured it kind of matched. I don't
have a super strong opinion on this as long as we try to keep it consistent. I
know we use __xxx__ for various internal things - files directories - but
didn't think we had any for configs. thoughts?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this
[GitHub] [spark] AmplabJenkins removed a comment on issue #26190: [SPARK-29532][SQL] simplify interval string parsing
AmplabJenkins removed a comment on issue #26190: [SPARK-29532][SQL] simplify interval string parsing URL: https://github.com/apache/spark/pull/26190#issuecomment-545590640 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112552/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26190: [SPARK-29532][SQL] simplify interval string parsing
AmplabJenkins commented on issue #26190: [SPARK-29532][SQL] simplify interval string parsing URL: https://github.com/apache/spark/pull/26190#issuecomment-545590640 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112552/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26232: [SPARK-29093][PYTHON][ML] Remove automatically generated param setters in _shared_params_code_gen.py
AmplabJenkins commented on issue #26232: [SPARK-29093][PYTHON][ML] Remove automatically generated param setters in _shared_params_code_gen.py URL: https://github.com/apache/spark/pull/26232#issuecomment-545590513 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112559/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26189: [SPARK-29533][SQL][TEST] Benchmark casting strings to intervals
dongjoon-hyun commented on issue #26189: [SPARK-29533][SQL][TEST] Benchmark casting strings to intervals URL: https://github.com/apache/spark/pull/26189#issuecomment-545590584 Hi, All. This was good but I'll make a follow up to regenerate on EC2 in order to compare with the other PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26190: [SPARK-29532][SQL] simplify interval string parsing
AmplabJenkins commented on issue #26190: [SPARK-29532][SQL] simplify interval string parsing URL: https://github.com/apache/spark/pull/26190#issuecomment-545590627 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26232: [SPARK-29093][PYTHON][ML] Remove automatically generated param setters in _shared_params_code_gen.py
SparkQA commented on issue #26232: [SPARK-29093][PYTHON][ML] Remove automatically generated param setters in _shared_params_code_gen.py URL: https://github.com/apache/spark/pull/26232#issuecomment-545590481 **[Test build #112559 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112559/testReport)** for PR 26232 at commit [`a67163e`](https://github.com/apache/spark/commit/a67163e285876651dbb7eeb1d0ee66aa30dbce31). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26232: [SPARK-29093][PYTHON][ML] Remove automatically generated param setters in _shared_params_code_gen.py
AmplabJenkins commented on issue #26232: [SPARK-29093][PYTHON][ML] Remove automatically generated param setters in _shared_params_code_gen.py URL: https://github.com/apache/spark/pull/26232#issuecomment-545590503 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26190: [SPARK-29532][SQL] simplify interval string parsing
SparkQA commented on issue #26190: [SPARK-29532][SQL] simplify interval string parsing URL: https://github.com/apache/spark/pull/26190#issuecomment-545589890 **[Test build #112552 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112552/testReport)** for PR 26190 at commit [`33ceedc`](https://github.com/apache/spark/commit/33ceedcedeafbe11070630ef3f47db37b5599ed0). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26190: [SPARK-29532][SQL] simplify interval string parsing
SparkQA removed a comment on issue #26190: [SPARK-29532][SQL] simplify interval string parsing URL: https://github.com/apache/spark/pull/26190#issuecomment-545526541 **[Test build #112552 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112552/testReport)** for PR 26190 at commit [`33ceedc`](https://github.com/apache/spark/commit/33ceedcedeafbe11070630ef3f47db37b5599ed0). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] rdblue commented on a change in pull request #25955: [SPARK-29277][SQL] Add early DSv2 filter and projection pushdown
rdblue commented on a change in pull request #25955: [SPARK-29277][SQL] Add
early DSv2 filter and projection pushdown
URL: https://github.com/apache/spark/pull/25955#discussion_r338225063
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
##
@@ -243,17 +247,36 @@ class FindDataSourceTable(sparkSession: SparkSession)
extends Rule[LogicalPlan]
override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
case i @ InsertIntoStatement(UnresolvedCatalogRelation(tableMeta), _, _,
_, _)
if DDLUtils.isDatasourceTable(tableMeta) =>
- i.copy(table = readDataSourceTable(tableMeta))
+ if (DataSource.isV2Provider(tableMeta.provider.get,
sparkSession.sessionState.conf)) {
Review comment:
Fair enough, I'll remove the refactor.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26232: [SPARK-29093][PYTHON][ML] Remove automatically generated param setters in _shared_params_code_gen.py
AmplabJenkins removed a comment on issue #26232: [SPARK-29093][PYTHON][ML] Remove automatically generated param setters in _shared_params_code_gen.py URL: https://github.com/apache/spark/pull/26232#issuecomment-545587231 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26078: [SPARK-29151][CORE] Support fractional resources for task resource scheduling
AmplabJenkins removed a comment on issue #26078: [SPARK-29151][CORE] Support fractional resources for task resource scheduling URL: https://github.com/apache/spark/pull/26078#issuecomment-545587162 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17525/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26078: [SPARK-29151][CORE] Support fractional resources for task resource scheduling
AmplabJenkins removed a comment on issue #26078: [SPARK-29151][CORE] Support fractional resources for task resource scheduling URL: https://github.com/apache/spark/pull/26078#issuecomment-545587159 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26232: [SPARK-29093][PYTHON][ML] Remove automatically generated param setters in _shared_params_code_gen.py
AmplabJenkins removed a comment on issue #26232: [SPARK-29093][PYTHON][ML] Remove automatically generated param setters in _shared_params_code_gen.py URL: https://github.com/apache/spark/pull/26232#issuecomment-545587242 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17524/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26232: [SPARK-29093][PYTHON][ML] Remove automatically generated param setters in _shared_params_code_gen.py
AmplabJenkins commented on issue #26232: [SPARK-29093][PYTHON][ML] Remove automatically generated param setters in _shared_params_code_gen.py URL: https://github.com/apache/spark/pull/26232#issuecomment-545587242 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17524/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26078: [SPARK-29151][CORE] Support fractional resources for task resource scheduling
AmplabJenkins commented on issue #26078: [SPARK-29151][CORE] Support fractional resources for task resource scheduling URL: https://github.com/apache/spark/pull/26078#issuecomment-545587162 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17525/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26078: [SPARK-29151][CORE] Support fractional resources for task resource scheduling
AmplabJenkins commented on issue #26078: [SPARK-29151][CORE] Support fractional resources for task resource scheduling URL: https://github.com/apache/spark/pull/26078#issuecomment-545587159 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26232: [SPARK-29093][PYTHON][ML] Remove automatically generated param setters in _shared_params_code_gen.py
AmplabJenkins commented on issue #26232: [SPARK-29093][PYTHON][ML] Remove automatically generated param setters in _shared_params_code_gen.py URL: https://github.com/apache/spark/pull/26232#issuecomment-545587231 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26078: [SPARK-29151][CORE] Support fractional resources for task resource scheduling
SparkQA commented on issue #26078: [SPARK-29151][CORE] Support fractional resources for task resource scheduling URL: https://github.com/apache/spark/pull/26078#issuecomment-545586648 **[Test build #112560 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112560/testReport)** for PR 26078 at commit [`f3ece52`](https://github.com/apache/spark/commit/f3ece52ece91bafdac556fc617a57182812a3361). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26232: [SPARK-29093][PYTHON][ML] Remove automatically generated param setters in _shared_params_code_gen.py
SparkQA commented on issue #26232: [SPARK-29093][PYTHON][ML] Remove automatically generated param setters in _shared_params_code_gen.py URL: https://github.com/apache/spark/pull/26232#issuecomment-545586632 **[Test build #112559 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112559/testReport)** for PR 26232 at commit [`a67163e`](https://github.com/apache/spark/commit/a67163e285876651dbb7eeb1d0ee66aa30dbce31). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao opened a new pull request #26232: [SPARK-29093][PYTHON][ML] Remove automatically generated param setters in _shared_params_code_gen.py
huaxingao opened a new pull request #26232: [SPARK-29093][PYTHON][ML] Remove automatically generated param setters in _shared_params_code_gen.py URL: https://github.com/apache/spark/pull/26232 ### What changes were proposed in this pull request? Remove automatically generated param setters in _shared_params_code_gen.py ### Why are the changes needed? To keep parity between scala and python ### Does this PR introduce any user-facing change? Yes Add some setters in Python ML XXXModels ### How was this patch tested? unit tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] tgravescs commented on issue #26078: [SPARK-29151][CORE] Support fractional resources for task resource scheduling
tgravescs commented on issue #26078: [SPARK-29151][CORE] Support fractional resources for task resource scheduling URL: https://github.com/apache/spark/pull/26078#issuecomment-545584181 test this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] tgravescs commented on issue #26078: [SPARK-29151][CORE] Support fractional resources for task resource scheduling
tgravescs commented on issue #26078: [SPARK-29151][CORE] Support fractional resources for task resource scheduling URL: https://github.com/apache/spark/pull/26078#issuecomment-545584129 Jenkins, add to whitelist This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26197: Implement p-value simulation and unit tests for chi2 test
AmplabJenkins removed a comment on issue #26197: Implement p-value simulation and unit tests for chi2 test URL: https://github.com/apache/spark/pull/26197#issuecomment-545581673 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112558/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver.
AmplabJenkins removed a comment on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver. URL: https://github.com/apache/spark/pull/26170#issuecomment-545581692 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112551/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver.
AmplabJenkins removed a comment on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver. URL: https://github.com/apache/spark/pull/26170#issuecomment-545581684 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26197: Implement p-value simulation and unit tests for chi2 test
AmplabJenkins commented on issue #26197: Implement p-value simulation and unit tests for chi2 test URL: https://github.com/apache/spark/pull/26197#issuecomment-545581658 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26197: Implement p-value simulation and unit tests for chi2 test
SparkQA removed a comment on issue #26197: Implement p-value simulation and unit tests for chi2 test URL: https://github.com/apache/spark/pull/26197#issuecomment-545580592 **[Test build #112558 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112558/testReport)** for PR 26197 at commit [`da1c6fa`](https://github.com/apache/spark/commit/da1c6fae18aa19d073fae5afc184a14912a50729). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26197: Implement p-value simulation and unit tests for chi2 test
AmplabJenkins removed a comment on issue #26197: Implement p-value simulation and unit tests for chi2 test URL: https://github.com/apache/spark/pull/26197#issuecomment-545581658 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26197: Implement p-value simulation and unit tests for chi2 test
SparkQA commented on issue #26197: Implement p-value simulation and unit tests for chi2 test URL: https://github.com/apache/spark/pull/26197#issuecomment-545581645 **[Test build #112558 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112558/testReport)** for PR 26197 at commit [`da1c6fa`](https://github.com/apache/spark/commit/da1c6fae18aa19d073fae5afc184a14912a50729). * This patch **fails build dependency tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class ChiSqTestSuite extends SparkFunSuite with MLlibTestSparkContext with Logging ` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver.
AmplabJenkins commented on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver. URL: https://github.com/apache/spark/pull/26170#issuecomment-545581684 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26197: Implement p-value simulation and unit tests for chi2 test
AmplabJenkins commented on issue #26197: Implement p-value simulation and unit tests for chi2 test URL: https://github.com/apache/spark/pull/26197#issuecomment-545581673 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112558/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver.
AmplabJenkins commented on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver. URL: https://github.com/apache/spark/pull/26170#issuecomment-545581692 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112551/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26206: [SPARK-29551][CORE] Fix a bug about fetch failed when an executor is lost
dongjoon-hyun commented on a change in pull request #26206: [SPARK-29551][CORE]
Fix a bug about fetch failed when an executor is lost
URL: https://github.com/apache/spark/pull/26206#discussion_r338215381
##
File path:
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
##
@@ -537,6 +536,169 @@ class DAGSchedulerSuite extends SparkFunSuite with
LocalSparkContext with TimeLi
assert(mapStatus2(2).location.host === "hostB")
}
+ test("All shuffle files on the executor should be cleaned up when executor
lost " +
+"and then causes 'fetch failed'") {
+// whether to kill Executor or not before FetchFailed
+Seq(true, false).foreach { killExecutor => {
+ afterEach()
+ val conf = new SparkConf()
+ conf.set(config.SHUFFLE_SERVICE_ENABLED.key, "true")
+ conf.set("spark.files.fetchFailure.unRegisterOutputOnHost", "false")
+ init(conf)
+ runEvent(ExecutorAdded("exec-hostA1", "hostA"))
+ runEvent(ExecutorAdded("exec-hostA2", "hostA"))
+ runEvent(ExecutorAdded("exec-hostB", "hostB"))
+ val firstRDD = new MyRDD(sc, 3, Nil)
+ val firstShuffleDep = new ShuffleDependency(firstRDD, new
HashPartitioner(3))
+ val firstShuffleId = firstShuffleDep.shuffleId
+ val shuffleMapRdd = new MyRDD(sc, 3, List(firstShuffleDep))
+ val shuffleDep = new ShuffleDependency(shuffleMapRdd, new
HashPartitioner(3))
+ val secondShuffleId = shuffleDep.shuffleId
+ val reduceRdd = new MyRDD(sc, 1, List(shuffleDep))
+
+ submit(reduceRdd, Array(0))
+ // map stage1 completes successfully, with one task on each executor
+ complete(taskSets(0), Seq(
+(Success,
+ MapStatus(BlockManagerId("exec-hostA1", "hostA", 12345),
+Array.fill[Long](1)(2), mapTaskId = 5)),
+(Success,
+ MapStatus(BlockManagerId("exec-hostA2", "hostA", 12345),
+Array.fill[Long](1)(2), mapTaskId = 6)),
+(Success, makeMapStatus("hostB", 1, mapTaskId = 7))
+ ))
+ // map stage2 completes successfully, with one task on each executor
+ complete(taskSets(1), Seq(
+(Success,
+ MapStatus(BlockManagerId("exec-hostA1", "hostA", 12345),
+Array.fill[Long](1)(2), mapTaskId = 8)),
+(Success,
+ MapStatus(BlockManagerId("exec-hostA2", "hostA", 12345),
+Array.fill[Long](1)(2), mapTaskId = 9)),
+(Success, makeMapStatus("hostB", 1))
+ ))
+ // make sure our test setup is correct
+ val initialMapStatus1 =
mapOutputTracker.shuffleStatuses(firstShuffleId).mapStatuses
+ // val initialMapStatus1 = mapOutputTracker.mapStatuses.get(0).get
+ assert(initialMapStatus1.count(_ != null) === 3)
+ assert(initialMapStatus1.map {
+_.location.executorId
+ }.toSet ===
+Set("exec-hostA1", "exec-hostA2", "exec-hostB"))
+ val initialMapStatus2 =
mapOutputTracker.shuffleStatuses(secondShuffleId).mapStatuses
+ assert(initialMapStatus2.count(_ != null) === 3)
+ assert(initialMapStatus2.map {
+_.location.executorId
+ }.toSet ===
+Set("exec-hostA1", "exec-hostA2", "exec-hostB"))
+ // kill exec-hostA2
+ if (killExecutor) {
+runEvent(ExecutorLost("exec-hostA2", ExecutorKilled))
+ }
+ // reduce stage fails with a fetch failure from one host
+ complete(taskSets(2), Seq(
+(FetchFailed(BlockManagerId("exec-hostA2", "hostA", 12345),
+ secondShuffleId, 0L, 0, 0, "ignored"), null)
+ ))
+ // Here is the main assertion -- make sure that we de-register
+ // the map outputs for exec-hostA2
+ val mapStatus1 =
mapOutputTracker.shuffleStatuses(firstShuffleId).mapStatuses
+ assert(mapStatus1.count(_ != null) === 2)
+ assert(mapStatus1(0).location.executorId === "exec-hostA1")
+ assert(mapStatus1(0).location.host === "hostA")
+ assert(mapStatus1(2).location.executorId === "exec-hostB")
+ assert(mapStatus1(2).location.host === "hostB")
+
+ val mapStatus2 =
mapOutputTracker.shuffleStatuses(secondShuffleId).mapStatuses
+ assert(mapStatus2.count(_ != null) === 2)
+ assert(mapStatus2(0).location.executorId === "exec-hostA1")
+ assert(mapStatus2(0).location.host === "hostA")
+ assert(mapStatus2(2).location.executorId === "exec-hostB")
+ assert(mapStatus2(2).location.host === "hostB")
+}
+}
+ }
+
+ test("All shuffle files on the host should be cleaned up when host lost") {
Review comment:
The same comments for this test case (test case name, use config key instead
of string literal, indentation)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
[GitHub] [spark] SparkQA commented on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver.
SparkQA commented on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver. URL: https://github.com/apache/spark/pull/26170#issuecomment-545580952 **[Test build #112551 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112551/testReport)** for PR 26170 at commit [`5dd6afd`](https://github.com/apache/spark/commit/5dd6afdcbf373a1064d92b7fc5a658d3a11b66fc). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver.
SparkQA removed a comment on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver. URL: https://github.com/apache/spark/pull/26170#issuecomment-545519977 **[Test build #112551 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112551/testReport)** for PR 26170 at commit [`5dd6afd`](https://github.com/apache/spark/commit/5dd6afdcbf373a1064d92b7fc5a658d3a11b66fc). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26197: Implement p-value simulation and unit tests for chi2 test
SparkQA commented on issue #26197: Implement p-value simulation and unit tests for chi2 test URL: https://github.com/apache/spark/pull/26197#issuecomment-545580592 **[Test build #112558 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112558/testReport)** for PR 26197 at commit [`da1c6fa`](https://github.com/apache/spark/commit/da1c6fae18aa19d073fae5afc184a14912a50729). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26206: [SPARK-29551][CORE] Fix a bug about fetch failed when an executor is lost
dongjoon-hyun commented on a change in pull request #26206: [SPARK-29551][CORE]
Fix a bug about fetch failed when an executor is lost
URL: https://github.com/apache/spark/pull/26206#discussion_r338214733
##
File path:
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
##
@@ -537,6 +536,169 @@ class DAGSchedulerSuite extends SparkFunSuite with
LocalSparkContext with TimeLi
assert(mapStatus2(2).location.host === "hostB")
}
+ test("All shuffle files on the executor should be cleaned up when executor
lost " +
+"and then causes 'fetch failed'") {
+// whether to kill Executor or not before FetchFailed
+Seq(true, false).foreach { killExecutor => {
+ afterEach()
+ val conf = new SparkConf()
+ conf.set(config.SHUFFLE_SERVICE_ENABLED.key, "true")
+ conf.set("spark.files.fetchFailure.unRegisterOutputOnHost", "false")
+ init(conf)
+ runEvent(ExecutorAdded("exec-hostA1", "hostA"))
+ runEvent(ExecutorAdded("exec-hostA2", "hostA"))
+ runEvent(ExecutorAdded("exec-hostB", "hostB"))
+ val firstRDD = new MyRDD(sc, 3, Nil)
+ val firstShuffleDep = new ShuffleDependency(firstRDD, new
HashPartitioner(3))
+ val firstShuffleId = firstShuffleDep.shuffleId
+ val shuffleMapRdd = new MyRDD(sc, 3, List(firstShuffleDep))
+ val shuffleDep = new ShuffleDependency(shuffleMapRdd, new
HashPartitioner(3))
+ val secondShuffleId = shuffleDep.shuffleId
+ val reduceRdd = new MyRDD(sc, 1, List(shuffleDep))
+
+ submit(reduceRdd, Array(0))
+ // map stage1 completes successfully, with one task on each executor
+ complete(taskSets(0), Seq(
+(Success,
+ MapStatus(BlockManagerId("exec-hostA1", "hostA", 12345),
+Array.fill[Long](1)(2), mapTaskId = 5)),
+(Success,
+ MapStatus(BlockManagerId("exec-hostA2", "hostA", 12345),
+Array.fill[Long](1)(2), mapTaskId = 6)),
+(Success, makeMapStatus("hostB", 1, mapTaskId = 7))
+ ))
+ // map stage2 completes successfully, with one task on each executor
+ complete(taskSets(1), Seq(
+(Success,
+ MapStatus(BlockManagerId("exec-hostA1", "hostA", 12345),
+Array.fill[Long](1)(2), mapTaskId = 8)),
+(Success,
+ MapStatus(BlockManagerId("exec-hostA2", "hostA", 12345),
+Array.fill[Long](1)(2), mapTaskId = 9)),
+(Success, makeMapStatus("hostB", 1))
+ ))
+ // make sure our test setup is correct
+ val initialMapStatus1 =
mapOutputTracker.shuffleStatuses(firstShuffleId).mapStatuses
+ // val initialMapStatus1 = mapOutputTracker.mapStatuses.get(0).get
+ assert(initialMapStatus1.count(_ != null) === 3)
+ assert(initialMapStatus1.map {
+_.location.executorId
+ }.toSet ===
+Set("exec-hostA1", "exec-hostA2", "exec-hostB"))
+ val initialMapStatus2 =
mapOutputTracker.shuffleStatuses(secondShuffleId).mapStatuses
+ assert(initialMapStatus2.count(_ != null) === 3)
+ assert(initialMapStatus2.map {
+_.location.executorId
+ }.toSet ===
+Set("exec-hostA1", "exec-hostA2", "exec-hostB"))
+ // kill exec-hostA2
+ if (killExecutor) {
+runEvent(ExecutorLost("exec-hostA2", ExecutorKilled))
+ }
+ // reduce stage fails with a fetch failure from one host
+ complete(taskSets(2), Seq(
+(FetchFailed(BlockManagerId("exec-hostA2", "hostA", 12345),
+ secondShuffleId, 0L, 0, 0, "ignored"), null)
+ ))
+ // Here is the main assertion -- make sure that we de-register
+ // the map outputs for exec-hostA2
+ val mapStatus1 =
mapOutputTracker.shuffleStatuses(firstShuffleId).mapStatuses
+ assert(mapStatus1.count(_ != null) === 2)
+ assert(mapStatus1(0).location.executorId === "exec-hostA1")
+ assert(mapStatus1(0).location.host === "hostA")
+ assert(mapStatus1(2).location.executorId === "exec-hostB")
+ assert(mapStatus1(2).location.host === "hostB")
+
+ val mapStatus2 =
mapOutputTracker.shuffleStatuses(secondShuffleId).mapStatuses
+ assert(mapStatus2.count(_ != null) === 2)
+ assert(mapStatus2(0).location.executorId === "exec-hostA1")
+ assert(mapStatus2(0).location.host === "hostA")
+ assert(mapStatus2(2).location.executorId === "exec-hostB")
+ assert(mapStatus2(2).location.host === "hostB")
+}
+}
Review comment:
Indentation seems to be corrupted in some point before this line.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail:
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26206: [SPARK-29551][CORE] Fix a bug about fetch failed when an executor is lost
dongjoon-hyun commented on a change in pull request #26206: [SPARK-29551][CORE]
Fix a bug about fetch failed when an executor is lost
URL: https://github.com/apache/spark/pull/26206#discussion_r338214415
##
File path:
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
##
@@ -537,6 +536,169 @@ class DAGSchedulerSuite extends SparkFunSuite with
LocalSparkContext with TimeLi
assert(mapStatus2(2).location.host === "hostB")
}
+ test("All shuffle files on the executor should be cleaned up when executor
lost " +
+"and then causes 'fetch failed'") {
+// whether to kill Executor or not before FetchFailed
+Seq(true, false).foreach { killExecutor => {
+ afterEach()
+ val conf = new SparkConf()
+ conf.set(config.SHUFFLE_SERVICE_ENABLED.key, "true")
+ conf.set("spark.files.fetchFailure.unRegisterOutputOnHost", "false")
+ init(conf)
+ runEvent(ExecutorAdded("exec-hostA1", "hostA"))
+ runEvent(ExecutorAdded("exec-hostA2", "hostA"))
+ runEvent(ExecutorAdded("exec-hostB", "hostB"))
+ val firstRDD = new MyRDD(sc, 3, Nil)
+ val firstShuffleDep = new ShuffleDependency(firstRDD, new
HashPartitioner(3))
+ val firstShuffleId = firstShuffleDep.shuffleId
+ val shuffleMapRdd = new MyRDD(sc, 3, List(firstShuffleDep))
+ val shuffleDep = new ShuffleDependency(shuffleMapRdd, new
HashPartitioner(3))
+ val secondShuffleId = shuffleDep.shuffleId
+ val reduceRdd = new MyRDD(sc, 1, List(shuffleDep))
+
+ submit(reduceRdd, Array(0))
+ // map stage1 completes successfully, with one task on each executor
+ complete(taskSets(0), Seq(
+(Success,
+ MapStatus(BlockManagerId("exec-hostA1", "hostA", 12345),
+Array.fill[Long](1)(2), mapTaskId = 5)),
+(Success,
+ MapStatus(BlockManagerId("exec-hostA2", "hostA", 12345),
+Array.fill[Long](1)(2), mapTaskId = 6)),
+(Success, makeMapStatus("hostB", 1, mapTaskId = 7))
+ ))
+ // map stage2 completes successfully, with one task on each executor
+ complete(taskSets(1), Seq(
+(Success,
+ MapStatus(BlockManagerId("exec-hostA1", "hostA", 12345),
+Array.fill[Long](1)(2), mapTaskId = 8)),
+(Success,
+ MapStatus(BlockManagerId("exec-hostA2", "hostA", 12345),
+Array.fill[Long](1)(2), mapTaskId = 9)),
+(Success, makeMapStatus("hostB", 1))
+ ))
+ // make sure our test setup is correct
+ val initialMapStatus1 =
mapOutputTracker.shuffleStatuses(firstShuffleId).mapStatuses
+ // val initialMapStatus1 = mapOutputTracker.mapStatuses.get(0).get
Review comment:
Shall we remove this commented code?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26206: [SPARK-29551][CORE] Fix a bug about fetch failed when an executor is lost
dongjoon-hyun commented on a change in pull request #26206: [SPARK-29551][CORE]
Fix a bug about fetch failed when an executor is lost
URL: https://github.com/apache/spark/pull/26206#discussion_r338214048
##
File path:
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
##
@@ -537,6 +536,169 @@ class DAGSchedulerSuite extends SparkFunSuite with
LocalSparkContext with TimeLi
assert(mapStatus2(2).location.host === "hostB")
}
+ test("All shuffle files on the executor should be cleaned up when executor
lost " +
+"and then causes 'fetch failed'") {
+// whether to kill Executor or not before FetchFailed
+Seq(true, false).foreach { killExecutor => {
+ afterEach()
+ val conf = new SparkConf()
+ conf.set(config.SHUFFLE_SERVICE_ENABLED.key, "true")
+ conf.set("spark.files.fetchFailure.unRegisterOutputOnHost", "false")
Review comment:
nit. `"spark.files.fetchFailure.unRegisterOutputOnHost"` ->
`UNREGISTER_OUTPUT_ON_HOST_ON_FETCH_FAILURE.key`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] juliuszsompolski commented on issue #25721: [WIP][SPARK-29018][SQL] Implement Spark Thrift Server with it's own code base on PROTOCOL_VERSION_V9
juliuszsompolski commented on issue #25721: [WIP][SPARK-29018][SQL] Implement Spark Thrift Server with it's own code base on PROTOCOL_VERSION_V9 URL: https://github.com/apache/spark/pull/25721#issuecomment-545579648 > > 2. on top of that, all the Thriftserver code that is just translated from Java to Scala without changes - it has a new package now, so it will remain unused. > Translate all code seems too heavy. And we build it with protocol v11. we can't direct apply to v1.2.1/v2.3.5. > I prefer to build the framework based on protocol v11 such as common class and process like Operation.class, SessionManager.class, then fill in the details I am confused. It seems that in this PR you already did translation from Java to Scala of all Hive Thriftserver code? I assume it was mostly somehow autogenerated? A lot of these classes don't really change much from Hive to Spark, except for mechanical translation from Java to Scala, renaming package names, removing dependence on some Hive objects. Do I see correctly? I would commit these in a separate PR, to separate "mechanical" changes from places that were actually rewritten. I am also not sure whether we should translate those from Java to Scala at all. Maybe we should keep these in Java code, and only implement the Spark specific stuff in scala, removing Java Hive stuff that is not needed anymore. So e.g. - Keep CLIService.java, ThriftCLIService.java, ThriftHttpServlet.java, ... - all things that don't really get modified by Spark in Java - Do Spark specific implementation in scala, and remove the no longer needed Java thriftserver impl. E.g. (Spark)ExecuteStatementOperation.scala and remove ExecuteStatementOperation.java; (Spark)OperationManager.scala and remove OperationManager.java etc. etc. OR Translate all these Java files to scala, like is done in this PR? I think I would keep them in Java to avoid potential errors in translation, and also to see easier if we want to port any future Hive changes to them. @gatorsmile @rxin - what do you think? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26206: [SPARK-29551][CORE] Fix a bug about fetch failed when an executor is lost
dongjoon-hyun commented on a change in pull request #26206: [SPARK-29551][CORE]
Fix a bug about fetch failed when an executor is lost
URL: https://github.com/apache/spark/pull/26206#discussion_r338213188
##
File path:
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
##
@@ -510,7 +510,6 @@ class DAGSchedulerSuite extends SparkFunSuite with
LocalSparkContext with TimeLi
assert(initialMapStatus1.map{_.mapId}.toSet === Set(5, 6, 7))
val initialMapStatus2 =
mapOutputTracker.shuffleStatuses(secondShuffleId).mapStatuses
-// val initialMapStatus1 = mapOutputTracker.mapStatuses.get(0).get
Review comment:
Let's revert this change. This kind clean up is good in general of course,
but this makes the backport difficult.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26206: [SPARK-29551][CORE] Fix a bug about fetch failed when an executor is lost
dongjoon-hyun commented on a change in pull request #26206: [SPARK-29551][CORE]
Fix a bug about fetch failed when an executor is lost
URL: https://github.com/apache/spark/pull/26206#discussion_r338212624
##
File path:
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
##
@@ -537,6 +536,169 @@ class DAGSchedulerSuite extends SparkFunSuite with
LocalSparkContext with TimeLi
assert(mapStatus2(2).location.host === "hostB")
}
+ test("All shuffle files on the executor should be cleaned up when executor
lost " +
Review comment:
For the bug fix PR, we had better add `SPARK-29551` prefix for the test case
name.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26197: Implement p-value simulation and unit tests for chi2 test
AmplabJenkins commented on issue #26197: Implement p-value simulation and unit tests for chi2 test URL: https://github.com/apache/spark/pull/26197#issuecomment-545578161 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26197: Implement p-value simulation and unit tests for chi2 test
AmplabJenkins commented on issue #26197: Implement p-value simulation and unit tests for chi2 test URL: https://github.com/apache/spark/pull/26197#issuecomment-545578173 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17523/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26197: Implement p-value simulation and unit tests for chi2 test
AmplabJenkins removed a comment on issue #26197: Implement p-value simulation and unit tests for chi2 test URL: https://github.com/apache/spark/pull/26197#issuecomment-545578161 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26197: Implement p-value simulation and unit tests for chi2 test
AmplabJenkins removed a comment on issue #26197: Implement p-value simulation and unit tests for chi2 test URL: https://github.com/apache/spark/pull/26197#issuecomment-545578173 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17523/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26206: [SPARK-29551][CORE] Fix a bug about fetch failed when an executor is lost
dongjoon-hyun commented on a change in pull request #26206: [SPARK-29551][CORE] Fix a bug about fetch failed when an executor is lost URL: https://github.com/apache/spark/pull/26206#discussion_r338211766 ## File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ## @@ -175,6 +175,10 @@ private[spark] class DAGScheduler( // TODO: Garbage collect information about failure epochs when we know there are no more // stray messages to detect. private val failedEpoch = new HashMap[String, Long] + // There will be a regression when an executor lost and then causes 'fetch failed'. Review comment: `There will be a regression` is misleading because this mean this PR causes a regression. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] holdenk commented on a change in pull request #26197: Implement p-value simulation and unit tests for chi2 test
holdenk commented on a change in pull request #26197: Implement p-value
simulation and unit tests for chi2 test
URL: https://github.com/apache/spark/pull/26197#discussion_r338180024
##
File path: mllib/pom.xml
##
@@ -130,6 +130,11 @@
org.apache.spark
spark-tags_${scala.binary.version}
+
+ com.tdunning
Review comment:
Generally speaking we try and not pick up new dependencies for small
features, especially those which aren't maintained by a community. Looking at
tdunning's tdigest package there's been ~5 distinct contributors for 2019, and
if we expand the window up to 2017 that goes up to ~7.
Do we know how actively maintined this going to be and if he is going to
maintain the 3.X line once the 4.X release is out?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] holdenk commented on a change in pull request #26197: Implement p-value simulation and unit tests for chi2 test
holdenk commented on a change in pull request #26197: Implement p-value
simulation and unit tests for chi2 test
URL: https://github.com/apache/spark/pull/26197#discussion_r338180477
##
File path:
mllib/src/main/scala/org/apache/spark/mllib/stat/test/ChiSqTest.scala
##
@@ -151,6 +155,8 @@ private[spark] object ChiSqTest extends Logging {
*/
def chiSquared(observed: Vector,
expected: Vector = Vectors.dense(Array.empty[Double]),
+ simulatePValue: Boolean = false,
Review comment:
Spark MLLib is in maintaince mode, see
https://spark.apache.org/docs/latest/ml-guide.html . If we do want this I think
we need to make sure it is exposed in Spark ML.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] atronchi commented on issue #26197: Implement p-value simulation and unit tests for chi2 test
atronchi commented on issue #26197: Implement p-value simulation and unit tests for chi2 test URL: https://github.com/apache/spark/pull/26197#issuecomment-545577841 Last push integrated some offline feedback. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26229: [SPARK-21492][SQL][Follow Up] Reimplement UnsafeExternalRowSorter in database style iterator
AmplabJenkins removed a comment on issue #26229: [SPARK-21492][SQL][Follow Up] Reimplement UnsafeExternalRowSorter in database style iterator URL: https://github.com/apache/spark/pull/26229#issuecomment-545576440 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112547/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26229: [SPARK-21492][SQL][Follow Up] Reimplement UnsafeExternalRowSorter in database style iterator
AmplabJenkins removed a comment on issue #26229: [SPARK-21492][SQL][Follow Up] Reimplement UnsafeExternalRowSorter in database style iterator URL: https://github.com/apache/spark/pull/26229#issuecomment-545576433 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26197: Implement p-value simulation and unit tests for chi2 test
AmplabJenkins removed a comment on issue #26197: Implement p-value simulation and unit tests for chi2 test URL: https://github.com/apache/spark/pull/26197#issuecomment-545575920 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112557/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26229: [SPARK-21492][SQL][Follow Up] Reimplement UnsafeExternalRowSorter in database style iterator
AmplabJenkins commented on issue #26229: [SPARK-21492][SQL][Follow Up] Reimplement UnsafeExternalRowSorter in database style iterator URL: https://github.com/apache/spark/pull/26229#issuecomment-545576440 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112547/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26229: [SPARK-21492][SQL][Follow Up] Reimplement UnsafeExternalRowSorter in database style iterator
AmplabJenkins commented on issue #26229: [SPARK-21492][SQL][Follow Up] Reimplement UnsafeExternalRowSorter in database style iterator URL: https://github.com/apache/spark/pull/26229#issuecomment-545576433 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26197: Implement p-value simulation and unit tests for chi2 test
AmplabJenkins removed a comment on issue #26197: Implement p-value simulation and unit tests for chi2 test URL: https://github.com/apache/spark/pull/26197#issuecomment-545575912 Build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26190: [SPARK-29532][SQL] simplify interval string parsing
dongjoon-hyun commented on a change in pull request #26190: [SPARK-29532][SQL] simplify interval string parsing URL: https://github.com/apache/spark/pull/26190#discussion_r338209548 ## File path: sql/core/benchmarks/IntervalBenchmark-results.txt ## @@ -1,25 +1,25 @@ -Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.15 -Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz +Java HotSpot(TM) 64-Bit Server VM 1.8.0_161-b12 on Mac OS X 10.14 +Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz cast strings to intervals:Best Time(ms) Avg Time(ms) Stdev(ms)Rate(M/s) Per Row(ns) Relative -string w/ interval 420435 18 2.4 419.8 1.0X -string w/o interval 359365 10 2.8 358.7 1.2X -1 units w/ interval 752759 8 1.3 752.0 0.6X -1 units w/o interval762766 4 1.3 762.0 0.6X -2 units w/ interval 961970 8 1.0 960.7 0.4X -2 units w/o interval970976 9 1.0 970.2 0.4X -3 units w/ interval1130 1136 7 0.91130.4 0.4X -3 units w/o interval 1150 1158 9 0.91150.3 0.4X -4 units w/ interval1333 1336 3 0.71333.5 0.3X -4 units w/o interval 1354 1359 4 0.71354.5 0.3X -5 units w/ interval1523 1525 2 0.71523.3 0.3X -5 units w/o interval 1549 1551 3 0.61549.4 0.3X -6 units w/ interval1661 1663 2 0.61660.8 0.3X -6 units w/o interval 1691 1704 13 0.61691.2 0.2X -7 units w/ interval1811 1817 8 0.61810.6 0.2X -7 units w/o interval 1853 1854 1 0.51853.2 0.2X -8 units w/ interval2029 2037 8 0.52028.7 0.2X -8 units w/o interval 2075 2075 1 0.52074.5 0.2X -9 units w/ interval2170 2175 5 0.52170.0 0.2X -9 units w/o interval 2204 2212 8 0.52203.6 0.2X +prepare string w/ interval 403419 18 2.5 403.1 1.0X +prepare string w/o interval 341353 21 2.9 341.1 1.2X +1 units w/ interval5154 5159 8 0.25153.5 0.1X +1 units w/o interval 4818 4833 20 0.24817.6 0.1X +2 units w/ interval6191 6223 41 0.26190.6 0.1X +2 units w/o interval 6236 6264 25 0.26235.7 0.1X +3 units w/ interval7397 7567 170 0.17397.0 0.1X +3 units w/o interval 7280 7367 76 0.17279.6 0.1X +4 units w/ interval8197 8228 27 0.18197.3 0.0X +4 units w/o interval 7977 7989 17 0.17977.3 0.1X +5 units w/ interval9089 9192 101 0.19088.8 0.0X +5 units w/o interval 8853 8858 5 0.18852.8 0.0X +6 units w/ interval9696 9720 23 0.19695.6 0.0X +6 units w/o interval 9509 9518 9 0.19509.4 0.0X +7 units w/ interval 10738
[GitHub] [spark] SparkQA removed a comment on issue #26197: Implement p-value simulation and unit tests for chi2 test
SparkQA removed a comment on issue #26197: Implement p-value simulation and unit tests for chi2 test URL: https://github.com/apache/spark/pull/26197#issuecomment-545574388 **[Test build #112557 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112557/testReport)** for PR 26197 at commit [`34852e6`](https://github.com/apache/spark/commit/34852e62ecafb3573d4622ffabaab61fdcd253c0). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26229: [SPARK-21492][SQL][Follow Up] Reimplement UnsafeExternalRowSorter in database style iterator
SparkQA commented on issue #26229: [SPARK-21492][SQL][Follow Up] Reimplement UnsafeExternalRowSorter in database style iterator URL: https://github.com/apache/spark/pull/26229#issuecomment-545575754 **[Test build #112547 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112547/testReport)** for PR 26229 at commit [`5623c7a`](https://github.com/apache/spark/commit/5623c7a5a042827bd277853df0dd9e84e8ca1a9c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26197: Implement p-value simulation and unit tests for chi2 test
SparkQA commented on issue #26197: Implement p-value simulation and unit tests for chi2 test URL: https://github.com/apache/spark/pull/26197#issuecomment-545575899 **[Test build #112557 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112557/testReport)** for PR 26197 at commit [`34852e6`](https://github.com/apache/spark/commit/34852e62ecafb3573d4622ffabaab61fdcd253c0). * This patch **fails build dependency tests**. * This patch **does not merge cleanly**. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26229: [SPARK-21492][SQL][Follow Up] Reimplement UnsafeExternalRowSorter in database style iterator
SparkQA removed a comment on issue #26229: [SPARK-21492][SQL][Follow Up] Reimplement UnsafeExternalRowSorter in database style iterator URL: https://github.com/apache/spark/pull/26229#issuecomment-545474776 **[Test build #112547 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112547/testReport)** for PR 26229 at commit [`5623c7a`](https://github.com/apache/spark/commit/5623c7a5a042827bd277853df0dd9e84e8ca1a9c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26197: Implement p-value simulation and unit tests for chi2 test
AmplabJenkins commented on issue #26197: Implement p-value simulation and unit tests for chi2 test URL: https://github.com/apache/spark/pull/26197#issuecomment-545575912 Build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26197: Implement p-value simulation and unit tests for chi2 test
AmplabJenkins commented on issue #26197: Implement p-value simulation and unit tests for chi2 test URL: https://github.com/apache/spark/pull/26197#issuecomment-545575920 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112557/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26197: Implement p-value simulation and unit tests for chi2 test
AmplabJenkins removed a comment on issue #26197: Implement p-value simulation and unit tests for chi2 test URL: https://github.com/apache/spark/pull/26197#issuecomment-545574921 Build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26197: Implement p-value simulation and unit tests for chi2 test
AmplabJenkins commented on issue #26197: Implement p-value simulation and unit tests for chi2 test URL: https://github.com/apache/spark/pull/26197#issuecomment-545574921 Build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26197: Implement p-value simulation and unit tests for chi2 test
AmplabJenkins commented on issue #26197: Implement p-value simulation and unit tests for chi2 test URL: https://github.com/apache/spark/pull/26197#issuecomment-545574930 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17522/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26197: Implement p-value simulation and unit tests for chi2 test
AmplabJenkins removed a comment on issue #26197: Implement p-value simulation and unit tests for chi2 test URL: https://github.com/apache/spark/pull/26197#issuecomment-545574930 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17522/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26197: Implement p-value simulation and unit tests for chi2 test
SparkQA commented on issue #26197: Implement p-value simulation and unit tests for chi2 test URL: https://github.com/apache/spark/pull/26197#issuecomment-545574388 **[Test build #112557 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112557/testReport)** for PR 26197 at commit [`34852e6`](https://github.com/apache/spark/commit/34852e62ecafb3573d4622ffabaab61fdcd253c0). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26190: [SPARK-29532][SQL] simplify interval string parsing
dongjoon-hyun commented on a change in pull request #26190: [SPARK-29532][SQL] simplify interval string parsing URL: https://github.com/apache/spark/pull/26190#discussion_r338206250 ## File path: sql/core/benchmarks/IntervalBenchmark-results.txt ## @@ -1,25 +1,25 @@ -Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.15 -Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz +Java HotSpot(TM) 64-Bit Server VM 1.8.0_161-b12 on Mac OS X 10.14 +Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz cast strings to intervals:Best Time(ms) Avg Time(ms) Stdev(ms)Rate(M/s) Per Row(ns) Relative -string w/ interval 420435 18 2.4 419.8 1.0X -string w/o interval 359365 10 2.8 358.7 1.2X -1 units w/ interval 752759 8 1.3 752.0 0.6X -1 units w/o interval762766 4 1.3 762.0 0.6X -2 units w/ interval 961970 8 1.0 960.7 0.4X -2 units w/o interval970976 9 1.0 970.2 0.4X -3 units w/ interval1130 1136 7 0.91130.4 0.4X -3 units w/o interval 1150 1158 9 0.91150.3 0.4X -4 units w/ interval1333 1336 3 0.71333.5 0.3X -4 units w/o interval 1354 1359 4 0.71354.5 0.3X -5 units w/ interval1523 1525 2 0.71523.3 0.3X -5 units w/o interval 1549 1551 3 0.61549.4 0.3X -6 units w/ interval1661 1663 2 0.61660.8 0.3X -6 units w/o interval 1691 1704 13 0.61691.2 0.2X -7 units w/ interval1811 1817 8 0.61810.6 0.2X -7 units w/o interval 1853 1854 1 0.51853.2 0.2X -8 units w/ interval2029 2037 8 0.52028.7 0.2X -8 units w/o interval 2075 2075 1 0.52074.5 0.2X -9 units w/ interval2170 2175 5 0.52170.0 0.2X -9 units w/o interval 2204 2212 8 0.52203.6 0.2X +prepare string w/ interval 403419 18 2.5 403.1 1.0X +prepare string w/o interval 341353 21 2.9 341.1 1.2X +1 units w/ interval5154 5159 8 0.25153.5 0.1X +1 units w/o interval 4818 4833 20 0.24817.6 0.1X +2 units w/ interval6191 6223 41 0.26190.6 0.1X +2 units w/o interval 6236 6264 25 0.26235.7 0.1X +3 units w/ interval7397 7567 170 0.17397.0 0.1X +3 units w/o interval 7280 7367 76 0.17279.6 0.1X +4 units w/ interval8197 8228 27 0.18197.3 0.0X +4 units w/o interval 7977 7989 17 0.17977.3 0.1X +5 units w/ interval9089 9192 101 0.19088.8 0.0X +5 units w/o interval 8853 8858 5 0.18852.8 0.0X +6 units w/ interval9696 9720 23 0.19695.6 0.0X +6 units w/o interval 9509 9518 9 0.19509.4 0.0X +7 units w/ interval 10738
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26190: [SPARK-29532][SQL] simplify interval string parsing
dongjoon-hyun commented on a change in pull request #26190: [SPARK-29532][SQL] simplify interval string parsing URL: https://github.com/apache/spark/pull/26190#discussion_r338205376 ## File path: sql/core/benchmarks/IntervalBenchmark-results.txt ## @@ -1,25 +1,25 @@ -Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.15 -Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz Review comment: Oh, the original benchmark result is on MacOS. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on issue #26222: [SPARK-29504][WebUI] Toggle full job description on click
gengliangwang commented on issue #26222: [SPARK-29504][WebUI] Toggle full job description on click URL: https://github.com/apache/spark/pull/26222#issuecomment-545569595 @PavithraRamachandran I have updated the PR title and description for you. If my modification is OK to you, I will merge this one. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25641: [SPARK-28921][BUILD][K8S][2.4] Update kubernetes client to 4.4.2
dongjoon-hyun commented on issue #25641: [SPARK-28921][BUILD][K8S][2.4] Update kubernetes client to 4.4.2 URL: https://github.com/apache/spark/pull/25641#issuecomment-545566271 Until now, there is no plan. However, according to https://spark.apache.org/news, we can guess like the following. - 2.4.3: May 8, 2019 - 2.4.4: September 1, 2019 - 2.4.5: Jan 2020? (This is reasonable. At least, I can volunteer for the release manager on January.) 3.0.0 RC1 also have the similar ETA (https://spark.apache.org/versioning-policy.html). I believe you will have 3.0.0 and 2.4.5 together at least early 2020. BTW, `K8s` dev cycle is fast. Although Apache Spark 2.4.5 will get the latest version of `K8s client` at that time instead of `4.6.1`, that is just a best effort like `2.4.4`. For the missed things, Apache Spark 2.4.6 will catch up later. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] brkyvz commented on a change in pull request #25955: [SPARK-29277][SQL] Add early DSv2 filter and projection pushdown
brkyvz commented on a change in pull request #25955: [SPARK-29277][SQL] Add
early DSv2 filter and projection pushdown
URL: https://github.com/apache/spark/pull/25955#discussion_r338196463
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##
@@ -672,19 +672,23 @@ class Analyzer(
*/
object ResolveTables extends Rule[LogicalPlan] {
def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsUp {
- case u: UnresolvedRelation =>
-lookupV2Relation(u.multipartIdentifier)
- .getOrElse(u)
+ case u: UnresolvedV2Relation =>
+CatalogV2Util.loadRelation(u.catalog, u.tableName).getOrElse(u)
- case i @ InsertIntoStatement(u: UnresolvedRelation, _, _, _, _) if
i.query.resolved =>
-lookupV2Relation(u.multipartIdentifier)
+ case i @ InsertIntoStatement(u: UnresolvedV2Relation, _, _, _, _) if
i.query.resolved =>
+CatalogV2Util.loadRelation(u.catalog, u.tableName)
.map(v2Relation => i.copy(table = v2Relation))
.getOrElse(i)
- case u: UnresolvedV2Relation =>
-CatalogV2Util.loadTable(u.catalog, u.tableName).map { table =>
- DataSourceV2Relation.create(table)
-}.getOrElse(u)
+ case desc @ DescribeTable(u: UnresolvedV2Relation, _) =>
Review comment:
Thanks for incorporating these!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] brkyvz commented on issue #26214: [SPARK-29558][SQL] ResolveTables and ResolveRelations should be order-insensitive
brkyvz commented on issue #26214: [SPARK-29558][SQL] ResolveTables and ResolveRelations should be order-insensitive URL: https://github.com/apache/spark/pull/26214#issuecomment-545564432 While code unification is nice to have, I think we've been purposefully trying to keep v1 code paths and v2 code paths separate, to make it a lot easier in the future to potentially delete the v1 parts. I think this change could make that a bit harder? What do you think? If we're worried about ordering of rules in the Analyzer, we can ensure that `ResolveRelations` always calls `ResolveTables` first, by just calling the `ResolveTables.apply` method within `ResolveRelations`, and remove `ResolveTables` from the resolution rules in the Analyzer. But it would still maintain that clean separation. Do you think that's possible? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on issue #26222: [SPARK-29504][WebUI]show full decription when .click on ellipse
gengliangwang commented on issue #26222: [SPARK-29504][WebUI]show full decription when .click on ellipse URL: https://github.com/apache/spark/pull/26222#issuecomment-545564250 @PavithraRamachandran Sorry my browser loaded the cached script. I try `cmd + shift + R` on my MacBook and it works. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26231: [SPARK-29572][SQL] add v1 read fallback API in DS v2
AmplabJenkins removed a comment on issue #26231: [SPARK-29572][SQL] add v1 read fallback API in DS v2 URL: https://github.com/apache/spark/pull/26231#issuecomment-545562829 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112548/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26231: [SPARK-29572][SQL] add v1 read fallback API in DS v2
AmplabJenkins removed a comment on issue #26231: [SPARK-29572][SQL] add v1 read fallback API in DS v2 URL: https://github.com/apache/spark/pull/26231#issuecomment-545562818 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26231: [SPARK-29572][SQL] add v1 read fallback API in DS v2
AmplabJenkins commented on issue #26231: [SPARK-29572][SQL] add v1 read fallback API in DS v2 URL: https://github.com/apache/spark/pull/26231#issuecomment-545562818 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] brkyvz commented on a change in pull request #26214: [SPARK-29558][SQL] ResolveTables and ResolveRelations should be order-insensitive
brkyvz commented on a change in pull request #26214: [SPARK-29558][SQL]
ResolveTables and ResolveRelations should be order-insensitive
URL: https://github.com/apache/spark/pull/26214#discussion_r338193572
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##
@@ -720,13 +712,9 @@ class Analyzer(
// Note this is compatible with the views defined by older versions of
Spark(before 2.2), which
// have empty defaultDatabase and all the relations in viewText have
database part defined.
def resolveRelation(plan: LogicalPlan): LogicalPlan = plan match {
- case u @ UnresolvedRelation(AsTemporaryViewIdentifier(ident))
Review comment:
Wouldn't it be simpler to just call `ResolveTables.apply(plan) match {` here
than to embed all the logic within `lookupTableFromCatalog`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26231: [SPARK-29572][SQL] add v1 read fallback API in DS v2
SparkQA removed a comment on issue #26231: [SPARK-29572][SQL] add v1 read fallback API in DS v2 URL: https://github.com/apache/spark/pull/26231#issuecomment-545490379 **[Test build #112548 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112548/testReport)** for PR 26231 at commit [`034bc07`](https://github.com/apache/spark/commit/034bc07e34914c3ac9b70f34f233f0c562dfed6e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26231: [SPARK-29572][SQL] add v1 read fallback API in DS v2
SparkQA commented on issue #26231: [SPARK-29572][SQL] add v1 read fallback API in DS v2 URL: https://github.com/apache/spark/pull/26231#issuecomment-545562504 **[Test build #112548 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112548/testReport)** for PR 26231 at commit [`034bc07`](https://github.com/apache/spark/commit/034bc07e34914c3ac9b70f34f233f0c562dfed6e). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `trait V1Scan extends Scan ` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26231: [SPARK-29572][SQL] add v1 read fallback API in DS v2
AmplabJenkins commented on issue #26231: [SPARK-29572][SQL] add v1 read fallback API in DS v2 URL: https://github.com/apache/spark/pull/26231#issuecomment-545562829 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112548/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shivsood commented on issue #25344: [SPARK-28151][SQL] Mapped ByteType to TinyINT for MsSQLServerDialect
shivsood commented on issue #25344: [SPARK-28151][SQL] Mapped ByteType to TinyINT for MsSQLServerDialect URL: https://github.com/apache/spark/pull/25344#issuecomment-545562768 > @shivsood ping, are you there? @maropu Yes, Would handle this this week. Please note that ShortType issue is fixed SPARK-28152. Or are you pointing to a different issue? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #17280: [SPARK-19939] [ML] Add support for association rules in ML
AmplabJenkins removed a comment on issue #17280: [SPARK-19939] [ML] Add support for association rules in ML URL: https://github.com/apache/spark/pull/17280#issuecomment-545561963 Build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #17280: [SPARK-19939] [ML] Add support for association rules in ML
AmplabJenkins removed a comment on issue #17280: [SPARK-19939] [ML] Add support for association rules in ML URL: https://github.com/apache/spark/pull/17280#issuecomment-545561972 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17521/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #17280: [SPARK-19939] [ML] Add support for association rules in ML
AmplabJenkins commented on issue #17280: [SPARK-19939] [ML] Add support for association rules in ML URL: https://github.com/apache/spark/pull/17280#issuecomment-545561972 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17521/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #17280: [SPARK-19939] [ML] Add support for association rules in ML
AmplabJenkins commented on issue #17280: [SPARK-19939] [ML] Add support for association rules in ML URL: https://github.com/apache/spark/pull/17280#issuecomment-545561963 Build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile edited a comment on issue #26201: [SPARK-29543][SS][UI] Init structured streaming ui
gatorsmile edited a comment on issue #26201: [SPARK-29543][SS][UI] Init structured streaming ui URL: https://github.com/apache/spark/pull/26201#issuecomment-545561088 cc @gengliangwang @sarutak @xuanyuanking @zsxwing This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dilipbiswal commented on issue #25854: [SPARK-29145][SQL] Spark SQL cannot handle "NOT IN" condition when using "JOIN"
dilipbiswal commented on issue #25854: [SPARK-29145][SQL] Spark SQL cannot handle "NOT IN" condition when using "JOIN" URL: https://github.com/apache/spark/pull/25854#issuecomment-545561612 @AngersZh Great.. Thanks a lot for adding the UTs. Looks good to me. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on issue #26201: [SPARK-29543][SS][UI] Init structured streaming ui
gatorsmile commented on issue #26201: [SPARK-29543][SS][UI] Init structured streaming ui URL: https://github.com/apache/spark/pull/26201#issuecomment-545561088 cc @gengliangwang @sarutak This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26216: [SPARK-29453][WEBUI] Improve tooltips information for SQL tab.
AmplabJenkins removed a comment on issue #26216: [SPARK-29453][WEBUI] Improve tooltips information for SQL tab. URL: https://github.com/apache/spark/pull/26216#issuecomment-545558851 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17520/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26216: [SPARK-29453][WEBUI] Improve tooltips information for SQL tab.
AmplabJenkins commented on issue #26216: [SPARK-29453][WEBUI] Improve tooltips information for SQL tab. URL: https://github.com/apache/spark/pull/26216#issuecomment-545558851 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17520/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
