[GitHub] [spark] dongjoon-hyun commented on issue #26233: [SPARK-29533][SQL][TESTS][FOLLOWUP] Regenerate the result on EC2

2019-10-23 Thread GitBox
dongjoon-hyun commented on issue #26233: [SPARK-29533][SQL][TESTS][FOLLOWUP] 
Regenerate the result on EC2 
URL: https://github.com/apache/spark/pull/26233#issuecomment-545591973
 
 
   @MaxGekk , @HyukjinKwon , @cloud-fan 
   This is preparation for https://github.com/apache/spark/pull/26190 .


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #26232: [SPARK-29093][PYTHON][ML] Remove automatically generated param setters in _shared_params_code_gen.py

2019-10-23 Thread GitBox
AmplabJenkins removed a comment on issue #26232: [SPARK-29093][PYTHON][ML] 
Remove automatically generated param setters in _shared_params_code_gen.py
URL: https://github.com/apache/spark/pull/26232#issuecomment-545590513
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112559/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] dongjoon-hyun opened a new pull request #26233: [SPARK-29533][SQL][TESTS][FOLLOWUP] Regenerate the result on EC2

2019-10-23 Thread GitBox
dongjoon-hyun opened a new pull request #26233: 
[SPARK-29533][SQL][TESTS][FOLLOWUP] Regenerate the result on EC2 
URL: https://github.com/apache/spark/pull/26233
 
 
   ### What changes were proposed in this pull request?
   
   This regenerates the result on EC2.
   
   ### Why are the changes needed?
   
   This will be used for the other PR reviews.
   
   ### Does this PR introduce any user-facing change?
   
   No.
   
   ### How was this patch tested?
   
   N/A.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA removed a comment on issue #26232: [SPARK-29093][PYTHON][ML] Remove automatically generated param setters in _shared_params_code_gen.py

2019-10-23 Thread GitBox
SparkQA removed a comment on issue #26232: [SPARK-29093][PYTHON][ML] Remove 
automatically generated param setters in _shared_params_code_gen.py
URL: https://github.com/apache/spark/pull/26232#issuecomment-545586632
 
 
   **[Test build #112559 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112559/testReport)**
 for PR 26232 at commit 
[`a67163e`](https://github.com/apache/spark/commit/a67163e285876651dbb7eeb1d0ee66aa30dbce31).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #26232: [SPARK-29093][PYTHON][ML] Remove automatically generated param setters in _shared_params_code_gen.py

2019-10-23 Thread GitBox
AmplabJenkins removed a comment on issue #26232: [SPARK-29093][PYTHON][ML] 
Remove automatically generated param setters in _shared_params_code_gen.py
URL: https://github.com/apache/spark/pull/26232#issuecomment-545590503
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #26190: [SPARK-29532][SQL] simplify interval string parsing

2019-10-23 Thread GitBox
AmplabJenkins removed a comment on issue #26190: [SPARK-29532][SQL] simplify 
interval string parsing
URL: https://github.com/apache/spark/pull/26190#issuecomment-545590627
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] tgravescs commented on a change in pull request #26170: [SPARK-29397][core] Extend plugin interface to include the driver.

2019-10-23 Thread GitBox
tgravescs commented on a change in pull request #26170: [SPARK-29397][core] 
Extend plugin interface to include the driver.
URL: https://github.com/apache/spark/pull/26170#discussion_r338226990
 
 

 ##
 File path: 
core/src/main/scala/org/apache/spark/internal/plugin/PluginContainer.scala
 ##
 @@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.internal.plugin
+
+import scala.collection.JavaConverters._
+import scala.util.{Either, Left, Right}
+
+import org.apache.spark.{SparkContext, SparkEnv}
+import org.apache.spark.api.plugin._
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config._
+import org.apache.spark.util.Utils
+
+sealed abstract class PluginContainer {
+
+  def shutdown(): Unit
+
+}
+
+private class DriverPluginContainer(sc: SparkContext, plugins: 
Seq[SparkPlugin])
+  extends PluginContainer with Logging {
+
+  private val driverPlugins: Seq[(String, DriverPlugin)] = plugins.flatMap { p 
=>
+val driverPlugin = p.driverPlugin()
+if (driverPlugin != null) {
+  val name = p.getClass().getName()
+  val ctx = new PluginContextImpl(name, sc.env.rpcEnv, 
sc.env.metricsSystem, sc.conf,
+sc.env.executorId)
+
+  val extraConf = driverPlugin.init(sc, ctx)
+  if (extraConf != null) {
+extraConf.asScala.foreach { case (k, v) =>
+  sc.conf.set(s"${PluginContainer.EXTRA_CONF_PREFIX}$name.$k", v)
+}
+  }
+  ctx.registerMetrics()
+  logInfo(s"Initialized driver component for plugin $name.")
+  Some(p.getClass().getName() -> driverPlugin)
+} else {
+  None
+}
+  }
+
+  if (driverPlugins.nonEmpty) {
+sc.env.rpcEnv.setupEndpoint(classOf[PluginEndpoint].getName(),
+  new PluginEndpoint(driverPlugins.toMap, sc.env.rpcEnv))
+  }
+
+  override def shutdown(): Unit = {
+driverPlugins.foreach { case (name, plugin) =>
+  try {
+logDebug(s"Stopping plugin $name.")
+plugin.shutdown()
+  } catch {
+case t: Throwable =>
+  logInfo(s"Exception while shutting down plugin $name.", t)
+  }
+}
+  }
+
+}
+
+private class ExecutorPluginContainer(env: SparkEnv, plugins: Seq[SparkPlugin])
+  extends PluginContainer with Logging {
+
+  private val executorPlugins: Seq[(String, ExecutorPlugin)] = {
+val allExtraConf = 
env.conf.getAllWithPrefix(PluginContainer.EXTRA_CONF_PREFIX)
+
+plugins.flatMap { p =>
+  val executorPlugin = p.executorPlugin()
+  if (executorPlugin != null) {
+val name = p.getClass().getName()
+val prefix = name + "."
+val extraConf = allExtraConf
+  .filter { case (k, v) => k.startsWith(prefix) }
+  .map { case (k, v) => k.substring(prefix.length()) -> v }
+  .toMap
+  .asJava
+val ctx = new PluginContextImpl(name, env.rpcEnv, env.metricsSystem, 
env.conf,
+  env.executorId)
+executorPlugin.init(ctx, extraConf)
+ctx.registerMetrics()
+
+logInfo(s"Initialized executor component for plugin $name.")
+Some(p.getClass().getName() -> executorPlugin)
+  } else {
+None
+  }
+}
+  }
+
+  override def shutdown(): Unit = {
+executorPlugins.foreach { case (name, plugin) =>
+  try {
+logDebug(s"Stopping plugin $name.")
+plugin.shutdown()
+  } catch {
+case t: Throwable =>
+  logInfo(s"Exception while shutting down plugin $name.", t)
+  }
+}
+  }
+}
+
+object PluginContainer {
+
+  val EXTRA_CONF_PREFIX = "spark.plugins.__internal_conf__."
 
 Review comment:
   sorry I was thinking just spark.plugins.internal.conf.  We have the 
internal() option to config builder so figured it kind of matched.   I don't 
have a super strong opinion on this as long as we try to keep it consistent.  I 
know we use __xxx__ for various internal things - files directories -  but 
didn't think we had any for configs.  thoughts?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this 

[GitHub] [spark] AmplabJenkins removed a comment on issue #26190: [SPARK-29532][SQL] simplify interval string parsing

2019-10-23 Thread GitBox
AmplabJenkins removed a comment on issue #26190: [SPARK-29532][SQL] simplify 
interval string parsing
URL: https://github.com/apache/spark/pull/26190#issuecomment-545590640
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112552/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #26190: [SPARK-29532][SQL] simplify interval string parsing

2019-10-23 Thread GitBox
AmplabJenkins commented on issue #26190: [SPARK-29532][SQL] simplify interval 
string parsing
URL: https://github.com/apache/spark/pull/26190#issuecomment-545590640
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112552/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #26232: [SPARK-29093][PYTHON][ML] Remove automatically generated param setters in _shared_params_code_gen.py

2019-10-23 Thread GitBox
AmplabJenkins commented on issue #26232: [SPARK-29093][PYTHON][ML] Remove 
automatically generated param setters in _shared_params_code_gen.py
URL: https://github.com/apache/spark/pull/26232#issuecomment-545590513
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112559/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] dongjoon-hyun commented on issue #26189: [SPARK-29533][SQL][TEST] Benchmark casting strings to intervals

2019-10-23 Thread GitBox
dongjoon-hyun commented on issue #26189: [SPARK-29533][SQL][TEST] Benchmark 
casting strings to intervals
URL: https://github.com/apache/spark/pull/26189#issuecomment-545590584
 
 
   Hi, All.
   This was good but I'll make a follow up to regenerate on EC2 in order to 
compare with the other PR.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #26190: [SPARK-29532][SQL] simplify interval string parsing

2019-10-23 Thread GitBox
AmplabJenkins commented on issue #26190: [SPARK-29532][SQL] simplify interval 
string parsing
URL: https://github.com/apache/spark/pull/26190#issuecomment-545590627
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #26232: [SPARK-29093][PYTHON][ML] Remove automatically generated param setters in _shared_params_code_gen.py

2019-10-23 Thread GitBox
SparkQA commented on issue #26232: [SPARK-29093][PYTHON][ML] Remove 
automatically generated param setters in _shared_params_code_gen.py
URL: https://github.com/apache/spark/pull/26232#issuecomment-545590481
 
 
   **[Test build #112559 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112559/testReport)**
 for PR 26232 at commit 
[`a67163e`](https://github.com/apache/spark/commit/a67163e285876651dbb7eeb1d0ee66aa30dbce31).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #26232: [SPARK-29093][PYTHON][ML] Remove automatically generated param setters in _shared_params_code_gen.py

2019-10-23 Thread GitBox
AmplabJenkins commented on issue #26232: [SPARK-29093][PYTHON][ML] Remove 
automatically generated param setters in _shared_params_code_gen.py
URL: https://github.com/apache/spark/pull/26232#issuecomment-545590503
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #26190: [SPARK-29532][SQL] simplify interval string parsing

2019-10-23 Thread GitBox
SparkQA commented on issue #26190: [SPARK-29532][SQL] simplify interval string 
parsing
URL: https://github.com/apache/spark/pull/26190#issuecomment-545589890
 
 
   **[Test build #112552 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112552/testReport)**
 for PR 26190 at commit 
[`33ceedc`](https://github.com/apache/spark/commit/33ceedcedeafbe11070630ef3f47db37b5599ed0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA removed a comment on issue #26190: [SPARK-29532][SQL] simplify interval string parsing

2019-10-23 Thread GitBox
SparkQA removed a comment on issue #26190: [SPARK-29532][SQL] simplify interval 
string parsing
URL: https://github.com/apache/spark/pull/26190#issuecomment-545526541
 
 
   **[Test build #112552 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112552/testReport)**
 for PR 26190 at commit 
[`33ceedc`](https://github.com/apache/spark/commit/33ceedcedeafbe11070630ef3f47db37b5599ed0).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] rdblue commented on a change in pull request #25955: [SPARK-29277][SQL] Add early DSv2 filter and projection pushdown

2019-10-23 Thread GitBox
rdblue commented on a change in pull request #25955: [SPARK-29277][SQL] Add 
early DSv2 filter and projection pushdown
URL: https://github.com/apache/spark/pull/25955#discussion_r338225063
 
 

 ##
 File path: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
 ##
 @@ -243,17 +247,36 @@ class FindDataSourceTable(sparkSession: SparkSession) 
extends Rule[LogicalPlan]
   override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
 case i @ InsertIntoStatement(UnresolvedCatalogRelation(tableMeta), _, _, 
_, _)
 if DDLUtils.isDatasourceTable(tableMeta) =>
-  i.copy(table = readDataSourceTable(tableMeta))
+  if (DataSource.isV2Provider(tableMeta.provider.get, 
sparkSession.sessionState.conf)) {
 
 Review comment:
   Fair enough, I'll remove the refactor.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #26232: [SPARK-29093][PYTHON][ML] Remove automatically generated param setters in _shared_params_code_gen.py

2019-10-23 Thread GitBox
AmplabJenkins removed a comment on issue #26232: [SPARK-29093][PYTHON][ML] 
Remove automatically generated param setters in _shared_params_code_gen.py
URL: https://github.com/apache/spark/pull/26232#issuecomment-545587231
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #26078: [SPARK-29151][CORE] Support fractional resources for task resource scheduling

2019-10-23 Thread GitBox
AmplabJenkins removed a comment on issue #26078: [SPARK-29151][CORE] Support 
fractional resources for task resource scheduling
URL: https://github.com/apache/spark/pull/26078#issuecomment-545587162
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17525/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #26078: [SPARK-29151][CORE] Support fractional resources for task resource scheduling

2019-10-23 Thread GitBox
AmplabJenkins removed a comment on issue #26078: [SPARK-29151][CORE] Support 
fractional resources for task resource scheduling
URL: https://github.com/apache/spark/pull/26078#issuecomment-545587159
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #26232: [SPARK-29093][PYTHON][ML] Remove automatically generated param setters in _shared_params_code_gen.py

2019-10-23 Thread GitBox
AmplabJenkins removed a comment on issue #26232: [SPARK-29093][PYTHON][ML] 
Remove automatically generated param setters in _shared_params_code_gen.py
URL: https://github.com/apache/spark/pull/26232#issuecomment-545587242
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17524/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #26232: [SPARK-29093][PYTHON][ML] Remove automatically generated param setters in _shared_params_code_gen.py

2019-10-23 Thread GitBox
AmplabJenkins commented on issue #26232: [SPARK-29093][PYTHON][ML] Remove 
automatically generated param setters in _shared_params_code_gen.py
URL: https://github.com/apache/spark/pull/26232#issuecomment-545587242
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17524/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #26078: [SPARK-29151][CORE] Support fractional resources for task resource scheduling

2019-10-23 Thread GitBox
AmplabJenkins commented on issue #26078: [SPARK-29151][CORE] Support fractional 
resources for task resource scheduling
URL: https://github.com/apache/spark/pull/26078#issuecomment-545587162
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17525/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #26078: [SPARK-29151][CORE] Support fractional resources for task resource scheduling

2019-10-23 Thread GitBox
AmplabJenkins commented on issue #26078: [SPARK-29151][CORE] Support fractional 
resources for task resource scheduling
URL: https://github.com/apache/spark/pull/26078#issuecomment-545587159
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #26232: [SPARK-29093][PYTHON][ML] Remove automatically generated param setters in _shared_params_code_gen.py

2019-10-23 Thread GitBox
AmplabJenkins commented on issue #26232: [SPARK-29093][PYTHON][ML] Remove 
automatically generated param setters in _shared_params_code_gen.py
URL: https://github.com/apache/spark/pull/26232#issuecomment-545587231
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #26078: [SPARK-29151][CORE] Support fractional resources for task resource scheduling

2019-10-23 Thread GitBox
SparkQA commented on issue #26078: [SPARK-29151][CORE] Support fractional 
resources for task resource scheduling
URL: https://github.com/apache/spark/pull/26078#issuecomment-545586648
 
 
   **[Test build #112560 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112560/testReport)**
 for PR 26078 at commit 
[`f3ece52`](https://github.com/apache/spark/commit/f3ece52ece91bafdac556fc617a57182812a3361).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #26232: [SPARK-29093][PYTHON][ML] Remove automatically generated param setters in _shared_params_code_gen.py

2019-10-23 Thread GitBox
SparkQA commented on issue #26232: [SPARK-29093][PYTHON][ML] Remove 
automatically generated param setters in _shared_params_code_gen.py
URL: https://github.com/apache/spark/pull/26232#issuecomment-545586632
 
 
   **[Test build #112559 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112559/testReport)**
 for PR 26232 at commit 
[`a67163e`](https://github.com/apache/spark/commit/a67163e285876651dbb7eeb1d0ee66aa30dbce31).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] huaxingao opened a new pull request #26232: [SPARK-29093][PYTHON][ML] Remove automatically generated param setters in _shared_params_code_gen.py

2019-10-23 Thread GitBox
huaxingao opened a new pull request #26232: [SPARK-29093][PYTHON][ML] Remove 
automatically generated param setters in _shared_params_code_gen.py
URL: https://github.com/apache/spark/pull/26232
 
 
   
   
   ### What changes were proposed in this pull request?
   Remove automatically generated param setters in _shared_params_code_gen.py
   
   
   ### Why are the changes needed?
   To keep parity between scala and python
   
   
   ### Does this PR introduce any user-facing change?
   Yes
   Add some setters in Python ML XXXModels 
   
   
   ### How was this patch tested?
   unit tests
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] tgravescs commented on issue #26078: [SPARK-29151][CORE] Support fractional resources for task resource scheduling

2019-10-23 Thread GitBox
tgravescs commented on issue #26078: [SPARK-29151][CORE] Support fractional 
resources for task resource scheduling
URL: https://github.com/apache/spark/pull/26078#issuecomment-545584181
 
 
   test this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] tgravescs commented on issue #26078: [SPARK-29151][CORE] Support fractional resources for task resource scheduling

2019-10-23 Thread GitBox
tgravescs commented on issue #26078: [SPARK-29151][CORE] Support fractional 
resources for task resource scheduling
URL: https://github.com/apache/spark/pull/26078#issuecomment-545584129
 
 
   Jenkins, add to whitelist


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #26197: Implement p-value simulation and unit tests for chi2 test

2019-10-23 Thread GitBox
AmplabJenkins removed a comment on issue #26197: Implement p-value simulation 
and unit tests for chi2 test
URL: https://github.com/apache/spark/pull/26197#issuecomment-545581673
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112558/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver.

2019-10-23 Thread GitBox
AmplabJenkins removed a comment on issue #26170: [SPARK-29397][core] Extend 
plugin interface to include the driver.
URL: https://github.com/apache/spark/pull/26170#issuecomment-545581692
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112551/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver.

2019-10-23 Thread GitBox
AmplabJenkins removed a comment on issue #26170: [SPARK-29397][core] Extend 
plugin interface to include the driver.
URL: https://github.com/apache/spark/pull/26170#issuecomment-545581684
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #26197: Implement p-value simulation and unit tests for chi2 test

2019-10-23 Thread GitBox
AmplabJenkins commented on issue #26197: Implement p-value simulation and unit 
tests for chi2 test
URL: https://github.com/apache/spark/pull/26197#issuecomment-545581658
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA removed a comment on issue #26197: Implement p-value simulation and unit tests for chi2 test

2019-10-23 Thread GitBox
SparkQA removed a comment on issue #26197: Implement p-value simulation and 
unit tests for chi2 test
URL: https://github.com/apache/spark/pull/26197#issuecomment-545580592
 
 
   **[Test build #112558 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112558/testReport)**
 for PR 26197 at commit 
[`da1c6fa`](https://github.com/apache/spark/commit/da1c6fae18aa19d073fae5afc184a14912a50729).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #26197: Implement p-value simulation and unit tests for chi2 test

2019-10-23 Thread GitBox
AmplabJenkins removed a comment on issue #26197: Implement p-value simulation 
and unit tests for chi2 test
URL: https://github.com/apache/spark/pull/26197#issuecomment-545581658
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #26197: Implement p-value simulation and unit tests for chi2 test

2019-10-23 Thread GitBox
SparkQA commented on issue #26197: Implement p-value simulation and unit tests 
for chi2 test
URL: https://github.com/apache/spark/pull/26197#issuecomment-545581645
 
 
   **[Test build #112558 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112558/testReport)**
 for PR 26197 at commit 
[`da1c6fa`](https://github.com/apache/spark/commit/da1c6fae18aa19d073fae5afc184a14912a50729).
* This patch **fails build dependency tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
 * `class ChiSqTestSuite extends SparkFunSuite with MLlibTestSparkContext 
with Logging `


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver.

2019-10-23 Thread GitBox
AmplabJenkins commented on issue #26170: [SPARK-29397][core] Extend plugin 
interface to include the driver.
URL: https://github.com/apache/spark/pull/26170#issuecomment-545581684
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #26197: Implement p-value simulation and unit tests for chi2 test

2019-10-23 Thread GitBox
AmplabJenkins commented on issue #26197: Implement p-value simulation and unit 
tests for chi2 test
URL: https://github.com/apache/spark/pull/26197#issuecomment-545581673
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112558/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver.

2019-10-23 Thread GitBox
AmplabJenkins commented on issue #26170: [SPARK-29397][core] Extend plugin 
interface to include the driver.
URL: https://github.com/apache/spark/pull/26170#issuecomment-545581692
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112551/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26206: [SPARK-29551][CORE] Fix a bug about fetch failed when an executor is lost

2019-10-23 Thread GitBox
dongjoon-hyun commented on a change in pull request #26206: [SPARK-29551][CORE] 
Fix a bug about fetch failed when an executor is lost
URL: https://github.com/apache/spark/pull/26206#discussion_r338215381
 
 

 ##
 File path: 
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
 ##
 @@ -537,6 +536,169 @@ class DAGSchedulerSuite extends SparkFunSuite with 
LocalSparkContext with TimeLi
 assert(mapStatus2(2).location.host === "hostB")
   }
 
+  test("All shuffle files on the executor should be cleaned up when executor 
lost " +
+"and then causes 'fetch failed'") {
+// whether to kill Executor or not before FetchFailed
+Seq(true, false).foreach { killExecutor => {
+  afterEach()
+  val conf = new SparkConf()
+  conf.set(config.SHUFFLE_SERVICE_ENABLED.key, "true")
+  conf.set("spark.files.fetchFailure.unRegisterOutputOnHost", "false")
+  init(conf)
+  runEvent(ExecutorAdded("exec-hostA1", "hostA"))
+  runEvent(ExecutorAdded("exec-hostA2", "hostA"))
+  runEvent(ExecutorAdded("exec-hostB", "hostB"))
+  val firstRDD = new MyRDD(sc, 3, Nil)
+  val firstShuffleDep = new ShuffleDependency(firstRDD, new 
HashPartitioner(3))
+  val firstShuffleId = firstShuffleDep.shuffleId
+  val shuffleMapRdd = new MyRDD(sc, 3, List(firstShuffleDep))
+  val shuffleDep = new ShuffleDependency(shuffleMapRdd, new 
HashPartitioner(3))
+  val secondShuffleId = shuffleDep.shuffleId
+  val reduceRdd = new MyRDD(sc, 1, List(shuffleDep))
+
+  submit(reduceRdd, Array(0))
+  // map stage1 completes successfully, with one task on each executor
+  complete(taskSets(0), Seq(
+(Success,
+  MapStatus(BlockManagerId("exec-hostA1", "hostA", 12345),
+Array.fill[Long](1)(2), mapTaskId = 5)),
+(Success,
+  MapStatus(BlockManagerId("exec-hostA2", "hostA", 12345),
+Array.fill[Long](1)(2), mapTaskId = 6)),
+(Success, makeMapStatus("hostB", 1, mapTaskId = 7))
+  ))
+  // map stage2 completes successfully, with one task on each executor
+  complete(taskSets(1), Seq(
+(Success,
+  MapStatus(BlockManagerId("exec-hostA1", "hostA", 12345),
+Array.fill[Long](1)(2), mapTaskId = 8)),
+(Success,
+  MapStatus(BlockManagerId("exec-hostA2", "hostA", 12345),
+Array.fill[Long](1)(2), mapTaskId = 9)),
+(Success, makeMapStatus("hostB", 1))
+  ))
+  // make sure our test setup is correct
+  val initialMapStatus1 = 
mapOutputTracker.shuffleStatuses(firstShuffleId).mapStatuses
+  //  val initialMapStatus1 = mapOutputTracker.mapStatuses.get(0).get
+  assert(initialMapStatus1.count(_ != null) === 3)
+  assert(initialMapStatus1.map {
+_.location.executorId
+  }.toSet ===
+Set("exec-hostA1", "exec-hostA2", "exec-hostB"))
+  val initialMapStatus2 = 
mapOutputTracker.shuffleStatuses(secondShuffleId).mapStatuses
+  assert(initialMapStatus2.count(_ != null) === 3)
+  assert(initialMapStatus2.map {
+_.location.executorId
+  }.toSet ===
+Set("exec-hostA1", "exec-hostA2", "exec-hostB"))
+  // kill exec-hostA2
+  if (killExecutor) {
+runEvent(ExecutorLost("exec-hostA2", ExecutorKilled))
+  }
+  // reduce stage fails with a fetch failure from one host
+  complete(taskSets(2), Seq(
+(FetchFailed(BlockManagerId("exec-hostA2", "hostA", 12345),
+  secondShuffleId, 0L, 0, 0, "ignored"), null)
+  ))
+  // Here is the main assertion -- make sure that we de-register
+  // the map outputs for exec-hostA2
+  val mapStatus1 = 
mapOutputTracker.shuffleStatuses(firstShuffleId).mapStatuses
+  assert(mapStatus1.count(_ != null) === 2)
+  assert(mapStatus1(0).location.executorId === "exec-hostA1")
+  assert(mapStatus1(0).location.host === "hostA")
+  assert(mapStatus1(2).location.executorId === "exec-hostB")
+  assert(mapStatus1(2).location.host === "hostB")
+
+  val mapStatus2 = 
mapOutputTracker.shuffleStatuses(secondShuffleId).mapStatuses
+  assert(mapStatus2.count(_ != null) === 2)
+  assert(mapStatus2(0).location.executorId === "exec-hostA1")
+  assert(mapStatus2(0).location.host === "hostA")
+  assert(mapStatus2(2).location.executorId === "exec-hostB")
+  assert(mapStatus2(2).location.host === "hostB")
+}
+}
+  }
+
+  test("All shuffle files on the host should be cleaned up when host lost") {
 
 Review comment:
   The same comments for this test case (test case name, use config key instead 
of string literal, indentation)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,

[GitHub] [spark] SparkQA commented on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver.

2019-10-23 Thread GitBox
SparkQA commented on issue #26170: [SPARK-29397][core] Extend plugin interface 
to include the driver.
URL: https://github.com/apache/spark/pull/26170#issuecomment-545580952
 
 
   **[Test build #112551 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112551/testReport)**
 for PR 26170 at commit 
[`5dd6afd`](https://github.com/apache/spark/commit/5dd6afdcbf373a1064d92b7fc5a658d3a11b66fc).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA removed a comment on issue #26170: [SPARK-29397][core] Extend plugin interface to include the driver.

2019-10-23 Thread GitBox
SparkQA removed a comment on issue #26170: [SPARK-29397][core] Extend plugin 
interface to include the driver.
URL: https://github.com/apache/spark/pull/26170#issuecomment-545519977
 
 
   **[Test build #112551 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112551/testReport)**
 for PR 26170 at commit 
[`5dd6afd`](https://github.com/apache/spark/commit/5dd6afdcbf373a1064d92b7fc5a658d3a11b66fc).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #26197: Implement p-value simulation and unit tests for chi2 test

2019-10-23 Thread GitBox
SparkQA commented on issue #26197: Implement p-value simulation and unit tests 
for chi2 test
URL: https://github.com/apache/spark/pull/26197#issuecomment-545580592
 
 
   **[Test build #112558 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112558/testReport)**
 for PR 26197 at commit 
[`da1c6fa`](https://github.com/apache/spark/commit/da1c6fae18aa19d073fae5afc184a14912a50729).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26206: [SPARK-29551][CORE] Fix a bug about fetch failed when an executor is lost

2019-10-23 Thread GitBox
dongjoon-hyun commented on a change in pull request #26206: [SPARK-29551][CORE] 
Fix a bug about fetch failed when an executor is lost
URL: https://github.com/apache/spark/pull/26206#discussion_r338214733
 
 

 ##
 File path: 
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
 ##
 @@ -537,6 +536,169 @@ class DAGSchedulerSuite extends SparkFunSuite with 
LocalSparkContext with TimeLi
 assert(mapStatus2(2).location.host === "hostB")
   }
 
+  test("All shuffle files on the executor should be cleaned up when executor 
lost " +
+"and then causes 'fetch failed'") {
+// whether to kill Executor or not before FetchFailed
+Seq(true, false).foreach { killExecutor => {
+  afterEach()
+  val conf = new SparkConf()
+  conf.set(config.SHUFFLE_SERVICE_ENABLED.key, "true")
+  conf.set("spark.files.fetchFailure.unRegisterOutputOnHost", "false")
+  init(conf)
+  runEvent(ExecutorAdded("exec-hostA1", "hostA"))
+  runEvent(ExecutorAdded("exec-hostA2", "hostA"))
+  runEvent(ExecutorAdded("exec-hostB", "hostB"))
+  val firstRDD = new MyRDD(sc, 3, Nil)
+  val firstShuffleDep = new ShuffleDependency(firstRDD, new 
HashPartitioner(3))
+  val firstShuffleId = firstShuffleDep.shuffleId
+  val shuffleMapRdd = new MyRDD(sc, 3, List(firstShuffleDep))
+  val shuffleDep = new ShuffleDependency(shuffleMapRdd, new 
HashPartitioner(3))
+  val secondShuffleId = shuffleDep.shuffleId
+  val reduceRdd = new MyRDD(sc, 1, List(shuffleDep))
+
+  submit(reduceRdd, Array(0))
+  // map stage1 completes successfully, with one task on each executor
+  complete(taskSets(0), Seq(
+(Success,
+  MapStatus(BlockManagerId("exec-hostA1", "hostA", 12345),
+Array.fill[Long](1)(2), mapTaskId = 5)),
+(Success,
+  MapStatus(BlockManagerId("exec-hostA2", "hostA", 12345),
+Array.fill[Long](1)(2), mapTaskId = 6)),
+(Success, makeMapStatus("hostB", 1, mapTaskId = 7))
+  ))
+  // map stage2 completes successfully, with one task on each executor
+  complete(taskSets(1), Seq(
+(Success,
+  MapStatus(BlockManagerId("exec-hostA1", "hostA", 12345),
+Array.fill[Long](1)(2), mapTaskId = 8)),
+(Success,
+  MapStatus(BlockManagerId("exec-hostA2", "hostA", 12345),
+Array.fill[Long](1)(2), mapTaskId = 9)),
+(Success, makeMapStatus("hostB", 1))
+  ))
+  // make sure our test setup is correct
+  val initialMapStatus1 = 
mapOutputTracker.shuffleStatuses(firstShuffleId).mapStatuses
+  //  val initialMapStatus1 = mapOutputTracker.mapStatuses.get(0).get
+  assert(initialMapStatus1.count(_ != null) === 3)
+  assert(initialMapStatus1.map {
+_.location.executorId
+  }.toSet ===
+Set("exec-hostA1", "exec-hostA2", "exec-hostB"))
+  val initialMapStatus2 = 
mapOutputTracker.shuffleStatuses(secondShuffleId).mapStatuses
+  assert(initialMapStatus2.count(_ != null) === 3)
+  assert(initialMapStatus2.map {
+_.location.executorId
+  }.toSet ===
+Set("exec-hostA1", "exec-hostA2", "exec-hostB"))
+  // kill exec-hostA2
+  if (killExecutor) {
+runEvent(ExecutorLost("exec-hostA2", ExecutorKilled))
+  }
+  // reduce stage fails with a fetch failure from one host
+  complete(taskSets(2), Seq(
+(FetchFailed(BlockManagerId("exec-hostA2", "hostA", 12345),
+  secondShuffleId, 0L, 0, 0, "ignored"), null)
+  ))
+  // Here is the main assertion -- make sure that we de-register
+  // the map outputs for exec-hostA2
+  val mapStatus1 = 
mapOutputTracker.shuffleStatuses(firstShuffleId).mapStatuses
+  assert(mapStatus1.count(_ != null) === 2)
+  assert(mapStatus1(0).location.executorId === "exec-hostA1")
+  assert(mapStatus1(0).location.host === "hostA")
+  assert(mapStatus1(2).location.executorId === "exec-hostB")
+  assert(mapStatus1(2).location.host === "hostB")
+
+  val mapStatus2 = 
mapOutputTracker.shuffleStatuses(secondShuffleId).mapStatuses
+  assert(mapStatus2.count(_ != null) === 2)
+  assert(mapStatus2(0).location.executorId === "exec-hostA1")
+  assert(mapStatus2(0).location.host === "hostA")
+  assert(mapStatus2(2).location.executorId === "exec-hostB")
+  assert(mapStatus2(2).location.host === "hostB")
+}
+}
 
 Review comment:
   Indentation seems to be corrupted in some point before this line.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: 

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26206: [SPARK-29551][CORE] Fix a bug about fetch failed when an executor is lost

2019-10-23 Thread GitBox
dongjoon-hyun commented on a change in pull request #26206: [SPARK-29551][CORE] 
Fix a bug about fetch failed when an executor is lost
URL: https://github.com/apache/spark/pull/26206#discussion_r338214415
 
 

 ##
 File path: 
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
 ##
 @@ -537,6 +536,169 @@ class DAGSchedulerSuite extends SparkFunSuite with 
LocalSparkContext with TimeLi
 assert(mapStatus2(2).location.host === "hostB")
   }
 
+  test("All shuffle files on the executor should be cleaned up when executor 
lost " +
+"and then causes 'fetch failed'") {
+// whether to kill Executor or not before FetchFailed
+Seq(true, false).foreach { killExecutor => {
+  afterEach()
+  val conf = new SparkConf()
+  conf.set(config.SHUFFLE_SERVICE_ENABLED.key, "true")
+  conf.set("spark.files.fetchFailure.unRegisterOutputOnHost", "false")
+  init(conf)
+  runEvent(ExecutorAdded("exec-hostA1", "hostA"))
+  runEvent(ExecutorAdded("exec-hostA2", "hostA"))
+  runEvent(ExecutorAdded("exec-hostB", "hostB"))
+  val firstRDD = new MyRDD(sc, 3, Nil)
+  val firstShuffleDep = new ShuffleDependency(firstRDD, new 
HashPartitioner(3))
+  val firstShuffleId = firstShuffleDep.shuffleId
+  val shuffleMapRdd = new MyRDD(sc, 3, List(firstShuffleDep))
+  val shuffleDep = new ShuffleDependency(shuffleMapRdd, new 
HashPartitioner(3))
+  val secondShuffleId = shuffleDep.shuffleId
+  val reduceRdd = new MyRDD(sc, 1, List(shuffleDep))
+
+  submit(reduceRdd, Array(0))
+  // map stage1 completes successfully, with one task on each executor
+  complete(taskSets(0), Seq(
+(Success,
+  MapStatus(BlockManagerId("exec-hostA1", "hostA", 12345),
+Array.fill[Long](1)(2), mapTaskId = 5)),
+(Success,
+  MapStatus(BlockManagerId("exec-hostA2", "hostA", 12345),
+Array.fill[Long](1)(2), mapTaskId = 6)),
+(Success, makeMapStatus("hostB", 1, mapTaskId = 7))
+  ))
+  // map stage2 completes successfully, with one task on each executor
+  complete(taskSets(1), Seq(
+(Success,
+  MapStatus(BlockManagerId("exec-hostA1", "hostA", 12345),
+Array.fill[Long](1)(2), mapTaskId = 8)),
+(Success,
+  MapStatus(BlockManagerId("exec-hostA2", "hostA", 12345),
+Array.fill[Long](1)(2), mapTaskId = 9)),
+(Success, makeMapStatus("hostB", 1))
+  ))
+  // make sure our test setup is correct
+  val initialMapStatus1 = 
mapOutputTracker.shuffleStatuses(firstShuffleId).mapStatuses
+  //  val initialMapStatus1 = mapOutputTracker.mapStatuses.get(0).get
 
 Review comment:
   Shall we remove this commented code?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26206: [SPARK-29551][CORE] Fix a bug about fetch failed when an executor is lost

2019-10-23 Thread GitBox
dongjoon-hyun commented on a change in pull request #26206: [SPARK-29551][CORE] 
Fix a bug about fetch failed when an executor is lost
URL: https://github.com/apache/spark/pull/26206#discussion_r338214048
 
 

 ##
 File path: 
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
 ##
 @@ -537,6 +536,169 @@ class DAGSchedulerSuite extends SparkFunSuite with 
LocalSparkContext with TimeLi
 assert(mapStatus2(2).location.host === "hostB")
   }
 
+  test("All shuffle files on the executor should be cleaned up when executor 
lost " +
+"and then causes 'fetch failed'") {
+// whether to kill Executor or not before FetchFailed
+Seq(true, false).foreach { killExecutor => {
+  afterEach()
+  val conf = new SparkConf()
+  conf.set(config.SHUFFLE_SERVICE_ENABLED.key, "true")
+  conf.set("spark.files.fetchFailure.unRegisterOutputOnHost", "false")
 
 Review comment:
   nit. `"spark.files.fetchFailure.unRegisterOutputOnHost"` -> 
`UNREGISTER_OUTPUT_ON_HOST_ON_FETCH_FAILURE.key`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] juliuszsompolski commented on issue #25721: [WIP][SPARK-29018][SQL] Implement Spark Thrift Server with it's own code base on PROTOCOL_VERSION_V9

2019-10-23 Thread GitBox
juliuszsompolski commented on issue #25721: [WIP][SPARK-29018][SQL] Implement 
Spark Thrift Server with it's own code base on PROTOCOL_VERSION_V9
URL: https://github.com/apache/spark/pull/25721#issuecomment-545579648
 
 
   > > 2. on top of that, all the Thriftserver code that is just translated 
from Java to Scala without changes - it has a new package now, so it will 
remain unused.
   
   > Translate all code seems too heavy. And we build it with protocol v11. we 
can't direct apply to v1.2.1/v2.3.5.
   > I prefer to build the framework based on protocol v11 such as common class 
and process like Operation.class, SessionManager.class, then fill in the details
   
   I am confused. It seems that in this PR you already did translation from 
Java to Scala of all Hive Thriftserver code? I assume it was mostly somehow 
autogenerated?
   A lot of these classes don't really change much from Hive to Spark, except 
for mechanical translation from Java to Scala, renaming package names, removing 
dependence on some Hive objects. Do I see correctly?
   I would commit these in a separate PR, to separate "mechanical" changes from 
places that were actually rewritten.
   
   I am also not sure whether we should translate those from Java to Scala at 
all. Maybe we should keep these in Java code, and only implement the Spark 
specific stuff in scala, removing Java Hive stuff that is not needed anymore. 
So e.g.
   - Keep CLIService.java, ThriftCLIService.java, ThriftHttpServlet.java, ... - 
all things that don't really get modified by Spark in Java
   - Do Spark specific implementation in scala, and remove the no longer needed 
Java thriftserver impl. E.g. (Spark)ExecuteStatementOperation.scala and remove 
ExecuteStatementOperation.java; (Spark)OperationManager.scala and remove 
OperationManager.java etc. etc.
   OR
   Translate all these Java files to scala, like is done in this PR?
   
   I think I would keep them in Java to avoid potential errors in translation, 
and also to see easier if we want to port any future Hive changes to them.
   @gatorsmile @rxin - what do you think?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26206: [SPARK-29551][CORE] Fix a bug about fetch failed when an executor is lost

2019-10-23 Thread GitBox
dongjoon-hyun commented on a change in pull request #26206: [SPARK-29551][CORE] 
Fix a bug about fetch failed when an executor is lost
URL: https://github.com/apache/spark/pull/26206#discussion_r338213188
 
 

 ##
 File path: 
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
 ##
 @@ -510,7 +510,6 @@ class DAGSchedulerSuite extends SparkFunSuite with 
LocalSparkContext with TimeLi
 assert(initialMapStatus1.map{_.mapId}.toSet === Set(5, 6, 7))
 
 val initialMapStatus2 = 
mapOutputTracker.shuffleStatuses(secondShuffleId).mapStatuses
-//  val initialMapStatus1 = mapOutputTracker.mapStatuses.get(0).get
 
 Review comment:
   Let's revert this change. This kind clean up is good in general of course, 
but this makes the backport difficult.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26206: [SPARK-29551][CORE] Fix a bug about fetch failed when an executor is lost

2019-10-23 Thread GitBox
dongjoon-hyun commented on a change in pull request #26206: [SPARK-29551][CORE] 
Fix a bug about fetch failed when an executor is lost
URL: https://github.com/apache/spark/pull/26206#discussion_r338212624
 
 

 ##
 File path: 
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
 ##
 @@ -537,6 +536,169 @@ class DAGSchedulerSuite extends SparkFunSuite with 
LocalSparkContext with TimeLi
 assert(mapStatus2(2).location.host === "hostB")
   }
 
+  test("All shuffle files on the executor should be cleaned up when executor 
lost " +
 
 Review comment:
   For the bug fix PR, we had better add `SPARK-29551` prefix for the test case 
name.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #26197: Implement p-value simulation and unit tests for chi2 test

2019-10-23 Thread GitBox
AmplabJenkins commented on issue #26197: Implement p-value simulation and unit 
tests for chi2 test
URL: https://github.com/apache/spark/pull/26197#issuecomment-545578161
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #26197: Implement p-value simulation and unit tests for chi2 test

2019-10-23 Thread GitBox
AmplabJenkins commented on issue #26197: Implement p-value simulation and unit 
tests for chi2 test
URL: https://github.com/apache/spark/pull/26197#issuecomment-545578173
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17523/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #26197: Implement p-value simulation and unit tests for chi2 test

2019-10-23 Thread GitBox
AmplabJenkins removed a comment on issue #26197: Implement p-value simulation 
and unit tests for chi2 test
URL: https://github.com/apache/spark/pull/26197#issuecomment-545578161
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #26197: Implement p-value simulation and unit tests for chi2 test

2019-10-23 Thread GitBox
AmplabJenkins removed a comment on issue #26197: Implement p-value simulation 
and unit tests for chi2 test
URL: https://github.com/apache/spark/pull/26197#issuecomment-545578173
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17523/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26206: [SPARK-29551][CORE] Fix a bug about fetch failed when an executor is lost

2019-10-23 Thread GitBox
dongjoon-hyun commented on a change in pull request #26206: [SPARK-29551][CORE] 
Fix a bug about fetch failed when an executor is lost
URL: https://github.com/apache/spark/pull/26206#discussion_r338211766
 
 

 ##
 File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
 ##
 @@ -175,6 +175,10 @@ private[spark] class DAGScheduler(
   // TODO: Garbage collect information about failure epochs when we know there 
are no more
   //   stray messages to detect.
   private val failedEpoch = new HashMap[String, Long]
+  // There will be a regression when an executor lost and then causes 'fetch 
failed'.
 
 Review comment:
   `There will be a regression` is misleading because this mean this PR causes 
a regression.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] holdenk commented on a change in pull request #26197: Implement p-value simulation and unit tests for chi2 test

2019-10-23 Thread GitBox
holdenk commented on a change in pull request #26197: Implement p-value 
simulation and unit tests for chi2 test
URL: https://github.com/apache/spark/pull/26197#discussion_r338180024
 
 

 ##
 File path: mllib/pom.xml
 ##
 @@ -130,6 +130,11 @@
   org.apache.spark
   spark-tags_${scala.binary.version}
 
+
+  com.tdunning
 
 Review comment:
   Generally speaking we try and not pick up new dependencies for small 
features, especially those which aren't maintained by a community. Looking at 
tdunning's tdigest package there's been ~5 distinct contributors for 2019, and 
if we expand the window up to 2017 that goes up to ~7.
   
   Do we know how actively maintined this going to be and if he is going to 
maintain the 3.X line once the 4.X release is out?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] holdenk commented on a change in pull request #26197: Implement p-value simulation and unit tests for chi2 test

2019-10-23 Thread GitBox
holdenk commented on a change in pull request #26197: Implement p-value 
simulation and unit tests for chi2 test
URL: https://github.com/apache/spark/pull/26197#discussion_r338180477
 
 

 ##
 File path: 
mllib/src/main/scala/org/apache/spark/mllib/stat/test/ChiSqTest.scala
 ##
 @@ -151,6 +155,8 @@ private[spark] object ChiSqTest extends Logging {
*/
   def chiSquared(observed: Vector,
   expected: Vector = Vectors.dense(Array.empty[Double]),
+  simulatePValue: Boolean = false,
 
 Review comment:
   Spark MLLib is in maintaince mode, see 
https://spark.apache.org/docs/latest/ml-guide.html . If we do want this I think 
we need to make sure it is exposed in Spark ML.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] atronchi commented on issue #26197: Implement p-value simulation and unit tests for chi2 test

2019-10-23 Thread GitBox
atronchi commented on issue #26197: Implement p-value simulation and unit tests 
for chi2 test
URL: https://github.com/apache/spark/pull/26197#issuecomment-545577841
 
 
   Last push integrated some offline feedback.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #26229: [SPARK-21492][SQL][Follow Up] Reimplement UnsafeExternalRowSorter in database style iterator

2019-10-23 Thread GitBox
AmplabJenkins removed a comment on issue #26229: [SPARK-21492][SQL][Follow Up] 
Reimplement UnsafeExternalRowSorter in database style iterator
URL: https://github.com/apache/spark/pull/26229#issuecomment-545576440
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112547/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #26229: [SPARK-21492][SQL][Follow Up] Reimplement UnsafeExternalRowSorter in database style iterator

2019-10-23 Thread GitBox
AmplabJenkins removed a comment on issue #26229: [SPARK-21492][SQL][Follow Up] 
Reimplement UnsafeExternalRowSorter in database style iterator
URL: https://github.com/apache/spark/pull/26229#issuecomment-545576433
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #26197: Implement p-value simulation and unit tests for chi2 test

2019-10-23 Thread GitBox
AmplabJenkins removed a comment on issue #26197: Implement p-value simulation 
and unit tests for chi2 test
URL: https://github.com/apache/spark/pull/26197#issuecomment-545575920
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112557/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #26229: [SPARK-21492][SQL][Follow Up] Reimplement UnsafeExternalRowSorter in database style iterator

2019-10-23 Thread GitBox
AmplabJenkins commented on issue #26229: [SPARK-21492][SQL][Follow Up] 
Reimplement UnsafeExternalRowSorter in database style iterator
URL: https://github.com/apache/spark/pull/26229#issuecomment-545576440
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112547/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #26229: [SPARK-21492][SQL][Follow Up] Reimplement UnsafeExternalRowSorter in database style iterator

2019-10-23 Thread GitBox
AmplabJenkins commented on issue #26229: [SPARK-21492][SQL][Follow Up] 
Reimplement UnsafeExternalRowSorter in database style iterator
URL: https://github.com/apache/spark/pull/26229#issuecomment-545576433
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #26197: Implement p-value simulation and unit tests for chi2 test

2019-10-23 Thread GitBox
AmplabJenkins removed a comment on issue #26197: Implement p-value simulation 
and unit tests for chi2 test
URL: https://github.com/apache/spark/pull/26197#issuecomment-545575912
 
 
   Build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26190: [SPARK-29532][SQL] simplify interval string parsing

2019-10-23 Thread GitBox
dongjoon-hyun commented on a change in pull request #26190: [SPARK-29532][SQL] 
simplify interval string parsing
URL: https://github.com/apache/spark/pull/26190#discussion_r338209548
 
 

 ##
 File path: sql/core/benchmarks/IntervalBenchmark-results.txt
 ##
 @@ -1,25 +1,25 @@
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.15
-Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_161-b12 on Mac OS X 10.14
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
 cast strings to intervals:Best Time(ms)   Avg Time(ms)   
Stdev(ms)Rate(M/s)   Per Row(ns)   Relative
 

-string w/ interval  420435 
 18  2.4 419.8   1.0X
-string w/o interval 359365 
 10  2.8 358.7   1.2X
-1 units w/ interval 752759 
  8  1.3 752.0   0.6X
-1 units w/o interval762766 
  4  1.3 762.0   0.6X
-2 units w/ interval 961970 
  8  1.0 960.7   0.4X
-2 units w/o interval970976 
  9  1.0 970.2   0.4X
-3 units w/ interval1130   1136 
  7  0.91130.4   0.4X
-3 units w/o interval   1150   1158 
  9  0.91150.3   0.4X
-4 units w/ interval1333   1336 
  3  0.71333.5   0.3X
-4 units w/o interval   1354   1359 
  4  0.71354.5   0.3X
-5 units w/ interval1523   1525 
  2  0.71523.3   0.3X
-5 units w/o interval   1549   1551 
  3  0.61549.4   0.3X
-6 units w/ interval1661   1663 
  2  0.61660.8   0.3X
-6 units w/o interval   1691   1704 
 13  0.61691.2   0.2X
-7 units w/ interval1811   1817 
  8  0.61810.6   0.2X
-7 units w/o interval   1853   1854 
  1  0.51853.2   0.2X
-8 units w/ interval2029   2037 
  8  0.52028.7   0.2X
-8 units w/o interval   2075   2075 
  1  0.52074.5   0.2X
-9 units w/ interval2170   2175 
  5  0.52170.0   0.2X
-9 units w/o interval   2204   2212 
  8  0.52203.6   0.2X
+prepare string w/ interval  403419 
 18  2.5 403.1   1.0X
+prepare string w/o interval 341353 
 21  2.9 341.1   1.2X
+1 units w/ interval5154   5159 
  8  0.25153.5   0.1X
+1 units w/o interval   4818   4833 
 20  0.24817.6   0.1X
+2 units w/ interval6191   6223 
 41  0.26190.6   0.1X
+2 units w/o interval   6236   6264 
 25  0.26235.7   0.1X
+3 units w/ interval7397   7567 
170  0.17397.0   0.1X
+3 units w/o interval   7280   7367 
 76  0.17279.6   0.1X
+4 units w/ interval8197   8228 
 27  0.18197.3   0.0X
+4 units w/o interval   7977   7989 
 17  0.17977.3   0.1X
+5 units w/ interval9089   9192 
101  0.19088.8   0.0X
+5 units w/o interval   8853   8858 
  5  0.18852.8   0.0X
+6 units w/ interval9696   9720 
 23  0.19695.6   0.0X
+6 units w/o interval   9509   9518 
  9  0.19509.4   0.0X
+7 units w/ interval   10738  

[GitHub] [spark] SparkQA removed a comment on issue #26197: Implement p-value simulation and unit tests for chi2 test

2019-10-23 Thread GitBox
SparkQA removed a comment on issue #26197: Implement p-value simulation and 
unit tests for chi2 test
URL: https://github.com/apache/spark/pull/26197#issuecomment-545574388
 
 
   **[Test build #112557 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112557/testReport)**
 for PR 26197 at commit 
[`34852e6`](https://github.com/apache/spark/commit/34852e62ecafb3573d4622ffabaab61fdcd253c0).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #26229: [SPARK-21492][SQL][Follow Up] Reimplement UnsafeExternalRowSorter in database style iterator

2019-10-23 Thread GitBox
SparkQA commented on issue #26229: [SPARK-21492][SQL][Follow Up] Reimplement 
UnsafeExternalRowSorter in database style iterator
URL: https://github.com/apache/spark/pull/26229#issuecomment-545575754
 
 
   **[Test build #112547 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112547/testReport)**
 for PR 26229 at commit 
[`5623c7a`](https://github.com/apache/spark/commit/5623c7a5a042827bd277853df0dd9e84e8ca1a9c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #26197: Implement p-value simulation and unit tests for chi2 test

2019-10-23 Thread GitBox
SparkQA commented on issue #26197: Implement p-value simulation and unit tests 
for chi2 test
URL: https://github.com/apache/spark/pull/26197#issuecomment-545575899
 
 
   **[Test build #112557 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112557/testReport)**
 for PR 26197 at commit 
[`34852e6`](https://github.com/apache/spark/commit/34852e62ecafb3573d4622ffabaab61fdcd253c0).
* This patch **fails build dependency tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA removed a comment on issue #26229: [SPARK-21492][SQL][Follow Up] Reimplement UnsafeExternalRowSorter in database style iterator

2019-10-23 Thread GitBox
SparkQA removed a comment on issue #26229: [SPARK-21492][SQL][Follow Up] 
Reimplement UnsafeExternalRowSorter in database style iterator
URL: https://github.com/apache/spark/pull/26229#issuecomment-545474776
 
 
   **[Test build #112547 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112547/testReport)**
 for PR 26229 at commit 
[`5623c7a`](https://github.com/apache/spark/commit/5623c7a5a042827bd277853df0dd9e84e8ca1a9c).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #26197: Implement p-value simulation and unit tests for chi2 test

2019-10-23 Thread GitBox
AmplabJenkins commented on issue #26197: Implement p-value simulation and unit 
tests for chi2 test
URL: https://github.com/apache/spark/pull/26197#issuecomment-545575912
 
 
   Build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #26197: Implement p-value simulation and unit tests for chi2 test

2019-10-23 Thread GitBox
AmplabJenkins commented on issue #26197: Implement p-value simulation and unit 
tests for chi2 test
URL: https://github.com/apache/spark/pull/26197#issuecomment-545575920
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112557/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #26197: Implement p-value simulation and unit tests for chi2 test

2019-10-23 Thread GitBox
AmplabJenkins removed a comment on issue #26197: Implement p-value simulation 
and unit tests for chi2 test
URL: https://github.com/apache/spark/pull/26197#issuecomment-545574921
 
 
   Build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #26197: Implement p-value simulation and unit tests for chi2 test

2019-10-23 Thread GitBox
AmplabJenkins commented on issue #26197: Implement p-value simulation and unit 
tests for chi2 test
URL: https://github.com/apache/spark/pull/26197#issuecomment-545574921
 
 
   Build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #26197: Implement p-value simulation and unit tests for chi2 test

2019-10-23 Thread GitBox
AmplabJenkins commented on issue #26197: Implement p-value simulation and unit 
tests for chi2 test
URL: https://github.com/apache/spark/pull/26197#issuecomment-545574930
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17522/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #26197: Implement p-value simulation and unit tests for chi2 test

2019-10-23 Thread GitBox
AmplabJenkins removed a comment on issue #26197: Implement p-value simulation 
and unit tests for chi2 test
URL: https://github.com/apache/spark/pull/26197#issuecomment-545574930
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17522/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #26197: Implement p-value simulation and unit tests for chi2 test

2019-10-23 Thread GitBox
SparkQA commented on issue #26197: Implement p-value simulation and unit tests 
for chi2 test
URL: https://github.com/apache/spark/pull/26197#issuecomment-545574388
 
 
   **[Test build #112557 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112557/testReport)**
 for PR 26197 at commit 
[`34852e6`](https://github.com/apache/spark/commit/34852e62ecafb3573d4622ffabaab61fdcd253c0).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26190: [SPARK-29532][SQL] simplify interval string parsing

2019-10-23 Thread GitBox
dongjoon-hyun commented on a change in pull request #26190: [SPARK-29532][SQL] 
simplify interval string parsing
URL: https://github.com/apache/spark/pull/26190#discussion_r338206250
 
 

 ##
 File path: sql/core/benchmarks/IntervalBenchmark-results.txt
 ##
 @@ -1,25 +1,25 @@
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.15
-Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_161-b12 on Mac OS X 10.14
+Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
 cast strings to intervals:Best Time(ms)   Avg Time(ms)   
Stdev(ms)Rate(M/s)   Per Row(ns)   Relative
 

-string w/ interval  420435 
 18  2.4 419.8   1.0X
-string w/o interval 359365 
 10  2.8 358.7   1.2X
-1 units w/ interval 752759 
  8  1.3 752.0   0.6X
-1 units w/o interval762766 
  4  1.3 762.0   0.6X
-2 units w/ interval 961970 
  8  1.0 960.7   0.4X
-2 units w/o interval970976 
  9  1.0 970.2   0.4X
-3 units w/ interval1130   1136 
  7  0.91130.4   0.4X
-3 units w/o interval   1150   1158 
  9  0.91150.3   0.4X
-4 units w/ interval1333   1336 
  3  0.71333.5   0.3X
-4 units w/o interval   1354   1359 
  4  0.71354.5   0.3X
-5 units w/ interval1523   1525 
  2  0.71523.3   0.3X
-5 units w/o interval   1549   1551 
  3  0.61549.4   0.3X
-6 units w/ interval1661   1663 
  2  0.61660.8   0.3X
-6 units w/o interval   1691   1704 
 13  0.61691.2   0.2X
-7 units w/ interval1811   1817 
  8  0.61810.6   0.2X
-7 units w/o interval   1853   1854 
  1  0.51853.2   0.2X
-8 units w/ interval2029   2037 
  8  0.52028.7   0.2X
-8 units w/o interval   2075   2075 
  1  0.52074.5   0.2X
-9 units w/ interval2170   2175 
  5  0.52170.0   0.2X
-9 units w/o interval   2204   2212 
  8  0.52203.6   0.2X
+prepare string w/ interval  403419 
 18  2.5 403.1   1.0X
+prepare string w/o interval 341353 
 21  2.9 341.1   1.2X
+1 units w/ interval5154   5159 
  8  0.25153.5   0.1X
+1 units w/o interval   4818   4833 
 20  0.24817.6   0.1X
+2 units w/ interval6191   6223 
 41  0.26190.6   0.1X
+2 units w/o interval   6236   6264 
 25  0.26235.7   0.1X
+3 units w/ interval7397   7567 
170  0.17397.0   0.1X
+3 units w/o interval   7280   7367 
 76  0.17279.6   0.1X
+4 units w/ interval8197   8228 
 27  0.18197.3   0.0X
+4 units w/o interval   7977   7989 
 17  0.17977.3   0.1X
+5 units w/ interval9089   9192 
101  0.19088.8   0.0X
+5 units w/o interval   8853   8858 
  5  0.18852.8   0.0X
+6 units w/ interval9696   9720 
 23  0.19695.6   0.0X
+6 units w/o interval   9509   9518 
  9  0.19509.4   0.0X
+7 units w/ interval   10738  

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26190: [SPARK-29532][SQL] simplify interval string parsing

2019-10-23 Thread GitBox
dongjoon-hyun commented on a change in pull request #26190: [SPARK-29532][SQL] 
simplify interval string parsing
URL: https://github.com/apache/spark/pull/26190#discussion_r338205376
 
 

 ##
 File path: sql/core/benchmarks/IntervalBenchmark-results.txt
 ##
 @@ -1,25 +1,25 @@
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.15
-Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
 
 Review comment:
   Oh, the original benchmark result is on MacOS.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] gengliangwang commented on issue #26222: [SPARK-29504][WebUI] Toggle full job description on click

2019-10-23 Thread GitBox
gengliangwang commented on issue #26222: [SPARK-29504][WebUI] Toggle full job 
description on click
URL: https://github.com/apache/spark/pull/26222#issuecomment-545569595
 
 
   @PavithraRamachandran I have updated the PR title and description for you.
   If my modification is OK to you, I will merge this one.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] dongjoon-hyun commented on issue #25641: [SPARK-28921][BUILD][K8S][2.4] Update kubernetes client to 4.4.2

2019-10-23 Thread GitBox
dongjoon-hyun commented on issue #25641: [SPARK-28921][BUILD][K8S][2.4] Update 
kubernetes client to 4.4.2
URL: https://github.com/apache/spark/pull/25641#issuecomment-545566271
 
 
   Until now, there is no plan. However, according to 
https://spark.apache.org/news, we can guess like the following.
   - 2.4.3: May 8, 2019
   - 2.4.4: September 1, 2019
   - 2.4.5: Jan 2020? (This is reasonable. At least, I can volunteer for the 
release manager on January.)
   
   3.0.0 RC1 also have the similar ETA 
(https://spark.apache.org/versioning-policy.html). I believe you will have 
3.0.0 and 2.4.5 together at least early 2020.
   
   BTW, `K8s` dev cycle is fast. Although Apache Spark 2.4.5 will get the 
latest version of `K8s client` at that time instead of `4.6.1`, that is just a 
best effort like `2.4.4`. For the missed things, Apache Spark 2.4.6 will catch 
up later.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] brkyvz commented on a change in pull request #25955: [SPARK-29277][SQL] Add early DSv2 filter and projection pushdown

2019-10-23 Thread GitBox
brkyvz commented on a change in pull request #25955: [SPARK-29277][SQL] Add 
early DSv2 filter and projection pushdown
URL: https://github.com/apache/spark/pull/25955#discussion_r338196463
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
 ##
 @@ -672,19 +672,23 @@ class Analyzer(
*/
   object ResolveTables extends Rule[LogicalPlan] {
 def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsUp {
-  case u: UnresolvedRelation =>
-lookupV2Relation(u.multipartIdentifier)
-  .getOrElse(u)
+  case u: UnresolvedV2Relation =>
+CatalogV2Util.loadRelation(u.catalog, u.tableName).getOrElse(u)
 
-  case i @ InsertIntoStatement(u: UnresolvedRelation, _, _, _, _) if 
i.query.resolved =>
-lookupV2Relation(u.multipartIdentifier)
+  case i @ InsertIntoStatement(u: UnresolvedV2Relation, _, _, _, _) if 
i.query.resolved =>
+CatalogV2Util.loadRelation(u.catalog, u.tableName)
   .map(v2Relation => i.copy(table = v2Relation))
   .getOrElse(i)
 
-  case u: UnresolvedV2Relation =>
-CatalogV2Util.loadTable(u.catalog, u.tableName).map { table =>
-  DataSourceV2Relation.create(table)
-}.getOrElse(u)
+  case desc @ DescribeTable(u: UnresolvedV2Relation, _) =>
 
 Review comment:
   Thanks for incorporating these!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] brkyvz commented on issue #26214: [SPARK-29558][SQL] ResolveTables and ResolveRelations should be order-insensitive

2019-10-23 Thread GitBox
brkyvz commented on issue #26214: [SPARK-29558][SQL] ResolveTables and 
ResolveRelations should be order-insensitive
URL: https://github.com/apache/spark/pull/26214#issuecomment-545564432
 
 
   While code unification is nice to have, I think we've been purposefully 
trying to keep v1 code paths and v2 code paths separate, to make it a lot 
easier in the future to potentially delete the v1 parts. I think this change 
could make that a bit harder? What do you think?
   
   If we're worried about ordering of rules in the Analyzer, we can ensure that 
`ResolveRelations` always calls `ResolveTables` first, by just calling the 
`ResolveTables.apply` method within `ResolveRelations`, and remove 
`ResolveTables` from the resolution rules in the Analyzer. But it would still 
maintain that clean separation. Do you think that's possible?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] gengliangwang commented on issue #26222: [SPARK-29504][WebUI]show full decription when .click on ellipse

2019-10-23 Thread GitBox
gengliangwang commented on issue #26222: [SPARK-29504][WebUI]show full 
decription when  .click on ellipse
URL: https://github.com/apache/spark/pull/26222#issuecomment-545564250
 
 
   @PavithraRamachandran Sorry my browser loaded the cached script. I try `cmd 
+ shift + R` on my MacBook and it works.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #26231: [SPARK-29572][SQL] add v1 read fallback API in DS v2

2019-10-23 Thread GitBox
AmplabJenkins removed a comment on issue #26231: [SPARK-29572][SQL] add v1 read 
fallback API in DS v2
URL: https://github.com/apache/spark/pull/26231#issuecomment-545562829
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112548/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #26231: [SPARK-29572][SQL] add v1 read fallback API in DS v2

2019-10-23 Thread GitBox
AmplabJenkins removed a comment on issue #26231: [SPARK-29572][SQL] add v1 read 
fallback API in DS v2
URL: https://github.com/apache/spark/pull/26231#issuecomment-545562818
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #26231: [SPARK-29572][SQL] add v1 read fallback API in DS v2

2019-10-23 Thread GitBox
AmplabJenkins commented on issue #26231: [SPARK-29572][SQL] add v1 read 
fallback API in DS v2
URL: https://github.com/apache/spark/pull/26231#issuecomment-545562818
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] brkyvz commented on a change in pull request #26214: [SPARK-29558][SQL] ResolveTables and ResolveRelations should be order-insensitive

2019-10-23 Thread GitBox
brkyvz commented on a change in pull request #26214: [SPARK-29558][SQL] 
ResolveTables and ResolveRelations should be order-insensitive
URL: https://github.com/apache/spark/pull/26214#discussion_r338193572
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
 ##
 @@ -720,13 +712,9 @@ class Analyzer(
 // Note this is compatible with the views defined by older versions of 
Spark(before 2.2), which
 // have empty defaultDatabase and all the relations in viewText have 
database part defined.
 def resolveRelation(plan: LogicalPlan): LogicalPlan = plan match {
-  case u @ UnresolvedRelation(AsTemporaryViewIdentifier(ident))
 
 Review comment:
   Wouldn't it be simpler to just call `ResolveTables.apply(plan) match {` here 
than to embed all the logic within `lookupTableFromCatalog`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA removed a comment on issue #26231: [SPARK-29572][SQL] add v1 read fallback API in DS v2

2019-10-23 Thread GitBox
SparkQA removed a comment on issue #26231: [SPARK-29572][SQL] add v1 read 
fallback API in DS v2
URL: https://github.com/apache/spark/pull/26231#issuecomment-545490379
 
 
   **[Test build #112548 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112548/testReport)**
 for PR 26231 at commit 
[`034bc07`](https://github.com/apache/spark/commit/034bc07e34914c3ac9b70f34f233f0c562dfed6e).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #26231: [SPARK-29572][SQL] add v1 read fallback API in DS v2

2019-10-23 Thread GitBox
SparkQA commented on issue #26231: [SPARK-29572][SQL] add v1 read fallback API 
in DS v2
URL: https://github.com/apache/spark/pull/26231#issuecomment-545562504
 
 
   **[Test build #112548 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/112548/testReport)**
 for PR 26231 at commit 
[`034bc07`](https://github.com/apache/spark/commit/034bc07e34914c3ac9b70f34f233f0c562dfed6e).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
 * `trait V1Scan extends Scan `


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #26231: [SPARK-29572][SQL] add v1 read fallback API in DS v2

2019-10-23 Thread GitBox
AmplabJenkins commented on issue #26231: [SPARK-29572][SQL] add v1 read 
fallback API in DS v2
URL: https://github.com/apache/spark/pull/26231#issuecomment-545562829
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112548/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] shivsood commented on issue #25344: [SPARK-28151][SQL] Mapped ByteType to TinyINT for MsSQLServerDialect

2019-10-23 Thread GitBox
shivsood commented on issue #25344: [SPARK-28151][SQL] Mapped ByteType to 
TinyINT for MsSQLServerDialect
URL: https://github.com/apache/spark/pull/25344#issuecomment-545562768
 
 
   > @shivsood ping, are you there?
   
   @maropu Yes, Would handle this this week. Please note that ShortType issue 
is fixed SPARK-28152. Or are you pointing to a different issue?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #17280: [SPARK-19939] [ML] Add support for association rules in ML

2019-10-23 Thread GitBox
AmplabJenkins removed a comment on issue #17280: [SPARK-19939] [ML] Add support 
for association rules in ML
URL: https://github.com/apache/spark/pull/17280#issuecomment-545561963
 
 
   Build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #17280: [SPARK-19939] [ML] Add support for association rules in ML

2019-10-23 Thread GitBox
AmplabJenkins removed a comment on issue #17280: [SPARK-19939] [ML] Add support 
for association rules in ML
URL: https://github.com/apache/spark/pull/17280#issuecomment-545561972
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17521/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #17280: [SPARK-19939] [ML] Add support for association rules in ML

2019-10-23 Thread GitBox
AmplabJenkins commented on issue #17280: [SPARK-19939] [ML] Add support for 
association rules in ML
URL: https://github.com/apache/spark/pull/17280#issuecomment-545561972
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17521/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #17280: [SPARK-19939] [ML] Add support for association rules in ML

2019-10-23 Thread GitBox
AmplabJenkins commented on issue #17280: [SPARK-19939] [ML] Add support for 
association rules in ML
URL: https://github.com/apache/spark/pull/17280#issuecomment-545561963
 
 
   Build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] gatorsmile edited a comment on issue #26201: [SPARK-29543][SS][UI] Init structured streaming ui

2019-10-23 Thread GitBox
gatorsmile edited a comment on issue #26201: [SPARK-29543][SS][UI] Init 
structured streaming ui
URL: https://github.com/apache/spark/pull/26201#issuecomment-545561088
 
 
   cc @gengliangwang @sarutak @xuanyuanking @zsxwing 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] dilipbiswal commented on issue #25854: [SPARK-29145][SQL] Spark SQL cannot handle "NOT IN" condition when using "JOIN"

2019-10-23 Thread GitBox
dilipbiswal commented on issue #25854: [SPARK-29145][SQL] Spark SQL cannot 
handle "NOT IN" condition when using "JOIN"  
URL: https://github.com/apache/spark/pull/25854#issuecomment-545561612
 
 
   @AngersZh Great.. Thanks a lot for adding the UTs. Looks good to me.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] gatorsmile commented on issue #26201: [SPARK-29543][SS][UI] Init structured streaming ui

2019-10-23 Thread GitBox
gatorsmile commented on issue #26201: [SPARK-29543][SS][UI] Init structured 
streaming ui
URL: https://github.com/apache/spark/pull/26201#issuecomment-545561088
 
 
   cc @gengliangwang @sarutak 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #26216: [SPARK-29453][WEBUI] Improve tooltips information for SQL tab.

2019-10-23 Thread GitBox
AmplabJenkins removed a comment on issue #26216: [SPARK-29453][WEBUI] Improve 
tooltips information for SQL tab.
URL: https://github.com/apache/spark/pull/26216#issuecomment-545558851
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17520/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #26216: [SPARK-29453][WEBUI] Improve tooltips information for SQL tab.

2019-10-23 Thread GitBox
AmplabJenkins commented on issue #26216: [SPARK-29453][WEBUI] Improve tooltips 
information for SQL tab.
URL: https://github.com/apache/spark/pull/26216#issuecomment-545558851
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17520/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



  1   2   3   4   5   6   7   8   9   10   >