[GitHub] cloud-fan commented on a change in pull request #23353: [SPARK-26402][SQL] Canonicalization on GetStructField
cloud-fan commented on a change in pull request #23353: [SPARK-26402][SQL]
Canonicalization on GetStructField
URL: https://github.com/apache/spark/pull/23353#discussion_r243179452
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CanonicalizeSuite.scala
##
@@ -50,4 +59,44 @@ class CanonicalizeSuite extends SparkFunSuite {
assert(range.where(arrays1).sameResult(range.where(arrays2)))
assert(!range.where(arrays1).sameResult(range.where(arrays3)))
}
+
+ test("SPARK-26402: GetStructField with different optional names are
semantically equal") {
+val expId = NamedExpression.newExprId
+val qualifier = Seq.empty[String]
+val structType = StructType(
+ StructField("a", StructType(StructField("b", IntegerType, false) ::
Nil), false) :: Nil)
+
+val fieldB1 = GetStructField(
+ AttributeReference("data1", structType, false)(expId, qualifier),
+ 0, Some("b1"))
+val fieldB2 = GetStructField(
+ AttributeReference("data2", structType, false)(expId, qualifier),
+ 0, Some("b2"))
+assert(fieldB1.semanticEquals(fieldB2))
+
+val fieldA1 = GetStructField(
+ GetStructField(
+AttributeReference("data1", structType, false)(expId, qualifier),
+0, Some("a1")),
+ 0, Some("b1"))
+val fieldA2 = GetStructField(
+ GetStructField(
+AttributeReference("data2", structType, false)(expId, qualifier),
+0, Some("a2")),
+ 0, Some("b2"))
+assert(fieldA1.semanticEquals(fieldA2))
+
+// End-to-end test case
+val testRelation = LocalRelation('a.int)
Review comment:
This is not a real end-to-end test...
How about add the following test to SQLQuerySuite?
```
sql("create table t (s struct) using json")
sql("select s.I from t group by s.i")
```
currently it fials with
```
org.apache.spark.sql.AnalysisException: expression 'default.t.`s`' is
neither present in the group by, nor is it an aggregate function
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] tdas commented on issue #23324: [SPARK-26267][SS]Retry when detecting incorrect offsets from Kafka
tdas commented on issue #23324: [SPARK-26267][SS]Retry when detecting incorrect offsets from Kafka URL: https://github.com/apache/spark/pull/23324#issuecomment-448899155 LGTM. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23351: [SPARK-26216][SQL][followup] use abstract class instead of trait for UserDefinedFunction
AmplabJenkins removed a comment on issue #23351: [SPARK-26216][SQL][followup] use abstract class instead of trait for UserDefinedFunction URL: https://github.com/apache/spark/pull/23351#issuecomment-448891655 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23351: [SPARK-26216][SQL][followup] use abstract class instead of trait for UserDefinedFunction
AmplabJenkins removed a comment on issue #23351: [SPARK-26216][SQL][followup] use abstract class instead of trait for UserDefinedFunction URL: https://github.com/apache/spark/pull/23351#issuecomment-448891661 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100329/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23351: [SPARK-26216][SQL][followup] use abstract class instead of trait for UserDefinedFunction
AmplabJenkins commented on issue #23351: [SPARK-26216][SQL][followup] use abstract class instead of trait for UserDefinedFunction URL: https://github.com/apache/spark/pull/23351#issuecomment-448891661 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100329/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23351: [SPARK-26216][SQL][followup] use abstract class instead of trait for UserDefinedFunction
AmplabJenkins commented on issue #23351: [SPARK-26216][SQL][followup] use abstract class instead of trait for UserDefinedFunction URL: https://github.com/apache/spark/pull/23351#issuecomment-448891655 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23351: [SPARK-26216][SQL][followup] use abstract class instead of trait for UserDefinedFunction
AmplabJenkins removed a comment on issue #23351: [SPARK-26216][SQL][followup] use abstract class instead of trait for UserDefinedFunction URL: https://github.com/apache/spark/pull/23351#issuecomment-448833958 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6295/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23351: [SPARK-26216][SQL][followup] use abstract class instead of trait for UserDefinedFunction
SparkQA removed a comment on issue #23351: [SPARK-26216][SQL][followup] use abstract class instead of trait for UserDefinedFunction URL: https://github.com/apache/spark/pull/23351#issuecomment-448833881 **[Test build #100329 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100329/testReport)** for PR 23351 at commit [`df00dbd`](https://github.com/apache/spark/commit/df00dbd7d7a4b728fac18f91ddcc21e5829ea6e4). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23351: [SPARK-26216][SQL][followup] use abstract class instead of trait for UserDefinedFunction
SparkQA commented on issue #23351: [SPARK-26216][SQL][followup] use abstract class instead of trait for UserDefinedFunction URL: https://github.com/apache/spark/pull/23351#issuecomment-448891218 **[Test build #100329 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100329/testReport)** for PR 23351 at commit [`df00dbd`](https://github.com/apache/spark/commit/df00dbd7d7a4b728fac18f91ddcc21e5829ea6e4). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `sealed abstract class UserDefinedFunction ` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data source if it is convertible
AmplabJenkins removed a comment on issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data source if it is convertible URL: https://github.com/apache/spark/pull/22514#issuecomment-448890580 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100330/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data source if it is convertible
AmplabJenkins commented on issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data source if it is convertible URL: https://github.com/apache/spark/pull/22514#issuecomment-448890578 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data source if it is convertible
AmplabJenkins removed a comment on issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data source if it is convertible URL: https://github.com/apache/spark/pull/22514#issuecomment-448890578 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data source if it is convertible
AmplabJenkins commented on issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data source if it is convertible URL: https://github.com/apache/spark/pull/22514#issuecomment-448890580 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100330/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data source if it is convertible
SparkQA removed a comment on issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data source if it is convertible URL: https://github.com/apache/spark/pull/22514#issuecomment-448838786 **[Test build #100330 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100330/testReport)** for PR 22514 at commit [`839a6ce`](https://github.com/apache/spark/commit/839a6ce1732fa37b5f8ec9afa2d51730fc6ca691). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data source if it is convertible
SparkQA commented on issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data source if it is convertible URL: https://github.com/apache/spark/pull/22514#issuecomment-448890147 **[Test build #100330 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100330/testReport)** for PR 22514 at commit [`839a6ce`](https://github.com/apache/spark/commit/839a6ce1732fa37b5f8ec9afa2d51730fc6ca691). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23336: [SPARK-26378][SQL] Restore performance of queries against wide CSV tables
SparkQA commented on issue #23336: [SPARK-26378][SQL] Restore performance of queries against wide CSV tables URL: https://github.com/apache/spark/pull/23336#issuecomment-44700 **[Test build #100332 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100332/testReport)** for PR 23336 at commit [`6dce8ec`](https://github.com/apache/spark/commit/6dce8ec11fefd30c22c2a5f53d3bad9e45fecac5). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] bersprockets commented on issue #23336: [SPARK-26378][SQL] Restore performance of queries against wide CSV tables
bersprockets commented on issue #23336: [SPARK-26378][SQL] Restore performance of queries against wide CSV tables URL: https://github.com/apache/spark/pull/23336#issuecomment-44521 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] jomach commented on issue #21033: [SPARK-19320][MESOS]allow specifying a hard limit on number of gpus required in each spark executor when running on mesos
jomach commented on issue #21033: [SPARK-19320][MESOS]allow specifying a hard limit on number of gpus required in each spark executor when running on mesos URL: https://github.com/apache/spark/pull/21033#issuecomment-448887982 @susanxhuynh What about only exposing a number of gpus per executor and only in MesosCoarseGrainedSchedulerBackend ? At least for us would solve our problem to use spark with gpus at scale This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] asfgit closed pull request #23308: [SPARK-26308][SQL] Avoid cast of decimals for ScalaUDF
asfgit closed pull request #23308: [SPARK-26308][SQL] Avoid cast of decimals
for ScalaUDF
URL: https://github.com/apache/spark/pull/23308
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
index 133fa119b7aa6..1706b3eece6d7 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
@@ -879,6 +879,37 @@ object TypeCoercion {
}
}
e.withNewChildren(children)
+
+ case udf: ScalaUDF if udf.inputTypes.nonEmpty =>
+val children = udf.children.zip(udf.inputTypes).map { case (in,
expected) =>
+ implicitCast(in, udfInputToCastType(in.dataType,
expected)).getOrElse(in)
+}
+udf.withNewChildren(children)
+}
+
+private def udfInputToCastType(input: DataType, expectedType: DataType):
DataType = {
+ (input, expectedType) match {
+// SPARK-26308: avoid casting to an arbitrary precision and scale for
decimals. Please note
+// that precision and scale cannot be inferred properly for a ScalaUDF
because, when it is
+// created, it is not bound to any column. So here the precision and
scale of the input
+// column is used.
+case (in: DecimalType, _: DecimalType) => in
+case (ArrayType(dtIn, _), ArrayType(dtExp, nullableExp)) =>
+ ArrayType(udfInputToCastType(dtIn, dtExp), nullableExp)
+case (MapType(keyDtIn, valueDtIn, _), MapType(keyDtExp, valueDtExp,
nullableExp)) =>
+ MapType(udfInputToCastType(keyDtIn, keyDtExp),
+udfInputToCastType(valueDtIn, valueDtExp),
+nullableExp)
+case (StructType(fieldsIn), StructType(fieldsExp)) =>
+ val fieldTypes =
+fieldsIn.map(_.dataType).zip(fieldsExp.map(_.dataType)).map { case
(dtIn, dtExp) =>
+ udfInputToCastType(dtIn, dtExp)
+}
+ StructType(fieldsExp.zip(fieldTypes).map { case (field, newDt) =>
+field.copy(dataType = newDt)
+ })
+case (_, other) => other
+ }
}
/**
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala
index fae90caebf96c..a23aaa3a0b3ef 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala
@@ -52,7 +52,7 @@ case class ScalaUDF(
udfName: Option[String] = None,
nullable: Boolean = true,
udfDeterministic: Boolean = true)
- extends Expression with ImplicitCastInputTypes with NonSQLExpression with
UserDefinedExpression {
+ extends Expression with NonSQLExpression with UserDefinedExpression {
// The constructor for SPARK 2.1 and 2.2
def this(
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/UDFSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/UDFSuite.scala
index 20dcefa7e3cad..a26d306cff6b5 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/UDFSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/UDFSuite.scala
@@ -17,6 +17,8 @@
package org.apache.spark.sql
+import java.math.BigDecimal
+
import org.apache.spark.sql.api.java._
import org.apache.spark.sql.catalyst.plans.logical.Project
import org.apache.spark.sql.execution.QueryExecution
@@ -26,7 +28,7 @@ import
org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationComm
import org.apache.spark.sql.functions.{lit, udf}
import org.apache.spark.sql.test.SharedSQLContext
import org.apache.spark.sql.test.SQLTestData._
-import org.apache.spark.sql.types.{DataTypes, DoubleType}
+import org.apache.spark.sql.types._
import org.apache.spark.sql.util.QueryExecutionListener
@@ -420,4 +422,32 @@ class UDFSuite extends QueryTest with SharedSQLContext {
checkAnswer(df, Seq(Row("null1x"), Row(null), Row("N3null")))
}
}
+
+ test("SPARK-26308: udf with decimal") {
+val df1 = spark.createDataFrame(
+ sparkContext.parallelize(Seq(Row(new
BigDecimal("20110002456556",
+ StructType(Seq(StructField("col1", DecimalType(30, 0)
+val udf1 = org.apache.spark.sql.functions.udf((value: BigDecimal) => {
+ if (value == null) null else value.toBigInteger.toString
+})
+checkAnswer(df1.select(udf1(df1.col("col1"))),
Seq(Row("20110002456556")))
+ }
+
+ test("SPARK-26308: udf with complex types of decimal")
[GitHub] cloud-fan commented on issue #23308: [SPARK-26308][SQL] Avoid cast of decimals for ScalaUDF
cloud-fan commented on issue #23308: [SPARK-26308][SQL] Avoid cast of decimals for ScalaUDF URL: https://github.com/apache/spark/pull/23308#issuecomment-448886346 thanks, merging to master! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB
AmplabJenkins removed a comment on issue #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB URL: https://github.com/apache/spark/pull/23243#issuecomment-448878588 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB
AmplabJenkins removed a comment on issue #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB URL: https://github.com/apache/spark/pull/23243#issuecomment-448878593 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6298/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB
SparkQA commented on issue #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB URL: https://github.com/apache/spark/pull/23243#issuecomment-448878592 **[Test build #100331 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100331/testReport)** for PR 23243 at commit [`d13a2f2`](https://github.com/apache/spark/commit/d13a2f2b4a720af4fd5f55f6b2352623c2e63839). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB
AmplabJenkins commented on issue #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB URL: https://github.com/apache/spark/pull/23243#issuecomment-448878588 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB
AmplabJenkins commented on issue #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB URL: https://github.com/apache/spark/pull/23243#issuecomment-448878593 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6298/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] asfgit closed pull request #23271: [SPARK-26318][SQL] Deprecate Row.merge
asfgit closed pull request #23271: [SPARK-26318][SQL] Deprecate Row.merge
URL: https://github.com/apache/spark/pull/23271
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/Row.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/Row.scala
index e12bf9616e2de..4f5af9ac80b10 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/Row.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/Row.scala
@@ -57,6 +57,7 @@ object Row {
/**
* Merge multiple rows into a single row, one after another.
*/
+ @deprecated("This method is deprecated and will be removed in future
versions.", "3.0.0")
def merge(rows: Row*): Row = {
// TODO: Improve the performance of this if used in performance critical
part.
new GenericRow(rows.flatMap(_.toSeq).toArray)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon commented on issue #23271: [SPARK-26318][SQL] Deprecate Row.merge
HyukjinKwon commented on issue #23271: [SPARK-26318][SQL] Deprecate Row.merge URL: https://github.com/apache/spark/pull/23271#issuecomment-448877783 Merged to master. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23340: [SPARK-23431][CORE] Expose the new executor memory metrics at the stage level
AmplabJenkins removed a comment on issue #23340: [SPARK-23431][CORE] Expose the new executor memory metrics at the stage level URL: https://github.com/apache/spark/pull/23340#issuecomment-448877272 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100324/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23340: [SPARK-23431][CORE] Expose the new executor memory metrics at the stage level
AmplabJenkins commented on issue #23340: [SPARK-23431][CORE] Expose the new executor memory metrics at the stage level URL: https://github.com/apache/spark/pull/23340#issuecomment-448877272 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100324/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23340: [SPARK-23431][CORE] Expose the new executor memory metrics at the stage level
AmplabJenkins removed a comment on issue #23340: [SPARK-23431][CORE] Expose the new executor memory metrics at the stage level URL: https://github.com/apache/spark/pull/23340#issuecomment-448877267 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23340: [SPARK-23431][CORE] Expose the new executor memory metrics at the stage level
AmplabJenkins commented on issue #23340: [SPARK-23431][CORE] Expose the new executor memory metrics at the stage level URL: https://github.com/apache/spark/pull/23340#issuecomment-448877267 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] weixiuli commented on a change in pull request #23243: [SPARK-26288][CORE]add initRegisteredExecutorsDB
weixiuli commented on a change in pull request #23243: [SPARK-26288][CORE]add
initRegisteredExecutorsDB
URL: https://github.com/apache/spark/pull/23243#discussion_r243158391
##
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##
@@ -182,6 +182,12 @@ package object config {
private[spark] val SHUFFLE_SERVICE_ENABLED =
ConfigBuilder("spark.shuffle.service.enabled").booleanConf.createWithDefault(false)
+ private[spark] val SHUFFLE_SERVICE_DB_ENABLED =
+ConfigBuilder("spark.shuffle.service.db.enabled")
Review comment:
Hi，@rxin，There is no reason for that，I also think it's better left on by
default and I will fix it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23340: [SPARK-23431][CORE] Expose the new executor memory metrics at the stage level
SparkQA removed a comment on issue #23340: [SPARK-23431][CORE] Expose the new executor memory metrics at the stage level URL: https://github.com/apache/spark/pull/23340#issuecomment-448810261 **[Test build #100324 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100324/testReport)** for PR 23340 at commit [`3dd8424`](https://github.com/apache/spark/commit/3dd842494621288707b566ba8bf8984dd4a0f0ce). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23340: [SPARK-23431][CORE] Expose the new executor memory metrics at the stage level
SparkQA commented on issue #23340: [SPARK-23431][CORE] Expose the new executor memory metrics at the stage level URL: https://github.com/apache/spark/pull/23340#issuecomment-448876995 **[Test build #100324 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100324/testReport)** for PR 23340 at commit [`3dd8424`](https://github.com/apache/spark/commit/3dd842494621288707b566ba8bf8984dd4a0f0ce). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField
AmplabJenkins removed a comment on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField URL: https://github.com/apache/spark/pull/23353#issuecomment-448876805 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField
AmplabJenkins removed a comment on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField URL: https://github.com/apache/spark/pull/23353#issuecomment-448876806 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100325/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField
AmplabJenkins commented on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField URL: https://github.com/apache/spark/pull/23353#issuecomment-448876806 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100325/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField
AmplabJenkins commented on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField URL: https://github.com/apache/spark/pull/23353#issuecomment-448876805 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField
SparkQA removed a comment on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField URL: https://github.com/apache/spark/pull/23353#issuecomment-448820695 **[Test build #100325 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100325/testReport)** for PR 23353 at commit [`5f1cc66`](https://github.com/apache/spark/commit/5f1cc661d9b08adb8481c47afd473dee9180af15). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField
SparkQA commented on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField URL: https://github.com/apache/spark/pull/23353#issuecomment-448876565 **[Test build #100325 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100325/testReport)** for PR 23353 at commit [`5f1cc66`](https://github.com/apache/spark/commit/5f1cc661d9b08adb8481c47afd473dee9180af15). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField
AmplabJenkins removed a comment on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField URL: https://github.com/apache/spark/pull/23353#issuecomment-448876014 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100326/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField
AmplabJenkins commented on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField URL: https://github.com/apache/spark/pull/23353#issuecomment-448876014 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100326/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField
AmplabJenkins removed a comment on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField URL: https://github.com/apache/spark/pull/23353#issuecomment-448876011 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField
AmplabJenkins commented on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField URL: https://github.com/apache/spark/pull/23353#issuecomment-448876011 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField
SparkQA removed a comment on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField URL: https://github.com/apache/spark/pull/23353#issuecomment-448824272 **[Test build #100326 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100326/testReport)** for PR 23353 at commit [`a22d13e`](https://github.com/apache/spark/commit/a22d13ec4f4bc09c2d98c0139a82e807a559155b). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField
SparkQA commented on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField URL: https://github.com/apache/spark/pull/23353#issuecomment-448875603 **[Test build #100326 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100326/testReport)** for PR 23353 at commit [`a22d13e`](https://github.com/apache/spark/commit/a22d13ec4f4bc09c2d98c0139a82e807a559155b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23336: [SPARK-26378][SQL] Restore performance of queries against wide CSV tables
AmplabJenkins removed a comment on issue #23336: [SPARK-26378][SQL] Restore performance of queries against wide CSV tables URL: https://github.com/apache/spark/pull/23336#issuecomment-448869732 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100327/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23336: [SPARK-26378][SQL] Restore performance of queries against wide CSV tables
AmplabJenkins commented on issue #23336: [SPARK-26378][SQL] Restore performance of queries against wide CSV tables URL: https://github.com/apache/spark/pull/23336#issuecomment-448869727 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23336: [SPARK-26378][SQL] Restore performance of queries against wide CSV tables
SparkQA removed a comment on issue #23336: [SPARK-26378][SQL] Restore performance of queries against wide CSV tables URL: https://github.com/apache/spark/pull/23336#issuecomment-448826971 **[Test build #100327 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100327/testReport)** for PR 23336 at commit [`6dce8ec`](https://github.com/apache/spark/commit/6dce8ec11fefd30c22c2a5f53d3bad9e45fecac5). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23336: [SPARK-26378][SQL] Restore performance of queries against wide CSV tables
AmplabJenkins commented on issue #23336: [SPARK-26378][SQL] Restore performance of queries against wide CSV tables URL: https://github.com/apache/spark/pull/23336#issuecomment-448869732 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100327/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23336: [SPARK-26378][SQL] Restore performance of queries against wide CSV tables
AmplabJenkins removed a comment on issue #23336: [SPARK-26378][SQL] Restore performance of queries against wide CSV tables URL: https://github.com/apache/spark/pull/23336#issuecomment-448869727 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23336: [SPARK-26378][SQL] Restore performance of queries against wide CSV tables
SparkQA commented on issue #23336: [SPARK-26378][SQL] Restore performance of queries against wide CSV tables URL: https://github.com/apache/spark/pull/23336#issuecomment-448869262 **[Test build #100327 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100327/testReport)** for PR 23336 at commit [`6dce8ec`](https://github.com/apache/spark/commit/6dce8ec11fefd30c22c2a5f53d3bad9e45fecac5). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon commented on a change in pull request #23352: [SPARK-26409][SQL][TESTS] SQLConf should be serializable in test sessions
HyukjinKwon commented on a change in pull request #23352:
[SPARK-26409][SQL][TESTS] SQLConf should be serializable in test sessions
URL: https://github.com/apache/spark/pull/23352#discussion_r243151945
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/SerializationSuite.scala
##
@@ -27,4 +27,9 @@ class SerializationSuite extends SparkFunSuite with
SharedSQLContext {
val spark = SparkSession.builder.getOrCreate()
new JavaSerializer(new
SparkConf()).newInstance().serialize(spark.sqlContext)
}
+
+ test("[SPARK-26409] SQLConf should be serializable") {
Review comment:
Yea, it should be okay as is. Honestly I want to to reformat all of them
tho, it's obviously separate topic to discuss across the codebase.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon commented on issue #23349: [SPARK-26403][SQL] Support pivoting using array column for `pivot(column)` API
HyukjinKwon commented on issue #23349: [SPARK-26403][SQL] Support pivoting using array column for `pivot(column)` API URL: https://github.com/apache/spark/pull/23349#issuecomment-448860344 BTW, if we give the `values` by `pivot(column, values)` API, it works as described in JIRA. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] bersprockets commented on issue #23336: [SPARK-26378][SQL] Restore performance of queries against wide CSV tables
bersprockets commented on issue #23336: [SPARK-26378][SQL] Restore performance of queries against wide CSV tables URL: https://github.com/apache/spark/pull/23336#issuecomment-448855727 @rxin I added the link to the description. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] rxin commented on issue #23336: [SPARK-26378][SQL] Restore performance of queries against wide CSV tables
rxin commented on issue #23336: [SPARK-26378][SQL] Restore performance of queries against wide CSV tables URL: https://github.com/apache/spark/pull/23336#issuecomment-448852774 Do you have a link to the PR that caused the regression? We should call it out in your PR description. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] gengliangwang commented on a change in pull request #23352: [SPARK-26409][SQL][TESTS] SQLConf should be serializable in test sessions
gengliangwang commented on a change in pull request #23352:
[SPARK-26409][SQL][TESTS] SQLConf should be serializable in test sessions
URL: https://github.com/apache/spark/pull/23352#discussion_r243148690
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/internal/BaseSessionStateBuilder.scala
##
@@ -309,13 +309,14 @@ private[sql] trait WithTestConf { self:
BaseSessionStateBuilder =>
def overrideConfs: Map[String, String]
override protected lazy val conf: SQLConf = {
+val overrideConfigurations = overrideConfs
val conf = parentState.map(_.conf.clone()).getOrElse {
new SQLConf {
clear()
override def clear(): Unit = {
super.clear()
// Make sure we start with the default test configs even after clear
- overrideConfs.foreach { case (key, value) => setConfString(key,
value) }
+ overrideConfigurations.foreach { case (key, value) =>
setConfString(key, value) }
Review comment:
@cloud-fan agree.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] rxin edited a comment on issue #23208: [SPARK-25530][SQL] data source v2 API refactor (batch write)
rxin edited a comment on issue #23208: [SPARK-25530][SQL] data source v2 API refactor (batch write) URL: https://github.com/apache/spark/pull/23208#issuecomment-448851290 Honestly speaking, while I feel the need to have a great API, I feel the same way as @cloud-fan. Things have been moving very slowly. If we question every single little detail, especially when one has no context, including designs that've been settled years ago (e.g. query id), we'd never make progress. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] gengliangwang commented on issue #23352: [SPARK-26409][SQL][TESTS] SQLConf should be serializable in test sessions
gengliangwang commented on issue #23352: [SPARK-26409][SQL][TESTS] SQLConf should be serializable in test sessions URL: https://github.com/apache/spark/pull/23352#issuecomment-448851550 @dongjoon-hyun I checked and the issue exists since 2.2.0. https://github.com/apache/spark/commit/ea361165e1ddce4d8aa0242ae3e878d7b39f1de2 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] rxin edited a comment on issue #23208: [SPARK-25530][SQL] data source v2 API refactor (batch write)
rxin edited a comment on issue #23208: [SPARK-25530][SQL] data source v2 API refactor (batch write) URL: https://github.com/apache/spark/pull/23208#issuecomment-448851290 Honestly speaking, while I feel the need to have a great API, I feel the same way as @cloud-fan. Things have been moving very slowly. If we question every single little detail, especially when one has no context, including ones that've been settled years ago (e.g. query id), we'd never make progress. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] rxin commented on issue #23208: [SPARK-25530][SQL] data source v2 API refactor (batch write)
rxin commented on issue #23208: [SPARK-25530][SQL] data source v2 API refactor (batch write) URL: https://github.com/apache/spark/pull/23208#issuecomment-448851290 Honestly speaking, while I feel the need to have a great API, I feel the same way as @cloud-fan. Things have been moving very slowly. If we question every single little detail, including ones that've been settled years ago (e.g. query id), we'd never make progress. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23337: [SPARK-26019][PYSPARK] Allow insecure py4j gateways
AmplabJenkins removed a comment on issue #23337: [SPARK-26019][PYSPARK] Allow insecure py4j gateways URL: https://github.com/apache/spark/pull/23337#issuecomment-448848002 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100320/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23337: [SPARK-26019][PYSPARK] Allow insecure py4j gateways
AmplabJenkins removed a comment on issue #23337: [SPARK-26019][PYSPARK] Allow insecure py4j gateways URL: https://github.com/apache/spark/pull/23337#issuecomment-448848000 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23337: [SPARK-26019][PYSPARK] Allow insecure py4j gateways
AmplabJenkins commented on issue #23337: [SPARK-26019][PYSPARK] Allow insecure py4j gateways URL: https://github.com/apache/spark/pull/23337#issuecomment-448848000 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23337: [SPARK-26019][PYSPARK] Allow insecure py4j gateways
AmplabJenkins commented on issue #23337: [SPARK-26019][PYSPARK] Allow insecure py4j gateways URL: https://github.com/apache/spark/pull/23337#issuecomment-448848002 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100320/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23337: [SPARK-26019][PYSPARK] Allow insecure py4j gateways
SparkQA removed a comment on issue #23337: [SPARK-26019][PYSPARK] Allow insecure py4j gateways URL: https://github.com/apache/spark/pull/23337#issuecomment-448772604 **[Test build #100320 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100320/testReport)** for PR 23337 at commit [`9cc545b`](https://github.com/apache/spark/commit/9cc545b278406fa57502af89bd4a4eb6a0ea2f04). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23337: [SPARK-26019][PYSPARK] Allow insecure py4j gateways
SparkQA commented on issue #23337: [SPARK-26019][PYSPARK] Allow insecure py4j gateways URL: https://github.com/apache/spark/pull/23337#issuecomment-448847583 **[Test build #100320 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100320/testReport)** for PR 23337 at commit [`9cc545b`](https://github.com/apache/spark/commit/9cc545b278406fa57502af89bd4a4eb6a0ea2f04). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] cloud-fan commented on a change in pull request #23352: [SPARK-26409][SQL][TESTS] SQLConf should be serializable in test sessions
cloud-fan commented on a change in pull request #23352:
[SPARK-26409][SQL][TESTS] SQLConf should be serializable in test sessions
URL: https://github.com/apache/spark/pull/23352#discussion_r243146082
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/internal/BaseSessionStateBuilder.scala
##
@@ -309,13 +309,14 @@ private[sql] trait WithTestConf { self:
BaseSessionStateBuilder =>
def overrideConfs: Map[String, String]
override protected lazy val conf: SQLConf = {
+val overrideConfigurations = overrideConfs
val conf = parentState.map(_.conf.clone()).getOrElse {
new SQLConf {
clear()
override def clear(): Unit = {
super.clear()
// Make sure we start with the default test configs even after clear
- overrideConfs.foreach { case (key, value) => setConfString(key,
value) }
+ overrideConfigurations.foreach { case (key, value) =>
setConfString(key, value) }
Review comment:
maybe we should move this trait to `src/test`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] rdblue commented on issue #23208: [SPARK-25530][SQL] data source v2 API refactor (batch write)
rdblue commented on issue #23208: [SPARK-25530][SQL] data source v2 API refactor (batch write) URL: https://github.com/apache/spark/pull/23208#issuecomment-448844351 I'm not suggesting we change the streaming design. But, where it influences the design of this API, I think we should consider all of our options. I also want to understand what those influences are. It wasn't obvious why a query ID would need to be passed in and it wasn't called out in the design doc. I naturally ask questions about things like that. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField
AmplabJenkins removed a comment on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField URL: https://github.com/apache/spark/pull/23353#issuecomment-448843984 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField
AmplabJenkins removed a comment on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField URL: https://github.com/apache/spark/pull/23353#issuecomment-448843989 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100321/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField
AmplabJenkins commented on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField URL: https://github.com/apache/spark/pull/23353#issuecomment-448843989 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100321/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField
AmplabJenkins commented on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField URL: https://github.com/apache/spark/pull/23353#issuecomment-448843984 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField
SparkQA removed a comment on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField URL: https://github.com/apache/spark/pull/23353#issuecomment-448784826 **[Test build #100321 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100321/testReport)** for PR 23353 at commit [`a5998bb`](https://github.com/apache/spark/commit/a5998bb251c51cd1b4c648de259e2607e8641db1). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField
SparkQA commented on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField URL: https://github.com/apache/spark/pull/23353#issuecomment-448843680 **[Test build #100321 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100321/testReport)** for PR 23353 at commit [`a5998bb`](https://github.com/apache/spark/commit/a5998bb251c51cd1b4c648de259e2607e8641db1). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class CanonicalizeSuite extends SparkFunSuite with ExpressionEvalHelper with PlanTest ` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] gengliangwang commented on a change in pull request #23352: [SPARK-26409][SQL][TESTS] SQLConf should be serializable in test sessions
gengliangwang commented on a change in pull request #23352:
[SPARK-26409][SQL][TESTS] SQLConf should be serializable in test sessions
URL: https://github.com/apache/spark/pull/23352#discussion_r243145326
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/SerializationSuite.scala
##
@@ -27,4 +27,9 @@ class SerializationSuite extends SparkFunSuite with
SharedSQLContext {
val spark = SparkSession.builder.getOrCreate()
new JavaSerializer(new
SparkConf()).newInstance().serialize(spark.sqlContext)
}
+
+ test("[SPARK-26409] SQLConf should be serializable") {
Review comment:
It is following the other test case in the test suite. I can change it to
either way. What do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField
AmplabJenkins removed a comment on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField URL: https://github.com/apache/spark/pull/23353#issuecomment-448842983 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100323/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField
AmplabJenkins removed a comment on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField URL: https://github.com/apache/spark/pull/23353#issuecomment-448842979 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField
AmplabJenkins commented on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField URL: https://github.com/apache/spark/pull/23353#issuecomment-448842979 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField
AmplabJenkins commented on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField URL: https://github.com/apache/spark/pull/23353#issuecomment-448842983 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100323/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField
SparkQA removed a comment on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField URL: https://github.com/apache/spark/pull/23353#issuecomment-448793604 **[Test build #100323 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100323/testReport)** for PR 23353 at commit [`4f21a36`](https://github.com/apache/spark/commit/4f21a369a639c0ccb337fa7dd4f5f62e7a61a102). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField
SparkQA commented on issue #23353: [SPARK-26402][SQL] Canonicalization on GetStructField URL: https://github.com/apache/spark/pull/23353#issuecomment-448842786 **[Test build #100323 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100323/testReport)** for PR 23353 at commit [`4f21a36`](https://github.com/apache/spark/commit/4f21a369a639c0ccb337fa7dd4f5f62e7a61a102). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23308: [SPARK-26308][SQL] Avoid cast of decimals for ScalaUDF
AmplabJenkins removed a comment on issue #23308: [SPARK-26308][SQL] Avoid cast of decimals for ScalaUDF URL: https://github.com/apache/spark/pull/23308#issuecomment-448842308 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23308: [SPARK-26308][SQL] Avoid cast of decimals for ScalaUDF
AmplabJenkins removed a comment on issue #23308: [SPARK-26308][SQL] Avoid cast of decimals for ScalaUDF URL: https://github.com/apache/spark/pull/23308#issuecomment-448842311 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100322/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23308: [SPARK-26308][SQL] Avoid cast of decimals for ScalaUDF
AmplabJenkins commented on issue #23308: [SPARK-26308][SQL] Avoid cast of decimals for ScalaUDF URL: https://github.com/apache/spark/pull/23308#issuecomment-448842311 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100322/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23308: [SPARK-26308][SQL] Avoid cast of decimals for ScalaUDF
AmplabJenkins commented on issue #23308: [SPARK-26308][SQL] Avoid cast of decimals for ScalaUDF URL: https://github.com/apache/spark/pull/23308#issuecomment-448842308 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23308: [SPARK-26308][SQL] Avoid cast of decimals for ScalaUDF
SparkQA removed a comment on issue #23308: [SPARK-26308][SQL] Avoid cast of decimals for ScalaUDF URL: https://github.com/apache/spark/pull/23308#issuecomment-448786636 **[Test build #100322 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100322/testReport)** for PR 23308 at commit [`50b1499`](https://github.com/apache/spark/commit/50b1499c50809bfdbb2b419ff54738cf7837b1a3). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23308: [SPARK-26308][SQL] Avoid cast of decimals for ScalaUDF
SparkQA commented on issue #23308: [SPARK-26308][SQL] Avoid cast of decimals for ScalaUDF URL: https://github.com/apache/spark/pull/23308#issuecomment-448841927 **[Test build #100322 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100322/testReport)** for PR 23308 at commit [`50b1499`](https://github.com/apache/spark/commit/50b1499c50809bfdbb2b419ff54738cf7837b1a3). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] asfgit closed pull request #22514: [SPARK-25271][SQL] Hive ctas commands should use data source if it is convertible
asfgit closed pull request #22514: [SPARK-25271][SQL] Hive ctas commands should
use data source if it is convertible
URL: https://github.com/apache/spark/pull/22514
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala
index e1faecedd20ed..096481f68275d 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala
@@ -820,6 +820,14 @@ object DDLUtils {
table.provider.isDefined && table.provider.get.toLowerCase(Locale.ROOT) !=
HIVE_PROVIDER
}
+ def readHiveTable(table: CatalogTable): HiveTableRelation = {
+HiveTableRelation(
+ table,
+ // Hive table columns are always nullable.
+ table.dataSchema.asNullable.toAttributes,
+ table.partitionSchema.asNullable.toAttributes)
+ }
+
/**
* Throws a standard error for actions that require partitionProvider = hive.
*/
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
index b304e2da6e1cf..b5cf8c9515bfb 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
@@ -244,27 +244,19 @@ class FindDataSourceTable(sparkSession: SparkSession)
extends Rule[LogicalPlan]
})
}
- private def readHiveTable(table: CatalogTable): LogicalPlan = {
-HiveTableRelation(
- table,
- // Hive table columns are always nullable.
- table.dataSchema.asNullable.toAttributes,
- table.partitionSchema.asNullable.toAttributes)
- }
-
override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
case i @ InsertIntoTable(UnresolvedCatalogRelation(tableMeta), _, _, _, _)
if DDLUtils.isDatasourceTable(tableMeta) =>
i.copy(table = readDataSourceTable(tableMeta))
case i @ InsertIntoTable(UnresolvedCatalogRelation(tableMeta), _, _, _, _)
=>
- i.copy(table = readHiveTable(tableMeta))
+ i.copy(table = DDLUtils.readHiveTable(tableMeta))
case UnresolvedCatalogRelation(tableMeta) if
DDLUtils.isDatasourceTable(tableMeta) =>
readDataSourceTable(tableMeta)
case UnresolvedCatalogRelation(tableMeta) =>
- readHiveTable(tableMeta)
+ DDLUtils.readHiveTable(tableMeta)
}
}
diff --git
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala
index 5823548a8063c..03f4b8d83e353 100644
---
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala
+++
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala
@@ -17,6 +17,8 @@
package org.apache.spark.sql.hive
+import java.util.Locale
+
import scala.util.control.NonFatal
import com.google.common.util.concurrent.Striped
@@ -29,6 +31,8 @@ import org.apache.spark.sql.catalyst.{QualifiedTableName,
TableIdentifier}
import org.apache.spark.sql.catalyst.catalog._
import org.apache.spark.sql.catalyst.plans.logical._
import org.apache.spark.sql.execution.datasources._
+import org.apache.spark.sql.execution.datasources.parquet.{ParquetFileFormat,
ParquetOptions}
+import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.internal.SQLConf.HiveCaseSensitiveInferenceMode._
import org.apache.spark.sql.types._
@@ -113,7 +117,44 @@ private[hive] class HiveMetastoreCatalog(sparkSession:
SparkSession) extends Log
}
}
- def convertToLogicalRelation(
+ // Return true for Apache ORC and Hive ORC-related configuration names.
+ // Note that Spark doesn't support configurations like
`hive.merge.orcfile.stripe.level`.
+ private def isOrcProperty(key: String) =
+key.startsWith("orc.") || key.contains(".orc.")
+
+ private def isParquetProperty(key: String) =
+key.startsWith("parquet.") || key.contains(".parquet.")
+
+ def convert(relation: HiveTableRelation): LogicalRelation = {
+val serde =
relation.tableMeta.storage.serde.getOrElse("").toLowerCase(Locale.ROOT)
+
+// Consider table and storage properties. For properties existing in both
sides, storage
+// properties will supersede table properties.
+if (serde.contains("parquet")) {
+ val options =
relation.tableMeta.properties.filterKeys(isParquetProperty) ++
+relation.tableMeta.storage.properties + (ParquetOptions.MERGE_SCHEMA ->
+
[GitHub] AmplabJenkins removed a comment on issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data source if it is convertible
AmplabJenkins removed a comment on issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data source if it is convertible URL: https://github.com/apache/spark/pull/22514#issuecomment-448839600 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data source if it is convertible
AmplabJenkins removed a comment on issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data source if it is convertible URL: https://github.com/apache/spark/pull/22514#issuecomment-448839606 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6297/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data source if it is convertible
AmplabJenkins commented on issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data source if it is convertible URL: https://github.com/apache/spark/pull/22514#issuecomment-448839600 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data source if it is convertible
AmplabJenkins commented on issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data source if it is convertible URL: https://github.com/apache/spark/pull/22514#issuecomment-448839606 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6297/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] cloud-fan commented on issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data source if it is convertible
cloud-fan commented on issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data source if it is convertible URL: https://github.com/apache/spark/pull/22514#issuecomment-448839376 The last commit is only updating comment, I'm merging it to master, thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] asfgit closed pull request #23213: [SPARK-26262][SQL] Runs SQLQueryTestSuite on mixed config sets: WHOLESTAGE_CODEGEN_ENABLED and CODEGEN_FACTORY_MODE
asfgit closed pull request #23213: [SPARK-26262][SQL] Runs SQLQueryTestSuite on
mixed config sets: WHOLESTAGE_CODEGEN_ENABLED and CODEGEN_FACTORY_MODE
URL: https://github.com/apache/spark/pull/23213
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/sql/core/src/test/resources/sql-tests/inputs/group-by.sql
b/sql/core/src/test/resources/sql-tests/inputs/group-by.sql
index ec263ea70bd4a..7e81ff1aba37b 100644
--- a/sql/core/src/test/resources/sql-tests/inputs/group-by.sql
+++ b/sql/core/src/test/resources/sql-tests/inputs/group-by.sql
@@ -141,8 +141,3 @@ SELECT every("true");
SELECT k, v, every(v) OVER (PARTITION BY k ORDER BY v) FROM test_agg;
SELECT k, v, some(v) OVER (PARTITION BY k ORDER BY v) FROM test_agg;
SELECT k, v, any(v) OVER (PARTITION BY k ORDER BY v) FROM test_agg;
-
--- simple explain of queries having every/some/any agregates. Optimized
--- plan should show the rewritten aggregate expression.
-EXPLAIN EXTENDED SELECT k, every(v), some(v), any(v) FROM test_agg GROUP BY k;
-
diff --git a/sql/core/src/test/resources/sql-tests/inputs/inline-table.sql
b/sql/core/src/test/resources/sql-tests/inputs/inline-table.sql
index 41d316444ed6b..b3ec956cd178e 100644
--- a/sql/core/src/test/resources/sql-tests/inputs/inline-table.sql
+++ b/sql/core/src/test/resources/sql-tests/inputs/inline-table.sql
@@ -49,6 +49,3 @@ select * from values ("one", count(1)), ("two", 2) as data(a,
b);
-- string to timestamp
select * from values (timestamp('1991-12-06 00:00:00.0'),
array(timestamp('1991-12-06 01:00:00.0'), timestamp('1991-12-06 12:00:00.0')))
as data(a, b);
-
--- cross-join inline tables
-EXPLAIN EXTENDED SELECT * FROM VALUES ('one', 1), ('three', null) CROSS JOIN
VALUES ('one', 1), ('three', null);
diff --git a/sql/core/src/test/resources/sql-tests/inputs/operators.sql
b/sql/core/src/test/resources/sql-tests/inputs/operators.sql
index 37f9cd44da7f2..ba14789d48db6 100644
--- a/sql/core/src/test/resources/sql-tests/inputs/operators.sql
+++ b/sql/core/src/test/resources/sql-tests/inputs/operators.sql
@@ -29,27 +29,6 @@ select 2 * 5;
select 5 % 3;
select pmod(-7, 3);
--- check operator precedence.
--- We follow Oracle operator precedence in the table below that lists the
levels of precedence
--- among SQL operators from high to low:
---
--- Operator Operation
---
--- +, - identity, negation
--- *, / multiplication, division
--- +, -, || addition, subtraction,
concatenation
--- =, !=, <, >, <=, >=, IS NULL, LIKE, BETWEEN, IN comparison
--- NOT exponentiation, logical
negation
--- AND conjunction
--- ORdisjunction
---
-explain select 'a' || 1 + 2;
-explain select 1 - 2 || 'b';
-explain select 2 * 4 + 3 || 'b';
-explain select 3 + 1 || 'a' || 4 / 2;
-explain select 1 == 1 OR 'a' || 'b' == 'ab';
-explain select 'a' || 'c' == 'ac' AND 2 == 3;
-
-- math functions
select cot(1);
select cot(null);
diff --git
a/sql/core/src/test/resources/sql-tests/inputs/sql-compatibility-functions.sql
b/sql/core/src/test/resources/sql-tests/inputs/sql-compatibility-functions.sql
index f1461032065ad..1ae49c8bfc76a 100644
---
a/sql/core/src/test/resources/sql-tests/inputs/sql-compatibility-functions.sql
+++
b/sql/core/src/test/resources/sql-tests/inputs/sql-compatibility-functions.sql
@@ -12,11 +12,6 @@ SELECT nullif(1, 2.1d), nullif(1, 1.0d);
SELECT nvl(1, 2.1d), nvl(null, 2.1d);
SELECT nvl2(null, 1, 2.1d), nvl2('n', 1, 2.1d);
--- explain for these functions; use range to avoid constant folding
-explain extended
-select ifnull(id, 'x'), nullif(id, 'x'), nvl(id, 'x'), nvl2(id, 'x', 'y')
-from range(2);
-
-- SPARK-16730 cast alias functions for Hive compatibility
SELECT boolean(1), tinyint(1), smallint(1), int(1), bigint(1);
SELECT float(1), double(1), decimal(1);
diff --git a/sql/core/src/test/resources/sql-tests/inputs/string-functions.sql
b/sql/core/src/test/resources/sql-tests/inputs/string-functions.sql
index 2effb43183d75..fbc231627e36f 100644
--- a/sql/core/src/test/resources/sql-tests/inputs/string-functions.sql
+++ b/sql/core/src/test/resources/sql-tests/inputs/string-functions.sql
@@ -5,10 +5,6 @@ select format_string();
-- A pipe operator for string concatenation
select 'a' || 'b' || 'c';
[GitHub] SparkQA commented on issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data source if it is convertible
SparkQA commented on issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data source if it is convertible URL: https://github.com/apache/spark/pull/22514#issuecomment-448838786 **[Test build #100330 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100330/testReport)** for PR 22514 at commit [`839a6ce`](https://github.com/apache/spark/commit/839a6ce1732fa37b5f8ec9afa2d51730fc6ca691). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] cloud-fan commented on issue #23213: [SPARK-26262][SQL] Runs SQLQueryTestSuite on mixed config sets: WHOLESTAGE_CODEGEN_ENABLED and CODEGEN_FACTORY_MODE
cloud-fan commented on issue #23213: [SPARK-26262][SQL] Runs SQLQueryTestSuite on mixed config sets: WHOLESTAGE_CODEGEN_ENABLED and CODEGEN_FACTORY_MODE URL: https://github.com/apache/spark/pull/23213#issuecomment-448838509 thanks, merging to master! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] cloud-fan commented on issue #23208: [SPARK-25530][SQL] data source v2 API refactor (batch write)
cloud-fan commented on issue #23208: [SPARK-25530][SQL] data source v2 API refactor (batch write) URL: https://github.com/apache/spark/pull/23208#issuecomment-448837287 > but it affects the API because it may change the structure of the builder or may be a reason to use a different pattern. According to the dicussion in the doc, seems you are challenging the current streaming engine in Spark. I don't think we should block the API refactor because of it. If we do need to change something fundamental in the streaming engine, that's a lot of work and it doesn't matter if the API refactor is done before or after it. > can you put the pseudocode for Spark's invocation sequence Sure, I'll follow the read side doc and put it in the write side doc This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23351: [SPARK-26216][SQL][followup] use abstract class instead of trait for UserDefinedFunction
AmplabJenkins removed a comment on issue #23351: [SPARK-26216][SQL][followup] use abstract class instead of trait for UserDefinedFunction URL: https://github.com/apache/spark/pull/23351#issuecomment-448834961 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23351: [SPARK-26216][SQL][followup] use abstract class instead of trait for UserDefinedFunction
AmplabJenkins removed a comment on issue #23351: [SPARK-26216][SQL][followup] use abstract class instead of trait for UserDefinedFunction URL: https://github.com/apache/spark/pull/23351#issuecomment-448834965 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6296/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
