[GitHub] spark issue #21226: [SPARK-24169][SQL] JsonToStructs should not access SQLCo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21226 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21226: [SPARK-24169][SQL] JsonToStructs should not access SQLCo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21226 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2857/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20973: [SPARK-20114][ML] spark.ml parity for sequential pattern...
Github user WeichenXu123 commented on the issue: https://github.com/apache/spark/pull/20973 Jenkins, test this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21226: [SPARK-24169][SQL] JsonToStructs should not access SQLCo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21226 **[Test build #90114 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90114/testReport)** for PR 21226 at commit [`e1491f0`](https://github.com/apache/spark/commit/e1491f0aeb62d1eda8cd8c55f890c8f87eec5761). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21190: [SPARK-22938][SQL][followup] Assert that SQLConf....
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21190#discussion_r185750124
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
---
@@ -102,22 +105,32 @@ object TypeCoercion {
case (_: TimestampType, _: DateType) | (_: DateType, _: TimestampType)
=>
Some(TimestampType)
-case (t1 @ StructType(fields1), t2 @ StructType(fields2)) if
t1.sameType(t2) =>
- Some(StructType(fields1.zip(fields2).map { case (f1, f2) =>
-// Since `t1.sameType(t2)` is true, two StructTypes have the same
DataType
-// except `name` (in case of `spark.sql.caseSensitive=false`) and
`nullable`.
-// - Different names: use f1.name
-// - Different nullabilities: `nullable` is true iff one of them
is nullable.
-val dataType = findTightestCommonType(f1.dataType, f2.dataType).get
-StructField(f1.name, dataType, nullable = f1.nullable ||
f2.nullable)
- }))
+case (t1 @ StructType(fields1), t2 @ StructType(fields2)) =>
+ val isSameType = if (caseSensitive) {
+t1.sameType(t2)
--- End diff --
Call `DataType.equalsIgnoreNullability` here for better show the difference
between the call of `DataType.equalsIgnoreCaseAndNullability` below?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21226: [SPARK-24169][SQL] JsonToStructs should not access SQLCo...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21226 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21225: [SPARK-24168][SQL] WindowExec should not access SQLConf ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21225 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2856/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21225: [SPARK-24168][SQL] WindowExec should not access SQLConf ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21225 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21225: [SPARK-24168][SQL] WindowExec should not access SQLConf ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21225 **[Test build #90113 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90113/testReport)** for PR 21225 at commit [`c5ab777`](https://github.com/apache/spark/commit/c5ab77710de805eab3a0a815790a58cd2de56cb2). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21050: [SPARK-23912][SQL]add array_distinct
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21050#discussion_r185731095
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
---
@@ -1059,3 +1059,96 @@ case class Flatten(child: Expression) extends
UnaryExpression {
override def prettyName: String = "flatten"
}
+
+/**
+ * Removes duplicate values from the array.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(array) - Removes duplicate values from the array.",
+ examples = """
+Examples:
+ > SELECT _FUNC_(array(1, 2, 3, null, 3));
+ [1,2,3,null]
+ """, since = "2.4.0")
+case class ArrayDistinct(child: Expression)
+ extends UnaryExpression with ExpectsInputTypes {
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(ArrayType)
+
+ override def dataType: DataType = child.dataType
+
+ override def nullSafeEval(array: Any): Any = {
+val elementType = child.dataType.asInstanceOf[ArrayType].elementType
+val data =
array.asInstanceOf[ArrayData].toArray[AnyRef](elementType).distinct
+new GenericArrayData(data.asInstanceOf[Array[Any]])
+ }
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+val elementType = dataType.asInstanceOf[ArrayType].elementType
+nullSafeCodeGen(ctx, ev, (array) => {
+ val arrayClass = classOf[GenericArrayData].getName
+ val distinctArray = ctx.freshName("distinctArray")
+ val i = ctx.freshName("i")
+ val j = ctx.freshName("j")
+ val pos = ctx.freshName("arrayPosition")
+ val getValue1 = CodeGenerator.getValue(array, elementType, i)
+ val getValue2 = CodeGenerator.getValue(array, elementType, j)
+ s"""
+ |int $pos = 0;
+ |for (int $i = 0; $i < $array.numElements(); $i ++) {
--- End diff --
I guess nested loop must be very slow. Can you use a hash table or
something to check duplication? Maybe `OpenHashSet`?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21050: [SPARK-23912][SQL]add array_distinct
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21050#discussion_r185739950
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
---
@@ -1059,3 +1059,96 @@ case class Flatten(child: Expression) extends
UnaryExpression {
override def prettyName: String = "flatten"
}
+
+/**
+ * Removes duplicate values from the array.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(array) - Removes duplicate values from the array.",
+ examples = """
+Examples:
+ > SELECT _FUNC_(array(1, 2, 3, null, 3));
+ [1,2,3,null]
+ """, since = "2.4.0")
+case class ArrayDistinct(child: Expression)
+ extends UnaryExpression with ExpectsInputTypes {
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(ArrayType)
+
+ override def dataType: DataType = child.dataType
+
+ override def nullSafeEval(array: Any): Any = {
+val elementType = child.dataType.asInstanceOf[ArrayType].elementType
+val data =
array.asInstanceOf[ArrayData].toArray[AnyRef](elementType).distinct
--- End diff --
Do we need to keep the element order? If so, does `distinct` keep the order?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21225: [SPARK-24168][SQL] WindowExec should not access SQLConf ...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21225 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21226: [SPARK-24169][SQL] JsonToStructs should not access SQLCo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21226 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90102/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21226: [SPARK-24169][SQL] JsonToStructs should not access SQLCo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21226 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21226: [SPARK-24169][SQL] JsonToStructs should not access SQLCo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21226 **[Test build #90102 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90102/testReport)** for PR 21226 at commit [`e1491f0`](https://github.com/apache/spark/commit/e1491f0aeb62d1eda8cd8c55f890c8f87eec5761). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21225: [SPARK-24168][SQL] WindowExec should not access SQLConf ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21225 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90103/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21225: [SPARK-24168][SQL] WindowExec should not access SQLConf ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21225 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21225: [SPARK-24168][SQL] WindowExec should not access SQLConf ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21225 **[Test build #90103 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90103/testReport)** for PR 21225 at commit [`c5ab777`](https://github.com/apache/spark/commit/c5ab77710de805eab3a0a815790a58cd2de56cb2). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21225: [SPARK-24168][SQL] WindowExec should not access SQLConf ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21225 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21225: [SPARK-24168][SQL] WindowExec should not access SQLConf ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21225 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90106/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21225: [SPARK-24168][SQL] WindowExec should not access SQLConf ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21225 **[Test build #90106 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90106/testReport)** for PR 21225 at commit [`c5ab777`](https://github.com/apache/spark/commit/c5ab77710de805eab3a0a815790a58cd2de56cb2). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21069: [SPARK-23920][SQL]add array_remove to remove all ...
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21069#discussion_r185726735
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala ---
@@ -691,6 +691,30 @@ class DataFrameFunctionsSuite extends QueryTest with
SharedSQLContext {
}
}
+ test("array remove") {
+val df = Seq(
+ (Array[Int](2, 1, 2, 3), Array("a", "b", "c", "a"), Array("", "")),
+ (Array.empty[Int], Array.empty[String], Array.empty[String]),
+ (null, null, null)
+).toDF("a", "b", "c")
+checkAnswer(
+ df.select(array_remove(df("a"), 2), array_remove(df("b"), "a"),
array_remove(df("c"), "")),
--- End diff --
maybe `$"a"` form is preferred to `df("a")`.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21069: [SPARK-23920][SQL]add array_remove to remove all ...
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21069#discussion_r185723637
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
---
@@ -883,3 +883,70 @@ case class Concat(children: Seq[Expression]) extends
Expression {
override def sql: String = s"concat(${children.map(_.sql).mkString(",
")})"
}
+
+/**
+ * Remove all elements that equal to element from the given array
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(array, element) - Remove all elements that equal to
element from array.",
+ examples = """
+Examples:
+ > SELECT _FUNC_(array(1, 2, 3, null, 3), 3);
+ [1,2,null]
+ """, since = "2.4.0")
+case class ArrayRemove(left: Expression, right: Expression)
+ extends BinaryExpression with ImplicitCastInputTypes with
CodegenFallback {
+
+ override def dataType: DataType = left.dataType
+
+ override def inputTypes: Seq[AbstractDataType] =
+Seq(ArrayType, left.dataType.asInstanceOf[ArrayType].elementType)
+
+ override def nullSafeEval(arr: Any, value: Any): Any = {
+val elementType = left.dataType.asInstanceOf[ArrayType].elementType
+val data =
arr.asInstanceOf[ArrayData].toArray[AnyRef](elementType).filter(_ != value)
+new GenericArrayData(data.asInstanceOf[Array[Any]])
+ }
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+val elementType = left.dataType.asInstanceOf[ArrayType].elementType
+nullSafeCodeGen(ctx, ev, (arr, value) => {
+ val arrayClass = classOf[GenericArrayData].getName
+ val values = ctx.freshName("values")
+ val i = ctx.freshName("i")
+ val pos = ctx.freshName("arrayPosition")
+ val numsToRemove = ctx.freshName("newArrLen")
+ val getValue = CodeGenerator.getValue(arr, right.dataType, i)
+ s"""
+ |int $pos = 0;
+ |int $numsToRemove = 0;
+ |Object[] $values;
+ |
+ |for (int $i = 0; $i < $arr.numElements(); $i ++) {
+ | if (!$arr.isNullAt($i) && ${ctx.genEqual(right.dataType,
value, getValue)}) {
+ |$numsToRemove = $numsToRemove + 1;
+ | }
+ |}
+ |$values = new Object[$arr.numElements() - $numsToRemove];
+ |for (int $i = 0; $i < $arr.numElements(); $i ++) {
+ | if ($arr.isNullAt($i)) {
+ | $values[$pos] = null;
+ | $pos = $pos + 1;
+ | }
+ | else {
+ | if (${ctx.genEqual(right.dataType, value, getValue)}) {
+ | ;
+ | }
+ | else {
+ | $values[$pos] = ${CodeGenerator.getValue(arr, elementType,
s"$i")};
--- End diff --
`$values[$pos] = $getValue;`?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21069: [SPARK-23920][SQL]add array_remove to remove all ...
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21069#discussion_r185721700
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
---
@@ -280,4 +280,35 @@ class CollectionExpressionsSuite extends SparkFunSuite
with ExpressionEvalHelper
checkEvaluation(Concat(Seq(aa0, aa1)), Seq(Seq("a", "b"), Seq("c"),
Seq("d"), Seq("e", "f")))
}
+
+ test("Array remove") {
+val a0 = Literal.create(Seq(1, 2, 3, 2, 2, 5), ArrayType(IntegerType))
+val a1 = Literal.create(Seq("b", "a", "a", "c", "b"),
ArrayType(StringType))
+val a2 = Literal.create(Seq[String](null, "", null, ""),
ArrayType(StringType))
+val a3 = Literal.create(Seq.empty[Integer], ArrayType(IntegerType))
+val a4 = Literal.create(null, ArrayType(StringType))
+val a5 = Literal.create(Seq(1, null, 8, 9, null),
ArrayType(IntegerType))
+val a6 = Literal.create(Seq(true, false, false, true),
ArrayType(BooleanType))
+
+checkEvaluation(ArrayRemove(a0, Literal(0)), Seq(1, 2, 3, 2, 2, 5))
+checkEvaluation(ArrayRemove(a0, Literal(1)), Seq(2, 3, 2, 2, 5))
+checkEvaluation(ArrayRemove(a0, Literal(2)), Seq(1, 3, 5))
+checkEvaluation(ArrayRemove(a0, Literal(3)), Seq(1, 2, 2, 2, 5))
+checkEvaluation(ArrayRemove(a0, Literal(5)), Seq(1, 2, 3, 2, 2))
--- End diff --
Can you add a case for something like `ArrayRemove(a0, Literal(10))` to
check no value is removed with not contained value?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21069: [SPARK-23920][SQL]add array_remove to remove all ...
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21069#discussion_r185720852
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
---
@@ -883,3 +883,70 @@ case class Concat(children: Seq[Expression]) extends
Expression {
override def sql: String = s"concat(${children.map(_.sql).mkString(",
")})"
}
+
+/**
+ * Remove all elements that equal to element from the given array
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(array, element) - Remove all elements that equal to
element from array.",
+ examples = """
+Examples:
+ > SELECT _FUNC_(array(1, 2, 3, null, 3), 3);
+ [1,2,null]
+ """, since = "2.4.0")
+case class ArrayRemove(left: Expression, right: Expression)
+ extends BinaryExpression with ImplicitCastInputTypes with
CodegenFallback {
+
+ override def dataType: DataType = left.dataType
+
+ override def inputTypes: Seq[AbstractDataType] =
+Seq(ArrayType, left.dataType.asInstanceOf[ArrayType].elementType)
+
+ override def nullSafeEval(arr: Any, value: Any): Any = {
+val elementType = left.dataType.asInstanceOf[ArrayType].elementType
+val data =
arr.asInstanceOf[ArrayData].toArray[AnyRef](elementType).filter(_ != value)
+new GenericArrayData(data.asInstanceOf[Array[Any]])
+ }
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+val elementType = left.dataType.asInstanceOf[ArrayType].elementType
+nullSafeCodeGen(ctx, ev, (arr, value) => {
+ val arrayClass = classOf[GenericArrayData].getName
+ val values = ctx.freshName("values")
+ val i = ctx.freshName("i")
+ val pos = ctx.freshName("arrayPosition")
+ val numsToRemove = ctx.freshName("newArrLen")
+ val getValue = CodeGenerator.getValue(arr, right.dataType, i)
+ s"""
+ |int $pos = 0;
+ |int $numsToRemove = 0;
+ |Object[] $values;
+ |
+ |for (int $i = 0; $i < $arr.numElements(); $i ++) {
+ | if (!$arr.isNullAt($i) && ${ctx.genEqual(right.dataType,
value, getValue)}) {
+ |$numsToRemove = $numsToRemove + 1;
+ | }
+ |}
+ |$values = new Object[$arr.numElements() - $numsToRemove];
--- End diff --
Can you specialize creating `ArrayData` for primitive types to avoid boxing?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21069: [SPARK-23920][SQL]add array_remove to remove all ...
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21069#discussion_r185715072
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
---
@@ -287,3 +287,44 @@ case class ArrayContains(left: Expression, right:
Expression)
override def prettyName: String = "array_contains"
}
+
+/**
+ * Remove all elements that equal to element from the given array
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(array, element) - Remove all elements that equal to
element from array.",
+ examples = """
+Examples:
+ > SELECT _FUNC_(array(1, 2, 3, null, 3), 3);
+ [1,2,null]
+ """, since = "2.4.0")
+case class ArrayRemove(left: Expression, right: Expression)
+ extends BinaryExpression with ImplicitCastInputTypes with
CodegenFallback {
--- End diff --
Now that there is a codegen, we can remove `CodegenFallback`.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21069: [SPARK-23920][SQL]add array_remove to remove all ...
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21069#discussion_r185722053
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
---
@@ -280,4 +280,35 @@ class CollectionExpressionsSuite extends SparkFunSuite
with ExpressionEvalHelper
checkEvaluation(Concat(Seq(aa0, aa1)), Seq(Seq("a", "b"), Seq("c"),
Seq("d"), Seq("e", "f")))
}
+
+ test("Array remove") {
+val a0 = Literal.create(Seq(1, 2, 3, 2, 2, 5), ArrayType(IntegerType))
+val a1 = Literal.create(Seq("b", "a", "a", "c", "b"),
ArrayType(StringType))
+val a2 = Literal.create(Seq[String](null, "", null, ""),
ArrayType(StringType))
+val a3 = Literal.create(Seq.empty[Integer], ArrayType(IntegerType))
+val a4 = Literal.create(null, ArrayType(StringType))
+val a5 = Literal.create(Seq(1, null, 8, 9, null),
ArrayType(IntegerType))
+val a6 = Literal.create(Seq(true, false, false, true),
ArrayType(BooleanType))
+
+checkEvaluation(ArrayRemove(a0, Literal(0)), Seq(1, 2, 3, 2, 2, 5))
+checkEvaluation(ArrayRemove(a0, Literal(1)), Seq(2, 3, 2, 2, 5))
+checkEvaluation(ArrayRemove(a0, Literal(2)), Seq(1, 3, 5))
+checkEvaluation(ArrayRemove(a0, Literal(3)), Seq(1, 2, 2, 2, 5))
+checkEvaluation(ArrayRemove(a0, Literal(5)), Seq(1, 2, 3, 2, 2))
+
+checkEvaluation(ArrayRemove(a1, Literal("")), Seq("b", "a", "a", "c",
"b"))
+checkEvaluation(ArrayRemove(a1, Literal("a")), Seq("b", "c", "b"))
+checkEvaluation(ArrayRemove(a1, Literal("b")), Seq("a", "a", "c"))
+checkEvaluation(ArrayRemove(a1, Literal("c")), Seq("b", "a", "a", "b"))
+
+checkEvaluation(ArrayRemove(a2, Literal("")), Seq(null, null))
+checkEvaluation(ArrayRemove(a2, Literal.create(null, StringType)),
null)
+
+checkEvaluation(ArrayRemove(a3, Literal(1)), Seq.empty[Integer])
+
+checkEvaluation(ArrayRemove(a4, Literal("a")), null)
+
+checkEvaluation(ArrayRemove(a5, Literal(9)), Seq(1, null, 8, null))
+checkEvaluation(ArrayRemove(a6, Literal(false)), Seq(true, true))
--- End diff --
Can you add a case for something like `ArrayRemove(a0, Literal(null,
IntegerType))`?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21069: [SPARK-23920][SQL]add array_remove to remove all ...
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21069#discussion_r185716705
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
---
@@ -883,3 +883,70 @@ case class Concat(children: Seq[Expression]) extends
Expression {
override def sql: String = s"concat(${children.map(_.sql).mkString(",
")})"
}
+
+/**
+ * Remove all elements that equal to element from the given array
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(array, element) - Remove all elements that equal to
element from array.",
+ examples = """
+Examples:
+ > SELECT _FUNC_(array(1, 2, 3, null, 3), 3);
+ [1,2,null]
+ """, since = "2.4.0")
+case class ArrayRemove(left: Expression, right: Expression)
+ extends BinaryExpression with ImplicitCastInputTypes with
CodegenFallback {
+
+ override def dataType: DataType = left.dataType
+
+ override def inputTypes: Seq[AbstractDataType] =
+Seq(ArrayType, left.dataType.asInstanceOf[ArrayType].elementType)
+
+ override def nullSafeEval(arr: Any, value: Any): Any = {
+val elementType = left.dataType.asInstanceOf[ArrayType].elementType
+val data =
arr.asInstanceOf[ArrayData].toArray[AnyRef](elementType).filter(_ != value)
+new GenericArrayData(data.asInstanceOf[Array[Any]])
+ }
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+val elementType = left.dataType.asInstanceOf[ArrayType].elementType
+nullSafeCodeGen(ctx, ev, (arr, value) => {
+ val arrayClass = classOf[GenericArrayData].getName
+ val values = ctx.freshName("values")
+ val i = ctx.freshName("i")
+ val pos = ctx.freshName("arrayPosition")
+ val numsToRemove = ctx.freshName("newArrLen")
+ val getValue = CodeGenerator.getValue(arr, right.dataType, i)
--- End diff --
`CodeGenerator.getValue(arr, elementType, i)`?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21069: [SPARK-23920][SQL]add array_remove to remove all ...
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21069#discussion_r185716774
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
---
@@ -883,3 +883,70 @@ case class Concat(children: Seq[Expression]) extends
Expression {
override def sql: String = s"concat(${children.map(_.sql).mkString(",
")})"
}
+
+/**
+ * Remove all elements that equal to element from the given array
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(array, element) - Remove all elements that equal to
element from array.",
+ examples = """
+Examples:
+ > SELECT _FUNC_(array(1, 2, 3, null, 3), 3);
+ [1,2,null]
+ """, since = "2.4.0")
+case class ArrayRemove(left: Expression, right: Expression)
+ extends BinaryExpression with ImplicitCastInputTypes with
CodegenFallback {
+
+ override def dataType: DataType = left.dataType
+
+ override def inputTypes: Seq[AbstractDataType] =
+Seq(ArrayType, left.dataType.asInstanceOf[ArrayType].elementType)
+
+ override def nullSafeEval(arr: Any, value: Any): Any = {
+val elementType = left.dataType.asInstanceOf[ArrayType].elementType
+val data =
arr.asInstanceOf[ArrayData].toArray[AnyRef](elementType).filter(_ != value)
+new GenericArrayData(data.asInstanceOf[Array[Any]])
+ }
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+val elementType = left.dataType.asInstanceOf[ArrayType].elementType
+nullSafeCodeGen(ctx, ev, (arr, value) => {
+ val arrayClass = classOf[GenericArrayData].getName
+ val values = ctx.freshName("values")
+ val i = ctx.freshName("i")
+ val pos = ctx.freshName("arrayPosition")
+ val numsToRemove = ctx.freshName("newArrLen")
+ val getValue = CodeGenerator.getValue(arr, right.dataType, i)
+ s"""
+ |int $pos = 0;
+ |int $numsToRemove = 0;
+ |Object[] $values;
+ |
+ |for (int $i = 0; $i < $arr.numElements(); $i ++) {
+ | if (!$arr.isNullAt($i) && ${ctx.genEqual(right.dataType,
value, getValue)}) {
--- End diff --
`ctx.genEqual(elementType, value, getValue)`?
Btw, this expression is used twice, so we should extract it.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21069: [SPARK-23920][SQL]add array_remove to remove all ...
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21069#discussion_r185716615
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
---
@@ -883,3 +883,70 @@ case class Concat(children: Seq[Expression]) extends
Expression {
override def sql: String = s"concat(${children.map(_.sql).mkString(",
")})"
}
+
+/**
+ * Remove all elements that equal to element from the given array
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(array, element) - Remove all elements that equal to
element from array.",
+ examples = """
+Examples:
+ > SELECT _FUNC_(array(1, 2, 3, null, 3), 3);
+ [1,2,null]
+ """, since = "2.4.0")
+case class ArrayRemove(left: Expression, right: Expression)
+ extends BinaryExpression with ImplicitCastInputTypes with
CodegenFallback {
+
+ override def dataType: DataType = left.dataType
+
+ override def inputTypes: Seq[AbstractDataType] =
+Seq(ArrayType, left.dataType.asInstanceOf[ArrayType].elementType)
+
+ override def nullSafeEval(arr: Any, value: Any): Any = {
+val elementType = left.dataType.asInstanceOf[ArrayType].elementType
+val data =
arr.asInstanceOf[ArrayData].toArray[AnyRef](elementType).filter(_ != value)
+new GenericArrayData(data.asInstanceOf[Array[Any]])
+ }
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+val elementType = left.dataType.asInstanceOf[ArrayType].elementType
+nullSafeCodeGen(ctx, ev, (arr, value) => {
+ val arrayClass = classOf[GenericArrayData].getName
+ val values = ctx.freshName("values")
+ val i = ctx.freshName("i")
+ val pos = ctx.freshName("arrayPosition")
+ val numsToRemove = ctx.freshName("newArrLen")
--- End diff --
`ctx.freshName("numsToRemove")`?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20761: [SPARK-20327][CORE][YARN] Add CLI support for YARN custo...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/20761 Cool, thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21210: [SPARK-23489][SQL][TEST] HiveExternalCatalogVersionsSuit...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/21210 Thank you for merging, @cloud-fan. And thank you, all. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21219: [SPARK-24160] ShuffleBlockFetcherIterator should fail if...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21219 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2855/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21219: [SPARK-24160] ShuffleBlockFetcherIterator should fail if...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21219 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21219: [SPARK-24160] ShuffleBlockFetcherIterator should fail if...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21219 **[Test build #90112 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90112/testReport)** for PR 21219 at commit [`41d06e1`](https://github.com/apache/spark/commit/41d06e13d0f95f1dd146b6b512a0becc88eb2caa). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21219: [SPARK-24160] ShuffleBlockFetcherIterator should fail if...
Github user JoshRosen commented on the issue: https://github.com/apache/spark/pull/21219 jenkins retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21215: [SPARK-24148][SQL] Overloading array function to support...
Github user lokm01 commented on the issue:
https://github.com/apache/spark/pull/21215
Hey @maropu,
So we've encountered a number of issues with casting:
1. Casting an empty array to an array of primitive types caused an
exception on 2.2.1, but works on 2.3.0+ so that's sorted
2. We're still facing an issue on 2.3.0 when we try to cast an empty array
to an array of complex types. See the following example:
`
case class Outer(a: List[Inner])
case class Inner(b: Int, c: String)
object App4 extends App {
val spark =
SparkSession.builder().appName("").master("local[*]").getOrCreate()
import spark.implicits._
import org.apache.spark.sql.functions._
val df = spark.createDataFrame(Seq[Outer]())
val r = spark.range(100).select(array().cast(df.schema("a").dataType))
r.printSchema()
r.show
}
`
This code produces
> Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot
resolve 'array()' due to data type mismatch: cannot cast array to
array>;;
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21224: [SPARK-24167][SQL] ParquetFilters should not access SQLC...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21224 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21224: [SPARK-24167][SQL] ParquetFilters should not access SQLC...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21224 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2854/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21201: [SPARK-24128][SQL] Mention configuration option in impli...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21201 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21201: [SPARK-24128][SQL] Mention configuration option in impli...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21201 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2853/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21145: [SPARK-24073][SQL]: Rename DataReaderFactory to ReadTask...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21145 I agree this is not a real factory, but it's not a real task either. I feel something like `DataReaderHolder` or `ReadTaskDescriptor` should be better. also cc @rxin @marmbrus --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21169: [SPARK-23715][SQL] the input of to/from_utc_timestamp ca...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21169 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2852/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21169: [SPARK-23715][SQL] the input of to/from_utc_timestamp ca...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21169 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21224: [SPARK-24167][SQL] ParquetFilters should not access SQLC...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21224 **[Test build #90110 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90110/testReport)** for PR 21224 at commit [`d7dc8a8`](https://github.com/apache/spark/commit/d7dc8a85489122e5b91cf5bc7cc0190f4d474a2c). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21222 **[Test build #90111 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90111/testReport)** for PR 21222 at commit [`c1ad1c5`](https://github.com/apache/spark/commit/c1ad1c557e6165455457adb6f148d6d9616548a1). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21223: [SPARK-24166][SQL] InMemoryTableScanExec should not acce...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21223 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21193: [SPARK-24121][SQL][WIP] Add API for handling expression ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21193 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21193: [SPARK-24121][SQL][WIP] Add API for handling expression ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21193 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2851/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21223: [SPARK-24166][SQL] InMemoryTableScanExec should not acce...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21223 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2850/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21210: [SPARK-23489][SQL][TEST] HiveExternalCatalogVersi...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21210 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21201: [SPARK-24128][SQL] Mention configuration option in impli...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21201 **[Test build #90109 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90109/testReport)** for PR 21201 at commit [`42d0241`](https://github.com/apache/spark/commit/42d0241feef21c71be1a5223ff7c359b7ad4c27b). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21222: [SPARK-24161][SS] Enable debug package feature on struct...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21222 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21210: [SPARK-23489][SQL][TEST] HiveExternalCatalogVersionsSuit...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21210 thanks, merging to master/2.3! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21201: [SPARK-24128][SQL] Mention configuration option in impli...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21201 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21225: [SPARK-24168][SQL] WindowExec should not access SQLConf ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21225 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2849/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21225: [SPARK-24168][SQL] WindowExec should not access SQLConf ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21225 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21225: [SPARK-24168][SQL] WindowExec should not access SQLConf ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21225 **[Test build #90106 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90106/testReport)** for PR 21225 at commit [`c5ab777`](https://github.com/apache/spark/commit/c5ab77710de805eab3a0a815790a58cd2de56cb2). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21223: [SPARK-24166][SQL] InMemoryTableScanExec should not acce...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21223 **[Test build #90107 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90107/testReport)** for PR 21223 at commit [`d900b4c`](https://github.com/apache/spark/commit/d900b4c04f5def9bac6cec33c1c7753761a19658). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21169: [SPARK-23715][SQL] the input of to/from_utc_timestamp ca...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21169 **[Test build #90108 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90108/testReport)** for PR 21169 at commit [`b6d91db`](https://github.com/apache/spark/commit/b6d91db2fd71b50389cf3647a31eefc83d5dbc44). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21226: [SPARK-24169][SQL] JsonToStructs should not access SQLCo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21226 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2848/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21226: [SPARK-24169][SQL] JsonToStructs should not access SQLCo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21226 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21193: [SPARK-24121][SQL][WIP] Add API for handling expression ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21193 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2847/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21193: [SPARK-24121][SQL][WIP] Add API for handling expression ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21193 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21225: [SPARK-24168][SQL] WindowExec should not access SQLConf ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21225 **[Test build #90103 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90103/testReport)** for PR 21225 at commit [`c5ab777`](https://github.com/apache/spark/commit/c5ab77710de805eab3a0a815790a58cd2de56cb2). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20761: [SPARK-20327][CORE][YARN] Add CLI support for YARN custo...
Github user szyszy commented on the issue: https://github.com/apache/spark/pull/20761 Hey! Exactly, I almost addressed all of Marcelo's comments so please expect a PR update in the coming days. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21193: [SPARK-24121][SQL][WIP] Add API for handling expression ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21193 **[Test build #90104 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90104/testReport)** for PR 21193 at commit [`5fe425c`](https://github.com/apache/spark/commit/5fe425c2d2837f00bdfe9ba5e6f446829fba32c1). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21226: [SPARK-24169][SQL] JsonToStructs should not access SQLCo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21226 **[Test build #90102 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90102/testReport)** for PR 21226 at commit [`e1491f0`](https://github.com/apache/spark/commit/e1491f0aeb62d1eda8cd8c55f890c8f87eec5761). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21169: [SPARK-23715][SQL] the input of to/from_utc_timestamp ca...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21169 **[Test build #90105 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90105/testReport)** for PR 21169 at commit [`b6d91db`](https://github.com/apache/spark/commit/b6d91db2fd71b50389cf3647a31eefc83d5dbc44). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21169: [SPARK-23715][SQL] the input of to/from_utc_timestamp ca...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21169 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21223: [SPARK-24166][SQL] InMemoryTableScanExec should not acce...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21223 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21169: [SPARK-23715][SQL] the input of to/from_utc_timestamp ca...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21169 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21226: [SPARK-24169][SQL] JsonToStructs should not access SQLCo...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21226 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21073 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21193: [SPARK-24121][SQL][WIP] Add API for handling expression ...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21193 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21073 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90091/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21225: [SPARK-24168][SQL] WindowExec should not access SQLConf ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21225 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90097/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21219: [SPARK-24160] ShuffleBlockFetcherIterator should fail if...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21219 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90093/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21169: [SPARK-23715][SQL] the input of to/from_utc_timestamp ca...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21169 **[Test build #90099 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90099/testReport)** for PR 21169 at commit [`b6d91db`](https://github.com/apache/spark/commit/b6d91db2fd71b50389cf3647a31eefc83d5dbc44). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21219: [SPARK-24160] ShuffleBlockFetcherIterator should fail if...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21219 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21193: [SPARK-24121][SQL][WIP] Add API for handling expression ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21193 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21225: [SPARK-24168][SQL] WindowExec should not access SQLConf ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21225 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21193: [SPARK-24121][SQL][WIP] Add API for handling expression ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21193 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90092/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21226: [SPARK-24169][SQL] JsonToStructs should not access SQLCo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21226 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21223: [SPARK-24166][SQL] InMemoryTableScanExec should not acce...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21223 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90095/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21073 **[Test build #90091 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90091/testReport)** for PR 21073 at commit [`77ae014`](https://github.com/apache/spark/commit/77ae014bea3d9fbc20fbd11d5508c3606e26343d). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21225: [SPARK-24168][SQL] WindowExec should not access SQLConf ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21225 **[Test build #90097 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90097/testReport)** for PR 21225 at commit [`c5ab777`](https://github.com/apache/spark/commit/c5ab77710de805eab3a0a815790a58cd2de56cb2). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21226: [SPARK-24169][SQL] JsonToStructs should not access SQLCo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21226 **[Test build #90098 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90098/testReport)** for PR 21226 at commit [`e1491f0`](https://github.com/apache/spark/commit/e1491f0aeb62d1eda8cd8c55f890c8f87eec5761). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21219: [SPARK-24160] ShuffleBlockFetcherIterator should fail if...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21219 **[Test build #90093 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90093/testReport)** for PR 21219 at commit [`41d06e1`](https://github.com/apache/spark/commit/41d06e13d0f95f1dd146b6b512a0becc88eb2caa). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21201: [SPARK-24128][SQL] Mention configuration option in impli...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21201 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90101/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21226: [SPARK-24169][SQL] JsonToStructs should not access SQLCo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21226 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90098/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21169: [SPARK-23715][SQL] the input of to/from_utc_timestamp ca...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21169 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90099/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21169: [SPARK-23715][SQL] the input of to/from_utc_timestamp ca...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21169 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21223: [SPARK-24166][SQL] InMemoryTableScanExec should not acce...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21223 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21201: [SPARK-24128][SQL] Mention configuration option in impli...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21201 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21193: [SPARK-24121][SQL][WIP] Add API for handling expression ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21193 **[Test build #90092 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90092/testReport)** for PR 21193 at commit [`5fe425c`](https://github.com/apache/spark/commit/5fe425c2d2837f00bdfe9ba5e6f446829fba32c1). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class ExprCode(var code: Block, var isNull: ExprValue, var value: ExprValue)` * `trait Block extends JavaCode ` * ` implicit class BlockHelper(val sc: StringContext) extends AnyVal ` * `case class CodeBlock(codeParts: Seq[String], exprValues: Seq[Any]) extends Block ` * `case class Blocks(blocks: Seq[Block]) extends Block ` * `trait ExprValue extends JavaCode ` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21201: [SPARK-24128][SQL] Mention configuration option in impli...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21201 **[Test build #90101 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90101/testReport)** for PR 21201 at commit [`42d0241`](https://github.com/apache/spark/commit/42d0241feef21c71be1a5223ff7c359b7ad4c27b). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21223: [SPARK-24166][SQL] InMemoryTableScanExec should not acce...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21223 **[Test build #90095 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90095/testReport)** for PR 21223 at commit [`d900b4c`](https://github.com/apache/spark/commit/d900b4c04f5def9bac6cec33c1c7753761a19658). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21187: [SPARK-24035][SQL] SQL syntax for Pivot
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21187 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90094/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21187: [SPARK-24035][SQL] SQL syntax for Pivot
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21187 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
