This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new 4353f7d [SPARK-30894][SQL][2.4] Make Size's nullable independent from
SQL config changes
4353f7d is described below
commit 4353f7d961aba7f1f65066245215b08817663701
Author: Maxim Gekk <[email protected]>
AuthorDate: Thu Oct 15 14:00:38 2020 -0700
[SPARK-30894][SQL][2.4] Make Size's nullable independent from SQL config
changes
This is a backport of https://github.com/apache/spark/pull/27658
### What changes were proposed in this pull request?
In the PR, I propose to add the `legacySizeOfNull ` parameter to the `Size`
expression, and pass the value of `spark.sql.legacy.sizeOfNull` if
`legacySizeOfNull` is not provided on creation of `Size`.
### Why are the changes needed?
This allows to avoid the issue when the configuration change between
different phases of planning, and this can silently break a query plan which
can lead to crashes or data corruption.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
By `CollectionExpressionsSuite`.
Closes #30058 from anuragmantri/SPARK-30894-2.4.
Authored-by: Maxim Gekk <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../spark/sql/catalyst/expressions/collectionOperations.scala | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
index 6d74f45..c8bc1e7 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
@@ -89,9 +89,10 @@ trait BinaryArrayExpressionWithImplicitCast extends
BinaryExpression
> SELECT _FUNC_(NULL);
-1
""")
-case class Size(child: Expression) extends UnaryExpression with
ExpectsInputTypes {
+case class Size(child: Expression, legacySizeOfNull: Boolean)
+ extends UnaryExpression with ExpectsInputTypes {
- val legacySizeOfNull = SQLConf.get.legacySizeOfNull
+ def this(child: Expression) = this(child, SQLConf.get.legacySizeOfNull)
override def dataType: DataType = IntegerType
override def inputTypes: Seq[AbstractDataType] =
Seq(TypeCollection(ArrayType, MapType))
@@ -123,6 +124,10 @@ case class Size(child: Expression) extends UnaryExpression
with ExpectsInputType
}
}
+object Size {
+ def apply(child: Expression): Size = new Size(child)
+}
+
/**
* Returns an unordered array containing the keys of the map.
*/
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
