This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 8f968b4 [SPARK-26730][SQL] Strip redundant AssertNotNull for
ExpressionEncoder's serializer
8f968b4 is described below
commit 8f968b4c064558d6a4c38e7e2592e49d88889507
Author: wuyi <ngone_5...@163.com>
AuthorDate: Fri Feb 1 10:48:37 2019 +0800
[SPARK-26730][SQL] Strip redundant AssertNotNull for ExpressionEncoder's
serializer
## What changes were proposed in this pull request?
For types like Product, we've already add AssertNotNull when we construct
serializer(see code below), so we could strip redundant AssertNotNull for those
types.
```
val fieldValue = Invoke(
AssertNotNull(inputObject, walkedTypePath), fieldName,
dataTypeFor(fieldType),
returnNullable = !fieldType.typeSymbol.asClass.isPrimitive)
```
## How was this patch tested?
Existed.
Closes #23651 from
Ngone51/dev-strip-redundant-assertnotnull-for-ecnoder-serializer.
Authored-by: wuyi <ngone_5...@163.com>
Signed-off-by: Wenchen Fan <wenc...@databricks.com>
---
.../main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala
index 1b06835..d5af91a 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala
@@ -631,8 +631,11 @@ object ScalaReflection extends ScalaReflection {
"cannot be used as field name\n" + walkedTypePath.mkString("\n"))
}
- val fieldValue = Invoke(
- AssertNotNull(inputObject, walkedTypePath), fieldName,
dataTypeFor(fieldType),
+ // SPARK-26730 inputObject won't be null with If's guard below. And
KnownNotNul
+ // is necessary here. Because for a nullable nested inputObject with
struct data
+ // type, e.g. StructType(IntegerType, StringType), it will return
nullable=true
+ // for IntegerType without KnownNotNull. And that's what we do not
expect to.
+ val fieldValue = Invoke(KnownNotNull(inputObject), fieldName,
dataTypeFor(fieldType),
returnNullable = !fieldType.typeSymbol.asClass.isPrimitive)
val clsName = getClassNameFromType(fieldType)
val newPath = s"""- field (class: "$clsName", name: "$fieldName")"""
+: walkedTypePath
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
