Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/22732
For Scala 2.11, we should not introduce any behavior change and also keep
binary and source compatibility.
```
scala> import org.apache.spark.sql.types.DataTypes
import org.apache.spark.sql.types.DataTypes
scala> val f2 = udf({(x: Int) => x}, DataTypes.IntegerType)
f2: org.apache.spark.sql.expressions.UserDefinedFunction =
UserDefinedFunction(<function1>,IntegerType,None)
scala> spark.range(3).select(f2('id + null)).show()
+----------------+
|UDF((id + null))|
+----------------+
| null|
| null|
| null|
+----------------+
```
For Scala 2.12, since we are unable to know the type nullability in a few
APIs, we issue a warning message in these cases. Below is the example which
will generate a different answer:
```
scala> import org.apache.spark.sql.types.DataTypes
import org.apache.spark.sql.types.DataTypes
scala> val f2 = udf({(x: Int) => x}, DataTypes.IntegerType)
f2: org.apache.spark.sql.expressions.UserDefinedFunction =
UserDefinedFunction($Lambda$2801/26868055@5eb35a26,IntegerType,None)
scala> spark.range(3).select(f2('id + null)).show()
18/10/18 23:07:02 WARN ScalaReflection: Scala version 2.12.7 cannot get
type nullability correctly via reflection, thus Spark cannot add proper input
null check for UDF. To avoid this problem, use the typed UDF interfaces instead.
+----------------+
|UDF((id + null))|
+----------------+
| 0|
| 0|
| 0|
+----------------+
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
