Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/20637
We need to detect the correctly written test with a wrong result. Let us
think about the following `map_zip_with` without #22126.
In the following example, `map_zip_with` without #22126 return `expr` that
has `Map(1 -> -10, 4 -> null)` with `MapType(IntegerType, IntegerType,
valueContainsNull = false)`. The `DataTypeP is not incorrect since `Map(...)`
includes `null`. Thus, the test must be failed.
As described in [the
comment](https://github.com/apache/spark/pull/20637#discussion_r212817682),
* With `asNullable`: the test fails with incorrect evaluation (it is not
passing expectedly)
* Without `asNullable`: the test fails with NullPointerException (it is not
passing unexpectedly).
Since #22126 has been merged, the test is passed. If someone would
unintentionally generate incorrect dataType, we must detect the mistake by
failing the test without an exception. To avoid the exception, we need
`asNullable`.
Is this an answer to your question?
```
val miia = Literal.create(Map(1 -> 10),
MapType(IntegerType, IntegerType, valueContainsNull = false))
val miib = Literal.create(Map(1 -> -1, 4 -> -4),
MapType(IntegerType, IntegerType, valueContainsNull = false))
val expr = map_zip_with(miia, miib, multiplyKeyWithValues)
checkEvaluation(expr, Map(1 -> -10, 4 -> null))
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
