Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/14118
FYI, before
[SPARK-14143](https://issues.apache.org/jira/browse/SPARK-14143), null values
had been handled this way: :
```scala
if (datum == options.nullValue && nullable &&
(!castType.isInstanceOf[StringType]))
```
Then in [SPARK-14143](https://issues.apache.org/jira/browse/SPARK-14143),
it was first broken down into numeric data types in
https://github.com/apache/spark/pull/11947/commits/93ac6bb3eb63efb775b48af090a37a6cbe4f30c4
to handle byte-specific null value, short-specific null value, int-specific
null value, ... :
```scala
case _: ByteType => if (datum == params.byteNullValue && nullable) null
else datum.toByte
case _: ShortType => if (datum == params.shortNullValue && nullable) null
else datum.toShort
case _: IntegerType => if (datum == params.integerNullValue && nullable)
null else datum.toInt
...
```
then in
https://github.com/apache/spark/pull/11947/commits/698b4b41baa1ebd5d66ea6242bcb39bcd0887f8b
byte-specific null value, short-specific null value, int-specific null value,
... were reduced back to one single null value:
```scala
case _: ByteType => if (datum == params.nullValue && nullable) null else
datum.toByte
case _: ShortType => if (datum == params.nullValue && nullable) null else
datum.toShort
case _: IntegerType => if (datum == params.nullValue && nullable) null else
datum.toInt
```
Along with that change, we had introduced regression handling non-numeric
data types like `BooleanType` etc. Since we don't need to handle type-specific
null values, this patch switchs back to the way we handled null values in the
1.6 days (and thus fixes the regression):
```scala
if (datum == options.nullValue && nullable &&
(!castType.isInstanceOf[StringType]))
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
