sadikovi commented on PR #40699:
URL: https://github.com/apache/spark/pull/40699#issuecomment-1517271113
Actually, this is a breaking change for CAST expression. It affects the
string representation of complex types and not just `df.show` which could cause
issues when comparing the string representation of a row:
```sql
select cast(struct(null) as string) as c0, null as c1
```
```
+------+----+
| c0| c1|
+------+----+
|{NULL}|NULL|
+------+----+
List([{NULL},null])
```
It also affects CSV writes:
```scala
val df = spark.sql(
"""
select cast(array(1,2) as string) as c0, 1 as c1
union all
select cast(array(3,null) as string) as c0, null as c1
"""
).coalesce(1)
df.write.csv("file:/tmp/ivan_test.csv")
```
Before this PR:
```
"[1, 2]",1
"[3, null]",
```
After this PR:
```
"[1, 2]",1
"[3, NULL]",
```
And there is no option to re-enable null format.
On a broader side of things: It would be good to clarify why spark-sql shell
is expected to be visually compatible with spark-shell, those are two different
REPLs with their own ways of displaying data. For example, we don't display a
table the same way in those REPLs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
