NestDream commented on PR #59038:
URL: https://github.com/apache/spark/pull/59038#issuecomment-5830556960
cc @HyukjinKwon @zhengruifeng, this adds a row length check to
`_create_converter.convert_struct` and `StructType.toInternal`: both use `zip`,
so on Classic a row longer than the inferred or declared schema was silently
truncated whenever a field needed a converter, while the no-converter path,
`verifySchema=True` and Connect all raise for the same rows. Could you take a
look when you have a moment?
Quick repro on master, no config needed:
```python
from pyspark.sql import Row
sc = spark.sparkContext
spark.createDataFrame(sc.parallelize([("a", 1), ("b", 2, 3)])).collect()
# [Row(_1='a', _2=1), Row(_1='b', _2=2)] the 3 is gone
spark.createDataFrame(sc.parallelize([Row(a="x", b=1), Row(c=3, a="y",
b=2)])).collect()
# [Row(a='x', b=1), Row(a='3', b=None)] wrong values, not just a dropped
tail
spark.createDataFrame(sc.parallelize([(1, 1), (2, 2, 3)])).collect()
# STRUCT_ARRAY_LENGTH_MISMATCH same rows, only long columns,
so no Python converter: the JVM catches it
```
With this PR the first two raise `FIELD_STRUCT_LENGTH_MISMATCH`, the same
error `verifySchema=True` already gives for these rows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
