Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20280
^ Yup, let's leave the performance issue out. I think we might have to
raise an error too but it's kind of a radical change.
As a note, sorted fields are documented:
https://github.com/apache/spark/blob/3e40eb3f1ffac3d2f49459a801e3ce171ed34091/python/pyspark/sql/types.py#L1451-L1452
My only main concern is:
>... the field name must be a subset of what is in each row.
>... field names are not important and can be changed.
I think this is kind of a breaking change because we will basically now
disallow the names given by user explicitly IIUC?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
