Github user viirya commented on the issue:
https://github.com/apache/spark/pull/17096
In `StructType.add`, if the given field name is a `basestring`, we will
directly use it as key in `names` property.
If we pass in a `StructField`, we take `StructField.name` as key in
`names`. `But StructField` will encode field name as utf-8 if it is not a `str`.
And `StructType.__getitem__` will find matched field by comparing each
`StructField.name` with given search key. So it is unable to find the field
back with the unicode field name.
from pyspark.sql.types import StructType, StringType, StructField
struct1 = StructType().add(u"a", "string", True)
struct2 = StructType([StructField(u"a", StringType(), True)])
self.assertTrue(struct1 == struct2) # pass
self.assertTrue(struct1[u"a"] == struct2[u"a"]) #pass
struct1 = StructType().add(u"測試", "string", True)
struct2 = StructType([StructField(u"測試", StringType(), True)])
self.assertTrue(struct1 == struct2) # fail
self.assertTrue(struct1[u"測試"] == struct2[u"測試"]) # fail,
you can't find the field with key u"測試"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
