Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19664#discussion_r149564523
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/ScalaReflectionSuite.scala
---
@@ -335,4 +338,17 @@ class ScalaReflectionSuite extends SparkFunSuite {
assert(linkedHashMapDeserializer.dataType ==
ObjectType(classOf[LHMap[_, _]]))
}
+ test("SPARK-22442: Generate correct field names for special characters")
{
+ val serializer = serializerFor[SpecialCharAsFieldData](BoundReference(
+ 0, ObjectType(classOf[SpecialCharAsFieldData]), nullable = false))
+ val deserializer = deserializerFor[SpecialCharAsFieldData]
+ assert(serializer.dataType(0).name == "field.1")
+ assert(serializer.dataType(1).name == "field 2")
+
+ val argumentsFields =
deserializer.asInstanceOf[NewInstance].arguments.flatMap { _.collect {
+ case UpCast(u: UnresolvedAttribute, _, _) => u.name
+ }}
+ assert(argumentsFields(0) == "`field.1`")
--- End diff --
We need to deliberately wrap backticks around a field name such as
`field.1` because of the dot character. Otherwise `UnresolvedAttribute` will
parse it as two name parts `Seq("field", "1")` and then fail resolving later.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
