Johan Lasperas created SPARK-43217:
--------------------------------------
Summary: Correctly recurse into maps of maps and arrays of arrays
in StructType.findNestedField
Key: SPARK-43217
URL: https://issues.apache.org/jira/browse/SPARK-43217
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 3.4.0
Reporter: Johan Lasperas
[StructType.findNestedField|https://github.com/apache/spark/blob/db2625c70a8c3aff64e6a9466981c8dd49a4ca51/sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala#L325]
is unable to reach nested field below two directly nested maps or arrays.
Whenever it reaches a map or an array, it'll throw an `invalidFieldName`
exception if the child is not a struct.
The following throws 'Field name `a`.`element`.element`.`i` is invalid:
`a`.`element`.`element` is not a struct.', even though the access path is valid:
```
val schema = new StructType()
.add("a", ArrayType(ArrayType(
new StructType().add("i", "int"))))
findNestedField(Seq("a", "element", "element", "i"), schema)
```
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
