Johan Lasperas created SPARK-43217: -------------------------------------- Summary: Correctly recurse into maps of maps and arrays of arrays in StructType.findNestedField Key: SPARK-43217 URL: https://issues.apache.org/jira/browse/SPARK-43217 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 3.4.0 Reporter: Johan Lasperas
[StructType.findNestedField|https://github.com/apache/spark/blob/db2625c70a8c3aff64e6a9466981c8dd49a4ca51/sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala#L325] is unable to reach nested field below two directly nested maps or arrays. Whenever it reaches a map or an array, it'll throw an `invalidFieldName` exception if the child is not a struct. The following throws 'Field name `a`.`element`.element`.`i` is invalid: `a`.`element`.`element` is not a struct.', even though the access path is valid: ``` val schema = new StructType() .add("a", ArrayType(ArrayType( new StructType().add("i", "int")))) findNestedField(Seq("a", "element", "element", "i"), schema) ``` -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org