[ https://issues.apache.org/jira/browse/SPARK-43217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Johan Lasperas updated SPARK-43217: ----------------------------------- Description: [StructType.findNestedField|https://github.com/apache/spark/blob/db2625c70a8c3aff64e6a9466981c8dd49a4ca51/sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala#L325] is unable to reach nested field below two directly nested maps or arrays. Whenever it reaches a map or an array, it'll throw an `invalidFieldName` exception if the child is not a struct. The following throws 'Field name `a`.`element`.element`.`i` is invalid: `a`.`element`.`element` is not a struct.', even though the access path is valid: {code:java} val schema = new StructType() .add("a", ArrayType(ArrayType( new StructType().add("i", "int")))) findNestedField(Seq("a", "element", "element", "i"), schema) {code} was: [StructType.findNestedField|https://github.com/apache/spark/blob/db2625c70a8c3aff64e6a9466981c8dd49a4ca51/sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala#L325] is unable to reach nested field below two directly nested maps or arrays. Whenever it reaches a map or an array, it'll throw an `invalidFieldName` exception if the child is not a struct. The following throws 'Field name `a`.`element`.element`.`i` is invalid: `a`.`element`.`element` is not a struct.', even though the access path is valid: ``` val schema = new StructType() .add("a", ArrayType(ArrayType( new StructType().add("i", "int")))) findNestedField(Seq("a", "element", "element", "i"), schema) ``` > Correctly recurse into maps of maps and arrays of arrays in > StructType.findNestedField > -------------------------------------------------------------------------------------- > > Key: SPARK-43217 > URL: https://issues.apache.org/jira/browse/SPARK-43217 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 3.4.0 > Reporter: Johan Lasperas > Priority: Minor > > [StructType.findNestedField|https://github.com/apache/spark/blob/db2625c70a8c3aff64e6a9466981c8dd49a4ca51/sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala#L325] > is unable to reach nested field below two directly nested maps or arrays. > Whenever it reaches a map or an array, it'll throw an `invalidFieldName` > exception if the child is not a struct. > The following throws 'Field name `a`.`element`.element`.`i` is invalid: > `a`.`element`.`element` is not a struct.', even though the access path is > valid: > {code:java} > val schema = new StructType() > .add("a", ArrayType(ArrayType( > new StructType().add("i", "int")))) > findNestedField(Seq("a", "element", "element", "i"), schema) {code} > -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org