yihua commented on code in PR #19123:
URL: https://github.com/apache/hudi/pull/19123#discussion_r3825868512
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestFileGroupReaderPartitionColumn.scala:
##########
@@ -155,4 +155,132 @@ class TestFileGroupReaderPartitionColumn extends
SparkClientFunctionalTestHarnes
assertEquals("IN", rows(12L), "id=12 partition column must be IN")
assertEquals("IN", rows(14L), "id=14 partition column must be IN")
}
+
+ /**
+ * Regression test for reading a table partitioned on a nested column when
that nested column is
+ * also a mandatory field (here, the precombine/ordering field). Being
mandatory, the file group
+ * reader requests the partition column as a top-level field; for a nested
path
+ * ("nested_record.level") this previously failed in
`buildReaderWithPartitionValues` with
+ * `HoodieSchemaException: Illegal character in: nested_record.level` when
converting the
+ * StructType into an Avro-backed HoodieSchema.
+ *
+ * The nested partition value is never a flat top-level column in the data
file — it must be
+ * materialized from the partition path — so the fix keeps it out of the
file-read schema and
+ * appends it from the path. The existing
`TestCOWDataSource#testNestedFieldPartition` covers the
+ * common (non-mandatory) path where the field is already appended from the
path and does not hit
+ * this case.
+ */
+ @Test
+ def testReadTablePartitionedOnNestedColumnThatIsAlsoPrecombine(): Unit = {
+ val nestedSchema = StructType(Array(
+ StructField("nested_int", IntegerType, nullable = false),
+ StructField("level", StringType, nullable = false)
+ ))
+ val schema = StructType(Array(
+ StructField("id", LongType, nullable = false),
+ StructField("name", StringType, nullable = true),
+ StructField("nested_record", nestedSchema, nullable = true)
+ ))
+
+ val opts = Map(
+ "hoodie.table.name" -> "test_nested_partition_precombine",
+ "hoodie.datasource.write.table.type" -> "COPY_ON_WRITE",
+ "hoodie.datasource.write.recordkey.field" -> "id",
+ // partition column is ALSO the precombine field -> it becomes mandatory
(read from file),
+ // which is what drives the nested-name conversion the fix guards
against.
+ "hoodie.datasource.write.partitionpath.field" -> "nested_record.level",
+ "hoodie.datasource.write.precombine.field" -> "nested_record.level",
Review Comment:
nit: could `hoodie.datasource.write.precombine.field` use a different nested
field instead of the same partition path field?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
