hudi-agent commented on code in PR #19123:
URL: https://github.com/apache/hudi/pull/19123#discussion_r3502669372
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedFileFormat.scala:
##########
Review Comment:
🤖 **Line 429:** Following up on the earlier thread — I don't think this call
site passes the full `partitionSchema`. At line 341 `appendPartitionAndProject`
is invoked with `remainingPartitionSchema` as its `partitionSchema` arg, so
here `partitionSchema` is the *remaining* schema, whereas `readBaseFile` (line
532) passes the full one. With a mix of one top-level mandatory field (read
from file) + one nested field (appended), e.g. partition
`country,nested_record.level` both ordering fields:
remaining=[nested_record.level] (len 1), indices={1},
partitionValues.numFields=2 → line 424 is false →
`getFixedPartitionValues(partitionValues, remaining, {1})` does
`toSeq(remaining)` which materializes only ordinal 0 (country's value, as
StringType) then filters to {1} → empty, so the nested value is dropped. The
new tests are COW-only so they hit `readBaseFile` (correct) and never this MOR
branch. Could you confirm whether this should pass the full `partitionSchema`
to `getFixe
dPartitionValues` here (while keeping `remainingPartitionSchema` for the join
projection)?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
