voonhous opened a new issue, #19594: URL: https://github.com/apache/hudi/issues/19594
**Describe the problem you faced** On the legacy MOR read path (`HoodieMergeOnReadRDDV2`, reached when `hoodie.file.group.reader.enabled=false`), the file-group-reader branch of `compute` sources every projected column from the data files and never appends partition values. `SparkFileFormatInternalRowReaderContext` builds its `PartitionedFile` with `InternalRow.empty` and passes an empty partition schema to the base file reader, and the builder's `withPartitionPath` only feeds the bootstrap merge. So for a table whose partition columns are not persisted in the data files -- `hoodie.datasource.write.drop.partition.columns=true`, `hoodie.datasource.read.extract.partition.values.from.path=true`, or a bootstrap data-queries-only read -- every split taking that branch returns NULL for the partition columns. The skip-merging fast path does not have this problem: `requiredSchemaReaderSkipMerging` is built with `appendPartitionValues = shouldExtractPartitionValuesFromPartitionPath`, and the split carries the parsed values. Scope: only splits that take the file-group-reader branch, i.e. anything with log files, plus base-only splits re-routed there. Batch MOR reads are unaffected, since they always scan through `HoodieFileGroupReaderBasedFileFormat`, which appends partition values itself. The metadata table is unaffected: it has no partition columns. **To Reproduce** MOR table written with `drop.partition.columns=true`, read back through the legacy relation with `hoodie.file.group.reader.enabled=false` (the streaming source, or a directly-constructed `MergeOnReadSnapshotRelation` / `MergeOnReadIncrementalRelationV1/V2`), over a file slice that has log files. The partition columns come back null. Covered by `TestLegacyParquetReadPath#testMorSnapshotReadWithDroppedPartitionColumns`, which fails without the fix: the rows served from the partition-column-less base file come back with a null partition instead of their real value. **Expected behavior** The file-group-reader branch fills the partition columns with the values encoded in the partition path, matching the skip-merging reader and the file-format path. **Environment Description** * Hudi version: master (1.3.0-SNAPSHOT) * Spark version: any * Table type: MERGE_ON_READ * Running on Docker: no **Additional context** Found while reviewing #19583, which re-routes base-only splits with shredded variant columns onto this same branch. Fixed there rather than deferred, since that PR would otherwise have widened the gap to base-only splits: the branch now carries the parsed values on the split and binds them at their existing ordinals with a per-split projection. The output row shape was already correct, because `TableSchemaResolver` re-appends dropped partition columns to the table schema, so the reader was only filling them with nulls. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
