[
https://issues.apache.org/jira/browse/HUDI-8031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Vexler closed HUDI-8031.
---------------------------------
Resolution: Duplicate
> Allow read partition path field directly from file in new filegroup reader
> --------------------------------------------------------------------------
>
> Key: HUDI-8031
> URL: https://issues.apache.org/jira/browse/HUDI-8031
> Project: Apache Hudi
> Issue Type: Improvement
> Components: reader-core, spark
> Reporter: Jonathan Vexler
> Assignee: Jonathan Vexler
> Priority: Major
>
> Currently for spark, we append the same partition path value to the end of
> every record. If you use timestamp based keygen for example, your partition
> field can differ for every record.
> Idea for how to implement: In default source / hadoopfs factory, we figure
> out if the partition cols are going to all be the same values or not. If they
> are not, we set the partition schema as empty. 1 thing to think about is that
> in the file index, we need to look through the data filters and move them to
> partition filters if they are on the partition column.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
