[
https://issues.apache.org/jira/browse/HUDI-860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raymond Xu updated HUDI-860:
----------------------------
Fix Version/s: 0.12.0
(was: 0.11.0)
> Ability to do small file handling without need for caching
> ----------------------------------------------------------
>
> Key: HUDI-860
> URL: https://issues.apache.org/jira/browse/HUDI-860
> Project: Apache Hudi
> Issue Type: Task
> Components: writer-core
> Reporter: Vinoth Chandar
> Assignee: Alexey Kudinkin
> Priority: Blocker
> Fix For: 0.12.0
>
>
> As of now, in upsert path,
> * hudi builds a workloadProfile to understand total inserts and updates(with
> location info)
> * Following which, small files info are populated
> * Then buckets are populated with above info.
> * These buckets are later used when getPartition(Object key) is invoked in
> UpsertPartitioner.
> In step1: to build global workload profile, we had to do an action on entire
> JavaRDD<HoodieRecord>s in the driver and hudi does save the workload profile
> as well.
> For large write intensive batch jobs(COW types), caching this incurs
> additional overhead. So, this effort is trying to see if we can avoid doing
> this by some means.
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
