[
https://issues.apache.org/jira/browse/HUDI-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
vinoyang closed HUDI-1557.
--------------------------
Fix Version/s: 0.8.0
Resolution: Done
Done via master branch: 4c5b6923ccfaaa6616a934a3f690b1a795a42d41
> Make Flink write pipeline write task scalable
> ---------------------------------------------
>
> Key: HUDI-1557
> URL: https://issues.apache.org/jira/browse/HUDI-1557
> Project: Apache Hudi
> Issue Type: Sub-task
> Components: DeltaStreamer
> Reporter: Danny Chen
> Assignee: Danny Chen
> Priority: Major
> Labels: pull-request-available
> Fix For: 0.8.0
>
>
> This issue introduces a BucketAssigner that assigns bucket ID (partition path
> & fileID) for each stream record.
> There is no need to look up index and partition the records anymore in the
> following pipeline for these records,
> we actually decide the write target location before the write and each record
> computes its location when the BucketAssigner receives it, thus, the indexing
> is with streaming style.
> Computing locations for a batch of records all at a time is resource
> consuming so a pressure to the engine,
> we should avoid that in streaming system.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
