[
https://issues.apache.org/jira/browse/HUDI-6864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ethan Guo updated HUDI-6864:
----------------------------
Description:
See HUDI-6863 and
https://github.com/apache/hudi/pull/6802#issuecomment-1455802492
I think we need to make sure that the dedup parallelism is only applied to the
dedup stage, not affecting subsequent stages, which may require better
parallelism control by repartitioning with right parallelism before workload
profiling.
> Auto-tune dedup parallelism without affecting write parallelism
> ---------------------------------------------------------------
>
> Key: HUDI-6864
> URL: https://issues.apache.org/jira/browse/HUDI-6864
> Project: Apache Hudi
> Issue Type: Improvement
> Reporter: Ethan Guo
> Priority: Major
> Fix For: 0.14.1
>
>
> See HUDI-6863 and
> https://github.com/apache/hudi/pull/6802#issuecomment-1455802492
> I think we need to make sure that the dedup parallelism is only applied to
> the dedup stage, not affecting subsequent stages, which may require better
> parallelism control by repartitioning with right parallelism before workload
> profiling.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
