Danny Chen created HUDI-7574:
--------------------------------
Summary: Auto-pilot for Flink Hudi sink tasks
Key: HUDI-7574
URL: https://issues.apache.org/jira/browse/HUDI-7574
Project: Apache Hudi
Issue Type: Improvement
Components: flink
Reporter: Danny Chen
Assignee: Danny Chen
Fix For: 1.0.0
Currently the flink write task parallelism is set up throughÂ
{code:java}
write.tasks{code}
it is kind of a fixed number during the lifecycle of the ingestion pipeline,
while for streaming, there are always fluctuation of the workload, it is great
if we can tune the parallelism of write tasks based on the job load profile
dynamically.
On K8s, Flink provides a
[https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/autoscaler/]
which is suitable for the purpose, which deserves a further investigation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
