[
https://issues.apache.org/jira/browse/HUDI-6259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen closed HUDI-6259.
----------------------------
Fix Version/s: 0.14.0
Resolution: Fixed
Fixed via master branch: 91623814132c34f76e46841e6c369356de5df2f1
> deltastreamer support configuration hot update
> ----------------------------------------------
>
> Key: HUDI-6259
> URL: https://issues.apache.org/jira/browse/HUDI-6259
> Project: Apache Hudi
> Issue Type: New Feature
> Components: deltastreamer
> Reporter: Kong Wei
> Assignee: Kong Wei
> Priority: Major
> Labels: pull-request-available
> Fix For: 0.14.0
>
> Attachments: image-2023-05-24-14-26-45-826.png
>
>
> In our company, hudi is used as a platform. We provide deltastreamer (run in
> continuous mode) to write to a large number of sources (including mysql &
> tidb) as a long time service. We often need to update the hudi configuration,
> but we don’t want to restart deltastreamer to achieve it, which may be too
> heavy for our job scheduler server based on livy/yarn. Therefore, we provide
> deltastreamer configuration hot update function. It is possible to update
> some common parameters instantly, and these parameters will take effect at
> the next sync of deltastreamer. These parameters include:
> * hoodie.bulkinsert.shuffle.parallelism (used only in bulkinsert)
> * hoodie.upsert.shuffle.parallelism
> * hoodie.deltastreamer.kafka.source.maxEvents
> * hoodie.memory.merge.max.size
> * hoodie.memory.compaction.max.size
> * hoodie.datasource.hive_sync.*
> * hoodie.compact.inline.max.delta.commits
> * hoodie.compaction.strategy
> * hoodie.compaction.target.io
> !image-2023-05-24-14-26-45-826.png!
> So this JIRA ticket is to add the configuration hot update function to
> deltastreamer
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
