Kong Wei created HUDI-6259:
------------------------------
Summary: deltastreamer support configuration hot update
Key: HUDI-6259
URL: https://issues.apache.org/jira/browse/HUDI-6259
Project: Apache Hudi
Issue Type: New Feature
Components: deltastreamer
Reporter: Kong Wei
Assignee: Kong Wei
Attachments: image-2023-05-24-14-26-45-826.png
In our company, hudi is used as a platform. We provide deltastreamer (run in
continuous mode) to write to a large number of sources (including mysql & tidb)
as a long time service. We often need to update the hudi configuration, but we
don’t want to restart deltastreamer to achieve it, which may be too heavy for
our job scheduler server based on livy/yarn. Therefore, we provide
deltastreamer configuration hot update function. It is possible to update some
common parameters instantly, and these parameters will take effect at the next
sync of deltastreamer. These parameters include:
* hoodie.bulkinsert.shuffle.parallelism (used only in bulkinsert)
* hoodie.upsert.shuffle.parallelism
* hoodie.deltastreamer.kafka.source.maxEvents
* hoodie.memory.merge.max.size
* hoodie.memory.compaction.max.size
* hoodie.datasource.hive_sync.*
* hoodie.compact.inline.max.delta.commits
* hoodie.compaction.strategy
* hoodie.compaction.target.io
!image-2023-05-24-14-26-45-826.png!
So this JIRA ticket is to add the configuration hot update function to
deltastreamer
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
