Hi, Sivabalan,
Great to hear from you. Then I will create a JIRA ticket to track this feature Best Regards At 2023-05-24 02:22:36, "Sivabalan" <n.siv...@gmail.com> wrote: >I could not see the image you have attached. But I do get your ask here. >Will definitely benefit continuous deltastreamer use-cases. One possible >option is to keep track of the last mod time of the property file that we >feed in for detlastreamer top level config and before every batch, we can >check if it has changed. Optionally re-instantiate write config and other >components (write client etc) if applicable. If not, proceed as usual. >Should not be hard to add the support. > > > > >On Mon, 22 May 2023 at 00:05, 孔维 <18701146...@163.com> wrote: > >> Hi team, >> >> I am thinking about whether it is necessary to add the feature of >> configuration hot update to deltastreamer. >> >> In our company, hudi is used as a platform. We provide deltastreamer (run >> in continuous mode) to write to a large number of sources (including mysql >> & tidb) as a long time service. We often need to update the hudi >> configuration, but we don’t want to restart deltastreamer to achieve it, >> which may be too heavy for our job scheduler server based on livy/yarn. >> Therefore, we provide deltastreamer configuration hot update function. It >> is possible to update some common parameters instantly, and these >> parameters will take effect at the next sync of deltastreamer. These >> parameters include: >> >> - hoodie.bulkinsert.shuffle.parallelism (used only in bulkinsert) >> - hoodie.upsert.shuffle.parallelism >> - hoodie.deltastreamer.kafka.source.maxEvents >> - hoodie.memory.merge.max.size >> - hoodie.memory.compaction.max.size >> - hoodie.datasource.hive_sync.* >> - hoodie.compact.inline.max.delta.commits >> - hoodie.compaction.strategy >> - hoodie.compaction.target.io >> >> The whole flow chart is as follows: >> >> > >-- >Regards, >-Sivabalan
