sure. sg. thanks! On Tue, 23 May 2023 at 23:22, 孔维 <18701146...@163.com> wrote: > > Hi, Sivabalan, > > > Great to hear from you. Then I will create a JIRA ticket to track this feature > > > Best Regards > > At 2023-05-24 02:22:36, "Sivabalan" <n.siv...@gmail.com> wrote: > >I could not see the image you have attached. But I do get your ask here. > >Will definitely benefit continuous deltastreamer use-cases. One possible > >option is to keep track of the last mod time of the property file that we > >feed in for detlastreamer top level config and before every batch, we can > >check if it has changed. Optionally re-instantiate write config and other > >components (write client etc) if applicable. If not, proceed as usual. > >Should not be hard to add the support. > > > > > > > > > >On Mon, 22 May 2023 at 00:05, 孔维 <18701146...@163.com> wrote: > > > >> Hi team, > >> > >> I am thinking about whether it is necessary to add the feature of > >> configuration hot update to deltastreamer. > >> > >> In our company, hudi is used as a platform. We provide deltastreamer (run > >> in continuous mode) to write to a large number of sources (including mysql > >> & tidb) as a long time service. We often need to update the hudi > >> configuration, but we don’t want to restart deltastreamer to achieve it, > >> which may be too heavy for our job scheduler server based on livy/yarn. > >> Therefore, we provide deltastreamer configuration hot update function. It > >> is possible to update some common parameters instantly, and these > >> parameters will take effect at the next sync of deltastreamer. These > >> parameters include: > >> > >> - hoodie.bulkinsert.shuffle.parallelism (used only in bulkinsert) > >> - hoodie.upsert.shuffle.parallelism > >> - hoodie.deltastreamer.kafka.source.maxEvents > >> - hoodie.memory.merge.max.size > >> - hoodie.memory.compaction.max.size > >> - hoodie.datasource.hive_sync.* > >> - hoodie.compact.inline.max.delta.commits > >> - hoodie.compaction.strategy > >> - hoodie.compaction.target.io > >> > >> The whole flow chart is as follows: > >> > >> > > > >-- > >Regards, > >-Sivabalan
-- Regards, -Sivabalan
