Re: [DISCUSS] Should we support a service to manage all deltastreamer jobs?

[Vinoth Chandar]

+1 there are RFCs on table management services, but not specific to
deltastreamer itself.

Are you proposing building something specific to that?

On Wed, Jun 14, 2023 at 8:26 AM Pratyaksh Sharma 
wrote:

> Hi,
>
> Personally I am in favour of creating such a UI where monitoring and
> managing configurations is just a click away. That makes life a lot easier
> for users. So +1 on the proposal.
>
> I remember the work for it had started long back around 2019. You can check
> this RFC
> <
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=130027233
> >
> for your reference. I am not sure why this work could not continue though.
>
> On Wed, Jun 14, 2023 at 4:28 PM 孔维 <18701146...@163.com> wrote:
>
> > Hi, team,
> >
> >
> > Background:
> > More and more hudi accesses use deltastreamer, resulting in a large
> number
> > of deltastreamer jobs that need to be managed. In our company, we also
> > manage a large number of deltastreamer jobs by ourselves, and there is a
> > lot of operation and maintenance management and monitoring work.
> > If we can provide such a deltastreamer service to create, manage, and
> > monitor all tasks in a unified manner, it can greatly reduce the
> management
> > pressure of deltastreamer, and at the same time lower the threshold for
> > using deltastreamer, which is conducive to the promotion and use of
> > deltastreamer.
> > At the same time, considering that deltastreamer already supports
> > configuration hot update capability [
> > https://github.com/apache/hudi/pull/8807], we can offer configuration
> hot
> > update capability based on the feature, and make configuration changes
> > without restarting the job.
> >
> >
> > We hope to provide:
> > Provides a web UI to support creation, management and monitoring of
> > deltastreamer tasks
> > Using configuration hot update capability to provide timely configuration
> > change capability
> >
> >
> > I don't know whether such a service is in line with the evolution of the
> > community, and I hope to receive your reply!
> >
> >
> > Best Regards
>

Re: [DISCUSS] Should we support a service to manage all deltastreamer jobs?

[Pratyaksh Sharma]

Hi,

Personally I am in favour of creating such a UI where monitoring and
managing configurations is just a click away. That makes life a lot easier
for users. So +1 on the proposal.

I remember the work for it had started long back around 2019. You can check
this RFC

for your reference. I am not sure why this work could not continue though.

On Wed, Jun 14, 2023 at 4:28 PM 孔维 <18701146...@163.com> wrote:

> Hi, team,
>
>
> Background:
> More and more hudi accesses use deltastreamer, resulting in a large number
> of deltastreamer jobs that need to be managed. In our company, we also
> manage a large number of deltastreamer jobs by ourselves, and there is a
> lot of operation and maintenance management and monitoring work.
> If we can provide such a deltastreamer service to create, manage, and
> monitor all tasks in a unified manner, it can greatly reduce the management
> pressure of deltastreamer, and at the same time lower the threshold for
> using deltastreamer, which is conducive to the promotion and use of
> deltastreamer.
> At the same time, considering that deltastreamer already supports
> configuration hot update capability [
> https://github.com/apache/hudi/pull/8807], we can offer configuration hot
> update capability based on the feature, and make configuration changes
> without restarting the job.
>
>
> We hope to provide:
> Provides a web UI to support creation, management and monitoring of
> deltastreamer tasks
> Using configuration hot update capability to provide timely configuration
> change capability
>
>
> I don't know whether such a service is in line with the evolution of the
> community, and I hope to receive your reply!
>
>
> Best Regards

[DISCUSS] Should we support a service to manage all deltastreamer jobs?

[孔维]

Hi, team,


Background:
More and more hudi accesses use deltastreamer, resulting in a large number of 
deltastreamer jobs that need to be managed. In our company, we also manage a 
large number of deltastreamer jobs by ourselves, and there is a lot of 
operation and maintenance management and monitoring work. 
If we can provide such a deltastreamer service to create, manage, and monitor 
all tasks in a unified manner, it can greatly reduce the management pressure of 
deltastreamer, and at the same time lower the threshold for using 
deltastreamer, which is conducive to the promotion and use of deltastreamer.
At the same time, considering that deltastreamer already supports configuration 
hot update capability [https://github.com/apache/hudi/pull/8807], we can offer 
configuration hot update capability based on the feature, and make 
configuration changes without restarting the job.


We hope to provide:
Provides a web UI to support creation, management and monitoring of 
deltastreamer tasks
Using configuration hot update capability to provide timely configuration 
change capability


I don't know whether such a service is in line with the evolution of the 
community, and I hope to receive your reply!


Best Regards