Re: When using the HoodieDeltaStreamer, is there a corresponding parameter that can control the number of cycles? For example, if I cycle 5 times, I stop accessing data

[Sivabalan]

Can you try with latest master. We have fixed
 this recently.

On Mon, 3 Apr 2023 at 19:49, lee  wrote:

>
> Should we stop SparkContext?
> 李杰
> leedd1...@163.com
>
> 
>  Replied Message 
> From lee 
> Date 4/3/2023 11:09
> To Sivabalan 
> Cc dev@hudi.apache.org 
> Subject Re: When using the HoodieDeltaStreamer, is there a corresponding
> parameter that can control the number of cycles? For example, if I cycle 5
> times, I stop accessing data
> I tried using the
> 'org.apache.hudi.utilities.deltastreamer.NoNewDataTerminationStrategy' to
> stop the task, but it didn't seem to meet my expectations. I think that
> after it stops ExecutorService, the subsequent SparkContext will also stop,
> but now SparkContext will always be started and no subsequent logs will be
> visible.
>
>
>
>
> 李杰
> leedd1...@163.com
>
> 
>  Replied Message 
> From Sivabalan 
> Date 4/1/2023 01:07
> To  
> Subject Re: When using the HoodieDeltaStreamer, is there a corresponding
> parameter that can control the number of cycles? For example, if I cycle 5
> times, I stop accessing data
> We do have Graceful termination possibility w/ deltastreamer
> continuous mode. Please check here
> <
> https://hudi.apache.org/releases/release-0.12.2#deltastreamer-termination-strategy
> >
> for post write termination strategy. You can implement your own termination
> strategy. Hope that helps.
>
> On Thu, 30 Mar 2023 at 20:16, Vinoth Chandar  wrote:
>
> I believe there is no control today. You could hack a precommit validator
> and call System.exit if you want ;) (ugly, I know)
>
> But maybe we could introduce some abstraction to do a check between loops?
> or allow users to plugin some logic to decide whether to continue or exit?
>
> Love to understand the use-case more here.
>
> On Wed, Mar 29, 2023 at 7:32 AM lee  wrote:
>
> When I use the HoodieDeltaStreamer, the "-- continuous" parameter: "Delta
> Streamer runs in continuous mode running source match ->Transform ->Hudi
> Write in loop". So I would like to ask if there are any corresponding
> parameters that can control the number of cycles, such as stopping
> accessing data when I cycle 5 times.
>
>
>
> 李杰
> leedd1...@163.com
>
> <
>
>
> https://dashi.163.com/projects/signature-manager/detail/index.html?ftlId=1=%E6%9D%8E%E6%9D%B0=leedd1912%40163.com=https%3A%2F%2Fmail-online.nosdn.127.net%2Fsmc4215b668fdb6b5ca355a1c3319c4a0e.jpg=%5B%22leedd1912%40163.com%22%5D
>
>
>
>
>
>
> --
> Regards,
> -Sivabalan
>


-- 
Regards,
-Sivabalan

Re: When using the HoodieDeltaStreamer, is there a corresponding parameter that can control the number of cycles? For example, if I cycle 5 times, I stop accessing data

[lee]

Should we stop SparkContext?
| |
李杰
|
|
leedd1...@163.com
|
 Replied Message 
| From | lee |
| Date | 4/3/2023 11:09 |
| To | Sivabalan |
| Cc | dev@hudi.apache.org |
| Subject | Re: When using the HoodieDeltaStreamer, is there a corresponding 
parameter that can control the number of cycles? For example, if I cycle 5 
times, I stop accessing data |
I tried using the 
'org.apache.hudi.utilities.deltastreamer.NoNewDataTerminationStrategy' to stop 
the task, but it didn't seem to meet my expectations. I think that after it 
stops ExecutorService, the subsequent SparkContext will also stop, but now 
SparkContext will always be started and no subsequent logs will be visible.








| |
李杰
|
|
leedd1...@163.com
|
 Replied Message 
| From | Sivabalan |
| Date | 4/1/2023 01:07 |
| To |  |
| Subject | Re: When using the HoodieDeltaStreamer, is there a corresponding 
parameter that can control the number of cycles? For example, if I cycle 5 
times, I stop accessing data |
We do have Graceful termination possibility w/ deltastreamer
continuous mode. Please check here

for post write termination strategy. You can implement your own termination
strategy. Hope that helps.

On Thu, 30 Mar 2023 at 20:16, Vinoth Chandar  wrote:

I believe there is no control today. You could hack a precommit validator
and call System.exit if you want ;) (ugly, I know)

But maybe we could introduce some abstraction to do a check between loops?
or allow users to plugin some logic to decide whether to continue or exit?

Love to understand the use-case more here.

On Wed, Mar 29, 2023 at 7:32 AM lee  wrote:

When I use the HoodieDeltaStreamer, the "-- continuous" parameter: "Delta
Streamer runs in continuous mode running source match ->Transform ->Hudi
Write in loop". So I would like to ask if there are any corresponding
parameters that can control the number of cycles, such as stopping
accessing data when I cycle 5 times.



李杰
leedd1...@163.com

<
https://dashi.163.com/projects/signature-manager/detail/index.html?ftlId=1=%E6%9D%8E%E6%9D%B0=leedd1912%40163.com=https%3A%2F%2Fmail-online.nosdn.127.net%2Fsmc4215b668fdb6b5ca355a1c3319c4a0e.jpg=%5B%22leedd1912%40163.com%22%5D





--
Regards,
-Sivabalan

Re: When using the HoodieDeltaStreamer, is there a corresponding parameter that can control the number of cycles? For example, if I cycle 5 times, I stop accessing data

[lee]

I tried using the 
'org.apache.hudi.utilities.deltastreamer.NoNewDataTerminationStrategy' to stop 
the task, but it didn't seem to meet my expectations. I think that after it 
stops ExecutorService, the subsequent SparkContext will also stop, but now 
SparkContext will always be started and no subsequent logs will be visible.








| |
李杰
|
|
leedd1...@163.com
|
 Replied Message 
| From | Sivabalan |
| Date | 4/1/2023 01:07 |
| To |  |
| Subject | Re: When using the HoodieDeltaStreamer, is there a corresponding 
parameter that can control the number of cycles? For example, if I cycle 5 
times, I stop accessing data |
We do have Graceful termination possibility w/ deltastreamer
continuous mode. Please check here

for post write termination strategy. You can implement your own termination
strategy. Hope that helps.

On Thu, 30 Mar 2023 at 20:16, Vinoth Chandar  wrote:

I believe there is no control today. You could hack a precommit validator
and call System.exit if you want ;) (ugly, I know)

But maybe we could introduce some abstraction to do a check between loops?
or allow users to plugin some logic to decide whether to continue or exit?

Love to understand the use-case more here.

On Wed, Mar 29, 2023 at 7:32 AM lee  wrote:

When I use the HoodieDeltaStreamer, the "-- continuous" parameter: "Delta
Streamer runs in continuous mode running source match ->Transform ->Hudi
Write in loop". So I would like to ask if there are any corresponding
parameters that can control the number of cycles, such as stopping
accessing data when I cycle 5 times.



李杰
leedd1...@163.com

<
https://dashi.163.com/projects/signature-manager/detail/index.html?ftlId=1=%E6%9D%8E%E6%9D%B0=leedd1912%40163.com=https%3A%2F%2Fmail-online.nosdn.127.net%2Fsmc4215b668fdb6b5ca355a1c3319c4a0e.jpg=%5B%22leedd1912%40163.com%22%5D





--
Regards,
-Sivabalan

Re: When using the HoodieDeltaStreamer, is there a corresponding parameter that can control the number of cycles? For example, if I cycle 5 times, I stop accessing data

[Sivabalan]

We do have Graceful termination possibility w/ deltastreamer
continuous mode. Please check here

for post write termination strategy. You can implement your own termination
strategy. Hope that helps.

On Thu, 30 Mar 2023 at 20:16, Vinoth Chandar  wrote:

> I believe there is no control today. You could hack a precommit validator
> and call System.exit if you want ;) (ugly, I know)
>
> But maybe we could introduce some abstraction to do a check between loops?
> or allow users to plugin some logic to decide whether to continue or exit?
>
> Love to understand the use-case more here.
>
> On Wed, Mar 29, 2023 at 7:32 AM lee  wrote:
>
> > When I use the HoodieDeltaStreamer, the "-- continuous" parameter: "Delta
> > Streamer runs in continuous mode running source match ->Transform ->Hudi
> > Write in loop". So I would like to ask if there are any corresponding
> > parameters that can control the number of cycles, such as stopping
> > accessing data when I cycle 5 times.
> >
> >
> >
> > 李杰
> > leedd1...@163.com
> >
> > <
> https://dashi.163.com/projects/signature-manager/detail/index.html?ftlId=1=%E6%9D%8E%E6%9D%B0=leedd1912%40163.com=https%3A%2F%2Fmail-online.nosdn.127.net%2Fsmc4215b668fdb6b5ca355a1c3319c4a0e.jpg=%5B%22leedd1912%40163.com%22%5D
> >
> >
>


-- 
Regards,
-Sivabalan

Re: When using the HoodieDeltaStreamer, is there a corresponding parameter that can control the number of cycles? For example, if I cycle 5 times, I stop accessing data

[Vinoth Chandar]

I believe there is no control today. You could hack a precommit validator
and call System.exit if you want ;) (ugly, I know)

But maybe we could introduce some abstraction to do a check between loops?
or allow users to plugin some logic to decide whether to continue or exit?

Love to understand the use-case more here.

On Wed, Mar 29, 2023 at 7:32 AM lee  wrote:

> When I use the HoodieDeltaStreamer, the "-- continuous" parameter: "Delta
> Streamer runs in continuous mode running source match ->Transform ->Hudi
> Write in loop". So I would like to ask if there are any corresponding
> parameters that can control the number of cycles, such as stopping
> accessing data when I cycle 5 times.
>
>
>
> 李杰
> leedd1...@163.com
>
> 
>

When using the HoodieDeltaStreamer, is there a corresponding parameter that can control the number of cycles? For example, if I cycle 5 times, I stop accessing data

[lee]

When I use the HoodieDeltaStreamer, the "-- continuous" parameter: "Delta 
Streamer runs in continuous mode running source match ->Transform ->Hudi Write 
in loop". So I would like to ask if there are any corresponding parameters that 
can control the number of cycles, such as stopping accessing data when I cycle 
5 times.







| |
李杰
|
|
leedd1...@163.com
|