Re: [Streams] partition expansion of input streams

[Pegerto Fernandez Torres]

Hello Matthias,

No is not too late, I understand the complexity of change of modify the
topic size.

The approach to oversize the partition is implemented from day one, but
having multiple topics one each domain subject, I want to consider the case
of someone miss sizing a specific domain.

I think the approach you mention of having 2 application ids is the same
topic is the safer, and even if costly it's a valid approach for other use
cases and not only partition resizing.

Thanks for the advice

On 9 Feb 2018 6:56 pm, "Matthias J. Sax"  wrote:

Changing source topic partitions for stateful application is very difficult.

The best thing is to overpartition source topics to avoid this issue in
the first place. (Too late for you I guess.)

If you really need to change the partitions count, the cleanest solution
is, to create a second input topic with the new number of partitions and
start a new copy of the application that reads from the new source topic
and uses a different application.id. You run both applications in
parallel until the new application is ready to take over (the new app
should write to different output topics, too). During this time, your
producers should write to both topic.

It would also be possible, to copy the data from the old to the new
topic (new app builder.stream("old").to("new")) to let the new
application catch up quicker. This requires a syncronized switch of the
producer applications from old to new topic though Ie, run the "copy
app" until it reaches the end of the log. Stop Producers; let "copy app"
finish to copy the latest writes; restart producers and point to new topic.

If you can accept application downtime, it's of course simpler. You can
stop the app, reset it via the reset tool and clean local state. Change
number of partitions. Restart the app.

All approaches are difficult... There might be other approaches too. But
you cannot change the number of input topic partitions on the fly while
the application is running. It would crash.


-Matthias

On 2/9/18 6:51 AM, Pegerto Fernandez Torres wrote:
> Hello,
>
> I start to think in the problem of expanding our source partitions, it
> looks like a stateless stream will handle the expansion properly after
> Rebalancing.
>
> In the case of a state-full then the problem seems different, first
problem
> is the changelog itself, the expected definition will miss match.
>
> The second problem is the locality of the data, the results as new
> partition will ben process in a task that dont have the associated state.
>
>  Have someone design some work around or is there any additional
> documentation I should look at?
>

-- 


Check out our new blog, "Get your -aas in gear: Privatelink and PaaS 
democratisation on AWS" by Guy Richardson  


opencredo.com  . Twitter 
 . LinkedIn 



OpenCredo Ltd -- Delivering Software Innovation

Registered Office:  5-11 Lavington Street, London, SE1 0NZ
Registered in UK. No 3943999

Re: [Streams] partition expansion of input streams

[Matthias J. Sax]

Changing source topic partitions for stateful application is very difficult.

The best thing is to overpartition source topics to avoid this issue in
the first place. (Too late for you I guess.)

If you really need to change the partitions count, the cleanest solution
is, to create a second input topic with the new number of partitions and
start a new copy of the application that reads from the new source topic
and uses a different application.id. You run both applications in
parallel until the new application is ready to take over (the new app
should write to different output topics, too). During this time, your
producers should write to both topic.

It would also be possible, to copy the data from the old to the new
topic (new app builder.stream("old").to("new")) to let the new
application catch up quicker. This requires a syncronized switch of the
producer applications from old to new topic though Ie, run the "copy
app" until it reaches the end of the log. Stop Producers; let "copy app"
finish to copy the latest writes; restart producers and point to new topic.

If you can accept application downtime, it's of course simpler. You can
stop the app, reset it via the reset tool and clean local state. Change
number of partitions. Restart the app.

All approaches are difficult... There might be other approaches too. But
you cannot change the number of input topic partitions on the fly while
the application is running. It would crash.


-Matthias

On 2/9/18 6:51 AM, Pegerto Fernandez Torres wrote:
> Hello,
> 
> I start to think in the problem of expanding our source partitions, it
> looks like a stateless stream will handle the expansion properly after
> Rebalancing.
> 
> In the case of a state-full then the problem seems different, first problem
> is the changelog itself, the expected definition will miss match.
> 
> The second problem is the locality of the data, the results as new
> partition will ben process in a task that dont have the associated state.
> 
>  Have someone design some work around or is there any additional
> documentation I should look at?
> 



signature.asc
Description: OpenPGP digital signature

[Streams] partition expansion of input streams

[Pegerto Fernandez Torres]

Hello,

I start to think in the problem of expanding our source partitions, it
looks like a stateless stream will handle the expansion properly after
Rebalancing.

In the case of a state-full then the problem seems different, first problem
is the changelog itself, the expected definition will miss match.

The second problem is the locality of the data, the results as new
partition will ben process in a task that dont have the associated state.

 Have someone design some work around or is there any additional
documentation I should look at?

-- 

Pegerto Fernandez

-- 


Check out our new blog, "Get your -aas in gear: Privatelink and PaaS 
democratisation on AWS" by Guy Richardson  


opencredo.com  . Twitter 
 . LinkedIn 



OpenCredo Ltd -- Delivering Software Innovation

Registered Office:  5-11 Lavington Street, London, SE1 0NZ
Registered in UK. No 3943999