Changing source topic partitions for stateful application is very difficult.

The best thing is to overpartition source topics to avoid this issue in
the first place. (Too late for you I guess.)

If you really need to change the partitions count, the cleanest solution
is, to create a second input topic with the new number of partitions and
start a new copy of the application that reads from the new source topic
and uses a different application.id. You run both applications in
parallel until the new application is ready to take over (the new app
should write to different output topics, too). During this time, your
producers should write to both topic.

It would also be possible, to copy the data from the old to the new
topic (new app builder.stream("old").to("new")) to let the new
application catch up quicker. This requires a syncronized switch of the
producer applications from old to new topic though.... Ie, run the "copy
app" until it reaches the end of the log. Stop Producers; let "copy app"
finish to copy the latest writes; restart producers and point to new topic.

If you can accept application downtime, it's of course simpler. You can
stop the app, reset it via the reset tool and clean local state. Change
number of partitions. Restart the app.

All approaches are difficult... There might be other approaches too. But
you cannot change the number of input topic partitions on the fly while
the application is running. It would crash.


-Matthias

On 2/9/18 6:51 AM, Pegerto Fernandez Torres wrote:
> Hello,
> 
> I start to think in the problem of expanding our source partitions, it
> looks like a stateless stream will handle the expansion properly after
> Rebalancing.
> 
> In the case of a state-full then the problem seems different, first problem
> is the changelog itself, the expected definition will miss match.
> 
> The second problem is the locality of the data, the results as new
> partition will ben process in a task that dont have the associated state.
> 
>  Have someone design some work around or is there any additional
> documentation I should look at?
> 

Attachment: signature.asc
Description: OpenPGP digital signature

Reply via email to