[
https://issues.apache.org/jira/browse/CAMEL-15562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17502840#comment-17502840
]
Otavio Rodolfo Piske commented on CAMEL-15562:
----------------------------------------------
[~davsclaus] +1 from me. I'll close it and will create a new one.
> repository & strategy to store and retrieve the latest state of a consumer
> and allow to resume upon restart
> -----------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-15562
> URL: https://issues.apache.org/jira/browse/CAMEL-15562
> Project: Camel
> Issue Type: New Feature
> Components: camel-core-api, camel-core-engine
> Reporter: Luca Burgazzoli
> Assignee: Otavio Rodolfo Piske
> Priority: Major
> Fix For: 3.16.0
>
>
> A number of component support resuming operation from a given point in time
> or index but each component implement its own logic and it is currently not
> possible to use an external system to store such information.
> Because of that, even if the source support resuming from a point, we
> sometime still need to use an idempotent consumer which is ok but inefficient
> because the consumer may need to re-process events from the source as it
> lacks a way to retrieve from where it left.
> We should provide a generic mechanism to implement such logic so that:
> # it should be possible to restart a camel application without having to
> re-process events from the source even without the need of an idempotent
> repository (for sources that support resuming)
> # in case of sources that support partitioning, it should be possible to
> consume data from an elastic pool of camel consumers as the information about
> the consumer state is stored in an external repository so the system can
> rebalance the load anc new consumer can start where others have left
> *InProgressRepository*
> This repository is similar to the IndempotentRepository except it should
> allow to query the status of the entries.
> *InProgressStrategy*
> A simple strategy that automatically update the InProgressRepository
> according to some headers like:
> # _CamelInProgressID_: the key of the event
> # _CamelInProgressValue_: the value (opaque)
> # _CamelInProgressStatus_: the status of the consumer for the given key (i.e.
> could be InProgress when the consumer is processing a file and become Done
> when the file is fully processed)
> We should also implement a mechanism to be able to consistently determine the
> "partition key" according to the active consumers.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
