[
https://issues.apache.org/jira/browse/CAMEL-15955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claus Ibsen updated CAMEL-15955:
--------------------------------
Fix Version/s: (was: 3.10.0)
3.11.0
> camel-core - API and SPI for Camel components to resume from offset
> -------------------------------------------------------------------
>
> Key: CAMEL-15955
> URL: https://issues.apache.org/jira/browse/CAMEL-15955
> Project: Camel
> Issue Type: New Feature
> Components: camel-core
> Reporter: Claus Ibsen
> Assignee: Claus Ibsen
> Priority: Major
> Fix For: 3.11.0
>
>
> We need a API and SPI for components to optionally support some way of offset
> management. Meaning that a component is able to resume from a given offset to
> continue from last known state. For example if a Camel application was
> stopped/crashed, then it can restart and resume from last known offset.
> Something like Apache Kafka offset has.
> This benefits Camel in general, and Camel Kafka Connector also.
> An use-case is the famous big files. So the file component is reading a file,
> and process it line by line via a splitter. Then we would like to store an
> offset how far it has processed. And in case of crash/restart etc then it can
> continue from last known position. Today it would restart from top.
> Same for SQL,JDBC components with SQL ResultSet and so on.
> The offset management should support different quality levels and a SPI so
> different implementations can plugin. One could be to store to a plain file
> and for each step. And other can have a batch to store after N rows or N
> seconds. Or like in Kafka with every N seconds.
> Then each Camel component that can support this, can then implement the API
> and logic for handling this. For the file component we can maybe store an
> offset in the input stream, so when resuming we then forward the stream to
> that position.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
