[
https://issues.apache.org/jira/browse/CAMEL-23239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18103595#comment-18103595
]
Raymond commented on CAMEL-23239:
---------------------------------
Another comment as I keep coming across this use case where I would like to
store state. I've used several solutions so far, but I think to store data in a
key-value store (embedded or external like Redis) than keep in memory, store it
on disk or send it to a classical database (like Postgres) make more sense.
For now, I have always written separate bindings for Redis, MongoDB,
Elasticsearch, ActiveMQ and MapDB. However, it would be nice if there was an
abstraction layer (some sort of SPI/plugin system) with two possibilities:
1. Use a key-value embedded storage (datastore). Currently, I use MapDB, which
is relatively straightforward. However, there are other options, such as
Chronicle Map, H2 MV, Apache Ignite and JetBrains Xodus. There are also
non-Java options with Java APIs, such as RocksDB (via RocksJava), LMDB (via
LmdbJava) and DuckDB.
2. Use external datastores such as Redis, DynamoDB, MemCached, etcd, Hazelcast,
EHcache and Valkey.
Imagine that you configure/define the type of data you want to store and then
use the SPI to decide whether to store it embedded or elsewhere. I think this
aligns with the idea described in the issue description.
Camel State Store
│
┌────────────────┼────────────────┐
│ │ │
Value state Collection Exchange state
│ │ │
variables ErrorRegistry MessageHistory
counters deduplication AggregateRepository
tokens registries Claim Check
Data that I would like to store:
1. Store collections (like SimpleRegistry, ErrorRegistry, EndpointRegistry and
ServiceRegistry).
2. Store exchanges (EventNotifier, MessageHistory view/replay, Tracer and
BacklogTracer).
3. Store queue messages (SEDA).
4. Store variables (VariableRepository).
5. Store metrics (OpenTelemetry, JMX).
6. Store/cache (routes).
7. Store EIP state (AggregateRepository, Resequencer, Throttler, SAGA)
8. Store security (OAuth/token)
9. Store application properties (like key-value in main)
Not all things in the list are excellent fits for a K/V storage, but I just
wanted to be complete.
Goals I would like to achieve:
1. Off-heap data in memory / Reduce memory pressure
2. Offload internal Camel data to external systems.
3. Persistence (ensuring survival of a CamelContext or Container restart).
4. Endpoint: Use the store as an endpoint in routes.
5. Replacement extension of the IdempotentRepository
6. Decouple state lifecycle from CamelContext lifecycle
7. Better observability
To be honest, I long didn't want to store state (as this is kind of an old best
practice in data integration that says
that messages need to go through integrations independently and fast, however
especially to manage and observe these
integrations, it becomes increasingly important.
You see now that a lot of big Camel users all are implementing (sometimes
subpar) solutions on their own that is not SPI based but tied to one datastore.
At the end it's about a uniform, flexible and easy-to-use solution
> Add camel-state-store component with pluggable key-value store
> --------------------------------------------------------------
>
> Key: CAMEL-23239
> URL: https://issues.apache.org/jira/browse/CAMEL-23239
> Project: Camel
> Issue Type: New Feature
> Components: camel-core
> Reporter: Guillaume Nodet
> Assignee: Guillaume Nodet
> Priority: Major
> Fix For: 4.23.0
>
>
> h2. Motivation
> Camel provides dedicated components for specific caching/store technologies
> (Caffeine, Redis, Infinispan, etc.), each with its own API surface. This
> works well when users need the full feature set of a specific technology, but
> it creates friction when the actual need is simple: store and retrieve
> key-value pairs.
> The {{camel-state-store}} component follows the same "choose the problem, not
> the technology" pattern that Camel already uses successfully in other areas:
> * {{camel-sql}} / {{camel-jdbc}} — generic SQL over any JDBC database,
> without locking into a vendor
> * {{camel-jms}} — generic messaging over any JMS provider (with
> {{camel-activemq}}, {{camel-amqp}} as pre-configured variants)
> * {{camel-jcache}} — generic caching via JSR-107 over any compliant
> implementation
> Similarly, {{camel-state-store}} provides a unified key-value API where:
> * Users choose the capability first ("I need a key-value store") rather than
> a specific technology
> * The backend is swappable without changing route logic — develop with
> in-memory, deploy with Redis or Infinispan
> * The API surface is intentionally minimal (put, get, delete, contains, keys,
> clear) — unlike the full-featured technology-specific components
> h2. Difference from existing cache components
> || || camel-state-store || camel-caffeine-cache / camel-infinispan / etc. ||
> | *Focus* | Simple key-value store abstraction | Full feature set of a
> specific technology |
> | *Backend* | Pluggable via {{StateStoreBackend}} interface | Fixed to one
> technology |
> | *API* | Minimal: put, get, delete, contains, keys, clear | Rich: queries,
> events, statistics, pub/sub, etc. |
> | *Use case* | Portability, simplicity, migration from MuleSoft Object Store
> | Deep integration with a specific product |
> h2. Features
> * New {{camel-state-store}} component providing a simple, unified key-value
> store API with pluggable backends
> * Supports operations: {{put}}, {{putIfAbsent}}, {{get}}, {{delete}},
> {{contains}}, {{keys}}, {{size}}, {{clear}}
> * Per-entry TTL with endpoint-level default and per-message override via
> {{CamelStateStoreTtl}} header
> * *Auto-discovery*: if a single {{StateStoreBackend}} bean is in the
> registry, it is used automatically — no {{backend=#beanName}} needed on
> endpoints. Logs a WARN when multiple backends are found.
> * *Property-based configuration*: backends can be fully configured via
> {{application.properties}} using {{camel.beans.*}} syntax — no Java code
> required
> * Multi-module structure with pluggable backends:
> ** {{camel-state-store}}: core + in-memory backend ({{ConcurrentHashMap}},
> lazy TTL)
> ** {{camel-state-store-caffeine}}: Caffeine cache with per-entry variable
> expiry
> ** {{camel-state-store-redis}}: Redisson {{RMapCache}} with native TTL
> ** {{camel-state-store-infinispan}}: Hot Rod client with lifespan TTL
> * Custom backends via {{StateStoreBackend}} interface and bean references
> * Thread-safe backend lifecycle: {{start()}} called once per backend,
> idempotent guards in all backends
> h2. Route examples
> h3. Simple put/get with in-memory backend
> {code:java}
> from("direct:store")
> .setHeader(StateStoreConstants.KEY, constant("user-123"))
> .to("state-store:sessions?operation=put");
> from("direct:lookup")
> .setHeader(StateStoreConstants.KEY, constant("user-123"))
> .to("state-store:sessions?operation=get")
> .log("Found: ${body}");
> {code}
> h3. Caching HTTP responses with TTL
> {code:java}
> from("timer:poll?period=60000")
> .setHeader(StateStoreConstants.KEY, constant("weather"))
> .to("state-store:cache?operation=get")
> .choice()
> .when(body().isNull())
> .to("https://api.weather.com/current";)
> .setHeader(StateStoreConstants.KEY, constant("weather"))
> .setHeader(StateStoreConstants.TTL, constant(300000L))
> .to("state-store:cache?operation=put")
> .end()
> .to("direct:process-weather");
> {code}
> h3. Idempotent deduplication pattern
> {code:java}
> from("kafka:orders")
> .setHeader(StateStoreConstants.KEY, simple("${header.orderId}"))
> .to("state-store:processed?operation=putIfAbsent")
> .choice()
> .when(body().isNotNull())
> .log("Duplicate order ${header.orderId}, skipping")
> .stop()
> .end()
> .to("direct:process-order");
> {code}
> h3. YAML DSL with property-configured Redis backend
> {code:title=application.properties}
> camel.beans.redisBackend =
> #class:org.apache.camel.component.statestore.redis.RedisStateStoreBackend
> camel.beans.redisBackend.redisUrl = redis://redis:6379
> camel.beans.redisBackend.mapName = app-state
> {code}
> {code:yaml}
> - route:
> from:
> uri: direct:save
> steps:
> - setHeader:
> name: CamelStateStoreKey
> simple: "${header.userId}"
> - to:
> uri: state-store:preferences?operation=put
> - route:
> from:
> uri: direct:load
> steps:
> - setHeader:
> name: CamelStateStoreKey
> simple: "${header.userId}"
> - to:
> uri: state-store:preferences?operation=get
> {code}
> h2. Configuration examples
> h3. Java bean registration
> {code:java}
> @BindToRegistry("caffeineBackend")
> public CaffeineStateStoreBackend caffeine() {
> CaffeineStateStoreBackend backend = new CaffeineStateStoreBackend();
> backend.setMaximumSize(50_000);
> return backend;
> }
> {code}
> h3. Property-based configuration (no Java required)
> {code:title=Caffeine}
> camel.beans.caffeineBackend =
> #class:org.apache.camel.component.statestore.caffeine.CaffeineStateStoreBackend
> camel.beans.caffeineBackend.maximumSize = 50000
> {code}
> {code:title=Redis}
> camel.beans.redisBackend =
> #class:org.apache.camel.component.statestore.redis.RedisStateStoreBackend
> camel.beans.redisBackend.redisUrl = redis://myhost:6379
> camel.beans.redisBackend.mapName = my-app-state
> {code}
> {code:title=Infinispan}
> camel.beans.infinispanBackend =
> #class:org.apache.camel.component.statestore.infinispan.InfinispanStateStoreBackend
> camel.beans.infinispanBackend.hosts = myhost:11222
> camel.beans.infinispanBackend.cacheName = my-cache
> {code}
> h2. Design decision: StateStoreBackend interface location
> We considered moving the {{StateStoreBackend}} interface to {{camel-api}}
> alongside existing SPIs ({{IdempotentRepository}}, {{AggregationRepository}},
> {{StateRepository}}). We decided against it because:
> * The existing {{camel-api}} SPIs are there because core EIPs consume them
> (idempotent consumer, aggregator). {{StateStoreBackend}} is only consumed by
> the component itself.
> * The existing SPIs have different semantics (two-phase confirm/rollback,
> Exchange serialization) that don't map to a simple key-value store.
> * Moving it would add API surface with stricter stability guarantees for no
> practical benefit today.
> If a future core EIP needs a generic key-value store, the interface can be
> promoted then.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
