> * Add an API to the Java client that makes it easier to maintain a consistent
> Share State between instances of an application.
> * Provide some ready to use recipes, like a simple key-value store
>
> It is not a goal to implement a Pulsar backed Database system
While the first use case for Pulsar was indeed to be the
messaging/replication platform for a distributed database, and it has
been working in production for many years, I'm not convinced to add
this level of API as part of the Pulsar client API.
Pulsar API has been designed to be high-level and easy to use (and
reason about), with in mind the use cases of application developers. I
don't think that a "storage" level API fits well with the rest of
abstractions.
> public interface PulsarMap<K,V> extends AutoCloseable {
> ..
> CompletableFuture<V> put(K key, V value)
If all the logic is implemented in the client side, when there are
multiple clients sharing the same, how can any of them mutate the
state, since we actually enforce that there is a single exclusive
producer? Would a user get an error if there's already a different
client writing?
My impression is that, while looking convenient, a shared Map
interface is not the best abstraction for either case:
* If you're actually building a DB, you will definitely need access
to the log itself rather than a Map interface
* If you want to share some state across multiple processes without
using a DB, there are many tricky API, consistency and semantic
problems to solve, many of which are just pushed down to the
application which will need to be aware and understand them. At that
point, I would seriously recommend using a DB, or if the question is:
"I don't want to use an additional external system", then to use the
BK TableService component.
I think this feature should be best viewed as a recipe, as it doesn't
depend on or benefits from any internal broker support. If there are
enough interest and concrete use cases it can be then included later.
--
Matteo Merli
<matteo.me...@gmail.com>
On Fri, Dec 24, 2021 at 1:53 AM Enrico Olivelli <eolive...@gmail.com> wrote:
>
> Hello everyone,
> I want to start a discussion about PIP-124 Pulsar Client Shared State API
>
> This is the PIP document
> https://github.com/apache/pulsar/issues/13490
>
> This is a demo implementation (a proof-of-concept):
> https://github.com/eolivelli/pulsar-shared-state-manager
>
> Please take a look and share your thoughts
>
> I believe that this will unlock the potential of the Exclusive
> Producer and it will also make easier the life of many developers who
> are using Pulsar and need some API to share configuration, metadata,
> or any simple key-value data structure without adding a Database or
> other components to their library, Pulsar IO connector or Pulsar
> Protocol Handler.
>
> Thanks
> Enrico
