Hi Navneeth,
Yes, you are correct. I think there are some opportunities for improvement
there, but there are also reasons for it to be serialized in the in-memory
store.
Off the top of my head, we need to serialize stored data anyway to send it to
the changelog. Also, even though the store
Hi All,
I have a question about the inMemoryKeyValue store. I was under the
assumption that in-memory stores would not serialize the objects but when I
looked into the implementation I see InMemoryKeyValueStore uses a
NavigableMap of bytes which indicates the user objects have to be
serialized
t;
> -Matthias
>
> On 1/27/20 6:05 AM, Igor Danis wrote:
>> Hi all,
>>
>> I have question about kafka-streams, particularly in-memory state-store
>> (/org.apache.kafka.streams.state.internals.InMemoryKeyValueStore/).
>>
>> I believe that topology
What you say is correct. This is a severe bug rendering standby tasks
useless for in-memory state stores. Can you pleas open a ticket so we
can fix it asap?
-Matthias
On 1/27/20 6:05 AM, Igor Danis wrote:
> Hi all,
>
> I have question about kafka-streams, particularly in-memory st
Hi all,
I have question about kafka-streams, particularly in-memory state-store
(/org.apache.kafka.streams.state.internals.InMemoryKeyValueStore/).
I believe that topology is irrelevant here, but let's say I have one
source topic with single partition
feeding data into one statefull processor
de with 3 patterns:
P1:persistent state store without cache / P2:persistent state store with
cache / P3:in-memory state store.
1. Set up a local broker
2. Create a source topic for 1 partition
3. Produce 10 records whose value changes with the same key to source topic
4. Execute the Kafk
cument [
http://docs.confluent.io/3.1.1/streams/developer-guide.html#processor-api].
I am using in-memory state store,
its key is the ID of log category, value is count.
All the changelogs are written to a broker by context.commit() for fault
tolerance,
but since the data I handle is large and th
Hi Tom,
That is right that currently for in-memory state store the directory is not
actually used, i.e. the directory will be created but no data will be
written. But the "state.dir" is not a required config in the sense that if
user do not specify a value, it will default to "/tm
Our use case for kafka-streams only needs to use the in memory state store but
the current code on trunk still requires a valid ’state.dir’ that the client
can write too. A lock file per partition and an offset file is created but the
offset file always contains zero values for offsets since
