By default, RocksDB is used. Ie, the following are the same:
builder.globalTable("topic");
builder.globalTable(
"topic",
Materialized.as(Stores.persistentKeyValueStore("store-name"))
);
-Matthias
On 5/22/20 4:31 AM, Pushkar Deole wrote:
> Matthias,
>
> I can see that when the Materialized parameter is provided with store-name,
> the store can be accessed through kafkaStreams.store(store-name). This
> returns ReadOnlyKeyValueStore which seems to hold the Map structure
> internally. So, the GlobalKTable API is just an abstraction and if I
> haven't provided Materialized parameter while constructing GlobalKTable
> then which store does it use?
>
> On Thu, May 14, 2020 at 11:49 PM Matthias J. Sax <mj...@apache.org> wrote:
>
>> Yeah, the current API doesn't make it very clear how to do it. You can
>> set an in-memory like this:
>>
>>> builder.globalTable("topic",
>> Materialized.as(Stores.inMemoryKeyValueStore("store-name")));
>>
>>
>> We are already working on an improved API via KIP-591:
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-591%3A+Add+Kafka+Streams+config+to+set+default+store+type
>>
>>
>>
>> -Matthias
>>
>>
>> On 5/13/20 3:40 AM, Pushkar Deole wrote:
>>> Matthias,
>>>
>>> For GlobalKTable, I am looking at the APIs provided by StreamsBuilder
>> and I
>>> don't see any option to mention in-memory store there: all these API
>>> documentation states that The resulting GlobalKTable
>>> <
>> https://kafka.apache.org/23/javadoc/org/apache/kafka/streams/kstream/GlobalKTable.html
>>>
>>> will
>>> be materialized in a local KeyValueStore
>>> <
>> https://kafka.apache.org/23/javadoc/org/apache/kafka/streams/state/KeyValueStore.html
>>>
>>> with
>>> an internal store name . It doesn't give an option whether in-memory or
>>> backed by DB
>>>
>>> globalTable
>>> <
>> https://kafka.apache.org/23/javadoc/org/apache/kafka/streams/StreamsBuilder.html#globalTable-java.lang.String-
>>>
>>> (String
>>> <
>> https://docs.oracle.com/javase/8/docs/api/java/lang/String.html?is-external=true
>>>
>>> topic)
>>> globalTable
>>> <
>> https://kafka.apache.org/23/javadoc/org/apache/kafka/streams/StreamsBuilder.html#globalTable-java.lang.String-org.apache.kafka.streams.kstream.Consumed-org.apache.kafka.streams.kstream.Materialized-
>>>
>>> (String
>>> <
>> https://docs.oracle.com/javase/8/docs/api/java/lang/String.html?is-external=true
>>>
>>> topic, Consumed
>>> <
>> https://kafka.apache.org/23/javadoc/org/apache/kafka/streams/kstream/Consumed.html
>>>
>>> <K,V> consumed, Materialized
>>> <
>> https://kafka.apache.org/23/javadoc/org/apache/kafka/streams/kstream/Materialized.html
>>>
>>> <K,V,KeyValueStore
>>> <
>> https://kafka.apache.org/23/javadoc/org/apache/kafka/streams/state/KeyValueStore.html
>>>
>>> <org.apache.kafka.common.utils.Bytes,byte[]>> materialized)
>>>
>>> On Tue, May 12, 2020 at 11:07 PM Matthias J. Sax <mj...@apache.org>
>> wrote:
>>>
>>>> By default, RocksDB is used. You can also change it to use an in-memory
>>>> store that is basically a HashMap.
>>>>
>>>>
>>>> -Matthias
>>>>
>>>> On 5/12/20 10:16 AM, Pushkar Deole wrote:
>>>>> Thanks Liam!
>>>>>
>>>>> On Tue, May 12, 2020, 15:12 Liam Clarke-Hutchinson <
>>>>> liam.cla...@adscale.co.nz> wrote:
>>>>>
>>>>>> Hi Pushkar,
>>>>>>
>>>>>> GlobalKTables and KTables can have whatever data structure you like,
>> if
>>>> you
>>>>>> provide the appropriate deserializers - for example, an Kafka Streams
>>>> app I
>>>>>> maintain stores model data (exported to a topic per entity from
>> Postgres
>>>>>> via Kafka Connect's JDBC Source) as a GlobalKTable of Jackson
>>>> ObjectNode's
>>>>>> keyed by entity id
>>>>>>
>>>>>> If you're worried about efficiency, just treat KTables/GlobalKTables
>> as
>>>> a
>>>>>> HashMap<K, V> to and you're pretty much there. In terms of efficiency,
>>>>>> we're joining model data to about 7 - 10 TB of transactional data a
>>>> day,
>>>>>> and on average, run about 5 - 10 instances of our enrichment app with
>>>> about
>>>>>> 2GB max heap.
>>>>>>
>>>>>> Kind regards,
>>>>>>
>>>>>> Liam "Not a part of the Confluent team, but happy to help"
>>>>>> Clarke-Hutchinson
>>>>>>
>>>>>> On Tue, May 12, 2020 at 9:35 PM Pushkar Deole <pdeole2...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hello confluent team,
>>>>>>>
>>>>>>> Could you provide some information on what data structures are used
>>>>>>> internally by GlobalKTable and KTables. The application that I am
>>>> working
>>>>>>> on has a requirement to read cached data from GlobalKTable on every
>>>>>>> incoming event, so the reads from GlobalKTable need to be efficient.
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>
signature.asc
Description: OpenPGP digital signature
