thanks.. yes that would help
On Thu, May 14, 2020 at 11:49 PM Matthias J. Sax <mj...@apache.org> wrote:
> Yeah, the current API doesn't make it very clear how to do it. You can
> set an in-memory like this:
>
> > builder.globalTable("topic",
> Materialized.as(Stores.inMemoryKeyValueStore("store-name")));
>
>
> We are already working on an improved API via KIP-591:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-591%3A+Add+Kafka+Streams+config+to+set+default+store+type
>
>
>
> -Matthias
>
>
> On 5/13/20 3:40 AM, Pushkar Deole wrote:
> > Matthias,
> >
> > For GlobalKTable, I am looking at the APIs provided by StreamsBuilder
> and I
> > don't see any option to mention in-memory store there: all these API
> > documentation states that The resulting GlobalKTable
> > <
> https://kafka.apache.org/23/javadoc/org/apache/kafka/streams/kstream/GlobalKTable.html
> >
> > will
> > be materialized in a local KeyValueStore
> > <
> https://kafka.apache.org/23/javadoc/org/apache/kafka/streams/state/KeyValueStore.html
> >
> > with
> > an internal store name . It doesn't give an option whether in-memory or
> > backed by DB
> >
> > globalTable
> > <
> https://kafka.apache.org/23/javadoc/org/apache/kafka/streams/StreamsBuilder.html#globalTable-java.lang.String-
> >
> > (String
> > <
> https://docs.oracle.com/javase/8/docs/api/java/lang/String.html?is-external=true
> >
> > topic)
> > globalTable
> > <
> https://kafka.apache.org/23/javadoc/org/apache/kafka/streams/StreamsBuilder.html#globalTable-java.lang.String-org.apache.kafka.streams.kstream.Consumed-org.apache.kafka.streams.kstream.Materialized-
> >
> > (String
> > <
> https://docs.oracle.com/javase/8/docs/api/java/lang/String.html?is-external=true
> >
> > topic, Consumed
> > <
> https://kafka.apache.org/23/javadoc/org/apache/kafka/streams/kstream/Consumed.html
> >
> > <K,V> consumed, Materialized
> > <
> https://kafka.apache.org/23/javadoc/org/apache/kafka/streams/kstream/Materialized.html
> >
> > <K,V,KeyValueStore
> > <
> https://kafka.apache.org/23/javadoc/org/apache/kafka/streams/state/KeyValueStore.html
> >
> > <org.apache.kafka.common.utils.Bytes,byte[]>> materialized)
> >
> > On Tue, May 12, 2020 at 11:07 PM Matthias J. Sax <mj...@apache.org>
> wrote:
> >
> >> By default, RocksDB is used. You can also change it to use an in-memory
> >> store that is basically a HashMap.
> >>
> >>
> >> -Matthias
> >>
> >> On 5/12/20 10:16 AM, Pushkar Deole wrote:
> >>> Thanks Liam!
> >>>
> >>> On Tue, May 12, 2020, 15:12 Liam Clarke-Hutchinson <
> >>> liam.cla...@adscale.co.nz> wrote:
> >>>
> >>>> Hi Pushkar,
> >>>>
> >>>> GlobalKTables and KTables can have whatever data structure you like,
> if
> >> you
> >>>> provide the appropriate deserializers - for example, an Kafka Streams
> >> app I
> >>>> maintain stores model data (exported to a topic per entity from
> Postgres
> >>>> via Kafka Connect's JDBC Source) as a GlobalKTable of Jackson
> >> ObjectNode's
> >>>> keyed by entity id
> >>>>
> >>>> If you're worried about efficiency, just treat KTables/GlobalKTables
> as
> >> a
> >>>> HashMap<K, V> to and you're pretty much there. In terms of efficiency,
> >>>> we're joining model data to about 7 - 10 TB of transactional data a
> >> day,
> >>>> and on average, run about 5 - 10 instances of our enrichment app with
> >> about
> >>>> 2GB max heap.
> >>>>
> >>>> Kind regards,
> >>>>
> >>>> Liam "Not a part of the Confluent team, but happy to help"
> >>>> Clarke-Hutchinson
> >>>>
> >>>> On Tue, May 12, 2020 at 9:35 PM Pushkar Deole <pdeole2...@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> Hello confluent team,
> >>>>>
> >>>>> Could you provide some information on what data structures are used
> >>>>> internally by GlobalKTable and KTables. The application that I am
> >> working
> >>>>> on has a requirement to read cached data from GlobalKTable on every
> >>>>> incoming event, so the reads from GlobalKTable need to be efficient.
> >>>>>
> >>>>
> >>>
> >>
> >>
> >
>
>
