Re: RocksDBMapState example?

[Ted Yu]

For KeyedState, apart from https://ci.apache.org/projects/flink/flink-docs-
release-1.4/dev/stream/state/state.html#keyed-state-and-operator-state ,
you can refer to docs/dev/migration.md :

public void initializeState(FunctionInitializationContext context)
throws Exception {
counter = context.getKeyedStateStore().getState(
new ValueStateDescriptor<>("counter", Integer.class, 0));

FYI


On Tue, Apr 10, 2018 at 7:24 AM, NEKRASSOV, ALEXEI <an4...@att.com> wrote:

> Yes, I've read the documentation on working with state.
> It talks about MapState<UK, UV>. When I looked at Javadoc, I learned that
> MapState is an interface, with RocksDBMapState as one of the implementing
> classes.
>
> I'm not sure what you mean by KeyedState; I don't see a class with that
> name.
>
> I'm not clear how ValueState can be used to store key-value mapping. Can
> you please clarify?
>
> Thanks,
> Alex
>
> -Original Message-
> From: Dawid Wysakowicz [mailto:wysakowicz.da...@gmail.com]
> Sent: Tuesday, April 10, 2018 8:54 AM
> To: NEKRASSOV, ALEXEI <an4...@att.com>
> Cc: user@flink.apache.org
> Subject: Re: RocksDBMapState example?
>
> Hi Alexei,
>
> You should not use RocksDBMapState directly. Have you went through the doc
> page regarding working with state[1]?
> I think you want to use KeyedState, assuming the size of your keyspace.
> Probably a way to go would be to key your stream and then even ValueState
> (which will be scoped to that key) might be sufficient.
> You can configure flink further to use RocksDB as the underlying state
> backend[2]
>
> Regards,
> Dawid
>
> [1] https://ci.apache.org/projects/flink/flink-docs-
> release-1.4/dev/stream/state/state.html#working-with-state
> [2] https://ci.apache.org/projects/flink/flink-docs-
> master/ops/state/state_backends.html#state-backends
>
> > On 9 Apr 2018, at 17:41, NEKRASSOV, ALEXEI <an4...@att.com> wrote:
> >
> > Hi,
> >
> > I’d like to use RocksDB to store a key-value mapping table (with 45
> million keys).
> > Can someone please point me to an example of RocksDBMapState()
> constructor invocation? Or an explanation of constructor arguments?..
> >
> > Thanks,
> > Alex Nekrassov
> > nekras...@att.com
>
>

RE: RocksDBMapState example?

[NEKRASSOV, ALEXEI]

Yes, I've read the documentation on working with state.
It talks about MapState<UK, UV>. When I looked at Javadoc, I learned that 
MapState is an interface, with RocksDBMapState as one of the implementing 
classes.

I'm not sure what you mean by KeyedState; I don't see a class with that name.

I'm not clear how ValueState can be used to store key-value mapping. Can you 
please clarify?

Thanks,
Alex

-Original Message-
From: Dawid Wysakowicz [mailto:wysakowicz.da...@gmail.com] 
Sent: Tuesday, April 10, 2018 8:54 AM
To: NEKRASSOV, ALEXEI <an4...@att.com>
Cc: user@flink.apache.org
Subject: Re: RocksDBMapState example?

Hi Alexei,

You should not use RocksDBMapState directly. Have you went through the doc page 
regarding working with state[1]?
I think you want to use KeyedState, assuming the size of your keyspace. 
Probably a way to go would be to key your stream and then even ValueState 
(which will be scoped to that key) might be sufficient.
You can configure flink further to use RocksDB as the underlying state 
backend[2]

Regards,
Dawid

[1] 
https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/state/state.html#working-with-state
[2] 
https://ci.apache.org/projects/flink/flink-docs-master/ops/state/state_backends.html#state-backends

> On 9 Apr 2018, at 17:41, NEKRASSOV, ALEXEI <an4...@att.com> wrote:
> 
> Hi,
> 
> I’d like to use RocksDB to store a key-value mapping table (with 45 million 
> keys).
> Can someone please point me to an example of RocksDBMapState() constructor 
> invocation? Or an explanation of constructor arguments?..
> 
> Thanks,
> Alex Nekrassov
> nekras...@att.com

Re: RocksDBMapState example?

[Dawid Wysakowicz]

Hi Alexei,

You should not use RocksDBMapState directly. Have you went through the doc page 
regarding working with state[1]?
I think you want to use KeyedState, assuming the size of your keyspace. 
Probably a way to go would be to key your stream and then even ValueState 
(which will be scoped to that key) might be sufficient.
You can configure flink further to use RocksDB as the underlying state 
backend[2]

Regards,
Dawid

[1] 
https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/state/state.html#working-with-state
[2] 
https://ci.apache.org/projects/flink/flink-docs-master/ops/state/state_backends.html#state-backends

> On 9 Apr 2018, at 17:41, NEKRASSOV, ALEXEI  wrote:
> 
> Hi,
> 
> I’d like to use RocksDB to store a key-value mapping table (with 45 million 
> keys).
> Can someone please point me to an example of RocksDBMapState() constructor 
> invocation? Or an explanation of constructor arguments?..
> 
> Thanks,
> Alex Nekrassov
> nekras...@att.com



signature.asc
Description: Message signed with OpenPGP

RE: RocksDBMapState example?

[NEKRASSOV, ALEXEI]

I looked at that code, but I’m still not clear.

new RocksDBMapState<>(columnFamily, namespaceSerializer, stateDesc, this);

columnFamily is determined by 50-line function; is this necessary for a simple 
use case like mine? What should I use as state descriptor in that function?..
Last argument is set to “this”; does this mean I need to implement 
AbstractKeyedStateBackend, like RocksDBKeyedStateBackend does?

Again, I’m looking for a simple equivalent to
new HashMap();

or to
JedisPool pool = new JedisPool(new JedisPoolConfig(), redisHost);
Jedis jedis = pool.getResource();

Thanks,
Alex

From: Ted Yu [mailto:yuzhih...@gmail.com]
Sent: Monday, April 09, 2018 11:48 AM
To: user <user@flink.apache.org>
Subject: Re: RocksDBMapState example?

Hi,
Have you looked at the ctor call in :
flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java

around line 1261 ?

Cheers

Re: RocksDBMapState example?

[Ted Yu]

Hi,
Have you looked at the ctor call in :
flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java

around line 1261 ?

Cheers