Re: Storing large lists into state per key

Hi Jan, You could associate a key to each element of your Key's list (e.g., hashing the value), keep only the keys in heap (e.g., in a list) and the associated state key-value/s in an external store like RocksDB/Redis, but you will notice large overheads due to de/serializing - a huge penatly

Re: KeyGroupRangeAssignment ?

index in an >> Integer or IntValue and key off that field. >> >> On Tue, Feb 21, 2017 at 6:04 AM, Ovidiu-Cristian MARCU < >> ovidiu-cristian.ma...@inria.fr> wrote: >> >>> Hi, >>> >>> As in my example, each key is a window so I want to

Re: KeyGroupRangeAssignment ?

Filled in https://issues.apache.org/jira/browse/FLINK-5873 <https://issues.apache.org/jira/browse/FLINK-5873> Best, Ovidiu > On 21 Feb 2017, at 12:00, Ovidiu-Cristian MARCU > <ovidiu-cristian.ma...@inria.fr> wrote: > > Hi Till, > > I will look into fillin

Re: KeyGroupRangeAssignment ?

group sum up to 1024? this only works for the first case. What do the >> numbers mean then? >> >> Cheers, >> Till >> >> On Mon, Feb 20, 2017 at 3:45 PM, Ovidiu-Cristian MARCU < >> ovidiu-cristian.ma...@inria.fr> wrote: >> >>> H

Re: KeyGroupRangeAssignment ?

ch > group sum up to 1024? this only works for the first case. What do the > numbers mean then? > > Cheers, > Till > > On Mon, Feb 20, 2017 at 3:45 PM, Ovidiu-Cristian MARCU < > ovidiu-cristian.ma...@inria.fr <mailto:ovidiu-cristian.ma...@inria.fr>> wrote: &g

Re: KeyGroupRangeAssignment ?

to input.keyBy) could be easily tackled with some sort of local mapping but I am considering an use case with millions of keys. Best, Ovidiu > On 20 Feb 2017, at 15:45, Ovidiu-Cristian MARCU > <ovidiu-cristian.ma...@inria.fr> wrote: > > Hi, > > Thank you for clar

Re: KeyGroupRangeAssignment ?

via murmurHash(key.hashCode()) % maxParallelism. > > Alternatively if you don’t need a keyed stream, you could try to use a custom > partitioner via DataStream.partitionCustom. > > Cheers, > Till > > > On Mon, Feb 20, 2017 at 11:46 AM, Ovidiu-Cristian MARCU > <

RE: [FLINK-3035] Redis as State Backend

was using Redis as a state backend in my benchmarking Beam.It proved to be a bottleneck. Perhaps due to high frequency of updating state components.I replaced it with Java 8 ConcurrentHashmaps and it settled down tremendously.Amir- From: Ovidiu-Cristian MARCU <ovidiu-cristian.ma...@inria

RE: TopSpeedWindowing - in error: Could not forward element to next operator

all elements and the window function always sees an empty iterable. I have a fix for this in a recent PR: https://github.com/apache/flink/pull/2656 Cheers, Aljoscha On Fri, 21 Oct 2016 at 16:57 Ovidiu Cristian Marcu < ovidiu.cristian.ma...@huawei.com> wrote: > Hi > > I have th

RE: [FLINK-3035] Redis as State Backend

only used as an out-of-core hash table. When checkpointing we write everything from RocksDB to HDFS. When restoring we repopulate an empty local RocksDB instance from the data in HDFS. Cheers, Aljoscha On Fri, 21 Oct 2016 at 11:24 Ovidiu Cristian Marcu < ovidiu.cristian.ma...@huawei.

Re: [FLINK-3035] Redis as State Backend

om RocksDB to HDFS. When restoring we repopulate an empty local RocksDB > instance from the data in HDFS. > > Cheers, > Aljoscha > > On Fri, 21 Oct 2016 at 11:24 Ovidiu Cristian Marcu < > ovidiu.cristian.ma...@huawei.com> wrote: > >> Hi >> >> I missed yo

RE: TopSpeedWindowing - in error: Could not forward element to next operator

at 6:38 PM, Ovidiu Cristian Marcu < ovidiu.cristian.ma...@huawei.com> wrote: > Could you check the following issue on master? > > When running this example org.apache.flink.streaming.examples.windowing. > TopSpeedWindowing > With default configuration I have no errors. > &

RE: [FLINK-3035] Redis as State Backend

ointed out on the issue. The second Idea is problematic because there is no easy way to read all state for a given Flink operator from a running Redis instance to store it in HDFS. That's what I was getting at in my comment. Cheers, Aljoscha On Fri, 7 Oct 2016 at 17:19 Ovidiu Cristian Marcu < ovidi

TopSpeedWindowing - in error: Could not forward element to next operator

Could you check the following issue on master? When running this example org.apache.flink.streaming.examples.windowing. TopSpeedWindowing With default configuration I have no errors. When I change the state backend with RocksDB I receive this error: java.lang.RuntimeException: Could not

[FLINK-3035] Redis as State Backend

Hi Can you please expand the last comment: "I think, however, that for other reasons we will probably not be able to implement this well. The problem is that we have to somehow get at the state in redis for checkpointing. And if we use only one Redis instance for all states then this will be

Re: Evaluating Apache Flink

Hi Kevin, I have orchestrated an evaluation of Spark and Flink for various batch and graph processing workloads (no streaming, no sql) (this work has been accepted as a paper at Cluster and I will publish soon a report, for more details please contact me directly). Both engines did well,

Re: Side-effects of DataSet::count

Hi Stephan and all, Some reference to this may be https://issues.apache.org/jira/browse/FLINK-2250 ? I agree your priorities on streaming are very high, it will make a big +1 for the community to create a discussion/place for the design