Thanks Chesnay (and others).

That's what I was figuring out. Now let's go onto the follow up with my
exact use-case.

I have two streams A and B. A basically receives "rules" that the
processing of B should observe to process.

There is a "key" that allows me to know that a rule x coming in A is for
events with the same key coming in B.

I was planning to do (pseudo code):

      -> store in a ValueState the rule
      -> use the state to get the rule, transform the element according to
the rule, collect it

I think it should work, right, because the ValueState will be "per key" and
contain the rule for this key and so on?

Now, what I really care is not having all the elements of key1 in the same
parallelism, I just want to make sure key1 and key2 are isolated so I can
use the key state to store the corresponding rule and key2 rules are not
used for key1 and conversely.

So ideally instead of using 8 parallelisms, in order to use the full power
of my system, even with 8 keys I would like to use 16 parallelisms as I
don't care about all elements of key1 being in the same parallelism. All I
care is that the state contain the rule corresponding to this key.

What would be the recommended approach here?

Thanks again for your help,

On Thu, Apr 12, 2018 at 9:31 AM, Chesnay Schepler <>

> You will get 16 parallel executions since you specify a parallellism of
> 16, however 8 of these will not get any data.
> On 11.04.2018 23:29, Hao Sun wrote:
> From what I learnt, you have to control parallelism your self. You can set
> parallelism on operator or set default one through flink-config.yaml.
> I might be wrong.
> On Wed, Apr 11, 2018 at 2:16 PM Christophe Jolif <> wrote:
>> Hi all,
>> Imagine I have a default parallelism of 16 and I do something like
>> stream.keyBy("something").flatMap()
>> Now let's imagine I have less than 16 keys, maybe 8.
>> How many parallel executions of the flatMap function will I get? 8
>> because I have 8 keys, or 16 because I have default parallelism at 16?
>> (and I will have follow up questions depending on the answer I suspect ;))
>> Thanks,
>> --
>> Christophe

Reply via email to