Hi Pieter-Jan,
The second variant stores the elements of the broadcasted stream in operator
(thus non-keyed) state.
On the differences:
The Broadcast stream is not a keyed stream, so you are not in a keyed context,
thus you have no access to keyed state.
Given this, and assuming that you are implementing a CoProcessFunction, then
your function should also implement the
ListCheckpointed interface and store the broadcasted data as ListState.
From a systems perspective, in case of rescaling, then your function should
take care of removing duplicated manually
as in the case of scaling down for example, the state of 2 nodes may end up on
one. In addition, given that elements are
re-distributed randomly, you may even end up with some elements missing one
some nodes. This would be more evident
in case of scaling up. In this case, nodes will be missing data.
From an API perspective, assuming that the other stream is keyed and it has
state, accessing it in order to do some computation
would be impossible.
Cheers,
Kostas
> On Oct 9, 2018, at 2:52 PM, Pieter-Jan Van Aeken
> <pieterjan.vanae...@euranova.eu> wrote:
>
> Hi,
>
>
> I am not sure I fully understand the differences between doing something like
>
> dataStreamX.connect(dataStreamY.broadcast()).process(new CoProcessFunction{})
>
> and this
>
> dataStreamX.connect(dataStreamY.broadcast(descriptor).process(new
> BroadcastProcessFunction)
>
> Couldn't I manage the state exactly the same way using the first option? The
> only difference I could find was in read/write permissions for different
> states. But other than that, the difference in possibilities escapes me and I
> was hoping someone here could clarify.
>
>
> Kind regards,
>
> Pieter-Jan
> ___________________________
>
> Pieter-Jan Van Aeken
>
>
> Consultant - Data Engineer
> (M) +32 (0) 474 06 64 48
>
>
> EURA NOVA
>
> Rue Emile Francqui, 4
>
> 1435 Mont-Saint-Guibert
>
> (T) +32 10 75 02 00
>
>
> euranova.eu <http://euranova.eu/>
> research.euranova.eu <http://research.euranova.eu/>
> ♻ Be green, keep it on the screen
