Iterator of KeyValueGroupedDataset.flatMapGroupsWithState function

[Antonio Murgia - antonio.murg...@studio.unibo.it]

Hi all,

I'm currently developing a Spark Structured Streaming job and I'm performing 
flatMapGroupsWithState.

I'm concerned about the laziness of the Iterator[V] that is passed to my custom 
function (func: (K, Iterator[V], GroupState[S]) => Iterator[U]).

Is it ok to collect that iterator (with a toList, for example)? I have a logic 
that is practically impossible to perform on a Iterator, but I do not want to 
break Spark lazy chain, obviously.


Thank you in advance.


#A.M.