Github user senorcarbone commented on the pull request:
https://github.com/apache/flink/pull/1668#issuecomment-190279441
You can find an alternative version using `ListState` in the following
branch:
https://github.com/senorcarbone/flink/commits/ftloopsalt
So I noticed that this version is quite **slower** than the one with custom
operator state but it can support larger states apparently.
I am (ab)using the PartitionedState to store the ListState in the same key,
as @gyfora suggested since it is the only way to obtain the nice
representations at the moment. It would be nice to have them available for
operator state snapshots as well - @aljoscha have you thought about it? When
there is free time (after the release) it would be nice to see what @aljoscha
and @StephanEwen think of the two takes as well. No hurries, just take a look
when you have time!
The two annoying issues I noticed during testing and we need to check soon
are the following:
- The overhead of transmitting and finally delivering a barrier from the
`head` to its consumers increases in time (for each subsequent checkpoint).
That is due to having a single queue at the beginning of the iterative part of
the job. Events coming from the backedge are pushed further behind the input
queue. It would be nice to have take events in round robin among the two input
gates (iteration source, regular input). Otherwise, checkpoints in iterative
jobs can be really prolonged in time due to this.
- We need a proper way to deal with deadlocks. I removed the part where we
discard events in the tail upon timeout since that boils down to at most once
semantics. This PR is not solving deadlocks but I think we should find a
graceful way to tackle them. (@uce, any ideas? )
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
