Github user senorcarbone commented on the pull request:
https://github.com/apache/flink/pull/1668#issuecomment-186147597
Thanks for going through it Gyula!
I agree, the `ForwardingOneInputStreamTask` can be integrated in the
`StreamIterationTail`, I will probably do that. I just wanted to get the code
look less messy in the tail part but you are right.
About the operators, the `RecordPusher` was already there (even though it
was not initialised correctly). I just added the `UpstreamLogger` because I
wanted it to follow the operator lifecycle. I think the way it is we do not
need to override the `checkpointStatesInternal` to do the changes we discuss.
We just need to change the operator callback method and this could also be more
robust to changes in the StreamTask and operator interfaces and default
behavior, just my personal view but I see your point too.
I agree with the spill buffer logic. I am only confused a bit with the
output stream thing (the other part of the problem), is there something already
I can use? I haven't check recent changes. How does this work if we use
in-memory backend for example? The blob with all the messages will be anyway
packaged and sent within the stateHandle to the job manager in a single message
(potentially being over the limits), even if we use a stream API, or?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
