Hi Xinyu, Thanks for the proposal. Some requests for clarifications. Let's update the SEP directly instead of replying here.
E.g., in "For any following intermediate stream whose input streams are all end-of-stream, it will be marked as pending EOS" - Should clarify that (IIUC) something is injecting EOS messages in all intermediate stream partitions once it receives EOS from all input stream partitions it's consuming. Should also clarify what is that something. Same for "declare end of stream once all the EOS messages have been received." - What does this declaration involve and who is doing this? In pro for approach 2: Not clear what this means - "The watermark can conclude the input messages before this watermark have been complete." For the cons of approach 2: "Complicated failure scenario of the second job. It needs to checkpoint all the watermark messages received, so when it recovered from failure, it can still count." - How is this related to EOS? How is this related to the checkpoint watermark section? Also, what is the "more messages required to write.. " referring to? "Samza needs to reconcile based on the task counts." - Please explain what reconciliation means, why it needs to happen, and why we need to track the producer task and total task count in the watermark message to do this. Checkpoint watermarks section is also unclear. What problem are we trying to solve here? Should also move the message format and the watermark message interface sections to the bottom, since they depend on details in the event time and checkpoint watermark sections. Thanks, Prateek On Wed, May 24, 2017 at 11:30 AM, xinyu liu <xinyuliu...@gmail.com> wrote: > Hi all, > > I created SEP-6 for SAMZA-1260 > <https://issues.apache.org/jira/browse/SAMZA-1260>: Support Watermark > Across Intermediate Streams for Batch Processing. The link to the SEP is > here: > > https://cwiki.apache.org/confluence/display/SAMZA/SEP- > 6+Support+Watermark+Across+Intermediate+Streams+for+Batch+Processing > > Please review and comments are welcome! > > Thanks, > Xinyu >
