[
https://issues.apache.org/jira/browse/BEAM-849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890953#comment-15890953
]
Amit Sela commented on BEAM-849:
--------------------------------
A continually growing file is something else, which I agree on, but in that
case {{waitUntilFinish()}} would terminate after the file is done, right ? by
moving the WM to end-of-time ? so if that happens implicitly, why call
explicitly ?
All I'm saying is that in batch, beginning and end are known ahead of
execution, so blocking until termination is natural. In streaming however, the
end is unknown, so it's a bit awkward - some pipelines will behave the same,
like SDF-log-tail, and some won't like reading from Pubsub/Kafka.
I will agree that for the sake of a unified model it makes sense, but still a
bit un-natural, so that's why I think this ticket is for - to try and reason
about this and make "feel" more natural, no ?
As for "unbounded pipelines" not being a part of the model, it's a bit
confusing because it's all over the SDK.
> Redesign PipelineResult API
> ---------------------------
>
> Key: BEAM-849
> URL: https://issues.apache.org/jira/browse/BEAM-849
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
> Reporter: Pei He
>
> Current state:
> Jira https://issues.apache.org/jira/browse/BEAM-443 addresses
> waitUntilFinish() and cancel().
> However, there are additional work around PipelineResult:
> need clearly defined contract and verification across all runners
> need to revisit how to handle metrics/aggregators
> need to be able to get logs
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
