[
https://issues.apache.org/jira/browse/BEAM-7785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16894116#comment-16894116
]
Jan Lukavský edited comment on BEAM-7785 at 7/26/19 8:44 PM:
-------------------------------------------------------------
Hmmm ... looking into that - maybe the approach you suggest would not be that
complex as I suspected. Because QuiescenceDriver has reference to
EvaluationContext, _maybe_ it would be possible to copy WatermarkManager inside
the EvaluationContext before bundle and free it in bundle finish callback.
Would you find that possible?
was (Author: je-ik):
Hmmm ... looking into that - maybe the approach you suggest would not be that
complex as I suspected. Because QuiescenceDriver has reference on
EvaluationContext, _maybe_ it would be possible to copy WatermarkManager inside
the EvaluationContext before bundle and free it in bundle finish callback.
Would you find that possible?
> DirectRunner: watermarks are updated asynchronously from bundle processing
> --------------------------------------------------------------------------
>
> Key: BEAM-7785
> URL: https://issues.apache.org/jira/browse/BEAM-7785
> Project: Beam
> Issue Type: Bug
> Components: runner-direct
> Affects Versions: 2.13.0
> Reporter: Jan Lukavský
> Assignee: Jan Lukavský
> Priority: Major
> Time Spent: 4h
> Remaining Estimate: 0h
>
> Watermarks are updated in QuiescenceDriver (by calling fireTimers, which
> calls forceRefresh()) on WatermarkManager. This results in creating timer
> bundles, that are then processed asynchronously as DirectTransformExecutor.
> Because of that, watermarks (input watermarks mostly) might be updated while
> bundle is being processed. That violates assumption, that bundle processing
> should be atomical (with identical external conditions during processing of
> whole bundle).
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
