[
https://issues.apache.org/jira/browse/BEAM-7428?focusedWorklogId=264158&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264158
]
ASF GitHub Bot logged work on BEAM-7428:
----------------------------------------
Author: ASF GitHub Bot
Created on: 20/Jun/19 21:50
Start Date: 20/Jun/19 21:50
Worklog Time Spent: 10m
Work Description: jkff commented on issue #8741: [BEAM-7428] Output the
timestamp on elements in ReadAllViaFileBasedSource
URL: https://github.com/apache/beam/pull/8741#issuecomment-504212711
I see UnboundedSource, regular DoFn and SDF as the same thing and I think
they should behave identically w.r.t. watermarks (doesn't mean they should all
support the same capabilities, but there should be no contradictions between
the capabilities they *do* support).
All three take an input and produce outputs.
* In all three, produced outputs should be considered late if they're before
the output watermark.
* In all three, by default the output watermark equals timestamp of the
input element (this is not entirely accurate but I don't remember enough to
state this entirely accurately) representing the principle that you can't
output back in time. For the purposes of this statement, the "input element" of
UnboundedSource has timestamp -INF.
* We've considered cases where you *do* want to output back in time by a
limited amount, and there was some JIRA about adding a watermark shift
primitive, but we didn't get around to it. We kinda have DoFn.allowedLateness
for that.
* In some cases, "timestamp of the input element" is too slow, that's why
UnboundedSource and SDF allow you to advance the watermark faster. Regular
DoFn's don't allow you to do that because they output sufficiently few things
that it doesn't matter. I don't see SDF's updateWatermark() as a watermark hold
- we use a watermark hold under the hood to signal to the runner that
processing of the element isn't actually done, but if there was some way to
signal that differently (e.g. if a runner didn't use timers to schedule
continuations) then a hold would not be necessary and updateWatermark() would
only serve its true purpose of advancing the output watermark before the
element is done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 264158)
Time Spent: 5h 40m (was: 5.5h)
> ReadAllViaFileBasedSource does not output the timestamps of the read elements
> -----------------------------------------------------------------------------
>
> Key: BEAM-7428
> URL: https://issues.apache.org/jira/browse/BEAM-7428
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
> Reporter: Ismaël Mejía
> Assignee: Ismaël Mejía
> Priority: Minor
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
> This differs from the implementation of JavaReadViaImpulse that tackles a
> similar problem but does output the timestamps correctly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
