[
https://issues.apache.org/jira/browse/BEAM-10670?focusedWorklogId=489848&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-489848
]
ASF GitHub Bot logged work on BEAM-10670:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 23/Sep/20 21:45
Start Date: 23/Sep/20 21:45
Worklog Time Spent: 10m
Work Description: iemejia commented on pull request #12603:
URL: https://github.com/apache/beam/pull/12603#issuecomment-697989304
The phenomenon of microbatches producing results early I noticed it too in
the past when trying to enable the Read.Unbounded tests. I could not understand
why, and I thought it was probably due to some glitch in Spark implementation
or us screwing their scheduling but I struggled to debug the issue properly.
> Since watermark holds don't seem to be implemented, does the
GroupAlsoViaWindowSet hold back the watermark for elements that it currently
has buffered?
Probably, at least that may explain some of the inconsistencies.
> Can you explain how the GlobalWatermarkHolder works, can I register
anything as a sourceId?
In all honesty I am not so familiar with watermark handling on the Spark
runner. I took a look at the GlobalWatermarkHolder class and tried to figure
out but it was not really evident.
My impression is that the sourceId is aligned somehow with Spark's assigned
streamId, but I might be misinterpreting it.
https://github.com/apache/spark/blob/13664434387e338a5029e73a4388943f34e3fc07/streaming/src/main/scala/org/apache/spark/streaming/scheduler/InputInfoTracker.scala#L30
I wish I could help more but that part of the code is also not so well
documented. I doubt that the original authors of the code still remember the
details but maybe they remember at least the intentions of
`GlobalWatermarkHolder` and its use, and maybe if there were any open issues.
Just in case :crossed_fingers: maybe: @amitsela @aviemzur @staslev
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 489848)
Time Spent: 18h 50m (was: 18h 40m)
> Make non-portable Splittable DoFn the only option when executing Java "Read"
> transforms
> ---------------------------------------------------------------------------------------
>
> Key: BEAM-10670
> URL: https://issues.apache.org/jira/browse/BEAM-10670
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
> Reporter: Luke Cwik
> Assignee: Luke Cwik
> Priority: P2
> Time Spent: 18h 50m
> Remaining Estimate: 0h
>
> All runners seem to be capable of migrating to splittable DoFn for
> non-portable execution except for Dataflow runner v1 which will internalize
> the current primitive read implementation that is shared across runner
> implementations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
