[
https://issues.apache.org/jira/browse/BEAM-11105?focusedWorklogId=758131&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-758131
]
ASF GitHub Bot logged work on BEAM-11105:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 18/Apr/22 21:39
Start Date: 18/Apr/22 21:39
Worklog Time Spent: 10m
Work Description: lostluck commented on code in PR #17374:
URL: https://github.com/apache/beam/pull/17374#discussion_r852433361
##########
sdks/go/pkg/beam/pardo.go:
##########
@@ -64,9 +66,16 @@ func TryParDo(s Scope, dofn interface{}, col PCollection,
opts ...Option) ([]PCo
}
var rc *coder.Coder
+ // Sdfs will always encode restrictions as KV<restriction, watermark
state | nil>
if fn.IsSplittable() {
sdf := (*graph.SplittableDoFn)(fn)
- rc, err = inferCoder(typex.New(sdf.RestrictionT()))
+ restT := typex.New(sdf.RestrictionT())
+ // If no watermark estimator state, use boolean as a placeholder
Review Comment:
Outside of specifying a new custom coder, the coder for GlobalWindows
encodes them 0 size, as it's a mono value type.
I suppose this related to my question during design, about whether we're
using it all the time or not. A Watermark Estimator with no state along with an
SDF with type S should make the Restriction type S, rather than trying to make
up a type of WS to make KV<S, WS> valid.
Issue Time Tracking
-------------------
Worklog Id: (was: 758131)
Time Spent: 5h (was: 4h 50m)
> [Go SDK] Watermark Estimation
> -----------------------------
>
> Key: BEAM-11105
> URL: https://issues.apache.org/jira/browse/BEAM-11105
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-go
> Reporter: Robert Burke
> Assignee: Danny McCormick
> Priority: P3
> Time Spent: 5h
> Remaining Estimate: 0h
>
> Allow self checkpointing SplittableDoFns to specify a watermark estimator.
> (To be updated once [https://github.com/apache/beam/pull/13160] is merged and
> the programming guide updated with SDF content.)
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
