[
https://issues.apache.org/jira/browse/BEAM-11104?focusedWorklogId=766908&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-766908
]
ASF GitHub Bot logged work on BEAM-11104:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 05/May/22 20:11
Start Date: 05/May/22 20:11
Worklog Time Spent: 10m
Work Description: lostluck commented on code in PR #17558:
URL: https://github.com/apache/beam/pull/17558#discussion_r866277511
##########
sdks/go/pkg/beam/core/runtime/exec/pardo.go:
##########
@@ -174,17 +174,22 @@ func (n *ParDo) processSingleWindow(mainIn *MainInput)
(sdf.ProcessContinuation,
if err != nil {
return nil, n.fail(err)
}
- if mainIn.RTracker != nil && !mainIn.RTracker.IsDone() {
- return nil, rtErrHelper(mainIn.RTracker.GetError())
- }
// Forward direct output, if any. It is always a main output.
if val != nil {
+ // Check for incomplete processing of a restriction without a
checkpoint
+ if mainIn.RTracker != nil && !mainIn.RTracker.IsDone() &&
val.Continuation == nil {
+ return nil, rtErrHelper(mainIn.RTracker.GetError())
+ }
// We do not forward a ProcessContinuation on its own
if val.Elm == nil {
return val.Continuation, nil
}
return val.Continuation, n.Out[0].ProcessElement(n.ctx, val)
+ } else {
+ if mainIn.RTracker != nil && !mainIn.RTracker.IsDone() {
+ return nil, rtErrHelper(mainIn.RTracker.GetError())
+ }
}
Review Comment:
Style nit: If the previous clause ends with a return, drop the else.
Issue Time Tracking
-------------------
Worklog Id: (was: 766908)
Time Spent: 21h 20m (was: 21h 10m)
> [Go SDK] DoFn Self Checkpointing
> --------------------------------
>
> Key: BEAM-11104
> URL: https://issues.apache.org/jira/browse/BEAM-11104
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-go
> Reporter: Robert Burke
> Assignee: Jack McCluskey
> Priority: P3
> Time Spent: 21h 20m
> Remaining Estimate: 0h
>
> Allow SplittableDoFns to self checkpoint.
> Design doc:
> [https://docs.google.com/document/d/1_JbzjY9JR07ZK5v7PcZevUfzHPsqwzfV7W6AouNpMPk/edit?usp=sharing]
>
> Feature is written E2E and users will be able to return ProcessContinuations
> from SDFs as of 2.39.0 but the full behavior has not been fully validated. An
> integration test that validates self-checkpointing is working as-intended
> will need to be written and passing before the feature is no longer
> considered experimental and this ticket is marked as resolved.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
