[
https://issues.apache.org/jira/browse/BEAM-11104?focusedWorklogId=778227&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-778227
]
ASF GitHub Bot logged work on BEAM-11104:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 03/Jun/22 17:41
Start Date: 03/Jun/22 17:41
Worklog Time Spent: 10m
Work Description: jrmccluskey commented on code in PR #17956:
URL: https://github.com/apache/beam/pull/17956#discussion_r889194510
##########
website/www/site/content/en/documentation/programming-guide.md:
##########
@@ -6422,7 +6422,26 @@ resource utilization.
{{< /highlight >}}
{{< highlight go >}}
-This is not supported yet, see BEAM-11104.
+func (fn *splittableDoFn) ProcessElement(rt *sdf.LockRTracker, emit
func(Record)) sdf.ProcessContinuation {
+ position := rt.GetRestriction().(offsetrange.Restriction).Start
+ for {
+ records, err := fn.ExternalService.readNextRecords(position)
+ if err == fn.ExternalService.ThrottlingErr {
+ return sdf.ResumeProcessingIn(60 * time.Seconds)
+ }
+ if len(records) == 0 {
+ return sdf.ResumeProcessingIn(10 * time.Seconds)
Review Comment:
That's a fair note. Adding clarifying comments is always good for a
documentation snippet
Issue Time Tracking
-------------------
Worklog Id: (was: 778227)
Time Spent: 27h 40m (was: 27.5h)
> [Go SDK] DoFn Self Checkpointing
> --------------------------------
>
> Key: BEAM-11104
> URL: https://issues.apache.org/jira/browse/BEAM-11104
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-go
> Reporter: Robert Burke
> Assignee: Jack McCluskey
> Priority: P3
> Fix For: 2.40.0
>
> Time Spent: 27h 40m
> Remaining Estimate: 0h
>
> Allow SplittableDoFns to self checkpoint.
> Design doc:
> [https://docs.google.com/document/d/1_JbzjY9JR07ZK5v7PcZevUfzHPsqwzfV7W6AouNpMPk/edit?usp=sharing]
>
> Feature is written E2E and users will be able to return ProcessContinuations
> from SDFs as of 2.39.0 but the full behavior has not been fully validated. An
> integration test that validates self-checkpointing is working as-intended
> will need to be written and passing before the feature is no longer
> considered experimental and this ticket is marked as resolved.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
