[
https://issues.apache.org/jira/browse/BEAM-3836?focusedWorklogId=424566&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-424566
]
ASF GitHub Bot logged work on BEAM-3836:
----------------------------------------
Author: ASF GitHub Bot
Created on: 18/Apr/20 02:04
Start Date: 18/Apr/20 02:04
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #11448: [BEAM-3836]
Enable dynamic splitting/checkpointing within the Java SDK harness.
URL: https://github.com/apache/beam/pull/11448#discussion_r410553899
##########
File path:
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/ProcessBundleHandler.java
##########
@@ -340,7 +347,12 @@ private void
createRunnerAndConsumersForPTransformRecursively(
"Unable to find active bundle for instruction id %s.",
request.getProcessBundleSplit().getInstructionId()));
}
- throw new UnsupportedOperationException("TODO: BEAM-3836, support
splitting within SDK.");
+ BeamFnApi.ProcessBundleSplitResponse.Builder response =
+ BeamFnApi.ProcessBundleSplitResponse.newBuilder();
+ for (BeamFnDataReadRunner channelRoot : bundleProcessor.getChannelRoots())
{
+ channelRoot.trySplit(request.getProcessBundleSplit(), response);
Review comment:
Yes, each root gets the full split request and extracts out the desiredsplit
information for itself if there is any specific to it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 424566)
Time Spent: 1.5h (was: 1h 20m)
> Java SDK harness should understand a BundleSplitRequest and respond with a
> BundleSplit before bundle finishes
> -------------------------------------------------------------------------------------------------------------
>
> Key: BEAM-3836
> URL: https://issues.apache.org/jira/browse/BEAM-3836
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-java-harness
> Reporter: Eugene Kirpichov
> Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> This, like BEAM-3835, is more necessary for LS than for checkpointing, but is
> technically also part of the SDF splitting protocol.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
