[
https://issues.apache.org/jira/browse/BEAM-8619?focusedWorklogId=344481&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-344481
]
ASF GitHub Bot logged work on BEAM-8619:
----------------------------------------
Author: ASF GitHub Bot
Created on: 15/Nov/19 18:55
Start Date: 15/Nov/19 18:55
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10126: [BEAM-8619]
Tear down the DoFns upon the control service termination …
URL: https://github.com/apache/beam/pull/10126#discussion_r346962561
##########
File path:
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/ProcessBundleHandler.java
##########
@@ -215,6 +224,83 @@ private void
createRunnerAndConsumersForPTransformRecursively(
*/
public BeamFnApi.InstructionResponse.Builder
processBundle(BeamFnApi.InstructionRequest request)
throws Exception {
+ BeamFnApi.ProcessBundleResponse.Builder response =
BeamFnApi.ProcessBundleResponse.newBuilder();
+
+ BundleProcessor bundleProcessor =
+ bundleProcessors.computeIfAbsent(
+ request.getProcessBundle().getProcessBundleDescriptorId(),
Review comment:
I like where this is going but we can't run a DoFn instance in parallel
(since this is what we promise to users) so this lookup map needs to contain a
list of BundleProcessor instances that are free. If there isn't one that is
free then we need to create one.
Also if a BundleProcessor fails during execution we need to throw it away
since the DoFn's may be in a bad state.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 344481)
Time Spent: 0.5h (was: 20m)
> Tear down the DoFns upon the control service termination in Java SDK harness
> ----------------------------------------------------------------------------
>
> Key: BEAM-8619
> URL: https://issues.apache.org/jira/browse/BEAM-8619
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-harness
> Affects Versions: 2.18.0
> Reporter: sunjincheng
> Assignee: sunjincheng
> Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Per the discussion in the ML, the detail can be found [1], the teardown of
> DoFns should be supported in the portability framework. It happens at two
> places:
> 1) Upon the control service termination
> 2) Tear down the unused DoFns periodically
> The aim of this JIRA is to add support for teardown the DoFns upon the
> control service termination in Java SDK harness.
> [1]
> https://lists.apache.org/thread.html/0c4a4cf83cf2e35c3dfeb9d906e26cd82d3820968ba6f862f91739e4@%3Cdev.beam.apache.org%3E
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
