[
https://issues.apache.org/jira/browse/BEAM-2597?focusedWorklogId=98652&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98652
]
ASF GitHub Bot logged work on BEAM-2597:
----------------------------------------
Author: ASF GitHub Bot
Created on: 05/May/18 01:13
Start Date: 05/May/18 01:13
Worklog Time Spent: 10m
Work Description: bsidhom commented on issue #5285: [BEAM-2597] Flink
batch ExecutableStage operator
URL: https://github.com/apache/beam/pull/5285#issuecomment-386768754
I've left some TODOs in place as questions for reviewers.
Unfortunately, it turns out that we need a few more interfaces for bundle
management due to serialization. Flink requires that `Function` instances be
`Serializable` as this is how they are distributed within a cluster. In order
to make this unit-testable, I had to introduce (stateless) serializable
factories to produce the stateful, non-serializable instances that need to be
managed within the TaskManagers themselves.
The exact shape of `FlinkBundleFactory` is pending
[BEAM-3673](https://issues.apache.org/jira/browse/BEAM-3673)/[BEAM-3327](https://issues.apache.org/jira/browse/BEAM-3327).
`CloseableDistributedCache` will hopefully go away with the right abstractions
on top of ArtifactSource. I wasn't sure how we wanted that part to look as it's
not the interesting part of stage execution.
Additionally, since the state handler does not quite handle coders correctly
due to inconsistencies with ProcessbundleDescriptors/fusion, that part may be
better as a part of a follow-up PR. I'm happy to make that a skeleton/stub as I
did with `ExecutableStageFunction` originally.
R: @jkff @tgroh
CC: @axelmagn @angoenka @lukecwik
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 98652)
Time Spent: 20m (was: 10m)
> FlinkRunner ExecutableStage batch operator
> ------------------------------------------
>
> Key: BEAM-2597
> URL: https://issues.apache.org/jira/browse/BEAM-2597
> Project: Beam
> Issue Type: Sub-task
> Components: runner-flink
> Reporter: Kenneth Knowles
> Assignee: Ben Sidhom
> Priority: Major
> Labels: portability
> Time Spent: 20m
> Remaining Estimate: 0h
>
> This operator will execute user code in the context of an SDK harness by
> constructing a ProcessBundleDescriptor from an ExecutableStage (physical
> stage plan) and sending instructions/elements over the control and data
> planes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
