[
https://issues.apache.org/jira/browse/BEAM-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16953250#comment-16953250
]
Ning Kang edited comment on BEAM-646 at 10/16/19 10:51 PM:
-----------------------------------------------------------
Hi,
This is Ning. I'm working on a project that supports InteractiveRunner
(Python3+) when users use Beam in notebook environment.
Would apply() interception in runner still be useful as a hook for non-Beam
related but interactivity related features such as collecting IPython/notebook
cell information when a PTransfrom is applied?
Of course, we can also have all pipelines support interactivity by making the
interception inside pipelines themselves. But it's unlikely that all runners
would/could take a pipeline with interactivity logic at this moment. And those
ipython/notebook dependencies probably shouldn't be introduced into pipeline
itself.
Would there be any APIs that support invoking arbitrary external logic at
different stages of building a pipeline when pipeline is completely decoupled
from runner?
Thanks!
was (Author: ningk):
Hi,
This is Ning. I'm working a project that supports InteractiveRunner (Python3+)
when users use Beam in notebook environment.
Would apply() interception in runner still be useful as a hook for non-Beam
related but interactivity related features such as collecting IPython/notebook
cell information when a PTransfrom is applied?
Of course, we can also have all pipelines support interactivity by making the
interception inside pipelines themselves. But it's unlikely that all runners
would/could take a pipeline with interactivity logic at this moment. And those
ipython/notebook dependencies probably shouldn't be introduced into pipeline
itself.
Would there be any APIs that support invoking arbitrary external logic at
different stages of building a pipeline when pipeline is completely decoupled
from runner?
Thanks!
> Get runners out of the apply()
> ------------------------------
>
> Key: BEAM-646
> URL: https://issues.apache.org/jira/browse/BEAM-646
> Project: Beam
> Issue Type: Improvement
> Components: beam-model, sdk-java-core
> Reporter: Kenneth Knowles
> Assignee: Thomas Groh
> Priority: Major
> Labels: backwards-incompatible
> Fix For: 0.6.0
>
>
> Right now, the runner intercepts calls to apply() and replaces transforms as
> we go. This means that there is no "original" user graph. For portability and
> misc architectural benefits, we would like to build the original graph first,
> and have the runner override later.
> Some runners already work in this manner, but we could integrate it more
> smoothly, with more validation, via some handy APIs on e.g. the Pipeline
> object.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
