[
https://issues.apache.org/jira/browse/BEAM-8335?focusedWorklogId=342766&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-342766
]
ASF GitHub Bot logged work on BEAM-8335:
----------------------------------------
Author: ASF GitHub Bot
Created on: 13/Nov/19 18:26
Start Date: 13/Nov/19 18:26
Worklog Time Spent: 10m
Work Description: rohdesamuel commented on issue #9720: [BEAM-8335] Add
initial modules for interactive streaming support
URL: https://github.com/apache/beam/pull/9720#issuecomment-553536981
> > > Whether this is streaming or not has nothing to do with quiescence. By
spreading the call across several RPC calls the service has become stateful,
more complicated, and idempotent. Much better for the RPC to be "give me the
events of this test stream" than have a (one-time-use) multi-RPC protocol to
fetch them.
> >
> >
> > The current implementation in the DirectRunner is that the
TestStreamEvaluator reads a single event from the TestStream at a time. It is
stateful in nature as it has to keep its current index into the TestStream
events between calls. Remember that the DirectRunner creates a
TransformEvaluator per application of a bundle. In order to support that
semantic of requesting and sending a single element, the service will also
reflect that.
>
> We should not let an implementation, especially one that is hopefully
going away soon, dictate what our APIs should be. Instead, we should come up
with clean APIs that represent the abstraction well, and then figure out how
they can be implemented. Only if it's truly impossible (or undesirable) to
support a clean API should we mangle it to reflect the implementation.
Alright, I rewrote the API to return a stream of TestStream events back.
This also allowed to get rid of the ConnectRequest.
> > > This PR does three separate things that all fall under the heading of
"add initial modules." To facilitate getting this in, I suggest braking it up
in to multiple PRs.
> > > (1) Augment TestStream to support multiple output, with tests.
> > > (2) Add a service backing for TestStream.
> > > (3) Add the StreamingCache stream merging classes.
> >
> >
> > With all due respect, this should have been brought up at the start of
the review. With that in mind, I am inclined to finish this PR as-is, without
breaking it up. I will keep your recommendation in mind in future PRs.
>
> I don't see how splitting it up now is harder than it would have been at
the start (though, yes, had we done it sooner perhaps some pieces could have
been in already). But this is a huge change with many orthogonal features that
should be separated out, if just for ease of rollback in case something is
broken, and for clean history when trying to understand the code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 342766)
Time Spent: 23.5h (was: 23h 20m)
> Add streaming support to Interactive Beam
> -----------------------------------------
>
> Key: BEAM-8335
> URL: https://issues.apache.org/jira/browse/BEAM-8335
> Project: Beam
> Issue Type: Improvement
> Components: runner-py-interactive
> Reporter: Sam Rohde
> Assignee: Sam Rohde
> Priority: Major
> Time Spent: 23.5h
> Remaining Estimate: 0h
>
> This issue tracks the work items to introduce streaming support to the
> Interactive Beam experience. This will allow users to:
> * Write and run a streaming job in IPython
> * Automatically cache records from unbounded sources
> * Add a replay experience that replays all cached records to simulate the
> original pipeline execution
> * Add controls to play/pause/stop/step individual elements from the cached
> records
> * Add ability to inspect/visualize unbounded PCollections
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
