[
https://issues.apache.org/jira/browse/BEAM-4135?focusedWorklogId=95126&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95126
]
ASF GitHub Bot logged work on BEAM-4135:
----------------------------------------
Author: ASF GitHub Bot
Created on: 25/Apr/18 17:22
Start Date: 25/Apr/18 17:22
Worklog Time Spent: 10m
Work Description: tgroh opened a new pull request #5225: [BEAM-4135] Fork
the DirectRunner Engine
URL: https://github.com/apache/beam/pull/5225
For development velocity, the DirectRunner's two implementations can share a
large number of underlying implementations, but due to limitations on
intermediate
data type representations and the depth of the dependency chains, partial
migrations
end with extremely messy generic types. Forking the two implementations in
order
to minimize the need to modify the Java-SDK specific implementations of the
DirectRunner
is likely to significantly simplify initial development efforts, with the
ability to make
future components which maintain a high degree of complexity generic and
shared between
the two processing implementations.
This change is a fork + prune, removing as many classes as seems sensible.
Follow-up
changes will migrate Java SDK types (AppliedPTransform and PCollection being
the two
primary types) to the corresponding portable representations (PTransformNode
and PCollectionNode).
Some of the forked utilities (e.g. WatermarkManager) should be deduplicated,
as they are sufficiently generic at their current state. This will occur in
a follow-up change.
------------------------
Follow this checklist to help us incorporate your contribution quickly and
easily:
- [ ] Make sure there is a [JIRA
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the
change (usually before you start working on it). Trivial changes like typos do
not require a JIRA issue. Your pull request should address just this issue,
without pulling in other changes.
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA
issue.
- [ ] Write a pull request description that is detailed enough to
understand:
- [ ] What the pull request does
- [ ] Why it does it
- [ ] How it does it
- [ ] Why this approach
- [ ] Each commit in the pull request should have a meaningful subject line
and body.
- [ ] Run `./gradlew build` to make sure basic checks pass. A more thorough
check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 95126)
Time Spent: 2h (was: 1h 50m)
> Remove Use of Java SDK Types in the DirectRunner "engine"
> ---------------------------------------------------------
>
> Key: BEAM-4135
> URL: https://issues.apache.org/jira/browse/BEAM-4135
> Project: Beam
> Issue Type: New Feature
> Components: runner-direct
> Reporter: Thomas Groh
> Assignee: Thomas Groh
> Priority: Major
> Labels: portability
> Time Spent: 2h
> Remaining Estimate: 0h
>
> The "engine" consists of the components which determine where to schedule
> work and route it to the appropriate processors, such as WatermarkManager,
> DirectBundleProcessor, and associated.
>
> These engine components never inspect the actual characteristics of the
> packaged work (e.g. the PCollection is a token, rather than a rich object),
> so they should not require use of a PCollection directly - instead, they can
> be generic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
