GitHub user amitsela opened a pull request:
https://github.com/apache/incubator-beam/pull/1055
[BEAM-259] Enable RunnableOnService for batch.
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:
- [ ] Make sure the PR title is formatted like:
`[BEAM-<Jira issue #>] Description of pull request`
- [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
- [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
number, if there is one.
- [ ] If this contribution is large, please file an Apache
[Individual Contributor License
Agreement](https://www.apache.org/licenses/icla.txt).
---
Handle empty Flatten for bounded.
Spark will bubble out a SparkException for user code failure, so this won't
catch. Asserting on the
error message should be good enough.
outputWithTimestamp should handle start/finishBundle as well.
Explode WindowedValues before processing.
sideOutputWithTimestamp to address start/finishBundle.
SideInput with windows.
Unused for now, remove.
Take sideInput window startegy into account, for combine as well.
reduce code duplication.
Spark combine support.
reuse code where possible.
Expose sideInputs and insertDefault in Combine.Globally for direct
translation.
Direct translation of Combine.Globally into Spark's aggregate function.
Make default run with 4 cores by default - makes tests run with multiple
threads, but not too many.
SideInputReader for the Spark runner.
A common abstraction for Keyed and Global implementation.
Implement Combine.Globally via Spark's aggregate.
runnable-on-service profile doesn't need pluginManagement.
Removing test as it does not follow a deterministic combine implementation.
Context reuse is mostly for testing. To avoid a test failure that will stop
the context and fail all
following tests we need to recreate the context if it's stopped as well.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/amitsela/incubator-beam BEAM-259
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-beam/pull/1055.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1055
----
commit 53ed9951203da7f7e2cff0ef4d9ec64f37a0f29d
Author: Sela <[email protected]>
Date: 2016-09-23T10:32:28Z
Enable RunnableOnService for batch.
Handle empty Flatten for bounded.
Spark will bubble out a SparkException for user code failure, so this won't
catch. Asserting on the
error message should be good enough.
outputWithTimestamp should handle start/finishBundle as well.
Explode WindowedValues before processing.
sideOutputWithTimestamp to address start/finishBundle.
SideInput with windows.
Unused for now, remove.
Take sideInput window startegy into account, for combine as well.
reduce code duplication.
Spark combine support.
reuse code where possible.
Expose sideInputs and insertDefault in Combine.Globally for direct
translation.
Direct translation of Combine.Globally into Spark's aggregate function.
Make default run with 4 cores by default - makes tests run with multiple
threads, but not too many.
SideInputReader for the Spark runner.
A common abstraction for Keyed and Global implementation.
Implement Combine.Globally via Spark's aggregate.
runnable-on-service profile doesn't need pluginManagement.
Removing test as it does not follow a deterministic combine implementation.
Context reuse is mostly for testing. To avoid a test failure that will stop
the context and fail all
following tests we need to recreate the context if it's stopped as well.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
