GitHub user bastiliu reopened a pull request:
https://github.com/apache/beam/pull/3624
[BEAM-1899] Implementation of JStorm runner
Follow this checklist to help us incorporate your contribution quickly and
easily:
- [ ] Make sure there is a [JIRA
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the
change (usually before you start working on it). Trivial changes like typos do
not require a JIRA issue. Your pull request should address just this issue,
without pulling in other changes.
- [ ] Each commit in the pull request should have a meaningful subject
line and body.
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA
issue.
- [ ] Write a pull request description that is detailed enough to
understand what the pull request does, how, and why.
- [ ] Run `mvn clean verify` to make sure basic checks pass. A more
thorough check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
---
@kennknowles @peihe Could you please help to review the implementation of
JStorm runner?
Known issues:
- There are still two test suites which were failed(CountingSourceTest,
GenerateSequenceTest). I am looking on them.
- SetState, SplittableParDo, AttemptedMetric, CommittedMetrics, and
TestStream were not supported yet. So these parts are excluded in the
integration test.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/bastiliu/beam jstorm-runner
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/beam/pull/3624.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3624
----
commit 731555c1174aa4aaea70baf25e001e8bccd16142
Author: basti.lj <[email protected]>
Date: 2017-05-02T03:12:58Z
Refactor: move JStorm runner code to Beam repo.
commit ef81cd8e43cbc9379fee3a40a0c50bb8c267f9f9
Author: basti.lj <[email protected]>
Date: 2017-05-12T08:24:00Z
Support startBundle & finishBundle on jstorm batch mode
commit 0a5b7e69c6edc289052b44b9c9efd10d4ec1d5a8
Author: basti.lj <[email protected]>
Date: 2017-05-12T08:25:43Z
Update to internal version 0.7.0-jstorm
commit 4d0a594df51102578c86935a6db7bad853894ae1
Author: Pei He <[email protected]>
Date: 2017-05-17T08:28:16Z
Upgrade Beam to 2.0.0.
commit ec91b660e15e782016e2ec9518247c99921be31e
Author: Pei He <[email protected]>
Date: 2017-05-18T03:27:20Z
Internal: Upgrade fixup for imports.
commit 9d2ddb4553fb7347eb3c5900dee03a02d85cd925
Author: Pei He <[email protected]>
Date: 2017-05-18T03:30:36Z
jstorm-runner: 2.0.0 upgrade fixups.
commit c12c768e9f29b73d005adc367259bc11c8bdabfb
Author: Pei He <[email protected]>
Date: 2017-05-18T04:46:22Z
Internal: Upgrade fixup for states.
commit 9bd97cf35e0e1285fc6459708a5d062905befe74
Author: Pei He <[email protected]>
Date: 2017-05-18T05:03:40Z
Internal: Upgrade fixup for removing aggregators.
commit ff316f61132fd55914f7d19f320d244f4053832e
Author: Pei He <[email protected]>
Date: 2017-05-18T05:15:23Z
Internal: Upgrade fixup for others.
commit a09d761e3171ee62eda4718255144db9107d88cb
Author: Pei He <[email protected]>
Date: 2017-05-18T12:16:55Z
Internal: Implement Beam Metrics.
commit 1d85413c49335d9427bcc01c900bc500fdb7c5d4
Author: Pei He <[email protected]>
Date: 2017-05-18T12:22:07Z
Internal: workaround before calling finishBundle is fixed.
commit fdcd3e9fc4ed561d164f791c848c99c0f59742eb
Author: basti.lj <[email protected]>
Date: 2017-05-19T09:24:38Z
Fix that bug that state of timers were not persisted correctly.
commit 49968800bb731d1f34322615ddf35cc1e0be6f8c
Author: basti.lj <[email protected]>
Date: 2017-05-19T15:13:33Z
fix bug that startBundle/finishBundle was not called after fire timers by
updating watermark
commit 715512e92049fc2f018b56c044c2beb11fadebc5
Author: basti.lj <[email protected]>
Date: 2017-05-22T02:23:18Z
Revert "Internal: workaround before calling finishBundle is fixed."
This reverts commit c0a40cf4317a7fa63b401c7f5fecea6b17355b55.
commit d9cde0537ad1b80ca7455e5cc3af0332e208350b
Author: basti.lj <[email protected]>
Date: 2017-05-23T04:32:28Z
Map local TupleTag to external tag before output in DoFn
commit 3ab3e73fd10408c5ee212c179a1d8bc147a8dcbd
Author: basti.lj <[email protected]>
Date: 2017-05-23T04:39:13Z
Add combine test cases
commit e9775e4d10d6fd573496ba018dde0f4d68c6bd1d
Author: Pei He <[email protected]>
Date: 2017-05-25T09:54:52Z
Add WordCountBenchmark for JStorm runner.
commit 0b2b26df9060e2a961621c770e2fe07751c6acce
Author: Pei He <[email protected]>
Date: 2017-05-26T08:12:03Z
fixup: remove print messages.
commit 10dfe21738a2f2bd1c198c03b521b583d8af0a2b
Author: Pei He <[email protected]>
Date: 2017-05-27T09:47:27Z
Benchmark: add StateInternalsBenchmark.
commit fa9c498f30a690d5e64dcc48b28f58e15012f4b4
Author: basti.lj <[email protected]>
Date: 2017-06-02T02:39:04Z
1. Fix incorrect asserted value in stateInternalsTest
2. Deactivate source reader when closing spout
3. Fix typo in pom.xml
commit 343421d8ccf0a8ed4e63c64c816756400c706510
Author: basti.lj <[email protected]>
Date: 2017-06-06T08:17:36Z
1. Fix bug of state internal test
2. Improve performance of bag state by reducing duplicated wrtie/read ops
commit 7ee5ef4127ab849645bd6c136efe1c987fda3aa5
Author: basti.lj <[email protected]>
Date: 2017-06-06T09:27:52Z
Fix incorrect asserted result of CombineTest
commit ca79d9fc728a2ebf3f899d7c36b88945386ad954
Author: basti.lj <[email protected]>
Date: 2017-06-08T08:45:27Z
support Iterable in JStormBagState.read
commit 294d7933c4bbb2ec7ebf6242ef3793ce12a22938
Author: basti.lj <[email protected]>
Date: 2017-06-08T08:47:08Z
Build immutable list for "Iterable" type output value
commit 94e83eff35cc9cc7a6cacda225cc3725d2317ee4
Author: basti.lj <[email protected]>
Date: 2017-06-09T07:47:07Z
Support keys, values and entries interface of MapState
commit e4aa7c692fc14dfc67f57f924b82ad1f022bc134
Author: basti.lj <[email protected]>
Date: 2017-06-12T12:13:58Z
Replace immutable list with array list because "null" is a valid value
which is not permitted in immutable list.
commit 6c161da1c50e47362eb7c4dbd5ca0730bdc3cab3
Author: basti.lj <[email protected]>
Date: 2017-06-14T06:34:07Z
Add unit test for CoGroupByKey and adding massive data into BagState
commit 65d11974d15aca5293ff77c2169e6de798216635
Author: basti.lj <[email protected]>
Date: 2017-06-26T03:10:12Z
Support user to define parallelism number of composite PTransform
commit 1d5c3a8bca3c772b0cb88c50b0721f48e3660201
Author: basti.lj <[email protected]>
Date: 2017-06-26T03:21:56Z
Add missing file for last commit of "parallelism number support on
composite transform"
commit 2912db32ed4a7d5b0875fdf9eb351618c5384cee
Author: basti.lj <[email protected]>
Date: 2017-07-04T07:14:16Z
1. Fix the bug that different DoFn executors use same state instance
2. Support stateful keyed DoFn
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
