[
https://issues.apache.org/jira/browse/BEAM-1899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16097964#comment-16097964
]
ASF GitHub Bot commented on BEAM-1899:
--------------------------------------
GitHub user bastiliu reopened a pull request:
https://github.com/apache/beam/pull/3624
[BEAM-1899] Implementation of JStorm runner
Follow this checklist to help us incorporate your contribution quickly and
easily:
- [ ] Make sure there is a [JIRA
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the
change (usually before you start working on it). Trivial changes like typos do
not require a JIRA issue. Your pull request should address just this issue,
without pulling in other changes.
- [ ] Each commit in the pull request should have a meaningful subject
line and body.
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA
issue.
- [ ] Write a pull request description that is detailed enough to
understand what the pull request does, how, and why.
- [ ] Run `mvn clean verify` to make sure basic checks pass. A more
thorough check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
---
@kennknowles @peihe Could you please help to review the implementation of
JStorm runner?
Known issues:
- There are still two test suites which were failed(CountingSourceTest,
GenerateSequenceTest). I am looking on them.
- SetState, SplittableParDo, AttemptedMetric, CommittedMetrics, and
TestStream were not supported yet. So these parts are excluded in the
integration test.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/bastiliu/beam jstorm-runner
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/beam/pull/3624.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3624
----
commit 731555c1174aa4aaea70baf25e001e8bccd16142
Author: basti.lj <[email protected]>
Date: 2017-05-02T03:12:58Z
Refactor: move JStorm runner code to Beam repo.
commit ef81cd8e43cbc9379fee3a40a0c50bb8c267f9f9
Author: basti.lj <[email protected]>
Date: 2017-05-12T08:24:00Z
Support startBundle & finishBundle on jstorm batch mode
commit 0a5b7e69c6edc289052b44b9c9efd10d4ec1d5a8
Author: basti.lj <[email protected]>
Date: 2017-05-12T08:25:43Z
Update to internal version 0.7.0-jstorm
commit 4d0a594df51102578c86935a6db7bad853894ae1
Author: Pei He <[email protected]>
Date: 2017-05-17T08:28:16Z
Upgrade Beam to 2.0.0.
commit ec91b660e15e782016e2ec9518247c99921be31e
Author: Pei He <[email protected]>
Date: 2017-05-18T03:27:20Z
Internal: Upgrade fixup for imports.
commit 9d2ddb4553fb7347eb3c5900dee03a02d85cd925
Author: Pei He <[email protected]>
Date: 2017-05-18T03:30:36Z
jstorm-runner: 2.0.0 upgrade fixups.
commit c12c768e9f29b73d005adc367259bc11c8bdabfb
Author: Pei He <[email protected]>
Date: 2017-05-18T04:46:22Z
Internal: Upgrade fixup for states.
commit 9bd97cf35e0e1285fc6459708a5d062905befe74
Author: Pei He <[email protected]>
Date: 2017-05-18T05:03:40Z
Internal: Upgrade fixup for removing aggregators.
commit ff316f61132fd55914f7d19f320d244f4053832e
Author: Pei He <[email protected]>
Date: 2017-05-18T05:15:23Z
Internal: Upgrade fixup for others.
commit a09d761e3171ee62eda4718255144db9107d88cb
Author: Pei He <[email protected]>
Date: 2017-05-18T12:16:55Z
Internal: Implement Beam Metrics.
commit 1d85413c49335d9427bcc01c900bc500fdb7c5d4
Author: Pei He <[email protected]>
Date: 2017-05-18T12:22:07Z
Internal: workaround before calling finishBundle is fixed.
commit fdcd3e9fc4ed561d164f791c848c99c0f59742eb
Author: basti.lj <[email protected]>
Date: 2017-05-19T09:24:38Z
Fix that bug that state of timers were not persisted correctly.
commit 49968800bb731d1f34322615ddf35cc1e0be6f8c
Author: basti.lj <[email protected]>
Date: 2017-05-19T15:13:33Z
fix bug that startBundle/finishBundle was not called after fire timers by
updating watermark
commit 715512e92049fc2f018b56c044c2beb11fadebc5
Author: basti.lj <[email protected]>
Date: 2017-05-22T02:23:18Z
Revert "Internal: workaround before calling finishBundle is fixed."
This reverts commit c0a40cf4317a7fa63b401c7f5fecea6b17355b55.
commit d9cde0537ad1b80ca7455e5cc3af0332e208350b
Author: basti.lj <[email protected]>
Date: 2017-05-23T04:32:28Z
Map local TupleTag to external tag before output in DoFn
commit 3ab3e73fd10408c5ee212c179a1d8bc147a8dcbd
Author: basti.lj <[email protected]>
Date: 2017-05-23T04:39:13Z
Add combine test cases
commit e9775e4d10d6fd573496ba018dde0f4d68c6bd1d
Author: Pei He <[email protected]>
Date: 2017-05-25T09:54:52Z
Add WordCountBenchmark for JStorm runner.
commit 0b2b26df9060e2a961621c770e2fe07751c6acce
Author: Pei He <[email protected]>
Date: 2017-05-26T08:12:03Z
fixup: remove print messages.
commit 10dfe21738a2f2bd1c198c03b521b583d8af0a2b
Author: Pei He <[email protected]>
Date: 2017-05-27T09:47:27Z
Benchmark: add StateInternalsBenchmark.
commit fa9c498f30a690d5e64dcc48b28f58e15012f4b4
Author: basti.lj <[email protected]>
Date: 2017-06-02T02:39:04Z
1. Fix incorrect asserted value in stateInternalsTest
2. Deactivate source reader when closing spout
3. Fix typo in pom.xml
commit 343421d8ccf0a8ed4e63c64c816756400c706510
Author: basti.lj <[email protected]>
Date: 2017-06-06T08:17:36Z
1. Fix bug of state internal test
2. Improve performance of bag state by reducing duplicated wrtie/read ops
commit 7ee5ef4127ab849645bd6c136efe1c987fda3aa5
Author: basti.lj <[email protected]>
Date: 2017-06-06T09:27:52Z
Fix incorrect asserted result of CombineTest
commit ca79d9fc728a2ebf3f899d7c36b88945386ad954
Author: basti.lj <[email protected]>
Date: 2017-06-08T08:45:27Z
support Iterable in JStormBagState.read
commit 294d7933c4bbb2ec7ebf6242ef3793ce12a22938
Author: basti.lj <[email protected]>
Date: 2017-06-08T08:47:08Z
Build immutable list for "Iterable" type output value
commit 94e83eff35cc9cc7a6cacda225cc3725d2317ee4
Author: basti.lj <[email protected]>
Date: 2017-06-09T07:47:07Z
Support keys, values and entries interface of MapState
commit e4aa7c692fc14dfc67f57f924b82ad1f022bc134
Author: basti.lj <[email protected]>
Date: 2017-06-12T12:13:58Z
Replace immutable list with array list because "null" is a valid value
which is not permitted in immutable list.
commit 6c161da1c50e47362eb7c4dbd5ca0730bdc3cab3
Author: basti.lj <[email protected]>
Date: 2017-06-14T06:34:07Z
Add unit test for CoGroupByKey and adding massive data into BagState
commit 65d11974d15aca5293ff77c2169e6de798216635
Author: basti.lj <[email protected]>
Date: 2017-06-26T03:10:12Z
Support user to define parallelism number of composite PTransform
commit 1d5c3a8bca3c772b0cb88c50b0721f48e3660201
Author: basti.lj <[email protected]>
Date: 2017-06-26T03:21:56Z
Add missing file for last commit of "parallelism number support on
composite transform"
commit 2912db32ed4a7d5b0875fdf9eb351618c5384cee
Author: basti.lj <[email protected]>
Date: 2017-07-04T07:14:16Z
1. Fix the bug that different DoFn executors use same state instance
2. Support stateful keyed DoFn
----
> JStorm runner
> -------------
>
> Key: BEAM-1899
> URL: https://issues.apache.org/jira/browse/BEAM-1899
> Project: Beam
> Issue Type: New Feature
> Components: runner-ideas
> Reporter: Pei He
> Assignee: Basti Liu
>
> Implement a runner for JStorm https://github.com/alibaba/jstorm
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
