GitHub user bastiliu opened a pull request:
https://github.com/apache/beam/pull/3624
Jstorm runner
Follow this checklist to help us incorporate your contribution quickly and
easily:
- [ ] Make sure there is a [JIRA
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the
change (usually before you start working on it). Trivial changes like typos do
not require a JIRA issue. Your pull request should address just this issue,
without pulling in other changes.
- [ ] Each commit in the pull request should have a meaningful subject
line and body.
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA
issue.
- [ ] Write a pull request description that is detailed enough to
understand what the pull request does, how, and why.
- [ ] Run `mvn clean verify` to make sure basic checks pass. A more
thorough check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
---
@kennknowles @peihe Could you please help to review the implementation of
JStorm runner?
Known issues:
- There are still two test suites which were failed(CountingSourceTest,
GenerateSequenceTest). I am looking on them.
- SetState, SplittableParDo, AttemptedMetric, CommittedMetrics, and
TestStream were not supported yet. So these parts are excluded in the
integration test.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/bastiliu/beam jstorm-runner
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/beam/pull/3624.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3624
----
commit b0c2bb71a51d4354e4081884c09fb37609e5ede2
Author: basti.lj <[email protected]>
Date: 2017-05-02T03:12:58Z
Refactor: move JStorm runner code to Beam repo.
commit fc3194a974265186912be30617fdedf6c40a35c2
Author: basti.lj <[email protected]>
Date: 2017-05-12T08:24:00Z
Support startBundle & finishBundle on jstorm batch mode
commit 8aeaf7f3e65ed2b350731922f940db3f5efbe613
Author: basti.lj <[email protected]>
Date: 2017-05-12T08:25:43Z
Update to internal version 0.7.0-jstorm
commit d5d5775a0164fe5041d49a7f9588ad60f3428089
Author: Pei He <[email protected]>
Date: 2017-05-17T08:28:16Z
Upgrade Beam to 2.0.0.
commit 90a275a19550f27521f1b89a217221375a77c65a
Author: Pei He <[email protected]>
Date: 2017-05-18T03:27:20Z
Internal: Upgrade fixup for imports.
commit 230913f454458ff7d00daabdd8abee67767acf8f
Author: Pei He <[email protected]>
Date: 2017-05-18T03:30:36Z
jstorm-runner: 2.0.0 upgrade fixups.
commit 2abaf34864ffc2b797a10746a672dad6967b3e1b
Author: Pei He <[email protected]>
Date: 2017-05-18T04:46:22Z
Internal: Upgrade fixup for states.
commit 231d6f5bc349b540dea0c65a220e2ad0f53e05bf
Author: Pei He <[email protected]>
Date: 2017-05-18T05:03:40Z
Internal: Upgrade fixup for removing aggregators.
commit 953b3404e352399bcba24203aab84819acc3c2ac
Author: Pei He <[email protected]>
Date: 2017-05-18T05:15:23Z
Internal: Upgrade fixup for others.
commit 8fe54d84138c5aa8d4ec8499c47c6c6fdc7464d0
Author: Pei He <[email protected]>
Date: 2017-05-18T12:16:55Z
Internal: Implement Beam Metrics.
commit 2a26ddf909158572b2006c3279e5567e2bd78fa7
Author: Pei He <[email protected]>
Date: 2017-05-18T12:22:07Z
Internal: workaround before calling finishBundle is fixed.
commit eb18d2b08a51929ccf204a01e404e4e5e2033bb8
Author: basti.lj <[email protected]>
Date: 2017-05-19T09:24:38Z
Fix that bug that state of timers were not persisted correctly.
commit 7fc51a08e11bd38376954000d92b625eeba2f8bf
Author: basti.lj <[email protected]>
Date: 2017-05-19T15:13:33Z
fix bug that startBundle/finishBundle was not called after fire timers by
updating watermark
commit 57327622df1dd6624f12b7ce763484eb2ba7750e
Author: basti.lj <[email protected]>
Date: 2017-05-22T02:23:18Z
Revert "Internal: workaround before calling finishBundle is fixed."
This reverts commit c0a40cf4317a7fa63b401c7f5fecea6b17355b55.
commit 100bf43768259c2ecdd9cb20f717001655c5cc09
Author: basti.lj <[email protected]>
Date: 2017-05-23T04:32:28Z
Map local TupleTag to external tag before output in DoFn
commit 52f3531fa4e1abe059780598ee4f73cddf01d5fe
Author: basti.lj <[email protected]>
Date: 2017-05-23T04:39:13Z
Add combine test cases
commit 9feb84786f96f3594380295dbd81088a185e61f5
Author: Pei He <[email protected]>
Date: 2017-05-25T09:54:52Z
Add WordCountBenchmark for JStorm runner.
commit a7311d8c53faf731ac532ed5e3ed331a3da0286d
Author: Pei He <[email protected]>
Date: 2017-05-26T08:12:03Z
fixup: remove print messages.
commit 2c993706cd95ee02ebc4001d54549e34b6dfe2af
Author: Pei He <[email protected]>
Date: 2017-05-27T09:47:27Z
Benchmark: add StateInternalsBenchmark.
commit 795e84d5e489165bef6d77035a88cdaf13c86bde
Author: basti.lj <[email protected]>
Date: 2017-06-02T02:39:04Z
1. Fix incorrect asserted value in stateInternalsTest
2. Deactivate source reader when closing spout
3. Fix typo in pom.xml
commit 8c28ccca271a746a88f3ff865ab59b32c6cee779
Author: basti.lj <[email protected]>
Date: 2017-06-06T08:17:36Z
1. Fix bug of state internal test
2. Improve performance of bag state by reducing duplicated wrtie/read ops
commit 13ff43a58cfd01d54889a0f5586a10bdb6c35599
Author: basti.lj <[email protected]>
Date: 2017-06-06T09:27:52Z
Fix incorrect asserted result of CombineTest
commit 5004ac860825f4a29ae023cb3ac5e642740b4f8c
Author: basti.lj <[email protected]>
Date: 2017-06-08T08:45:27Z
support Iterable in JStormBagState.read
commit 8129cbc8aaeda09ef5c2c33bdda2e5f54e0397dd
Author: basti.lj <[email protected]>
Date: 2017-06-08T08:47:08Z
Build immutable list for "Iterable" type output value
commit 479df6bb5935964b3a97d5942815b153e21dcee5
Author: basti.lj <[email protected]>
Date: 2017-06-09T07:47:07Z
Support keys, values and entries interface of MapState
commit dc21f6f2684b1d38fade1082ccc8aac16f76e9db
Author: basti.lj <[email protected]>
Date: 2017-06-12T12:13:58Z
Replace immutable list with array list because "null" is a valid value
which is not permitted in immutable list.
commit 6a1c8c6100d99d2f9f46757403970cddcd10a7bf
Author: basti.lj <[email protected]>
Date: 2017-06-14T06:34:07Z
Add unit test for CoGroupByKey and adding massive data into BagState
commit 8762dc9b66e10e231d1bc8d82bd9551e86fa2007
Author: basti.lj <[email protected]>
Date: 2017-06-26T03:10:12Z
Support user to define parallelism number of composite PTransform
commit 28a353a8e49cf9abddce4623c061fc70b27b4ee6
Author: basti.lj <[email protected]>
Date: 2017-06-26T03:21:56Z
Add missing file for last commit of "parallelism number support on
composite transform"
commit 1de41762e662c5d6964e765684919717d6fdfdd8
Author: basti.lj <[email protected]>
Date: 2017-07-04T07:14:16Z
1. Fix the bug that different DoFn executors use same state instance
2. Support stateful keyed DoFn
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
