GitHub user amitsela opened a pull request:
https://github.com/apache/beam/pull/1987
[BEAM-920] Add support for Watermarks in the Spark runner.
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:
- [ ] Make sure the PR title is formatted like:
`[BEAM-<Jira issue #>] Description of pull request`
- [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
- [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
number, if there is one.
- [ ] If this contribution is large, please file an Apache
[Individual Contributor License
Agreement](https://www.apache.org/licenses/icla.txt).
---
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/amitsela/beam watermarks
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/beam/pull/1987.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1987
----
commit 33288a7c2a30f65886cdf93007f202dfcbdfb842
Author: Sela <ans...@paypal.com>
Date: 2017-02-12T16:28:19Z
A global Watermark holder to update and broadcast to workers.
commit a44c5de1a043d3aa14fc187b5621efad683d7015
Author: Sela <ans...@paypal.com>
Date: 2017-02-12T16:30:47Z
Ingest the input watermarks into the GlobalWatermarkHolder.
commit 656e2fb0ad35e11661a14a9dc2db489abf1880f3
Author: Sela <ans...@paypal.com>
Date: 2017-02-12T16:31:14Z
Advance watermarks onBatchCompleted hook.
commit 8300b3f363dbbffabeb7d7148198ae146a59bca4
Author: Sela <ans...@paypal.com>
Date: 2017-02-12T16:32:06Z
Handle QueuedStream (for testing) and track sources upstream.
commit 8aa0268dbd607037407aaae30e8968872c3e5146
Author: Sela <ans...@paypal.com>
Date: 2017-02-12T16:33:16Z
Refactor according to changes.
commit 0d1b1ce202c48763153cfab4147e28f48e4c28e1
Author: Sela <ans...@paypal.com>
Date: 2017-02-12T16:33:55Z
Exopse RegexMatcher as part of the SDK.
commit 4c8fec02f8b7181fcc192268ec48907dadca3c40
Author: Sela <ans...@paypal.com>
Date: 2017-02-12T16:34:30Z
Watermark tests.
commit 1150eb517c35352a62611444ca24030e3bdd9e7c
Author: Sela <ans...@paypal.com>
Date: 2017-02-12T17:23:28Z
Streaming sources tracking test.
commit 3d3f19566e98c684e68f34afffc6c01904852b0a
Author: Sela <ans...@paypal.com>
Date: 2017-02-12T17:47:00Z
This relied on a wrong functionality as descrined in BEAM-1444 and should
be revisited there.
commit e3593b5cc1d4004b49b2f5d44ca07c6ec63806b3
Author: Sela <ans...@paypal.com>
Date: 2017-02-12T18:08:40Z
fixup! licenese.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
