[GitHub] incubator-beam pull request #1678: [BEAM-1198, BEAM-846, BEAM-260] Refactor ...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1678 [BEAM-1198, BEAM-846, BEAM-260] Refactor Dataflow translator to decouple input and output graphs more easily Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- This is preparatory work to make it possible for the translator to have a more loosely coupled relationship between its input and output. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam Dataflow-Translator Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1678.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1678 commit 8ed4bb68660c537e4a12c1077ecfa104f9a82eaa Author: Kenneth Knowles <k...@google.com> Date: 2016-12-21T22:21:50Z Inline needless interface DataflowTranslator.TranslationContext The only implementation was DataflowTranslator.Translator. This class needs some updating and the extra layer of the interface simply obscures that work. commit 272d06d7507ad7162616dd1b613efa7c8f5f4069 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-21T22:34:27Z Explicitly pass Step to mutate in Dataflow translator Previously, there was always a "current" step that was the most recent step created. This makes it cumbersome or impossible to do things like translate one primitive transform into a small subgraph of steps. Thus we added hacks like CreatePCollectionView which are not actually part of the model at all - in fact, we should be able to add the needed CollectionToSingleton steps simply by looking at the side inputs of a ParDo node. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #662: [BEAM-115] WIP: JSON Schema definition of ...
GitHub user kennknowles reopened a pull request: https://github.com/apache/incubator-beam/pull/662 [BEAM-115] WIP: JSON Schema definition of pipeline This is a json-schema sketch of the concrete schema from the [Pipeline Runner API proposal document](https://s.apache.org/beam-runner-api). Because our [serialization tech discussion](http://mail-archives.apache.org/mod_mbox/beam-dev/201606.mbox/%3CCAN_Ypr2ZPQG3OgPWu==kf-zztg06k0v5i0ay3dabchjyver...@mail.gmail.com%3E) seemed to favor JSON on the front end and Proto on the backend, I made this quick port. The original Avro IDL definition is also on [a branch with a test](https://github.com/kennknowles/incubator-beam/blob/pipeline-model/model/pipeline/src/main/avro/org/apache/beam/model/pipeline/pipeline.avdl). Notes & Caveats: - I did not try to flesh out any more details; this was a straight port. There's plenty to add, but a PR seems like a place that will attract a desired kind of concrete discussion even in the current state. - Typing this makes my hands hurt. Luckily, it should change exceedingly rarely. There are many libraries that can generate json-schema in various ways, including Jackson itself, but I'm not so sure any of them are applicable. - Reading this makes my eyes hurt. This is a real problem. We need a readable spec, not just a test suite for validation. - I am not so sure that [the schema library](https://github.com/daveclayton/json-schema-validator) I've used to build my smoke test is a good long term choice. I chose it because it was Jackson-based. - I've left comments in the JSON even though that is frowned upon, and taken advantage of Jackson's feature to allow them. They can also go into `"description"` fields. - Perhaps we could write YAML and convert to json-schema with no loss of precision? Feel free to leave comments here about the schema or meta issues of e.g. where the schema should live and what libraries we might want to use. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam pipeline-json-schema Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/662.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #662 commit c5843ce10e782056c76157169eb5516bf18ed9e4 Author: Kenneth Knowles <k...@google.com> Date: 2016-06-10T15:51:02Z WIP: add JSON Schema definition of pipeline --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1673: [BEAM-27] Require TimeDomain to delete a ...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1673 [BEAM-27] Require TimeDomain to delete a timer Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: @aljoscha A bit of an oversight, I neglected the fact that runners generally store different sorts of timers in rather different ways. When a user sets a timer, the `DoFnSignature` is available, so this will be for free. And when system code deletes a timer, the domain will always be known. This will require a Dataflow update, so don't worry if Dataflow-specific integration tests don't pass. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam delete-by-domain Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1673.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1673 commit 46dfd0fb4d2a1533d3ed053983faee6537d3ccf0 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-21T04:09:25Z Require TimeDomain to delete a timer --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1670: [BEAM-25, BEAM-1117] Fixes for direct run...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1670 [BEAM-25, BEAM-1117] Fixes for direct runner expansion and evaluation of stateful ParDo Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: @tgroh also peeled off from the timers PR, these are fixes for the whole setup. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam DirectRunner-Stateful Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1670.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1670 commit 0615fc9749c3fd0012f4d5524ea8486413778636 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-20T21:58:29Z Fix windowing in direct runner Stateful ParDo commit 7bc23d6b53ed29ae565121df49180ad8d4aac653 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-20T23:59:45Z Actually propagate and commit state in direct runner --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1669: [BEAM-1117] Direct runner timers prereqs
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1669 [BEAM-1117] Direct runner timers prereqs Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- Per request, here are some commits from #1667 broken out. I am happy to trim off more, etc, whatever is easiest for review. R: @tgroh You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam DirectRunner-timers-prereqs Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1669.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1669 commit f64816e0cf2e4fcc9525f40ede01c2f8e4ecf28d Author: Kenneth Knowles <k...@google.com> Date: 2016-12-20T04:40:11Z Add informative Instant formatter to BoundedWindow commit 46c6a4f613629f09b48e3630aa344760b0ad46d4 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-20T04:40:47Z Use informative Instant formatter in WatermarkHold commit 92baa418fbe53c0e7c7afc81db31fc02ab7f3915 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-20T21:57:55Z Add static Window.withOutputTimeFn to match build method commit 7118c4ff85636a65431be54fa2e2f18fb52914cf Author: Kenneth Knowles <k...@google.com> Date: 2016-12-20T22:20:07Z Add UsesTestStream for use with JUnit @Category commit f667a3e8abcd95be7a235132219c936178ab6bc8 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-08T04:18:44Z Allow setting timer by ID in DirectTimerInternals commit 217e5245e59800d57aa36551fbbdb642a5b447a0 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-20T21:37:40Z Hold output watermark according to pending timers --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1667: [BEAM-1117] Support user timers for ParDo...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1667 [BEAM-1117] Support user timers for ParDo in the direct runner Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: @tgroh You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam DirectRunner-timers Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1667.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1667 commit a3ac176cd7edb18d4f633682ee0e6ff30ab76f64 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-08T04:18:44Z Allow setting timer by ID in DirectTimerInternals commit 445750d6cf36f1eda1094531541788260c3fe229 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-08T18:27:23Z No longer reject timers for ParDo in direct runner commit d428abe9e12ddd2609773512a180589ff960d954 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-08T23:18:44Z Deliver timers in the direct runner commit 6915bbc550ad692656e8eeb1ba7161213c9a6ce6 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-20T04:40:11Z Add informative Instant formatter to BoundedWindow commit 2af3f93602b5299cc33c876310a784fc82ff4941 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-20T04:40:47Z Use informative Instant formatter in WatermarkHold --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1666: [BEAM-362] Move ExecutionContext and rela...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1666 [BEAM-362] Move ExecutionContext and related classes to runners-core Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: @lukecwik This is built on top of #1665 and will require a new worker image. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam ExecutionContext Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1666.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1666 commit 03a85e82ca5f1dff5aae184907508d7c5309a404 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-16T04:13:25Z Remove deprecated AggregatorFactory from SDK commit 6d7a4b10ba74b1fc08d0ad6a759ca5e0ebffdbba Author: Kenneth Knowles <k...@google.com> Date: 2016-12-16T04:20:34Z Move ExecutionContext and related classes to runners-core --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1665: [BEAM-362] Remove deprecated AggregatorFa...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1665 [BEAM-362] Remove deprecated AggregatorFactory from SDK Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- The reason this change actually adds real value is that it is the last reference to `ExecutionContext` in the SDK. After this PR, we can move all that to runners-core, which in turn starts to unblock moving `StateInternals` and `TimerInternals` to runners-core. I haven't included that here since it will require a Dataflow worker process while this will go right in. R: @lukecwik (randomly selected committer) You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam AggregatorFactory Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1665.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1665 commit 7484ed9903b9b41dbb47c08c2c0eb8c45d3371ac Author: Kenneth Knowles <k...@google.com> Date: 2016-12-16T04:13:25Z Remove deprecated AggregatorFactory from SDK --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1662: Prune branches in Jenkins build
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1662 Prune branches in Jenkins build This avoids possible conflicts in evolution of the configuration of how remote branches are mapped to local branches. Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: @jasonkuster I believe this would have prevented breakage. Since branches are nothing but pointers, having them sit around to potentially be incompatible seems to serve no purpose. In other words, I feel this should be a default configuration, unless someone has a rebuttal. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam jenkins-prune Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1662.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1662 commit bd6db885836af40940fca20cf680833661180e67 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-20T04:02:48Z Prune branches in Jenkins build This avoids possible conflicts in evolution of the configuration of how remote branches are mapped to local branches. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1660: [BEAM-27] Support setting and deleting ti...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1660 [BEAM-27] Support setting and deleting timers by ID in InMemoryTimerInternals Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- This is build on top of the move to runners-core in #1652. Only the second commit contains nontrivial changes. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam InMemoryTimerInternals-dedup Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1660.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1660 commit 6effd6c93587d1c0a02ba9285b47d165ae5c517d Author: Kenneth Knowles <k...@google.com> Date: 2016-12-16T04:45:56Z Move InMemoryTimerInternals to runners-core commit 557d2d724c53233ad9d34c9239ff5cf77b754d73 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-17T04:22:59Z Restore SDK's InMemoryTimerInternals, deprecated commit 0ef0e3a3ecabafdd934f7d03f47f0a220187fb22 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-19T22:01:36Z Support set and delete of timer by ID in InMemoryTimerInternals --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1659: [BEAM-1143] More escaping in Jenkins time...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1659 [BEAM-1143] More escaping in Jenkins timestamp spec Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- The current timestamp spec is insufficiently escaped. The failure mode is that relative timestamps are shown instead of a readable standard datetime. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam jenkins-timestamps Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1659.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1659 commit 627ccb522d7d818aebe2256246e81af2d3062d11 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-19T19:39:29Z More escaping in Jenkins timestamp spec --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1658: Revert "Show timestamps on log lines in J...
Github user kennknowles closed the pull request at: https://github.com/apache/incubator-beam/pull/1658 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1658: Revert "Show timestamps on log lines in J...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1658 Revert "Show timestamps on log lines in Jenkins" This reverts commit 4d964734c719037dd6e7e2a3c256da574514758a. Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam revert-jenkins Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1658.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1658 commit 0ccfc0bd35a9b6279188db571dbc48579963e681 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-19T19:34:15Z Revert "Show timestamps on log lines in Jenkins" This reverts commit 4d964734c719037dd6e7e2a3c256da574514758a. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1657: Fix seed job fetch spec
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1657 Fix seed job fetch spec Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam jenkins-fetch Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1657.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1657 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1652: [BEAM-27, BEAM-362] Move InMemoryTimerInt...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1652 [BEAM-27, BEAM-362] Move InMemoryTimerInternals to runners-core; restore a deprecated copy in its place Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- This is a replay of ec0bf7b4023ff75f4ec6723d2e77ed507eb57c51 with an additional adapter commit that should enable integration tests to succeed. Note that this configuration of commits works will with the current capabilities of `git` and GitHub. This behavior is not baked into the history, but is calculated dynamically by both, so it may improve over time. Notes on what would happen if this were done in one commit: - `git log --follow` finds the history - `git diff --find-copies` finds the copy - `git blame --find-copies-harder` finds the copy, but `--find-copies` does not I don't recall which level GitHub's UI works with, but the most important two are `git log` and `git blame`. To make sure it works without any of the above, two commits is better. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam InMemoryTimerInternals Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1652.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1652 commit 6effd6c93587d1c0a02ba9285b47d165ae5c517d Author: Kenneth Knowles <k...@google.com> Date: 2016-12-16T04:45:56Z Move InMemoryTimerInternals to runners-core commit 557d2d724c53233ad9d34c9239ff5cf77b754d73 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-17T04:22:59Z Restore SDK's InMemoryTimerInternals, deprecated --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1647: [BEAM-1089] Replace "--none--" with a mes...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1647 [BEAM-1089] Replace "--none--" with a message in Jenkins comments Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: @jasonkuster (I'll grab a committer later) I believe there was some expectation that the magic value "--none--" would disable comments. Instead, though, Jenkins is writing that exact string to our pull requests. Until we figure out how to actually disable comments, we may as well leave informative messages. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam jenkins-messages Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1647.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1647 commit 814b029689606d9d37dd696e4599efa0646a204b Author: Kenneth Knowles <k...@google.com> Date: 2016-12-16T21:59:31Z Replace "--none--" with a message in Jenkins comments I believe there was some expectation that the magic value "--none--" would disable comments. Instead, though, Jenkins is writing that exact string to our pull requests. Until we figure out how to actually disable comments, we may as well leave informative messages. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1645: Add properties to skip tests for particul...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1645 Add properties to skip tests for particular modules Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- A discussion piece. Used pervasively, we might trim a lot of time and also improve usability. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam skip-all-the-tests Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1645.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1645 commit b5dea398cf14fe8717fa27ad166fa860c55ab7d1 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-16T06:55:20Z Add properties to skip tests for particular modules --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1642: Delete deprecated TimerCallback
Github user kennknowles closed the pull request at: https://github.com/apache/incubator-beam/pull/1642 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1644: [BEAM-362] Port runners to runners-core A...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1644 [BEAM-362] Port runners to runners-core AggregatoryFactory Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: @amitsela For context: `runners.core.AggregatorFactory` is a copy of `sdk.transforms.Aggregator.AggregatorFactory`. So I am just porting everything to the non-deprecated bit so I can delete it. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam runners-core-AggregatorFactory Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1644.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1644 commit 63ac16669e7884dd42d431b9948ed675f7af3f03 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-16T05:06:14Z Port runners to runners-core AggregatoryFactory --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1641: Remove deprecated AggregatorFactory from ...
Github user kennknowles closed the pull request at: https://github.com/apache/incubator-beam/pull/1641 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1643: [BEAM-362] Move InMemoryTimerInternals to...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1643 [BEAM-362] Move InMemoryTimerInternals to runners-core Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: @jkff (I'll grab a committer later, but your contributions are thing affected) The only substantive aspect of this change is the move of the timer internals directly into `ProcessFn`. I think this is right or at least the right compromise for many reasons. - `TimerInternals` is really runner-facing; we don't want that interface in the SDK. - Changes to `TimerInternals` incur worker compatibility concerns, so getting it into runners-core is a win. (I have to change it soon, so I am trying to make my life easier) - `DoFnTester` doesn't actually support timers at all, so it didn't make sense for them to be in there. - When `DoFnTester` does support timers, it is trivial to make its own priority queues, and it will also want to offer greater insight via a richer `TestingTimerInternals` sort of implementation, which needs not implement the same interface, etc. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam InMemoryTimerInternals Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1643.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1643 commit 5d0bf9895654c12e50410a98347eb9177de00b1d Author: Kenneth Knowles <k...@google.com> Date: 2016-12-16T04:45:56Z Move InMemoryTimerInternals to runners-core --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1642: Delete deprecated TimerCallback
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1642 Delete deprecated TimerCallback Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: @tweise I missed doing this completely, because I was focused on just a couple annoying methods. But now there really are no `@Deprecated` and `TimerCallback` is deleted. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam TimerCallback Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1642.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1642 commit 33a977a7a157f4f8dd3d2abc8facdbaa6a723143 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-16T04:26:40Z Delete deprecated TimerCallback --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1641: Remove deprecated AggregatorFactory from ...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1641 Remove deprecated AggregatorFactory from SDK Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: @tgroh easy one Analogous Dataflow worker changes are en route, not requiring a dance, but I thought I'd just open this since they won't affect the code review. This is a trivial port from one copy to another of this interface. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam AggregatorFactory Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1641.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1641 commit baa3e4997e061eed69d33fd1ffbfd2af832bf0f6 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-16T04:13:25Z Remove deprecated AggregatorFactory from SDK --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1640: Show timestamps on log lines in Jenkins
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1640 Show timestamps on log lines in Jenkins Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: @dhalperi This adds timestamp prefixes to Maven log messages. Unsure if we can just toggle this at the Jenkins level, which would be superior. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam jenkins-timestamps Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1640.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1640 commit e4179994549f6bcdd93f513ea21bac4551d07019 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-16T03:51:22Z Show timestamps on log lines in Jenkins --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1635: Remove deprecated methods of InMemoryTime...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1635 Remove deprecated methods of InMemoryTimerInternals Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: @tweise just choosing reviewer at random CC: @scwhittle for the deprecation which I think is now cleaned up (no JIRA because it is just cleanup) You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam remove-deprecated-TimerInternals Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1635.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1635 commit e26f4075af6f2c990e23dc9f8fc8be2233652a9f Author: Kenneth Knowles <k...@google.com> Date: 2016-12-16T00:02:23Z Remove deprecated methods of InMemoryTimerInternals --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1630: Cherry pick #1620 to release-0.4.0-incuba...
Github user kennknowles closed the pull request at: https://github.com/apache/incubator-beam/pull/1630 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1630: Cherry pick #1620 to release-0.4.0-incuba...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1630 Cherry pick #1620 to release-0.4.0-incubating Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- The tests on #1620 have not come back yet, so please wait to merge until that is confirmed and merged on `master`. Just starting this PR to have something to reference. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam cherry-pick-CombineFnWithSideInputs Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1630.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1630 commit 7294f227b7b9ff660bcf8dbbb196662820438e36 Author: Eugene Kirpichov <kirpic...@google.com> Date: 2016-12-14T22:29:30Z [BEAM-1154] Get side input from proper window in ReduceFn --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1629: Set Dataflow worker version to 0.4.0-incu...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1629 Set Dataflow worker version to 0.4.0-incubating Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam bump-0.4.0-incubating-worker Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1629.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1629 commit 21507b8da0bb74cc8d844ede985648ac4508df2b Author: Kenneth Knowles <k...@google.com> Date: 2016-12-15T18:17:49Z Set Dataflow worker version to 0.4.0-incubating --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1628: Bump 0.4.0 incubating worker
Github user kennknowles closed the pull request at: https://github.com/apache/incubator-beam/pull/1628 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1628: Bump 0.4.0 incubating worker
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1628 Bump 0.4.0 incubating worker Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam bump-0.4.0-incubating-worker Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1628.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1628 commit 2c1d4eb4d878d1a8b8adf68b10eb8744cf524b5c Author: Davor Bonaci <da...@google.com> Date: 2016-12-13T21:48:26Z Update Maven Archetype versions to pin them to 0.4.0-incubating to be released commit 9f3b0635971a45f8baf8fa68ba5ec2f8c47cec6b Author: Davor Bonaci <da...@google.com> Date: 2016-12-13T22:18:32Z Update Maven Archetype version; fixup for Java8. commit 8d89bfce251d27c7fc6ff522a2cfeb676ec2c633 Author: Eugene Kirpichov <kirpic...@google.com> Date: 2016-12-13T22:35:33Z [BEAM-1149] Explode windows when fn uses side inputs commit b2780881a6d735420a2d29c3270b781ea8a3b74f Author: Kenneth Knowles <k...@google.com> Date: 2016-12-13T23:47:41Z This closes #1602 commit f78d96069b428356609f219357cbf0702ec56c26 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-14T00:09:57Z Re-exclude UsesStatefulParDo tests for Dataflow commit c1af44fa27633fd2a9592a13579415f6b974cfe6 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-14T00:36:42Z Revert "Allow stateful DoFn in DataflowRunner" This reverts commit 42bb15d2df28b99b6788010450f41f2932095771. The Dataflow service has introduced a bug that was masked by various test disabling. commit 10bb4767a1f989a1a75778828c07d9c72c450495 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-14T01:08:42Z This closes #1607 commit 122f05131c803ecaa082dfbc9ad6876b0ca467b8 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-14T19:26:27Z SimpleDoFnRunner observes window if SideInputReader is nonempty commit d9f24b86c644ea85fd197eaab4c2d16b20a70d5f Author: Kenneth Knowles <k...@google.com> Date: 2016-12-14T21:12:43Z Fix NPE in StatefulParDoEvaluatorFactoryTest mocking commit ea0d7f5c26fd97e8528d0614cdce8215ea05557c Author: Kenneth Knowles <k...@google.com> Date: 2016-12-14T22:33:04Z This closes #1619 commit af3989b086ed2e1bf0a08358ce6b7620c334cf49 Author: Davor Bonaci <da...@google.com> Date: 2016-12-15T02:58:57Z [maven-release-plugin] prepare release v0.4.0-incubating-RC1 commit 7e234aa7938606d1a01064a6fdccf01bb90d4f2d Author: Davor Bonaci <da...@google.com> Date: 2016-12-15T02:59:29Z [maven-release-plugin] rollback changes from release preparation of v0.4.0-incubating-RC1 commit 21507b8da0bb74cc8d844ede985648ac4508df2b Author: Kenneth Knowles <k...@google.com> Date: 2016-12-15T18:17:49Z Set Dataflow worker version to 0.4.0-incubating --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1626: Fix typo in mvn command args in Apex post...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1626 Fix typo in mvn command args in Apex postcommit Jenkins job Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: anyone Tested that this actually runs so kicking the seed job should fix up the postcommit for Apex. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam apex-mvn Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1626.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1626 commit 8c00f63ca264e73c7ab7cfc760ef7748edbb6f60 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-15T18:20:03Z Fix mvn command args in Apex postcommit Jenkins job --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1619: [BEAM-1149] Cherry pick some fixes to rel...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1619 [BEAM-1149] Cherry pick some fixes to release-0.4.0 Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- This fixes the remaining failures in [BEAM-1149](). I have confirmed `mvn clean verify` and the Spark runner's local runnable-on-service tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam cherry-picks Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1619.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1619 commit 122f05131c803ecaa082dfbc9ad6876b0ca467b8 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-14T19:26:27Z SimpleDoFnRunner observes window if SideInputReader is nonempty commit d9f24b86c644ea85fd197eaab4c2d16b20a70d5f Author: Kenneth Knowles <k...@google.com> Date: 2016-12-14T21:12:43Z Fix NPE in StatefulParDoEvaluatorFactoryTest mocking --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1618: [BEAM-1149] Fix NPE in StatefulParDoEvalu...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1618 [BEAM-1149] Fix NPE in StatefulParDoEvaluatorFactoryTest mocking Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: @tgroh OR @amitsela Overlooked in hasty merge of DoFnRunner fix. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam StatefulParDoEvaluatorFactory Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1618.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1618 commit 00b961df2e6086ea9f5cb8b9b8fb747739d33670 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-14T21:12:43Z Fix NPE in StatefulParDoEvaluatorFactoryTest mocking --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1616: [BEAM-1149] SimpleDoFnRunner observes win...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1616 [BEAM-1149] SimpleDoFnRunner observes window if SideInputReader is nonempty Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: @amitsela This is a quick fix. There is some redundancy with `PushbackSideInputDoFnRunner`. Confirmed that the following run - which is only run on postcommit - fails on `master` and succeeds with this PR: ``` mvn --batch-mode --errors verify -pl runners/spark \ -Prunnable-on-service-tests -Plocal-runnable-on-service-tests \ -D test=ParDoTest \ -D mdep.analyze.skip=true \ -D failIfNoTests=false \ -D forkCount=0 \ -D runnableOnServicePipelineOptions='[ "--runner=TestSparkRunner", "--streaming=false", "--enableSparkMetricSinks=false" ]' ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam observesWindow Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1616.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1616 commit 9fac4ac1abed954136bb4ed5b6e9c1471c2d3c3c Author: Kenneth Knowles <k...@google.com> Date: 2016-12-14T19:26:27Z SimpleDoFnRunner observes window if SideInputReader is nonempty --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1612: [BEAM-27] Support timer setting and recei...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1612 [BEAM-27] Support timer setting and receiving in SimpleDoFnRunner Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- These are a couple commits peeled off that should enable each runner to begin fulfilling the timer API. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam SimpleDoFnRunner-timers Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1612.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1612 commit 95f954de00d20b27e127e41a0118a305e2cd8a94 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-08T04:09:06Z Make TimerSpec and StateSpec fields accessible commit 19ccebe6170b24af0067c7a04483d8e9156a0ef5 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-23T22:21:40Z Add timer support to DoFnRunner(s) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1607: Cherry pick disabling of state & state te...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1607 Cherry pick disabling of state & state tests for Dataflow runner Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam cherry-pick-state Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1607.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1607 commit f78d96069b428356609f219357cbf0702ec56c26 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-14T00:09:57Z Re-exclude UsesStatefulParDo tests for Dataflow commit c1af44fa27633fd2a9592a13579415f6b974cfe6 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-14T00:36:42Z Revert "Allow stateful DoFn in DataflowRunner" This reverts commit 42bb15d2df28b99b6788010450f41f2932095771. The Dataflow service has introduced a bug that was masked by various test disabling. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1606: Revert "Allow stateful DoFn in DataflowRu...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1606 Revert "Allow stateful DoFn in DataflowRunner" Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- This reverts commit 42bb15d2df28b99b6788010450f41f2932095771. The Dataflow service has introduced a bug that was masked by various test disabling. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam no-state-in-Dataflow Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1606.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1606 commit df97fe4836a669b11ddeb37ef467bbfee1d803ea Author: Kenneth Knowles <k...@google.com> Date: 2016-12-14T00:36:42Z Revert "Allow stateful DoFn in DataflowRunner" This reverts commit 42bb15d2df28b99b6788010450f41f2932095771. The Dataflow service has introduced a bug that was masked by various test disabling. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1605: Re-exclude UsesStatefulParDo tests for Da...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1605 Re-exclude UsesStatefulParDo tests for Dataflow Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: @dhalperi You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam sickbay-state Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1605.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1605 commit 82249abe6f4e3f37ac46f2eef07208b7ac97756a Author: Kenneth Knowles <k...@google.com> Date: 2016-12-14T00:09:57Z Re-exclude UsesStatefulParDo tests for Dataflow --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1603: Revert "This closes #1541": Move PerKeyCo...
Github user kennknowles closed the pull request at: https://github.com/apache/incubator-beam/pull/1603 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1602: [BEAM-1149] Cherry pick side inputs fix t...
Github user kennknowles closed the pull request at: https://github.com/apache/incubator-beam/pull/1602 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1598: [BEAM-967] Add Jenkins postcommit for Run...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1598 [BEAM-967] Add Jenkins postcommit for RunnableOnService in Apex runner Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: @davorbonaci @tweise @jasonkuster You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam apex-postcommit Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1598.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1598 commit 64ac5245bbfa1f8ab8f9f2418cb4bc0b2f0b4c59 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-13T19:12:11Z Add Jenkins postcommit for RunnableOnService in Apex runner --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1597: [BEAM-1139] Revert Apex+Kryo PR and fix v...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1597 [BEAM-1139] Revert Apex+Kryo PR and fix via deps Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- Reverts #1586 and moves forwards with a minor dependency hack. This may break something in the Spark runner's integration tests, since the conflict is real and this simply overrides and hopes for the best. So do not bother to review until Jenkins confirms integration test success. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam kryo Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1597.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1597 commit 49a52cdfed404407ada7fb778413cf15b4790197 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-13T17:48:25Z Revert 91cc606 "This closes #1586": Kryo + UBRFBS This commits merged the following commits: 47cc2dc Add Tests for Kryo Serialization of URFBS 3c2e550 Add no-arg constructor for UnboundedReadFromBoundedSource Kryo support is, in fact, not needed for UnboundedReadFromBoundedSource, but we have a dependency issue where the Apex runner's dependency on Kryo was being overridden with an older version that does not support the @Bind annotations. All runners and poms are independently correct except the examples pom which merges the dependencies from all. commit 3c862d931ad2b2c537e36f3098ed230605ac6114 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-13T17:59:07Z Alphabetize and comment runners examples/java integration tests commit 204d6d2bd1aa59d2b6ee22c53427966e9cd69bfa Author: Kenneth Knowles <k...@google.com> Date: 2016-12-13T18:34:47Z Move Apex Kryo version into global property commit ab2d5c0fda7b87286f8e53840a40d6d2421feb33 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-13T18:37:06Z Fix version of Kryo in examples/java apex-runner profile --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1592: [BEAM-362] Move ActiveWindowSet and imple...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1592 [BEAM-362] Move ActiveWindowSet and implementations to runners-core Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam ActiveWindowSet Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1592.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1592 commit 9a71e6e1558721dc5f80868eeb599ffedd025773 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-13T04:18:02Z Move ActiveWindowSet and implementations to runners-core --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1585: [BEAM-1139] Add no-arg constructor for Un...
Github user kennknowles closed the pull request at: https://github.com/apache/incubator-beam/pull/1585 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1585: [BEAM-1139] Add no-arg constructor for Un...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1585 [BEAM-1139] Add no-arg constructor for UnboundedReadFromBoundedSource Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- This allows Kryo to work with the type, currently required by the Apex runner. This seems to be failing in precommit reliably. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam build-break Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1585.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1585 commit 4124f9b5cd44c78b6e6a892ed5ecd6b13bc1a23a Author: Kenneth Knowles <k...@google.com> Date: 2016-12-12T22:59:36Z Add no-arg constructor for UnboundedReadFromBoundedSource This allows Kryo to work with the type, currently required by the Apex runner. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1556: [BEAM-23] Refactors to support key-preser...
Github user kennknowles closed the pull request at: https://github.com/apache/incubator-beam/pull/1556 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1581: [BEAM-1117] Port Direct runner stateful P...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1581 [BEAM-1117] Port Direct runner stateful ParDo to take KeyedWorkItem Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- This is based upon the required PR #1556 but adds the motivating use of key-preservation logic. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam DirectRunner-Stateful-KeyedWorkItem Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1581.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1581 commit 746a78f98615a51355a901c50c5fd4db7c504a79 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-08T19:44:48Z Propagate key through ParDo if DoFn is key-preserving commit 9ae20e5b8ff0b1365a324062bce2e74799fbb68e Author: Kenneth Knowles <k...@google.com> Date: 2016-12-08T19:49:15Z Add some key-preserving to KeyedPValueTrackingVisitor commit d4129c6405fdc17ace4defc027cc5dafd61f232c Author: Kenneth Knowles <k...@google.com> Date: 2016-12-08T21:28:44Z Move responsibility for knowing about keyedness into EvaluationContext This will allow transform evaluators to inquire about whether various collections are keyed. commit d590f7436ef09520a531fb1561cbdb1f1cad94af Author: Kenneth Knowles <k...@google.com> Date: 2016-12-08T19:29:21Z Port direct runner StatefulParDo to KeyedWorkItem --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam-site pull request #104: [BEAM-438] Rename PTransform apply to...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam-site/pull/104 [BEAM-438] Rename PTransform apply to expand throughout The term does not appear to have been used in the running text, so replacement in the code of the gaming examples is the only change outside of the Javadoc. This PR should perhaps wait until `0.4.0-incubating` is released, depending on what we point users to these days. R: @davorbonaci You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam-site PTransform-expand Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam-site/pull/104.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #104 commit b68b6805960b8c852a6551256827be33054598b9 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-09T22:29:44Z Rename PTransform apply to expand throughout --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1561: [BEAM-498] Remove misc uses of OldDoFn
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1561 [BEAM-498] Remove misc uses of OldDoFn Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: @amitsela Just wiping these out in my idle moments You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam wipe-OldDoFn Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1561.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1561 commit de8ed03621586314c09e39015e074e6a4a7894cc Author: Kenneth Knowles <k...@google.com> Date: 2016-12-09T07:33:40Z Remove misc uses of OldDoFn --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1555: [BEAM-27] Fix exclusion of UsesTimersInPa...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1555 [BEAM-27] Fix exclusion of UsesTimersInParDo for FlinkRunner streaming tests Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: anyone (fixing postcommit) ping @aljoscha @mxm You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam FlinkRunner Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1555.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1555 commit a11eb637f6f1c7bdc41b24154cb3893bb194b096 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-08T20:42:03Z Fix exclusion of UsesTimersInParDo for FlinkRunner streaming tests --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1539: [BEAM-498] Remove misc occurrences of Old...
Github user kennknowles closed the pull request at: https://github.com/apache/incubator-beam/pull/1539 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1550: [BEAM-27] Reject timers for ParDo in each...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1550 [BEAM-27] Reject timers for ParDo in each runner separately and exclude timer tests Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- There are many trivial commits spread across the runners, in two phases. 1. Add a JUnit category for timers and exclude it from all runners. I've included one trivial test just to see that it is working. (FWIW I have hacked the direct runner enough in later commits to get this test to pass "for the wrong reasons") 2. Remove the timer rejection code from `ParDo` and add it to every runner. R: @tgroh @aljoscha @amitsela @tweise I'll wait for all four to LGTM. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam UsesTimers Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1550.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1550 commit 5c99d2c3a573c3963fd6efa9e11ed860d26f0365 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-07T04:49:15Z Add JUnit category UsesTimersInParDo With this, various runners can disable tests for this capability until they support it. commit 604695f1b7cbd6dfeda94d82532fde535b3e3448 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-07T04:52:24Z Disable tests for timers in ParDo for Apex runner commit 58cac2f4960044e646249eb8a3db27b345e1 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-07T04:52:49Z Disable tests for timers in ParDo for Flink runner commit f27bdf9b3c2e609c54615db60f0682b34f6575c5 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-07T04:53:05Z Disables tests for timers in ParDo for Spark runner commit be9c04bf0f2af7b428a61d3b74b68ea18c4f0e03 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-07T04:53:16Z Disable tests for timers in ParDo for Dataflow runner commit b66c5cb4f92e2c0d834f44ba246b8186cc9d0291 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-08T04:24:34Z Disable tests for timers in ParDo for direct runner commit e4f062495878328359234da4e586d12cb42965f9 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-07T04:49:40Z Add basic test for timers in ParDoTest commit ad4cbbc18ffbbedaaa17415220d7ff2908931797 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-08T04:04:51Z No longer reject timers in ParDo commit 0703093767db2776965384b09b322f01f4b79b39 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-08T04:34:34Z Reject timers for ParDo in ApexRunner commit 4b312e8d01e324db77e5f2de62eae35efc153adb Author: Kenneth Knowles <k...@google.com> Date: 2016-12-08T04:34:59Z Reject timers for ParDo in FlinkRunner commit 2e2bc885deb5000a269800bdbead8f17dff5a3a9 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-08T04:35:08Z Reject timers for ParDo in SparkRunner commit 2d2d83f721c21f681f431974edc4e044d44828c5 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-08T04:37:33Z Reject timers for ParDo in DirectRunner --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1543: [BEAM-25] Move CopyOnAccessStateInternals...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1543 [BEAM-25] Move CopyOnAccessStateInternals to runners/direct Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: @tgroh this is only actually used by the direct runner. Not necessarily the greatest JIRA for this, but I'm not sure of a blanket for cleaning out the SDK's excessive surface area, so I went with the state ticket. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam CopyOnAccessStateInternals Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1543.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1543 commit 019612ba5e5a656e848d458617007d39be42b3e9 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-07T22:28:39Z Move CopyOnAccessStateInternals to runners/direct --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1541: [BEAM-498] Move PerKeyCombineFnRunner to ...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1541 [BEAM-498] Move PerKeyCombineFnRunner to runners-core Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: @peihe seems there is no reason this needs to stay, right? `PerKeyCombineFnRunners` (the static util class) is already moved. R: @lukecwik this has some references to `OldDoFn` so at least this gets them out of the SDK. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam PerKeyCombineFnRunner Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1541.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1541 commit 351f58567212e7a9d4664ca65a0bd4a10a77ed81 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-07T22:22:21Z Move PerKeyCombineFnRunner to runners-core --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1539: [BEAM-498] Remove misc occurrences of Old...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1539 [BEAM-498] Remove misc occurrences of OldDoFn Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: @lukecwik here's a few more You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam remove-OldDoFn Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1539.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1539 commit 4c178c4dbfbdbcb5e5e3100a14807be7dfda62be Author: Kenneth Knowles <k...@google.com> Date: 2016-12-07T22:17:01Z Remove misc occurrences of OldDoFn --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1538: [BEAM-438] Rename PTransform.apply to PTr...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1538 [BEAM-438] Rename PTransform.apply to PTransform.expand Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- Opening this to have a code pointer in dev list discussion. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam PTransform-expand Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1538.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1538 commit ff50e4d470d954c17d73358aa3e0c4c8b4123b87 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-07T21:33:04Z Rename PTransform.apply to PTransform.expand --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1536: [BEAM-664] Revise WindowedWordCount examp...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1536 [BEAM-664] Revise WindowedWordCount example to be more independent of runner and execution style Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- This removes the use of BigQuery from the WindowedWordCount example, replacing it with a somewhat hacky file-based write of the output, using the window as the idempotency key. In order to port the test and to benefit from recent improvements in `FileBasedCheckSumMatcher`, I've factored out the resiliency code from that into an internal-only minimal `ShardedFile` class with just enough API surface to write these tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam WindowedWordCount Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1536.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1536 commit bac7b192c9fefdf536e324f1fde73bac2cd903fa Author: Kenneth Knowles <k...@google.com> Date: 2016-11-04T03:44:45Z Add IntervalWindow coder to the standard registry commit aa09449379dfaedbdfb0b73bae85ffd334b838b1 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-03T21:37:26Z Revise WindowedWordCount for runner and execution mode portability commit 357732efb866e4a24d76a9aaafa10e6bc964fe7e Author: Kenneth Knowles <k...@google.com> Date: 2016-11-08T06:06:00Z Check the onSuccessMatcher in DirectRunner if isBlockOnRun is set commit 76d19716829cb98c4f0d4b34244fde8866b6d6a9 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-05T22:32:12Z Factor out ShardedFile from FileChecksumMatcher --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1529: [BEAM-498] Port ParDoTest from OldDoFn to...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1529 [BEAM-498] Port ParDoTest from OldDoFn to new DoFn Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: @aljoscha seems about to time to make this leap? You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam ParDoTest Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1529.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1529 commit 794b54ec6b7284ffabd0a117263a9cf0143c0b72 Author: Kenneth Knowles <k...@google.com> Date: 2016-08-05T22:21:23Z Add OldDoFn -> DoFn adapter and expose raw DoFn in ParDo Previously, ParDo.of(DoFn) would immediately wrap it into an OldDoFn so the new DoFn could not be recovered, since After this change, the new DoFn is temporarily available via #getNewFn(), which will eventually overwrite #getFn(). When ParDo is instantiated via ParDo.of(OldDoFn), a wrapper in the reverse direction will create a new DoFn that can be accessed via the same method. So a ParDo transform temporarily always carries both. commit 06e4df97210b44081aba5288beb25446a8802191 Author: Kenneth Knowles <k...@google.com> Date: 2016-08-08T20:27:00Z Deprecate more uses of OldDoFn --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1528: [BEAM-27] Add DoFn.OnTimerContext and sup...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1528 [BEAM-27] Add DoFn.OnTimerContext and support as a DoFn parameter Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam OnTimerContext Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1528.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1528 commit 44a7b915d502c318ffabaa6fb808207bb3ea15e8 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-07T04:10:06Z Add DoFn.OnTimerContext commit 1934b704411fed76a58cbc657c7dc8be666c3885 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-07T04:01:18Z Add support for OnTimerContext parameter in DoFnSignature commit 14d9a3794c4f8c68626075343a1dec1d4c017686 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-07T04:10:21Z Access to OnTimerContext via DoFnInvokers.ArgumentProvider --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1523: [BEAM-1038] Allow stateful DoFn in Datafl...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1523 [BEAM-1038] Allow stateful DoFn in DataflowRunner Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- Confirmed that the post commit should succeed, via the following command: ``` mvn verify \ --batch-mode \ --errors \ --projects runners/google-cloud-dataflow-java \ -DforkCount=0 \ -DfailIfNoTests=false \ -Dtest=org.apache.beam.sdk.transforms.ParDoTest \ -DrunnableOnServicePipelineOptions='[ "--runner=org.apache.beam.runners.dataflow.testing.TestDataflowRunner", "--project=...", "--tempRoot=..." ]' ``` with output ending in: ``` Running org.apache.beam.sdk.transforms.ParDoTest Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 273.736 sec - in org.apache.beam.sdk.transforms.ParDoTest ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam DataflowRunner-state Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1523.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1523 commit 9e4d40731ff53b2cd0f2a57d18c6f560db441483 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-06T21:51:19Z Allow stateful DoFn in DataflowRunner --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1488: Fix pom syntax for excludedGroups for Spl...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1488 Fix pom syntax for excludedGroups for SplittableParDo Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: anyone. Ping committers in other time zones, @aljoscha @mxm @amitsela CC: @jkff Confirmed with `-P local-runnable-on-service-tests` for those runners that support it. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam SDF Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1488.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1488 commit ffa81edd0ec4d9a8150280efdb6a6de412114743 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-02T05:03:04Z Fix pom syntax for excludedGroups for SplittableParDo --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1478: [BEAM-362] Moved KeyedWorkItem and relate...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1478 [BEAM-362] Moved KeyedWorkItem and related classes to runners-core Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: @aljoscha would you take a look? It is just a move of a utility class that I think has no place in the user-facing SDK. After an initial "LGTM" I will need to do a little dance with the Dataflow runner for integration tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam KeyedWorkItem Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1478.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1478 commit fb0768a943e2586532965d5593ccf8cf6ae11781 Author: Kenneth Knowles <k...@google.com> Date: 2016-12-01T04:46:04Z Moved KeyedWorkItem and related classes to runners-core --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1477: [BEAM-27] Add timerId to TimerData
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1477 [BEAM-27] Add timerId to TimerData Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- This `timerId` is generated to be identical to historical behavior, and to be unique per time domain and timestamp. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam timerId Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1477.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1477 commit d53275d3075d1491d8718837b00093be09f93354 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-23T22:30:57Z Add timerId to TimerData This timerId is generated to be identical to historical behavior, and to be unique per time domain and timestamp. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1474: [BEAM-986] Revert "Improvements to Reduce...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1474 [BEAM-986] Revert "Improvements to ReduceFnRunner prefetching" Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- This reverts commit 4282c67c5fa4dea2fe6c8695e0ea23f383c6457b, which contained some incompatibilities outside of runners-core. The PR is #1366, for reference. R: anyone, pinging @tgroh or @dhalperi CC: @scwhittle You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam rollback Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1474.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1474 commit aaa3b91e1e7b39dd585314a6017235cdd127e923 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-30T23:21:53Z Revert "Improvements to ReduceFnRunner prefetching" This reverts commit 4282c67c5fa4dea2fe6c8695e0ea23f383c6457b, which contained some incompatibilities outside of runners-core. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1462: [BEAM-1062] Shade the SDK more fully; eve...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1462 [BEAM-1062] Shade the SDK more fully; everything but a whitelist Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- This is a re-opening of a past change. Previously, issues with snappy's use of reflection blocked progress. Things have changed a lot since then so let's put a little more effort into this. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam shade-SDK Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1462.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1462 commit 78dd9e3583349d232c2514a0380fa59b0bbd2791 Author: Kenneth Knowles <k...@google.com> Date: 2016-07-15T20:47:41Z Upgrade snappy version and move to dependencyManagement commit 05214785fa254ee434e8fdb0ae728c8505cb1cdd Author: Kenneth Knowles <k...@google.com> Date: 2016-11-30T04:06:54Z Shade more in the SDK; everything except a whitelist --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1460: Revert "Remove WindowedValue.valueInEmpty...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1460 Revert "Remove WindowedValue.valueInEmptyWindows" Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- This reverts commit 0e49b150e83d85ae432c640da937a9497068e71b, which breaks some DataflowRunner integration tests in ways that may take a minute to fix. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam rollback Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1460.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1460 commit 54435de54eb7d4bff8db2ae944bea1f8e813ecbd Author: Kenneth Knowles <k...@google.com> Date: 2016-11-30T00:57:09Z Revert "Remove WindowedValue.valueInEmptyWindows" This reverts commit 0e49b150e83d85ae432c640da937a9497068e71b, which breaks some DataflowRunner integration tests. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1458: Restore a private ValueInEmptyWindows to ...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1458 Restore a private ValueInEmptyWindows to DataflowRunner Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- This fixes the postcommit break at https://builds.apache.org/job/beam_PostCommit_RunnableOnService_GoogleCloudDataflow/1681/consoleFull caused by https://github.com/apache/incubator-beam/pull/1440 R: @dhalperi You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam ISM Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1458.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1458 commit 1d2e6579c55408e1ecb5e3173e3387ef19d6ff30 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-29T23:23:37Z Restore a private ValueInEmptyWindows to DataflowRunner for special hacks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1440: [BEAM-230] Remove WindowedValue.valueInEm...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1440 [BEAM-230] Remove WindowedValue.valueInEmptyWindows Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- A value in empty windows expands to no values, so it can be dropped at any time, perhaps unintentionally. This has affected real runners. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam valueInEmptyWindows Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1440.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1440 commit 05fb97b7133d79f343d0d6d6b26a3a56f8ee848b Author: Kenneth Knowles <k...@google.com> Date: 2016-05-10T18:39:35Z Remove WindowedValue.valueInEmptyWindows A value in empty windows expands to no values, so it can be dropped at any time, perhaps unintentionally. This has bitten runner authors, including Spark & Dataflow. While creating such a thing in memory is not automatically problematic, it is also not really useful. So this change removes it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1430: [BEAM-498] Remove an OldDoFn and obsolete...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1430 [BEAM-498] Remove an OldDoFn and obsolete WindowingInternals#writePCollectionViewData Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam writePCollectionViewData Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1430.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1430 commit d13dd6d4ee4354df876a315097328354320ebba8 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-23T19:22:08Z Remove unused body of StreamingPCollectionViewWriterFn commit 5144de18bc10c902053cf0cbac41c8cfa8e182a4 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-23T19:23:07Z Remove unused WindowingInternals.writePCollectionViewData --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1424: Add input and output types to TransformRe...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1424 Add input and output types to TransformResult Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- This would likely have caught some hard-to-diagnose (for me, new to the runner) type safety errors during the development of `StatefulParDoEvaluatorFactory`, so adding it should hopefully catch similar bugs in the future. As we discussed `InputT` is more useful and reasonable than `OutputT` but there are a number of places where they each line up and might add a little something. It adds some boilerplate, to be sure, and is still mostly type unsafe since there are lots of methods with generics only on the return type. Opening this for consideration / discussion. R: @tgroh You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam typed-TransformResult Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1424.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1424 commit 4e8a1157a88d8b7ecd64e75c46b0cd6fbbba03ee Author: Kenneth Knowles <k...@google.com> Date: 2016-11-23T00:01:45Z Add input and output types to TransformResult This would likely have caught some hard-to-diagnose type safety errors during the development of StatefulParDoEvaluatorFactory, so adding it should hopefully catch similar bugs in the future. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1410: [BEAM-25] Temporarily reject stateful Par...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1410 [BEAM-25] Temporarily reject stateful ParDo in ApexRunner (until support is added) Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: @tweise At present, the user-facing API for stateful `DoFn` is in the codebase but prevented from use. It is `ParDo.of(...)` that rejects stateful `DoFn`. In #1399 I am removing this rejection, so I need to add it to all runners until they support the API. I have also added a JUnit category so runners can exclude this from their `RunnableOnService` test suites. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam ApexRunner-state Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1410.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1410 commit e85cea78253d2f316a18d95d65aabc1176448841 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-16T05:33:01Z Reject stateful DoFn in ApexRunner commit f8b6bb7f1ab8720ca4f2d766831d8f243dd27085 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-21T23:41:13Z Add JUnit category for stateful ParDo tests --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1411: [BEAM-25] Temporarily reject stateful Par...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1411 [BEAM-25] Temporarily reject stateful ParDo in ApexRunner (until support is added) Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: @bjchambers At present, the user-facing API for stateful `DoFn` is in the codebase but prevented from use. It is `ParDo.of(...)` that rejects stateful `DoFn`. In #1399 I am removing this rejection, so I need to add it to all runners until they support the API. I have also added a JUnit category so runners can exclude this from their `RunnableOnService` test suites. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam DataflowRunner-state Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1411.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1411 commit c9640ea08e8078752a51508430c3661555c4c8a7 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-16T05:35:03Z Reject stateful DoFn in DataflowRunner commit 7b7d2dd30ee7bf56fddaa221386f1046875176b1 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-21T23:41:13Z Add JUnit category for stateful ParDo tests --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1409: [BEAM-25] Temporarily reject stateful Par...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1409 [BEAM-25] Temporarily reject stateful ParDo in SparkRunner (until support is added) Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: @amitsela At present, the user-facing API for stateful `DoFn` is in the codebase but prevented from use. It is `ParDo.of(...)` that rejects stateful `DoFn`. In #1399 I am removing this rejection, so I need to add it to all runners until they support the API. I have also added a JUnit category so runners can exclude this from their `RunnableOnService` test suites. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam SparkRunner-state Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1409.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1409 commit 8d715689dd5283b7b180c0b9ec4e188abba140f5 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-21T23:41:13Z Add JUnit category for stateful ParDo tests commit b0d07d74f7805ee1d30fdedf54c089790d63d898 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-16T05:33:13Z Reject stateful DoFn in SparkRunner --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1399: [BEAM-25] Direct runner state
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1399 [BEAM-25] Direct runner state Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam DirectRunner-state Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1399.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1399 commit 9d35e9f715f2c76ae54f3b304e41d1dbdf8c0f92 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-16T05:33:01Z Reject stateful DoFn in ApexRunner commit aab81549a70d1ac3f6580f66759566f673c243b8 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-16T05:33:13Z Reject stateful DoFn in SparkRunner commit 0c2aa666ecaf7def9fa28c6dc81b90c3ad0dc873 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-16T05:33:28Z Reject stateful DoFn in FlinkRunner commit ec12dd3102d0a76e5b4cf7ad0a503b746e165d01 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-16T05:35:03Z Reject stateful DoFn in DataflowRunner commit a66e71fb92734f6d423ce9ee012812322992ba40 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-10T05:10:51Z Add State parameter support to SimpleDoFnRunner commit 8a058b14ab4084ee122d396f96ef2b76d86d3368 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-21T19:34:45Z Remove overspecified type in ParDoEvaluator commit 677155a6148ea383065c1792db49e283e466cef1 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-10T05:59:15Z Add support for Stateful ParDo in the Direct runner This adds overrides and new evaluators to ensure that state is accessed in a single-threaded manner per key and is cleaned up when a window expires. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1389: Restore StateContexts.windowOnly for temp...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1389 Restore StateContexts.windowOnly for temporary compatibility Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- This addresses the postcommit failure here: https://builds.apache.org/job/beam_PostCommit_RunnableOnService_GoogleCloudDataflow/1600/org.apache.beam$beam-runners-google-cloud-dataflow-java/testReport/junit/org.apache.beam.sdk.transforms/CombineTest/testSessionsCombineWithContext/ R: @davorbonaci @jkff You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam StateContexts Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1389.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1389 commit bf469d0f9e668b87fe276c9c9bc6b230dd931a6e Author: Kenneth Knowles <k...@google.com> Date: 2016-11-18T22:40:59Z Restore StateContexts.windowOnly for temporary compatibility --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1386: [BEAM-498] Move DoFn.ArgumentProvider to ...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1386 [BEAM-498] Move DoFn.ArgumentProvider to DoFnInvokers.ArgumentProvider Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- The arguments provided as a single object are an aspect of the DoFnInvoker, not the DoFn. The DoFn itself is a specification that may have other ways of being invoked, depending on the circumstance. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam DoFnInvokers-ArgumentProvider Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1386.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1386 commit 045f0edd61392ad761e130906742d309ea976d2f Author: Kenneth Knowles <k...@google.com> Date: 2016-11-18T07:04:55Z Move DoFn.ArgumentProvider to DoFnInvokers.ArgumentProvider The arguments provided as a single object are an aspect of the DoFnInvoker, not the DoFn. The DoFn itself is a specification that may have other ways of being invoked, depending on the circumstance. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1378: [BEAM-23] DirectRunner: rewrite single-ou...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1378 [BEAM-23] DirectRunner: rewrite single-output ParDo to multi-output Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- This is in service of simplifying the override and expansion for splittable and stateful `ParDo`. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam direct-ParDoSingleViaMulti Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1378.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1378 commit 0b7fcc8ede9d16d02d047f40f669606cbe00fc4e Author: Kenneth Knowles <k...@google.com> Date: 2016-11-17T18:14:07Z Separate ParDoTest cases and make them more flexible A number of excessively rigid name tests preclude runner expansion of ParDo. This change makes them into independent unit tests for better signal and makes them more accurate to the intent - the name should have the relevant information, but may have other content. commit efe582f0fbd4d2693867e5d3d3b3db21de16a9fa Author: Kenneth Knowles <k...@google.com> Date: 2016-11-17T18:56:36Z No longer execute OldDoFnTest with DirectRunner commit 60cb48479ee5b0b6f50100bf2f6edec3fa7a552f Author: Kenneth Knowles <k...@google.com> Date: 2016-11-16T23:43:47Z DirectRunner: Expand ParDo.Bound into ParDo.BoundMulti commit abc19d5a4672aa0f1334b674c91196c77bf07641 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-17T19:07:32Z Do not override type descriptor in WithTimestamps commit ef700c967e8589b54053bc6b5e4743f50e43ec87 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-17T19:08:48Z Use getNewFn for coder inferences in ParDo --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1347: [BEAM-25] Allow stateful DoFn in DirectRu...
Github user kennknowles closed the pull request at: https://github.com/apache/incubator-beam/pull/1347 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam-site pull request #87: Remove Apex runner from work in progre...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam-site/pull/87 Remove Apex runner from work in progress R: @davorbonaci You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam-site apex-link Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam-site/pull/87.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #87 commit a7c48d8704563c616b80b82381455e8eeb17ce1b Author: Kenneth Knowles <k...@google.com> Date: 2016-11-16T17:37:23Z Remove Apex runner from work in progress --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1368: [BEAM-498] Transmit new DoFn in Dataflow ...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1368 [BEAM-498] Transmit new DoFn in Dataflow translator Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam DataflowRunner-DoFnInfo Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1368.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1368 commit 4afbb0ddc86886a4402e403db2cdfea8df966c76 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-16T06:27:35Z Transmit new DoFn, not OldDoFn, in Dataflow translator --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1360: [BEAM-498] Rename DoFn.ExtraContextFactor...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1360 [BEAM-498] Rename DoFn.ExtraContextFactory to DoFn.Arguments Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: @bjchambers I think we've discussed this before. Big picture: now we can treat all dynamic methods uniformly, and use `DoFnSignatures` and `ParDo` to do the needed validation. I think since all arguments to methods fall into the "extra" category, we can save some typing and use more conventional vocabulary. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam Arguments Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1360.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1360 commit 4afc16fad9eeaeee1d640822000da6a1062c4056 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-15T06:19:35Z Rename DoFn.ExtraContextFactory to DoFn.Arguments --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1355: [BEAM-498] Treat ProcessContext and Conte...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1355 [BEAM-498] Treat ProcessContext and Context like other DoFn parameters Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- Before this change, `ProcessContext` and `Context` were special-cased, while other parameters were treated generically. After this change, all parameters receive the same dynamic treatment. Thus a user need not request these parameters, the runner can save the effort of constructing them, and we can migrate towards less "all-in-one" parameters. I changed only the analysis and code generation a bit to remove special casing, but did not write new delegations that would actually add dynamic parameter generation to methods that do not already have them, so only `@ProcessElement` and `@OnTimer` are affected. Posted for self-review & signposting. CC: @tgroh who has discussed this and @jkff who may have interest. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam ProcessContext Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1355.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1355 commit 959e906a7591eaa40c3c2451f0c3dcfdffc50c32 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-11T22:55:16Z Treat ProcessContext and Context like other DoFn parameters Before this change, ProcessContext and Context were special-cased, while other parameters were treated generically. After this change, all parameters receive the same dynamic treatment. Thus a user need not request these parameters, the runner can save the effort of constructing them, and we can migrate towards less "all-in-one" parameters. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1347: [BEAM-25] Allow stateful DoFn. Implement ...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1347 [BEAM-25] Allow stateful DoFn. Implement for DirectRunner Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam DirectRunner-state Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1347.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1347 commit e81390b39854f0835d22e9e0cdea6cb297d47643 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-10T05:10:51Z Add State parameter support to SimpleDoFnRunner commit 67bc4b2e0c2e75969b16a471a4aa79bd9209bd9b Author: Kenneth Knowles <k...@google.com> Date: 2016-11-10T05:59:15Z Schedule window expiration state purge in DirectRunner --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1344: [BEAM-194] Add Java 8 examples archetype ...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1344 [BEAM-194] Add Java 8 examples archetype including mobile gaming example, with generator script Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam moar-examples Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1344.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1344 commit f4782aa2da8e5bbbfa70cb80db07419f717458d7 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-10T22:47:37Z Add script to generate the java8-examples archetype commit f783c4790b69b3cc6e220df1beacbcc3a5642b69 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-10T22:47:56Z Create java8-examples archetype module commit ccaae328545f2492a8194c89a58011299d1164e6 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-10T22:48:59Z Generate java8-examples archetype contents --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1338: [BEAM-664] Update examples archetype
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1338 [BEAM-664] Update examples archetype Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- @davorbonaci You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam examples-archetype Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1338.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1338 commit 985df7591946c32875ebf09aef648b84758e0034 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-10T19:46:24Z Add script to update examples archetype automatically commit 87493d48ba3dc80211aba84d3d9105e7ee3e3688 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-10T20:06:42Z Update examples archetype to match examples --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1275: [BEAM-664] Revise WindowedWordCount to be...
Github user kennknowles closed the pull request at: https://github.com/apache/incubator-beam/pull/1275 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1320: [BEAM-947] Fixup breakage in WindowedWord...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1320 [BEAM-947] Fixup breakage in WindowedWordCountIT Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: anyone CC: @jasonkuster @davorbonaci You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam example-it-breakage Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1320.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1320 commit 53e7198c92f6e8c124d3ebc1bda10f78c4c80cce Author: Kenneth Knowles <k...@google.com> Date: 2016-11-09T18:22:51Z Fixup breakage in WindowedWordCountIT --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1318: [BEAM-922] Fix findbugs errors and re-ena...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1318 [BEAM-922] Fix findbugs errors and re-enable for runners-core Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: @dhalperi The error in `UnboundedReadFromBoundedSource` was a real coding error; please review my small edit there. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam runners-core-findbugs Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1318.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1318 commit 2a4f6fffcdcf1804b1c173d6e4ab43ffc0431cb7 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-08T22:49:09Z Fix findbugs errors and re-enable for runners-core --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1315: [BEAM-664] Examples not windowed
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1315 [BEAM-664] Examples not windowed Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- This PR breaks out all the non-`WindowedWordCount` examples from #1275 as their changes are much simpler and can go in immediately. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam examples-not-windowed Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1315.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1315 commit ef6033e34152ef860f2195361b915164a67c0a72 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-03T21:18:43Z Hardcode MinimalWordCount to the DirectRunner This makes it easy to immediately run, and removes various non-portable instructions and others that aren't the easiest for a "Getting Started" scenario. commit 9fc73d069c35d5012b673d3932f3277dae3d1c1e Author: Kenneth Knowles <k...@google.com> Date: 2016-11-03T21:19:47Z Revise WordCount example to be better cross-runner example commit 0996f90316478a205a220bebfb25c11a02da7873 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-03T21:50:02Z Revise DebuggingWordCount to be more portable commit 64326c2e2229a8ae559715cf981ddcb89a98f80a Author: Kenneth Knowles <k...@google.com> Date: 2016-11-03T23:17:18Z Add runner-specific profiles to the examples pom.xml --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1314: [BEAM-664] Add runner-specific profiles t...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1314 [BEAM-664] Add runner-specific profiles to the examples pom.xml Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: @davorbonaci Breaking this commit from #1275 as it adds value immediately and that PR is likely to be delayed fixing the integration testing. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam examples-profiles Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1314.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1314 commit 543835062797f986e1eb2afdbabb38281a580644 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-03T23:17:18Z Add runner-specific profiles to the examples pom.xml --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1307: [BEAM-27] Connect generated DoFnInvoker's...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1307 [BEAM-27] Connect generated DoFnInvoker's invokerOnTimer to generated OnTimerInvoker Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam DoFnInvokers-OnTimerInvokers Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1307.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1307 commit 3e759d3be013fe38cea2601fe4f7cfe95b41d42b Author: Kenneth Knowles <k...@google.com> Date: 2016-11-02T04:23:48Z Connect generated DoFnInvoker.invokerOnTimer to OnTimerInvoker --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1296: Initial fixups for Regex
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1296 Initial fixups for Regex Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: anyone, since build is broken CC: @eljefe6a Jesse, here is just fixups for checkstyle, which has broken the build, and I don't think we need the `Transform` suffix. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam regex Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1296.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1296 commit 6954abe4e28103f5ddd3e1eebe998b765cd9de11 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-07T18:10:32Z Rename RegexTransform to just Regex commit 79b04551c7c9f964908ab4a1d95119ef8a7fff84 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-07T18:10:59Z Format Regex according to style guidelines --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1282: [BEAM-25] DoFnInvoker support for State a...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1282 [BEAM-25] DoFnInvoker support for State and Timer params Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: @jkff The last three commits are new. Perhaps while prerequisites receive further review you could suggest some intresting tests? I have just a couple, which just make sure the parameter gets passed, and rely somewhat on compositionality to ensure that is sufficient. CC: @tgroh blocked on prior #1252 You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam State-Params Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1282.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1282 commit 349cb2c5d3c2d92bb4925a0c6dbc675c89364e26 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-01T04:30:40Z Refactor and reuse parameter analysis in DoFnSignatures commit bce430ca73f056365723c6ee19772e2523e166f0 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-01T21:48:54Z Switch DoFnSignature, etc, from TypeToken to TypeDescriptor commit a7849497b5d435b3ddb8fba1433164af112f8998 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-01T21:50:24Z DoFnSignature: Make TypeDescriptor-returning methods public commit 63f9a3076755d8309481fd13da3f7325f2155b6a Author: Kenneth Knowles <k...@google.com> Date: 2016-11-01T22:38:01Z Add TypeDescriptor#getTypes commit d0724d72584774712b2ea5ff945c65fc84b30d77 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-04T01:42:25Z Generalize extraction of DoFn parameters from context commit d90576d91fae041bd980a218344fc3f88a18293a Author: Kenneth Knowles <k...@google.com> Date: 2016-11-04T02:26:36Z Switch DoFnInvokersTest to use a mock ExtraContextFactory commit 5c504d4a6604458e169612a65f098202f7bb3fc7 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-04T02:18:24Z Add DoFnInvoker dispatch for State and Timer parameters --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1281: [BEAM-25] Generalized extra context facto...
Github user kennknowles closed the pull request at: https://github.com/apache/incubator-beam/pull/1281 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1281: [BEAM-25] Generalized extra context facto...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1281 [BEAM-25] Generalized extra context factory Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: @jkff Only the last commit is new. This makes it so each "extra" parameter gets to return a wholly arbitrary stack manipulation. This will be needed to pass the parameters for state and timers. I am proposing it now because it also cleans up the type casting, and will make downcasting of the window type a trivial act in parallel with other developments. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam Generalized-ExtraContextFactory Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1281.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1281 commit 880a538f4ad43d6399bd873c7b6cb7a786a13b6a Author: Kenneth Knowles <k...@google.com> Date: 2016-11-01T22:38:01Z Add TypeDescriptor#getTypes commit 340d922503c06d3a82e964ef0fdf1e04713e40b9 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-01T21:48:54Z Switch DoFnSignature, etc, from TypeToken to TypeDescriptor commit 3061436c8719b073a3d5176566ea5e6493224e9c Author: Kenneth Knowles <k...@google.com> Date: 2016-11-01T21:50:24Z DoFnSignature: Make TypeDescriptor-returning methods public commit e117ac82cfabcb8b9729e13c98647c799ea7caf3 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-01T04:30:40Z Refactor and reuse parameter analysis in DoFnSignatures commit 8610daa593cd4113ef033391c0d95e77dd513922 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-04T01:42:25Z Generalize extraction of DoFn parameters from context --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1275: [BEAM-664] Revise examples to be more ind...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1275 [BEAM-664] Revise examples to be more independent of runner and execution style Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam examples Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1275.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1275 commit aa6bdd87b7d53a8b6f9b3b2088225c4cbbcf2e76 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-03T21:18:43Z Hardcode MinimalWordCount to the DirectRunner This makes it easy to immediately run, and removes various non-portable instructions and others that aren't the easiest for a "Getting Started" scenario. commit fd0ddcc6d5a629e245a2bd043fbdabdba5f98dc9 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-03T21:19:47Z Revise WordCount example to be better cross-runner example commit 6fad0415aff2fe2dae0d000d55aa5cf32612ff22 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-03T21:37:26Z Revise WindowedWordCount for runner and execution mode portability commit 416d7444ade7bb2b77502e8380517ea5f0ed9b9a Author: Kenneth Knowles <k...@google.com> Date: 2016-11-03T21:50:02Z Revise DebuggingWordCount to be more portable --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1256: On timer invoker
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1256 On timer invoker Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: @jkff This comes after #1252 but is otherwise independent. You can review just the additional commit. I have a few more thoughts here that I would love feedback on. I am trying to avoid major overhauls. I will leave my concerns as a self-review so they are contextualized. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam OnTimerInvoker Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1256.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1256 commit 15a8901ade07c1649378a0b72238eb514877c61c Author: Kenneth Knowles <k...@google.com> Date: 2016-11-01T21:48:54Z Switch DoFnSignature, etc, from TypeToken to TypeDescriptor commit 6de216695c05e4d9b283f0842aed8713075be7aa Author: Kenneth Knowles <k...@google.com> Date: 2016-11-01T21:50:24Z DoFnSignature: Make TypeDescriptor-returning methods public commit 40bbfff3dba803951607ae15d5d2cf99fbe64a32 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-01T22:38:01Z Add TypeDescriptor#getTypes commit f2c2f0736d7904474041ed16f782a1c19ca6264e Author: Kenneth Knowles <k...@google.com> Date: 2016-11-01T04:30:40Z Refactor and reuse parameter analysis in DoFnSignatures commit a0f5a1521c3b0d8d195f63d15dcc1779640edd8b Author: Kenneth Knowles <k...@google.com> Date: 2016-11-02T02:58:35Z fixup! use mutable style for contexts commit aa31ebcfcb51078c39de715c63b01c33f8b0db46 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-01T02:26:38Z Add OnTimerInvoker(s), for invoking DoFn @OnTimer methods OnTimerInvoker encapsulates the dispatch from onTimer() to a call to the DoFn method annotated with @OnTimer(). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1252: [BEAM-27] Factor of parameter analysis in...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1252 [BEAM-27] Factor of parameter analysis in DoFnSignatures; use it for @OnTimer methods Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: @jkff This might be easiest to review as individual commits, though the last one is the goal of this PR. This PR is a prefix of ongoing work to invoke `@OnTimer` methods. I've peeled it off and fixed it up and added a smidge of testing. Since the code is entirely shared, just making sure the right path is taken provides moderate coverage. (a further refactor could make this clearer, by isolating the shared code into a class with its own unit tests, absolving its callers of proving they exercise the code path, but I thought this baby step was better) You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam DoFnSignatures Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1252.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1252 commit 5946ec175d55f9164f9b1b34a06539da3c24f5ae Author: Kenneth Knowles <k...@google.com> Date: 2016-11-01T04:30:40Z Refactor and reuse parameter analysis in DoFnSignatures commit 15a8901ade07c1649378a0b72238eb514877c61c Author: Kenneth Knowles <k...@google.com> Date: 2016-11-01T21:48:54Z Switch DoFnSignature, etc, from TypeToken to TypeDescriptor commit 6de216695c05e4d9b283f0842aed8713075be7aa Author: Kenneth Knowles <k...@google.com> Date: 2016-11-01T21:50:24Z DoFnSignature: Make TypeDescriptor-returning methods public commit 40bbfff3dba803951607ae15d5d2cf99fbe64a32 Author: Kenneth Knowles <k...@google.com> Date: 2016-11-01T22:38:01Z Add TypeDescriptor#getTypes --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1217: [BEAM-27] Add setTimer and deleteTimer by...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1217 [BEAM-27] Add setTimer and deleteTimer by ID in TimerInternals Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- I have pulled this commit out of my ongoing work to add timers to `DoFn`, specifically my branch where I add `DoFnInvoker.invokeOnTimer(, ...)`. For users, timers will have IDs. This is a step towards that API. I think it is also just better in most ways, especially for being able to delete timers without having to know exactly when it was set for. Before this change, a timer's ID is its `TimeDomain` + `Instant`. Inlining `TimerData` for this description, I think `setTimer(TimeDomain, Instant)` is OK for setting an anonymous "wake up" as we use it today, while `deleteTimer(TimeDomain, Instant)` should be removed. R: @tgroh and @aljoscha and @amitsela I am curious about runner-specific thoughts if you have any. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam TimerInternals Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1217.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1217 commit 932cb20e64199942369376748faabae548f72df4 Author: Kenneth Knowles <k...@google.com> Date: 2016-10-25T19:28:14Z Add setTimer and deleteTimer by ID in TimerInternals For users, timers will have IDs. This is a step towards that API; our current API treats the timer's timestamp as its ID, more-or-less. We can leave that API or not. This change adds the more general API. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1216: [BEAM-854] Directly implement ReifyTimest...
GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1216 [BEAM-854] Directly implement ReifyTimestampsAndWindows in SparkRunner Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: @amitsela I introduced a performance issue in [BEAM-854](https://issues.apache.org/jira/browse/BEAM-854) that affects the SparkRunner and DirectRunner. As per discussion there, `ReifyTimestampsAndWindows` can only use the multi-window `WindowedValue` representation if it is implemented as a primitive. The fix is pretty trivial, and actually just simpler anyhow. I ran `mvn integration-test -Plocal-runnable-on-service-tests -pl runners/spark` but feel free to pull this down and do it right if I have missed something, inline the class, etc, whatever you like. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam Spark-Reify Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1216.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1216 commit 72b634857bc660716a1ce5e8fa03b39138472652 Author: Kenneth Knowles <k...@google.com> Date: 2016-10-28T05:18:19Z Directly implement ReifyTimestampsAndWindows in SparkRunner --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
