[GitHub] incubator-beam pull request #1473: Adds TransformEvaluator.startBundle()
Github user jkff closed the pull request at: https://github.com/apache/incubator-beam/pull/1473 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1636: [BEAM-498] Moves OldDoFn to runners-core
GitHub user jkff opened a pull request: https://github.com/apache/incubator-beam/pull/1636 [BEAM-498] Moves OldDoFn to runners-core This will of course need the usual Dataflow worker surgery. R: @kennknowles You can merge this pull request into a Git repository by running: $ git pull https://github.com/jkff/incubator-beam move-old-do-fn Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1636.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1636 commit 374a450a66baf43786e2a22afb8e8832d3146441 Author: Eugene Kirpichov <kirpic...@google.com> Date: 2016-12-16T00:16:46Z Moves OldDoFn to runners-core --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1620: [BEAM-1154] Get side input from proper wi...
GitHub user jkff opened a pull request: https://github.com/apache/incubator-beam/pull/1620 [BEAM-1154] Get side input from proper window in ReduceFn R: @kennknowles CC: @peihe You can merge this pull request into a Git repository by running: $ git pull https://github.com/jkff/incubator-beam combine-side-input Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1620.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1620 commit 38bca062b99900f44321035d660fa9b63f5179b4 Author: Eugene Kirpichov <kirpic...@google.com> Date: 2016-12-14T22:29:30Z [BEAM-1154] Get side input from proper window in ReduceFn --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1601: [BEAM-1149] Explode windows when fn uses ...
GitHub user jkff opened a pull request: https://github.com/apache/incubator-beam/pull/1601 [BEAM-1149] Explode windows when fn uses side inputs This basically reverts https://github.com/apache/incubator-beam/commit/38f0b11cc9028cf347e3c96b6e6116e5a5a9972d and adds some comments and a test. R: @kennknowles You can merge this pull request into a Git repository by running: $ git pull https://github.com/jkff/incubator-beam multi-window-side-inputs Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1601.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1601 commit e2644e3a869c40bbb3614198dfed8a4b2ab35ab8 Author: Eugene Kirpichov <kirpic...@google.com> Date: 2016-12-13T22:35:33Z [BEAM-1149] Explode windows when fn uses side inputs --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1584: DO NOT REVIEW. Debugging MqttIO failures.
Github user jkff closed the pull request at: https://github.com/apache/incubator-beam/pull/1584 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1584: DO NOT REVIEW. Debugging MqttIO failures.
GitHub user jkff opened a pull request: https://github.com/apache/incubator-beam/pull/1584 DO NOT REVIEW. Debugging MqttIO failures. (can't reproduce the error locally, and don't know how to run something on Travis - so creating a PR just for the sake of getting a Travis run) You can merge this pull request into a Git repository by running: $ git pull https://github.com/jkff/incubator-beam mqttio-test Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1584.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1584 commit 234916835fcd83a3a43409e81c94880d0988dde9 Author: Jean-Baptiste Onofré <jbono...@apache.org> Date: 2016-09-12T16:49:36Z [BEAM-606] Create MqttIO commit 6ed144825fcfc978103c5cc8c96546ae283f86b2 Author: Jean-Baptiste Onofré <jbono...@apache.org> Date: 2016-09-23T21:27:40Z [BEAM-606] Use AutoValue, remove checkpoint, introduce BlockingQueue and improve tests. commit 9fbee3fc87f92d2bd43d5dc79cc277d8e648ebb4 Author: Jean-Baptiste Onofré <jbono...@apache.org> Date: 2016-10-05T10:28:44Z [BEAM-606] Set topic required. Move some subscriber specific MQTT code on the reader. Improve tests. commit 6f4713006f9062f42bad9643ce9dbf115c7af119 Author: Jean-Baptiste Onofré <jbono...@apache.org> Date: 2016-10-05T14:46:24Z [BEAM-606] Replace KV internal use with a regular POJO commit 6bb48572c8cb2099d3782de5465eddade0ff3326 Author: Jean-Baptiste Onofré <jbono...@apache.org> Date: 2016-10-05T14:55:43Z [BEAM-606] Use a better value for watermark commit e02b2a25d4698eafda492192c339246db9f670af Author: Jean-Baptiste Onofré <jbono...@apache.org> Date: 2016-10-10T19:12:37Z [BEAM-606] Validation is now performed only in create(), maxNumRecords and maxReadTime are now exclusive, use ByteArrayCoder instead of SerializableCoder, use poll instead of take on the blocking queue, add javadoc on the with* methods, code cleanup commit 775a066625f5f250404ad2571aabf6e278aed662 Author: Jean-Baptiste Onofré <jbono...@apache.org> Date: 2016-10-13T07:44:00Z [BEAM-606] Rename connection configuration, improve javadoc commit 352b5fb279fe7dfd2d5a80bc023cbe7c2106eb1c Author: Jean-Baptiste Onofré <jbono...@apache.org> Date: 2016-10-13T08:41:52Z [BEAM-606] Refactore read test to be more reliable commit 42ff71deaf0bdedf9ee62f25d21aeb5efff2ec9b Author: Jean-Baptiste Onofré <jbono...@apache.org> Date: 2016-10-21T13:50:44Z [BEAM-606] Update documentation, change the QoS in the test commit 532704604658ac081eb16611958e4ad7419b6855 Author: Jean-Baptiste Onofré <jbono...@apache.org> Date: 2016-11-02T06:01:39Z [BEAM-606] Start ActiveMQ broker on free network port in the tests commit dda9092d1e2fb17c80641d3490de5a7d4ae242a5 Author: Jean-Baptiste Onofré <jbono...@apache.org> Date: 2016-11-02T09:38:55Z [BEAM-606] Fix parent pom version commit b92eb296a8755f2f92bbfa3f1ad9acc12d6ea080 Author: Jean-Baptiste Onofré <jbono...@apache.org> Date: 2016-11-03T08:50:00Z [BEAM-606] Exclude support of QoS 2, cleanup in javadoc commit 75d24726ec9ffb71844c760a3f321ebab0e7a228 Author: Jean-Baptiste Onofré <jbono...@apache.org> Date: 2016-11-14T12:07:01Z [BEAM-606] Add core MQTT test commit fc1195ad917144ed2ac829ea8f8c3df620e0cbb4 Author: Jean-Baptiste Onofré <jbono...@apache.org> Date: 2016-11-22T16:58:14Z [BEAM-606] Improved checkpoint with manual ack, update comments about QoS (relationship between publisher and subsribers), auto generate clientId (reliability) commit f7063fbc9343101a921715082b7cc06da0184483 Author: Jean-Baptiste Onofré <jbono...@apache.org> Date: 2016-12-05T10:14:41Z [BEAM-606] Fix checkpoint mark commit 7d9b9cdcd155cdd35082095887728afa9372e500 Author: Jean-Baptiste Onofré <jbono...@apache.org> Date: 2016-12-09T17:14:07Z [BEAM-606] Only support QoS 1 in Write, updated to last SDK changes commit aed01b9696f2bb87eb01d8477b2d322b24fb7119 Author: Jean-Baptiste Onofré <jbono...@apache.org> Date: 2016-12-10T15:56:35Z [BEAM-606] Use ActiveMQ KahaDB in test for persistence adapter commit e77fea21d065cf926506c1a49f05f3c9d7101d21 Author: Eugene Kirpichov <kirpic...@google.com> Date: 2016-12-12T22:04:26Z For debugging: dont drop stack in UserCodeException --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1565: Practically removes OldDoFn from SDK
GitHub user jkff opened a pull request: https://github.com/apache/incubator-beam/pull/1565 Practically removes OldDoFn from SDK After this change, the only usages of OldDoFn in the SDK are in combine context stuff, which itself should be moved to runners-core (outside the scope of this change). TODO: untangle into separate commits. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jkff/incubator-beam old-do-fn Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1565.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1565 commit 20930d70890f69dab915f587bb4eccef13f3001c Author: Eugene Kirpichov <kirpic...@google.com> Date: 2016-12-09T20:58:54Z WIP --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1486: Adds a category for tests using splittabl...
GitHub user jkff opened a pull request: https://github.com/apache/incubator-beam/pull/1486 Adds a category for tests using splittable DoFn Excludes the category in all runners except direct. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jkff/incubator-beam sdf-category Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1486.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1486 commit a95eda80e4f314feef9949fee2aa13b5399afc57 Author: Eugene Kirpichov <kirpic...@google.com> Date: 2016-11-30T20:55:45Z Category for tests using splittable DoFn --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1473: Adds TransformEvaluator.startBundle()
GitHub user jkff opened a pull request: https://github.com/apache/incubator-beam/pull/1473 Adds TransformEvaluator.startBundle() For symmetry with finishBundle() and to allow decoupling creation of a ParDoEvaluator from the time it calls fn.startBundle(). R: @tgroh You can merge this pull request into a Git repository by running: $ git pull https://github.com/jkff/incubator-beam start-bundle Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1473.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1473 commit a3c2e1c755ae51d6ad7058b7a811006ab7455ace Author: Eugene Kirpichov <kirpic...@google.com> Date: 2016-11-30T23:23:26Z Adds TransformEvaluator.startBundle() For symmetry with finishBundle() and to allow decoupling creation of a ParDoEvaluator from the time it calls fn.startBundle(). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1471: Preserves compressed windows in PushbackS...
GitHub user jkff opened a pull request: https://github.com/apache/incubator-beam/pull/1471 Preserves compressed windows in PushbackSideInputDoFnRunner R: @tgroh You can merge this pull request into a Git repository by running: $ git pull https://github.com/jkff/incubator-beam pushback-runner Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1471.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1471 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1461: [BEAM-1060] Makes DoFnTester use new DoFn...
GitHub user jkff opened a pull request: https://github.com/apache/incubator-beam/pull/1461 [BEAM-1060] Makes DoFnTester use new DoFn internally. There were 2 remaining users of DoFnTester.of(OldDoFn): - SplittableParDo.ProcessElements: this is fixed in https://github.com/apache/incubator-beam/pull/1261 - GroupAlsoByWindowsProperties: this one is harder. Various GABWDoFn's use OldDoFn.windowingInternals, and we can't pass that through a new DoFn. So instead I removed usage of DoFnTester from GroupAlsoByWindowsProperties in favor of a tiny hand-coded solution. So after this PR and #1261 DoFnTester.of(OldDoFn) can be deleted. R: @kennknowles You can merge this pull request into a Git repository by running: $ git pull https://github.com/jkff/incubator-beam tester-new-dofn Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1461.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1461 commit 4cdccc044f6d7b7599860c3bc656a7b33905f79f Author: Eugene Kirpichov <kirpic...@google.com> Date: 2016-11-18T21:10:22Z Makes DoFnTester use new DoFn internally. There were 2 remaining users of DoFnTester.of(OldDoFn): - SplittableParDo.ProcessElements: this is fixed in https://github.com/apache/incubator-beam/pull/1261 - GroupAlsoByWindowsProperties: this one is harder. Various GABWDoFn's use OldDoFn.windowingInternals, and we can't pass that through a new DoFn. So instead I removed usage of DoFnTester from GroupAlsoByWindowsProperties in favor of a tiny hand-coded solution. So after #1261 DoFnTester.of(OldDoFn) can be deleted. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1373: Properly uses result of transform overrid...
GitHub user jkff opened a pull request: https://github.com/apache/incubator-beam/pull/1373 Properly uses result of transform override Previously direct runner would use the transform override to .apply(), but would keep the original transform in the pipeline, e.g. it would use the original transform to look up an evaluator. The current commit makes it use the result of override "for real" (including, potentially replacing it further recursively). **This change is in DirectRunner.java.** Additionally, makes InputProvider type-safe (discovered due to a run-time ClassCastException that happened while testing the code above). **This is the rest of the PR.** R: @tgroh You can merge this pull request into a Git repository by running: $ git pull https://github.com/jkff/incubator-beam apply-transform Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1373.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1373 commit dc89bc614cd38e4daa34bfd6618254874d082265 Author: Eugene Kirpichov <kirpic...@google.com> Date: 2016-11-16T23:40:08Z Properly uses result of transform override Previously direct runner would use the transform override to .apply(), but would keep the original transform in the pipeline, e.g. it would use the original transform to look up an evaluator. The current commit makes it use the result of override "for real" (including, potentially replacing it further recursively). Additionally, makes InputProvider type-safe (discovered due to a run-time ClassCastException that happened while testing the code above). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1367: Declare archetype starter dependency on s...
GitHub user jkff opened a pull request: https://github.com/apache/incubator-beam/pull/1367 Declare archetype starter dependency on slf4j Example failed log before this PR: https://api.travis-ci.org/jobs/176202650/log.txt?deansi=true R: @lukecwik You can merge this pull request into a Git repository by running: $ git pull https://github.com/jkff/incubator-beam declare-dependency Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1367.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1367 commit b6723eaa79b01727ebafa01aa3e72a113c1e369b Author: Eugene Kirpichov <kirpic...@google.com> Date: 2016-11-16T00:01:42Z Declare archetype dependency on slf4j --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1364: Fixes after #1332
GitHub user jkff opened a pull request: https://github.com/apache/incubator-beam/pull/1364 Fixes after #1332 #1332 introduced a few precommit failures. This PR fixes them and makes a couple of other minor changes noticed while doing the fixes. R: @amitsela You can merge this pull request into a Git repository by running: $ git pull https://github.com/jkff/incubator-beam fixup-1332 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1364.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1364 commit 97ae0b43d287a3715d43579f2001b0960e769a14 Author: Eugene Kirpichov <kirpic...@google.com> Date: 2016-11-15T22:25:51Z Fix a bunch of precommit errors from #1332 commit b19af6520312d52bf7fd39ecec7bc9d7a4a3febe Author: Eugene Kirpichov <kirpic...@google.com> Date: 2016-11-15T22:32:11Z Renames TestPipelineOptions to SparkTestPipelineOptions To avoid confusion with sdk.testing.TestPipelineOptions. Also, a couple of other minor fixes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1353: Liberates ReduceFnRunner from WindowingIn...
GitHub user jkff opened a pull request: https://github.com/apache/incubator-beam/pull/1353 Liberates ReduceFnRunner from WindowingInternals, and lets WindowingInternals do windowed side outputs - Introduces WindowingInternals.sideOutputWindowedValue (will be necessary for Splittable DoFn) - Implements it properly in all runners (required some minor refactoring in Spark and Flink ProcessContext implementations) - Introduces "OutputWindowedValue" interface and "SideInputAccess" interfaces, and uses them in ReduceFnRunner instead of directly using WindowingInternals. - Introduces adapters from WindowingInternals to these two interfaces, for use in OldDoFn contexts - Moves some StateContext functions into ReduceFnContextFactory, because they make more sense in runners-core than in sdk (because they are only invoked by different runners). The only remaining StateContexts function is nullContext, but I couldn't figure out an easy way to move it into runners-core and gave up (however in fact I'm not sure its current usages are correct at all...) R: @kennknowles (for bulk of the code and as committer) CC: @aljoscha @amitsela (for the minor refactorings in respective runners) You can merge this pull request into a Git repository by running: $ git pull https://github.com/jkff/incubator-beam reducefn-windowing-internals Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1353.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1353 commit 14dd49e02ab2eabb7c7e42cb1e9cab29f7cea44e Author: Eugene Kirpichov <kirpic...@google.com> Date: 2016-11-11T02:40:53Z Liberates ReduceFnRunner from WindowingInternals commit d6c62454bb822063c863d2bcd1f2436453b94864 Author: Eugene Kirpichov <kirpic...@google.com> Date: 2016-11-12T00:37:42Z Refactor FlinkProcessContext more cleanly into single- and multi-output versions commit 14e5847f738b3d89808d5d612684fcddffd51b7f Author: Eugene Kirpichov <kirpic...@google.com> Date: 2016-11-12T00:57:27Z Refactor SparkProcessContext more cleanly into single- and multi-output versions --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1036: Introduces the Redistribute transform
Github user jkff closed the pull request at: https://github.com/apache/incubator-beam/pull/1036 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1260: Reduces exposure of WindowedValue in the ...
GitHub user jkff opened a pull request:
https://github.com/apache/incubator-beam/pull/1260
Reduces exposure of WindowedValue in the SDK
- Introduces ValueInSingleWindow for purposes of PAssert
- Moves several classes that should be in runners-core
and that used WindowedValue, from sdks/ to runners-core:
KeyedWorkItem stuff and ExecutionContext stuff.
- Removes deprecated AggregatorFactory which was the only
remaining compilation blocker after the two above.
- Moves WindowMatchers{,Test} and PCollectionViewTesting
to runners-core
- Uses ValueInSingleWindow in parts of DoFnTester where possible
After this commit, the only places where WindowedValue is exposed in the
SDK API surface are:
- `WindowingInternals.writePCollectionViewData(TupleTag tag,
Iterable<WindowedValue> data, Coder elemCoder)`
- Deprecated methods of `PCollectionView` - it should probably be in
runners-core, but I got lost trying to move it there, so I'll defer it to
future work. I also don't understand what these methods do, maybe if they're
deprecated there's some easy way to remove them.
R: @kennknowles OR @tgroh
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jkff/incubator-beam reduce-windowed-value
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-beam/pull/1260.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1260
commit 7dd58ed5cb245fc2c6571e9b722aef53c6e590ea
Author: Eugene Kirpichov <kirpic...@google.com>
Date: 2016-10-31T22:46:25Z
Towards removing WindowedValue from SDK
- Introduces ValueInSingleWindow for purposes of PAssert
- Moves several classes that should be in runners-core
and that used WindowedValue, from sdks/ to runners-core:
KeyedWorkItem stuff and ExecutionContext stuff.
- Removes deprecated AggregatorFactory which was the only
remaining compilation blocker after the two above.
After this commit, WindowedValue does not appear in any public
APIs provided by sdks/.
commit 18548fd9c370349e89e8c5b48e8953ff2576f686
Author: Eugene Kirpichov <kirpic...@google.com>
Date: 2016-10-31T23:33:13Z
Some more liberation of SDK from WindowedValue
- Moved WindowMatchers{,Test} and PCollectionViewTesting
to runners-core
- Used ValueInSingleWindow in parts of DoFnTester where possible
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-beam pull request #1243: Fixes crash of shade plugin on SparkProce...
GitHub user jkff opened a pull request: https://github.com/apache/incubator-beam/pull/1243 Fixes crash of shade plugin on SparkProcessContext Error discovered by debugging the Maven plugin itself (using mvnDebug) and setting a breakpoint for the exception and then inspecting the call stack to spot a weird-looking method signature descriptor: `Lorg/apache/beam/sdk/transforms/windowing/WindowFn.AssignContext;` (note the exclamation mark, which is an invalid character AFAIK! I have no idea where it came from, but the current change makes it go away) R: @dhalperi CC: @amitsela You can merge this pull request into a Git repository by running: $ git pull https://github.com/jkff/incubator-beam spark-process-context-shade Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1243.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1243 commit fc8d49e86344ba97a032b19eed4f881c20f9805b Author: Eugene Kirpichov <kirpic...@google.com> Date: 2016-11-01T01:06:43Z Fixes crash of shade plugin on SparkProcessContext Error discovered by debugging the Maven plugin itself (using mvnDebug) and setting a breakpoint for the exception and then inspecting the call stack to spot a weird-looking method signature descriptor: `Lorg/apache/beam/sdk/transforms/windowing/WindowFn.AssignContext;` (note the exclamation mark, which is an invalid character AFAIK! I have no idea where it came from, but the current change makes it go away) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1203: Deduplicates ParDo{Single,Multi}Evaluator...
GitHub user jkff opened a pull request:
https://github.com/apache/incubator-beam/pull/1203
Deduplicates ParDo{Single,Multi}EvaluatorFactory
This is in preparation for adding a third one for a future ParDo-like
primitive
transform to be introduced inside SplittableParDo.
R: @tgroh
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jkff/incubator-beam pardo-evaluator
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-beam/pull/1203.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1203
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-beam pull request #1201: Makes PTransformOverrideFactory type-safe
GitHub user jkff opened a pull request: https://github.com/apache/incubator-beam/pull/1201 Makes PTransformOverrideFactory type-safe This removes a bunch of casting boilerplate from subclasses. Also, removes unused class `UncommittedBundleOutputManager` (seems not worth a separate PR). R: @tgroh You can merge this pull request into a Git repository by running: $ git pull https://github.com/jkff/incubator-beam typed-override-factory Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1201.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1201 commit c8b9786c61b4762cc20ff019e8fdad618c82648e Author: Eugene Kirpichov <kirpic...@google.com> Date: 2016-10-26T23:14:51Z Makes PTransformOverrideFactory type-safe --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1050: Makes FileBasedSink use a temporary direc...
GitHub user jkff opened a pull request: https://github.com/apache/incubator-beam/pull/1050 Makes FileBasedSink use a temporary directory When writing to `/path/to/foo`, temporary files would be written to `/path/too/foo-temp-$uid` (or something like that), i.e. as siblings of the final output. That could lead to issues like http://stackoverflow.com/q/39822859/278042 Now, temporary files are written to a path like: `/path/too/temp-beam-foo-$date/$uid`. This way, the temporary files won't match the same glob as the final output files (even though they may still fail to be deleted due to eventual consistency issues). You can merge this pull request into a Git repository by running: $ git pull https://github.com/jkff/incubator-beam file-sink-tmp-dir Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1050.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1050 commit 1c34acdaf4a0c0697c9646934ac163788133347b Author: Eugene Kirpichov <kirpic...@google.com> Date: 2016-10-04T22:23:27Z Makes FileBasedSink use a temporary directory When writing to /path/to/foo, temporary files would be written to /path/too/foo-temp-$uid (or something like that), i.e. as siblings of the final output. That could lead to issues like http://stackoverflow.com/q/39822859/278042 Now, temporary files are written to a path like: /path/too/temp-beam-foo-$date/$uid. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1048: Converts KafkaIO builders to @AutoValue
GitHub user jkff opened a pull request: https://github.com/apache/incubator-beam/pull/1048 Converts KafkaIO builders to @AutoValue This is in the same spirit as https://github.com/apache/incubator-beam/pull/1031, https://github.com/jbonofre/incubator-beam/pull/1 and https://github.com/apache/incubator-beam/pull/1033 . Semantics is unchanged AFAICT. The only user-visible change is that TypedRead and TypedWrite no longer exist (they were unnecessary in the first place) - see the trivial changes in test. R: @rangadi You can merge this pull request into a Git repository by running: $ git pull https://github.com/jkff/incubator-beam kafka-autovalue Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1048.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1048 commit ec3fdd030da1cc2ed2fdd0d16fad0396ef1855a9 Author: Eugene Kirpichov <kirpic...@google.com> Date: 2016-09-29T22:30:10Z Converts KafkaIO to AutoValue --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1036: Introduces the Rebundle transform
GitHub user jkff opened a pull request: https://github.com/apache/incubator-beam/pull/1036 Introduces the Rebundle transform It's similar to Reshuffle in that it prevents fusion of the surrounding transforms, however while Reshuffle requires the input collection to be KVs, Rebundle efficiently generates sufficiently unique keys itself. Also uses it in Datastore. The transform will be useful in JdbcIO. (I tried adapting it to also support the case of fixed number of bundles, as in the Write transform, but this has hairy semantics in the unbounded case, so I decided not to do it) R: @bjchambers CC: @jbonofre @dhalperi You can merge this pull request into a Git repository by running: $ git pull https://github.com/jkff/incubator-beam reparallelize Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1036.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1036 commit 58dc834c88b2a1249645f307276c97c9b9fa0e78 Author: Eugene Kirpichov <kirpic...@google.com> Date: 2016-09-30T18:18:38Z Introduces the Rebundle transform It's similar to Reshuffle in that it prevents fusion of the surrounding transforms, however while Reshuffle requires the input collection to be KVs, Rebundle efficiently generates sufficiently unique keys itself. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1031: Converts Datastore to use AutoValue
GitHub user jkff opened a pull request: https://github.com/apache/incubator-beam/pull/1031 Converts Datastore to use AutoValue This style saves a good amount of code and is easier to mimic in new connectors. I have a few more changes like this lined up. R: @dhalperi CC: @jbonofre You can merge this pull request into a Git repository by running: $ git pull https://github.com/jkff/incubator-beam datastore-autovalue Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1031.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1031 commit 7470d093a3252d65c8c6a26176e3f1e9279ee864 Author: Eugene Kirpichov <kirpic...@google.com> Date: 2016-09-29T00:29:11Z Converts Datastore to use AutoValue --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1028: Enables -Xlint:rawtypes
GitHub user jkff opened a pull request: https://github.com/apache/incubator-beam/pull/1028 Enables -Xlint:rawtypes Except for Spark and Flink runners and the microbenchmarks module. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jkff/incubator-beam xlint-rawtypes Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1028.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1028 commit af9f6d31c2db9754ca8f4194c41f379426183d30 Author: Eugene Kirpichov <kirpic...@google.com> Date: 2016-09-29T21:00:41Z Enables -Xlint:rawtypes Except for Spark and Flink runners and the microbenchmarks module --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1023: Deduplicates the 3 in-memory TimerInterna...
GitHub user jkff opened a pull request: https://github.com/apache/incubator-beam/pull/1023 Deduplicates the 3 in-memory TimerInternals There were 3 implementations of TimerInternals doing basically the same thing: - TestTimerInternals in ReduceFnTester - TestTimerInternals in TriggerTester (these two were nearly identical) - BatchTimerInternals (it was a subset of the above) There were also 2 copies of TestInMemoryStateInternals. This change deduplicates and reorganizes them: - Deduplicates the TestInMemoryStateInternals. It might make sense to pull its methods into InMemoryStateInternals and delete the class, I'm not sure. - Factors out the common timer stuff into InMemoryTimerInternals. - TriggerTester's implementation of TestTimerInternals used to (unnecessarily) access TestInMemoryStateInternals, presumably due to copy-paste. Now it uses the regular InMemoryTimerInternals. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jkff/incubator-beam dedup-timer-internals Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1023.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1023 commit d671a989a1ba8476ef146d589c589029e258f24b Author: Eugene Kirpichov <kirpic...@google.com> Date: 2016-09-28T22:52:27Z Deduplicates the 3 in-memory TimerInternals There were 3 implementations of TimerInternals doing basically the same thing: - TestTimerInternals in ReduceFnTester - TestTimerInternals in TriggerTester (these two were nearly identical) - BatchTimerInternals (it was a subset of the above) There were also 2 copies of TestInMemoryStateInternals. This change deduplicates and reorganizes them: - Deduplicates the TestInMemoryStateInternals. It might make sense to pull its methods into InMemoryStateInternals and delete the class, I'm not sure. - Factors out the common timer stuff into InMemoryTimerInternals. - TriggerTester's implementation of TestTimerInternals used to (unnecessarily) access TestInMemoryStateInternals, presumably due to copy-paste. Now it uses the regular InMemoryTimerInternals. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1011: Support for @Setup and @Teardown in DoFnT...
GitHub user jkff opened a pull request: https://github.com/apache/incubator-beam/pull/1011 Support for @Setup and @Teardown in DoFnTester - To allow testing DoFn reuse, adds a third cloning mode - CLONE_ONCE. The current CLONE is renamed to CLONE_PER_BUNDLE because it corresponds to the olden ways of DoFnTester. This rename should be harmless because it was the default, so probably nobody uses it (nobody uses it in the Beam codebase anyway), but I can rename it back if necessary. - Calls Setup lazily on the first interaction with the fn under test: depending on cloning mode, this is either a per-bundle clone, or the original fn, or a single clone of the original fn. - Calls Teardown either when the DoFnTester is closed, or when finishBundle'ing with CLONE_PER_BUNDLE. - Enforces that you have to configure the DoFnTester (setSideInputs, setCloningBehavior) before using it, to simplify the semantics. Makes setSideInputs consistent with setSideInput(without "s") in not resetting the state. - Allows processing a bundle after processing an element. This used to be disallowed, but it's weird, because processing a bundle after processing a bundle *was* allowed. Additionally, improves testing coverage. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jkff/incubator-beam do-fn-tester-lifecycle Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1011.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1011 commit 62ec3ed9bfa0bb94c41b0df4761d5d50cb4d196a Author: Eugene Kirpichov <kirpic...@google.com> Date: 2016-09-26T23:58:20Z Support for @Setup and @Teardown in DoFnTester --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #973: Deprecate TeardownPolicy for Dataflow serv...
GitHub user jkff opened a pull request: https://github.com/apache/incubator-beam/pull/973 Deprecate TeardownPolicy for Dataflow service We are moving towards supporting only TEARDOWN_ALWAYS. (This is a clone of https://github.com/pjesa/DataflowJavaSDK/commit/657ce1ff85caf44f7cf16d67b973055657980c2e on behalf of @pjesa) R: @dhalperi You can merge this pull request into a Git repository by running: $ git pull https://github.com/jkff/incubator-beam deprecate-teardown Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/973.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #973 commit 7509308c7eca146f94463fd3c6adc9e9b84c4ac2 Author: Eugene Kirpichov <kirpic...@google.com> Date: 2016-09-19T22:12:51Z Deprecate TeardownPolicy for Dataflow service We are moving towards supporting only TEARDOWN_ALWAYS. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #954: Enables BytecodeVerificationLocal on travi...
GitHub user jkff opened a pull request: https://github.com/apache/incubator-beam/pull/954 Enables BytecodeVerificationLocal on travis This is to achieve stricter verification of bytecode, mainly bytecode generated by DoFnInvokers dynamically. See https://issues.apache.org/jira/browse/BEAM-630 for an example situation that could have been caught earlier with this change. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jkff/incubator-beam bytecode-verify Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/954.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #954 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #953: [BEAM-630] Fixes wrong type name in genera...
GitHub user jkff opened a pull request: https://github.com/apache/incubator-beam/pull/953 [BEAM-630] Fixes wrong type name in generated code See https://issues.apache.org/jira/browse/BEAM-630 for details. We generated the wrong local variable type (with a semicolon - "descriptor" - rather than without a semicolon - "internal name"). It only failed when JVM was using sufficiently strict settings when verifying the bytecode. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jkff/incubator-beam fix-invokers Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/953.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #953 commit 2f825177c59add53fa259325c2fe17981a707ba1 Author: Eugene Kirpichov <kirpic...@google.com> Date: 2016-09-13T21:19:29Z [BEAM-630] Fixes wrong type name in generated code --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #926: More changes to DoFn{Signatures,Invokers}.
GitHub user jkff opened a pull request:
https://github.com/apache/incubator-beam/pull/926
More changes to DoFn{Signatures,Invokers}.
In preparation for Splittable DoFn.
* More generic code generation in DoFnInvokers:
supports methods with return values (thanks @bjchambers).
* Uses AutoValue builder in DoFnSignature.
* Contextual error reporting in DoFnSignatures parsing code.
* Rewrote DoFnInvokers tests to use Mockito.
* Changed DoFnSignatures tests to use local classes
and an "AnonymousMethod" class for testing analysis of
single methods.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jkff/incubator-beam do-fn-reflector
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-beam/pull/926.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #926
commit 9c88a510481df5cdfb229112cd35c364072c9e6f
Author: Eugene Kirpichov <kirpic...@google.com>
Date: 2016-08-12T00:13:53Z
More changes to DoFn{Signatures,Invokers}.
In preparation for Splittable DoFn.
* More generic code generation in DoFnInvokers:
supports methods with return values (thanks @bjchambers).
* Uses AutoValue builder in DoFnSignature.
* Contextual error reporting in DoFnSignatures parsing code.
* Rewrote DoFnInvokers tests to use Mockito.
* Changed DoFnSignatures tests to use local classes
and an "AnonymousMethod" class for testing analysis of
single methods.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-beam pull request #896: [BEAM-65] SplittableDoFn prototype.
GitHub user jkff opened a pull request: https://github.com/apache/incubator-beam/pull/896 [BEAM-65] SplittableDoFn prototype. Work in progress. Currently only runs in direct runner, and not ready for any use by real users. No tests for most of the code yet (will add later in this PR) but the rest is ready to take a look. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jkff/incubator-beam splittable-do-fn Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/896.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #896 commit f025018ec24f7b795886aee58c4df689f87a6481 Author: Eugene Kirpichov <kirpic...@google.com> Date: 2016-08-12T00:13:53Z [BEAM-65] SplittableDoFn prototype. Work in progress. Currently only runs in direct runner, and not ready for any use by real users. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #812: Rewrites DoFnReflector to go via DoFnSigna...
GitHub user jkff opened a pull request: https://github.com/apache/incubator-beam/pull/812 Rewrites DoFnReflector to go via DoFnSignature DoFnSignature encapsulates type information about a DoFn, in particular which arguments/features its methods actually use. Before this commit, DoFnReflector would parse/verify/generate code in one go; after this commit, these stages are separated: DoFnSignature encapsulates all information needed to generate the code. Additionally, removes the unnecessary genericity in the implementation of DoFnReflector's code generation for the very different methods processElement and start/finishBundle. The code is simpler if decomposed into utility functions, rather than attempting a uniform representation for different methods. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jkff/incubator-beam do-fn-reflector Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/812.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #812 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request: [BEAM-182] Bound trials in assertSpli...
GitHub user jkff opened a pull request: https://github.com/apache/incubator-beam/pull/144 [BEAM-182] Bound trials in assertSplitAtFraction At most 100 trials per item and 1000 trials total. This appears to be enough for tests such as AvroSourceTest in the majority of cases, which means if bugs exist, they will be found often enough. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jkff/incubator-beam source-test-utils Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/144.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #144 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
