[GitHub] incubator-beam pull request #1557: Add support for having an empty CountingI...
GitHub user lukecwik opened a pull request: https://github.com/apache/incubator-beam/pull/1557 Add support for having an empty CountingInput/CountingSource Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/lukecwik/incubator-beam counting_source Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1557.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1557 commit dff45ea3ce332a93f44c1ba511287ce900e9e6ef Author: Luke Cwik <lc...@google.com> Date: 2016-12-08T23:22:35Z Add support for having an empty CountingInput/CountingSource --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1308: [BEAM-939] Use the credentials from GcpOp...
GitHub user lukecwik opened a pull request: https://github.com/apache/incubator-beam/pull/1308 [BEAM-939] Use the credentials from GcpOptions instead of relying on BigtableOptions Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- This allows for us to create a version of the service depending on the PipelineOptions passed in at this time. This uses a function which takes in the contextual PipelineOptions and creates a service from it allowing us to not serialize the Credentials within the BigtableOptions. You can merge this pull request into a Git repository by running: $ git pull https://github.com/lukecwik/incubator-beam beam939 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1308.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1308 commit 5fd9cdc0c843cf126ce0f2e6c6c7563471d6b335 Author: Luke Cwik <lc...@google.com> Date: 2016-11-08T03:37:42Z [BEAM-939] Use the credentials from GcpOptions instead of relying on the BigtableOptions to provide them. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1246: [BEAM-626] Changes in AvroCoder serializa...
GitHub user lukecwik opened a pull request: https://github.com/apache/incubator-beam/pull/1246 [BEAM-626] Changes in AvroCoder serialization so it can serialize in Kryo Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/lukecwik/incubator-beam fix941 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1246.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1246 commit 1f770d400ca9e5506199783551c0780a1c46f77e Author: Aviem Zur <aviem...@gmail.com> Date: 2016-09-08T08:21:41Z Changes in AvroCoder serialization so it can serialize in Kryo commit 3df987f76467c1e78ce3738a0f2c0df9cd94f82c Author: Luke Cwik <lc...@google.com> Date: 2016-10-31T18:38:45Z !fixup Swap to use Serializable ThreadLocal with empty data block instead of memoization. commit 3358d3d6cc036cb06fa37c6c2547909586bf048f Author: Luke Cwik <lc...@google.com> Date: 2016-10-31T20:21:31Z fixup! Fix the memoization of the schema string to use readResolve/writeReplace with final fields for Java. Kryo ignores readResolve/writeReplace and serializes the Schema object directly. commit f92c39f14b49556f8e67c22c408520af0c204cce Author: Luke Cwik <lc...@google.com> Date: 2016-10-31T20:24:49Z fixup! minor whitespace clean up. commit a5f25e2e9edf9fe0c1fb59e594d5b450062ba77f Author: Luke Cwik <lc...@google.com> Date: 2016-10-31T20:36:14Z fixup! Add FB suppression on limitation of FB with Serializable objects containing writeReplace. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1124: Update Dataflow container image for curre...
GitHub user lukecwik opened a pull request: https://github.com/apache/incubator-beam/pull/1124 Update Dataflow container image for current version of Apache Beam. Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- Tested with WordCount Dataflow job 2016-10-17_22_16_00-3640376692187174806 in integration project. You can merge this pull request into a Git repository by running: $ git pull https://github.com/lukecwik/incubator-beam update_container Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1124.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1124 commit 3c47539c5954abdb00431bcfdf0ee223329a1a60 Author: Luke Cwik <lc...@google.com> Date: 2016-10-18T04:58:26Z Update Dataflow container image for current version of Apache Beam. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1109: "Intern" schemas and schema strings to pr...
GitHub user lukecwik opened a pull request: https://github.com/apache/incubator-beam/pull/1109 "Intern" schemas and schema strings to prevent out of memory issues Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- "Intern" schemas and schema strings to prevent out of memory issues when dealing with many many files in Avro sources. You can merge this pull request into a Git repository by running: $ git pull https://github.com/lukecwik/incubator-beam intern_avro_schema Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1109.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1109 commit 867b20751449a2a2686822e3418cbd4efc65cc7c Author: Luke Cwik <lc...@google.com> Date: 2016-10-14T18:46:37Z "Intern" schemas and schema strings to prevent out of memory issues when dealing with many many files in Avro sources. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1099: Move the step output ids to use a flat na...
GitHub user lukecwik opened a pull request: https://github.com/apache/incubator-beam/pull/1099 Move the step output ids to use a flat namespace. Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- Also add a logical mapping from tuple tag to the flat namespace for DoFns. Note that this is expected to break the Dataflow integration tests until the worker image is updated. You can merge this pull request into a Git repository by running: $ git pull https://github.com/lukecwik/incubator-beam flatid Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1099.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1099 commit 0f9b6cae01cead77369e336e2c6d1a39b69e8fe5 Author: Luke Cwik <lc...@google.com> Date: 2016-10-13T22:33:49Z Move the step output ids to use a flat namespace. Also add a logical mapping from tuple tag to the flat namespace for DoFns. commit f2d2ab825f4e378e45b8b24c773ecc8e40f83338 Author: Luke Cwik <lc...@google.com> Date: 2016-10-13T22:40:05Z !fixup Fix test to validate that ids are valid int64s. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1012: Update Dataflow worker image for Apache B...
GitHub user lukecwik opened a pull request: https://github.com/apache/incubator-beam/pull/1012 Update Dataflow worker image for Apache Beam Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/lukecwik/incubator-beam update_dataflow_vm_image Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1012.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1012 commit fad30d9b9b5c11098b61e8f73c1a5cfbdaa0bd85 Author: Luke Cwik <lc...@google.com> Date: 2016-09-27T02:25:39Z Update Dataflow worker image for Apache Beam --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #993: Minor clean-up of RunnableOnService defini...
GitHub user lukecwik opened a pull request: https://github.com/apache/incubator-beam/pull/993 Minor clean-up of RunnableOnService definition Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/lukecwik/incubator-beam fix_runnable_on_service Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/993.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #993 commit ff03b8726c982646a9eaf9d3756dacc8fb88c194 Author: Luke Cwik <lc...@google.com> Date: 2016-09-23T16:08:33Z Minor clean-up of RunnableOnService definition --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #986: Update grpc to 1.0.1 and protobuf to 3.0.0
GitHub user lukecwik opened a pull request: https://github.com/apache/incubator-beam/pull/986 Update grpc to 1.0.1 and protobuf to 3.0.0 Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/lukecwik/incubator-beam update_grpc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/986.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #986 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #934: [BEAM-544] Address comments during backpor...
GitHub user lukecwik reopened a pull request: https://github.com/apache/incubator-beam/pull/934 [BEAM-544] Address comments during backport Dataflow PR/423 for Apache Beam PR/918 Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/lukecwik/incubator-beam fixup-pr-918 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/934.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #934 commit fea99b0466646e553f773d1bfc0a6553c48b49b4 Author: Luke Cwik <lc...@google.com> Date: 2016-09-08T16:34:43Z Address comments during backport Dataflow PR/423 for Apache Beam PR/918 commit facabfca910cb215c7044b97becf58dcb4fd933b Author: Luke Cwik <lc...@google.com> Date: 2016-09-08T16:40:19Z !fixup Fix changes done by IDE. new line -> newline --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #934: [BEAM-544] Address comments during backpor...
Github user lukecwik closed the pull request at: https://github.com/apache/incubator-beam/pull/934 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #934: Address comments during backport Dataflow ...
GitHub user lukecwik opened a pull request: https://github.com/apache/incubator-beam/pull/934 Address comments during backport Dataflow PR/423 for Apache Beam PR/918 Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/lukecwik/incubator-beam fixup-pr-918 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/934.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #934 commit fea99b0466646e553f773d1bfc0a6553c48b49b4 Author: Luke Cwik <lc...@google.com> Date: 2016-09-08T16:34:43Z Address comments during backport Dataflow PR/423 for Apache Beam PR/918 commit facabfca910cb215c7044b97becf58dcb4fd933b Author: Luke Cwik <lc...@google.com> Date: 2016-09-08T16:40:19Z !fixup Fix changes done by IDE. new line -> newline --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #851: [BEAM-124] Modify example dependencies to ...
GitHub user lukecwik opened a pull request: https://github.com/apache/incubator-beam/pull/851 [BEAM-124] Modify example dependencies to only add runners as optional dependencies Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- Also support excluding runners by disabling the include-runners profile. Tested ITs BQ and WordCount pass on Dataflow WordCount passes on Flink and Spark Upfront setup: ``` mvn clean install -pl examples/java,examples/java8 -am ``` Dataflow: ``` mvn clean verify -pl examples/java -DskipITs=false -DintegrationTestPipelineOptions='[ "--tempRoot=gs://clouddfe-testing-temp-storage", "--runner=org.apache.beam.runners.dataflow.testing.TestDataflowRunner" ]' ``` Flink: ``` mvn clean verify -pl examples/java -DskipITs=false -DintegrationTestPipelineOptions='[ "--tempRoot=gs://clouddfe-testing-temp-storage", "--runner=org.apache.beam.runners.flink.TestFlinkRunner" ]' -Dit.test=WordCountIT ``` Spark: ``` mvn clean verify -pl examples/java -DskipITs=false -DintegrationTestPipelineOptions='[ "--tempRoot=/tmp", "--runner=org.apache.beam.runners.spark.SparkRunner" ]' -Dit.test=WordCountIT ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/lukecwik/incubator-beam examples-cleanup Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/851.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #851 commit 41273bcc40551d8bb36127ee775d2e21bf9194cd Author: Luke Cwik <lc...@google.com> Date: 2016-08-18T23:35:33Z Modify example dependencies to only add runners as optional dependencies. Also support excluding runners by disabling the include-runners profile. Tested ITs BQ and WordCount pass on Dataflow WordCount passes on Flink and Spark --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #740: Remove DataflowJUnitTestRunner as integrat...
GitHub user lukecwik opened a pull request: https://github.com/apache/incubator-beam/pull/740 Remove DataflowJUnitTestRunner as integration tests execute using surefire/failsafe Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/lukecwik/incubator-beam remove_junit_runner Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/740.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #740 commit d9b6903dbadd05e27fb3a2cb2a539374ef7fa487 Author: Luke Cwik <lc...@visitor-lcwik.wat.corp.google.com> Date: 2016-07-27T17:14:41Z Remove DataflowJUnitTestRunner as integration tests execute using surefire/failsafe. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #680: [BEAM-468] NullableCoder should not ask va...
GitHub user lukecwik opened a pull request: https://github.com/apache/incubator-beam/pull/680 [BEAM-468] NullableCoder should not ask valueCoder isRegisterByteSizeObserverCheap when value is null Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/lukecwik/incubator-beam beam468 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/680.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #680 commit c5030a0ebc09499a15704301703654cb3547 Author: Luke Cwik <lc...@google.com> Date: 2016-07-18T20:14:33Z [BEAM-468] NullableCoder should not ask valueCoder isRegisterByteSizeObserverCheap when value is null --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #677: [BEAM-462] Fix reference to old PrintHandl...
GitHub user lukecwik opened a pull request: https://github.com/apache/incubator-beam/pull/677 [BEAM-462] Fix reference to old PrintHandler in archetypes project Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- This fixes post commit failure: https://builds.apache.org/job/beam_PostCommit_MavenVerify/org.apache.beam$beam-sdks-java-maven-archetypes-examples/912/console You can merge this pull request into a Git repository by running: $ git pull https://github.com/lukecwik/incubator-beam logginghandler Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/677.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #677 commit f9dcca2f26add731b9412ddf3705d6bcb6726008 Author: Luke Cwik <lc...@google.com> Date: 2016-07-18T15:51:32Z [BEAM-462] Replace MonitoringUtil.PrintHandler with a handler that utilizes a Java logger commit d689610fd39841c3a72cabc3b87517a942c517d7 Author: Luke Cwik <lc...@google.com> Date: 2016-07-18T16:26:33Z !fixup Fix BlockingDataflowRunner javadoc commit c6bfa16430c9a2d99aa82634794e0fe31a935e08 Author: Luke Cwik <lc...@google.com> Date: 2016-07-18T18:39:39Z [BEAM-462] Fix archetype project to use logging handler. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #674: [BEAM-465] OutgoingMessageCoder should be ...
GitHub user lukecwik opened a pull request: https://github.com/apache/incubator-beam/pull/674 [BEAM-465] OutgoingMessageCoder should be an AtomicCoder Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/lukecwik/incubator-beam beam465 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/674.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #674 commit fbc5b5ea5c32deccfa62c2b60aa834d729136b12 Author: Luke Cwik <lc...@google.com> Date: 2016-07-18T16:08:14Z [BEAM-465] OutgoingMessageCoder should be an AtomicCoder --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #673: [BEAM-462] Replace MonitoringUtil.PrintHan...
GitHub user lukecwik opened a pull request: https://github.com/apache/incubator-beam/pull/673 [BEAM-462] Replace MonitoringUtil.PrintHandler with a handler that utilizes a Java logger Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/lukecwik/incubator-beam logginghandler Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/673.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #673 commit f9dcca2f26add731b9412ddf3705d6bcb6726008 Author: Luke Cwik <lc...@google.com> Date: 2016-07-18T15:51:32Z [BEAM-462] Replace MonitoringUtil.PrintHandler with a handler that utilizes a Java logger --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request: Verify one element per window for Dat...
GitHub user lukecwik opened a pull request: https://github.com/apache/incubator-beam/pull/317 Verify one element per window for DataflowPipelineRunner View.asSingleton Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- This changes the expansion of the DataflowPipelineRunner override for View.asSingleton to provide a useful error message to users if their PCollection contains more than one element per window. You can merge this pull request into a Git repository by running: $ git pull https://github.com/lukecwik/incubator-beam singleton_verify Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/317.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #317 commit e423cdce40bb11b39f254828791f15b5c0490b86 Author: Luke Cwik <lc...@google.com> Date: 2016-05-10T20:02:21Z Verify one element per window for DataflowPipelineRunner View.asSingleton This changes the expansion of the DataflowPipelineRunner override for View.asSingleton to provide a useful error message to users if there PCollection contains more than one element per window. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request: [BEAM-151] Move over some more Datafl...
GitHub user lukecwik opened a pull request: https://github.com/apache/incubator-beam/pull/243 [BEAM-151] Move over some more Dataflow specific classes. Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- Note that users should use proto ByteString instead of RandomAccessData since it provides a safer version of the same functionality. I hoped that I would be able to move over more of the *Cloud* classes and their helpers but they are embedded part of coders. Nothing more can be done here until there is an official Beam representation of a coder decoupled from Dataflow CloudKnownTypes. You can merge this pull request into a Git repository by running: $ git pull https://github.com/lukecwik/incubator-beam beam151 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/243.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #243 commit 305c002ecbbe031458dcc2da66b4a2a0b5f41174 Author: Luke Cwik <lc...@google.com> Date: 2016-04-26T18:43:26Z [BEAM-151] Move over some more Dataflow specific classes. Note that users should use proto ByteString instead of RandomAccessData since it provides a safer version of the same functionality. I hoped that I would be able to move over more of the *Cloud* classes and their helpers but they are embedded part of coders. Nothing more can be done here until there is an official Beam representation of a coder decoupled from Dataflow CloudKnownTypes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request: [BEAM-202] Clean-up *CoderBase classe...
GitHub user lukecwik opened a pull request: https://github.com/apache/incubator-beam/pull/194 [BEAM-202] Clean-up *CoderBase classes Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- We are on Jackson 2.7.0 now. You can merge this pull request into a Git repository by running: $ git pull https://github.com/lukecwik/incubator-beam remove_coder_base Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/194.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #194 commit 11e8bfed4c73ec21dcf5a671ff0381ec1f39d8d4 Author: Luke Cwik <lc...@google.com> Date: 2016-04-15T23:53:23Z [BEAM-202] Clean-up *CoderBase classes since we are on a newer version of Jackson --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request: [BEAM-151] Add support for RunnableOn...
GitHub user lukecwik opened a pull request: https://github.com/apache/incubator-beam/pull/173 [BEAM-151] Add support for RunnableOnService validation tests to runners module Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace "" in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- Default implementation for Dataflow requires you to specify the runnableOnServicePipelineOptions system property with a valid project/runner/... to be able to execute on Google Cloud Dataflow. Default implementation for Flink and Spark are to have it disabled. Note the whitespace formatting fix on spark pom.xml You can merge this pull request into a Git repository by running: $ git pull https://github.com/lukecwik/incubator-beam dataflow-runner-7 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/173.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #173 commit 60d5471f74df1da5979dc31a6a7ee11c6fe1a850 Author: Luke Cwik <lc...@google.com> Date: 2016-04-13T00:49:07Z [BEAM-151] Add support for RunnableOnService validation tests to runners package Default implementation for Flink and Spark are to have it disabled. Default implementation for Dataflow requires you to specify the runnableOnServicePipelineOptions system property with a valid project/runner/... to be able to execute on Google Cloud Dataflow. commit 86e61c5f91f02340b42e3608b0b949f6384ae3d9 Author: Luke Cwik <lc...@google.com> Date: 2016-04-13T02:50:49Z [BEAM-151] !fixup Add the runnable-on-service-tests execution commit c1b4f3350b830f726e16cfe6db0d6b337022 Author: Luke Cwik <lc...@google.com> Date: 2016-04-13T02:54:45Z [BEAM-151] !fixup Fix comment in TestPipeline --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request: [BEAM-151] Enable integration tests u...
Github user lukecwik closed the pull request at: https://github.com/apache/incubator-beam/pull/170 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request: [BEAM-151] Enable integration tests u...
GitHub user lukecwik opened a pull request: https://github.com/apache/incubator-beam/pull/170 [BEAM-151] Enable integration tests underneath google-cloud-dataflow-java Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [X] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [X] Replace "" in the title with the actual Jira issue number, if there is one. - [X] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- Added surefire.dependencyToScan and surefire.testPipelineOptions configuration options surefire.dependencyToScan adds an additional dependency used to find tests. This allows for java-sdk-all to be added to all runners for running the @Category(RunnableOnService.class) tests. surefire.testPipelineOptions configures the pipeline options used while constructing instances of a TestPipeline. A json list of arguments is expected, for example: [ "--runner=MyRunner", "--myRunnerOption=myRunnerValue" ] You can merge this pull request into a Git repository by running: $ git pull https://github.com/lukecwik/incubator-beam dataflow-runner-6 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/170.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #170 commit ec90f511c3e3308ceaa06a34666c2b654b7ef59d Author: Luke Cwik <lc...@google.com> Date: 2016-04-12T01:58:00Z [BEAM-151] Enable integration tests underneath google-cloud-dataflow-java Added surefire.dependencyToScan and surefire.testPipelineOptions configuration options surefire.dependencyToScan adds an additional dependency used to find tests. This allows for java-sdk-all to be added to all runners for running the @Category(RunnableOnService.class) tests. surefire.testPipelineOptions configures the pipeline options used while constructing instances of a TestPipeline. A json list of arguments is expected, for example: [ "--runner=MyRunner", "--myRunnerOption=myRunnerValue" ] --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request: [BEAM-151] Move Dataflow runner to ne...
GitHub user lukecwik opened a pull request: https://github.com/apache/incubator-beam/pull/165 [BEAM-151] Move Dataflow runner to new Dataflow runner maven module Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [X] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [X] Replace "" in the title with the actual Jira issue number, if there is one. - [X] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- Note that Flink runner and Java examples both depend on the Dataflow runner still. So I needed to make the Dataflow runner a top level component to satisfy these dependencies. Also, we are currently shading within the Dataflow runner and using the same namespace to reduce the jar size. You can merge this pull request into a Git repository by running: $ git pull https://github.com/lukecwik/incubator-beam dataflow-runner-5 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/165.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #165 commit 0c45173b206b97825b6a9c24d095efba110eb405 Author: Luke Cwik <lc...@google.com> Date: 2016-04-11T19:49:07Z [BEAM-151] Move a large portion of the Dataflow runner to separate maven module Note that Flink runner and Java examples both depend on the Dataflow runner still. So I needed to make the Dataflow runner a top level component to satisfy these dependencies. commit 31ca6c214e0e90a5acd0cfc895b01a97b20a165c Author: Luke Cwik <lc...@google.com> Date: 2016-04-11T19:55:53Z [BEAM-151] !fixup Add comment explaining build order change --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request: [BEAM-151] Move maven archetypes buil...
GitHub user lukecwik opened a pull request: https://github.com/apache/incubator-beam/pull/158 [BEAM-151] Move maven archetypes build order to be after runners Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace "" in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- sdks/java/maven-archtypes has several dependencies on the DataflowPipelineRunner. Until these are refactored out or a released artifact exists, we need to modify the build order. You can merge this pull request into a Git repository by running: $ git pull https://github.com/lukecwik/incubator-beam dataflow-runner-5 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/158.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #158 commit ba4cdc5f3f753d6a3fd26f6b6d4e2f9c3d8f7e70 Author: Luke Cwik <lc...@google.com> Date: 2016-04-09T01:37:46Z [BEAM-151] Move maven archetypes build order to be after runners Currently sdks/java/maven-archetypes has several dependencies on the DataflowPipelineRunner. This is mainly throw the utility class called DataflowExampleUtils. This is a temporary workaround till examples get updated to use only beam SDK. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request: [BEAM-151] Remove DataflowPipelineRun...
GitHub user lukecwik opened a pull request: https://github.com/apache/incubator-beam/pull/154 [BEAM-151] Remove DataflowPipelineRunner refs in TestPipeline/PAssert Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace "" in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/lukecwik/incubator-beam dataflow-runner-4 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/154.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #154 commit e403691b01e8deabd424bade0127e873004627f9 Author: Luke Cwik <lc...@google.com> Date: 2016-04-08T22:07:52Z [BEAM-151] Remove dependence on DataflowPipelineRunner in PAssert/TestPipeline Added the ability for TestDataflowPipelineRunner to throw an AssertionError when the pipeline fails using the first job error message. This allows for moving Dataflow to a new package. commit d84b921176e926cd71c540285f07fd37f921b0ec Author: Luke Cwik <lc...@google.com> Date: 2016-04-08T22:49:53Z [BEAM-151] Expand javadoc in TestPipeline explaining usage Update TestDataflowPipelineRunner to capture all the failure messages. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request: [BEAM-95] Use snappy compression on o...
GitHub user lukecwik opened a pull request: https://github.com/apache/incubator-beam/pull/22 [BEAM-95] Use snappy compression on objects serialized via SerializabUtils R: @davorbonaci You can merge this pull request into a Git repository by running: $ git pull https://github.com/lukecwik/incubator-beam snappy Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/22.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22 commit ba426f53d692e3fcb78e8ae5bda6895392af1fce Author: Luke Cwik <lc...@google.com> Date: 2016-03-04T18:13:15Z [BEAM-95] Use snappy compression on objects serialized via SerializableUtils commit d8ed89fa8fcb1e6d2f3ed7825b5ff1863b7b78bc Author: Luke Cwik <lc...@google.com> Date: 2016-03-04T18:15:59Z [BEAM-95] Add dependency explicitly to SDK pom --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
