[GitHub] incubator-beam pull request #1674: [BEAM-1146] Decrease spark runner startup...
GitHub user aviemzur opened a pull request: https://github.com/apache/incubator-beam/pull/1674 [BEAM-1146] Decrease spark runner startup overhead Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- Replace finding all `Source` and `Coder` implementations for serialization registration with wrapper classes. You can merge this pull request into a Git repository by running: $ git pull https://github.com/aviemzur/incubator-beam decrease-spark-runner-startup-overhead Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1674.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1674 commit 8501cdc88ee9c89f643120e34381ec9bc2562965 Author: Aviem Zur <aviem...@gmail.com> Date: 2016-12-21T15:49:34Z [BEAM-1146] Decrease spark runner startup overhead Replace finding all `Source` and `Coder` implementations for serialization registration with wrapper classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1574: [BEAM-1126] Expose UnboundedSource split ...
Github user aviemzur closed the pull request at: https://github.com/apache/incubator-beam/pull/1574 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1613: [BEAM-1144] Spark runner fails to deseria...
GitHub user aviemzur opened a pull request: https://github.com/apache/incubator-beam/pull/1613 [BEAM-1144] Spark runner fails to deserialize MicrobatchSource in cluster mode Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/aviemzur/incubator-beam cnf-deserialize-issue Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1613.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1613 commit 774d3b62a741ae892041a7ff40d7f78b3f6b2f3f Author: Aviem Zur <aviem...@gmail.com> Date: 2016-12-14T13:19:39Z [BEAM-1144] Spark runner fails to deserialize MicrobatchSource in cluster mode --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1594: [BEAM-1145] Remove classifier from shaded...
GitHub user aviemzur opened a pull request: https://github.com/apache/incubator-beam/pull/1594 [BEAM-1145] Remove classifier from shaded spark runner artifact Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/aviemzur/incubator-beam remove-spark-runner-shaded-artifact-classifier Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1594.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1594 commit 0b4a65374629b6490b68db3348de427bc2f14e0a Author: Aviem Zur <aviem...@gmail.com> Date: 2016-12-13T15:22:46Z [BEAM-1145] Remove classifier from shaded spark runner artifact --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1574: [BEAM-1126] Expose UnboundedSource split ...
GitHub user aviemzur opened a pull request: https://github.com/apache/incubator-beam/pull/1574 [BEAM-1126] Expose UnboundedSource split backlog in number of events Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/aviemzur/incubator-beam backlog-num-events Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1574.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1574 commit d550b9350bba3e7799f00ce09234243dc67a699c Author: Aviem Zur <aviem...@gmail.com> Date: 2016-12-10T17:06:06Z [BEAM-1126] Expose UnboundedSource split backlog in number of events --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1552: [BEAM-921] spark-runner: register sources...
GitHub user aviemzur opened a pull request: https://github.com/apache/incubator-beam/pull/1552 [BEAM-921] spark-runner: register sources and coders to serialize with java serializer Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/aviemzur/incubator-beam kryo-java-serialize-coders-sources Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1552.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1552 commit 53f2bbc9a26b8734cc76a3a6044849538f75cc29 Author: Aviem Zur <aviem...@gmail.com> Date: 2016-12-08T13:07:06Z [BEAM-921] spark-runner: register sources and coders to serialize with java serializer --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1450: [BEAM-1052] Add InputDStream id to Microb...
GitHub user aviemzur opened a pull request: https://github.com/apache/incubator-beam/pull/1450 [BEAM-1052] Add InputDStream id to MicrobatchSource hashcode. Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- Done to avoid collisions between splits of different sources. You can merge this pull request into a Git repository by running: $ git pull https://github.com/aviemzur/incubator-beam add-inputdstream-id-to-microbatchsource-hash-code Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1450.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1450 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #1291: [BEAM-762] Unify spark-runner EvaluationC...
GitHub user aviemzur opened a pull request: https://github.com/apache/incubator-beam/pull/1291 [BEAM-762] Unify spark-runner EvaluationContext and StreamingEvaluationContext Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/aviemzur/incubator-beam refactor-spark-runner-evaluation-context Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1291.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1291 commit 7f6851d6eb1a17de85c3d87407da3a678a281b83 Author: Aviem Zur <aviem...@gmail.com> Date: 2016-11-07T12:51:19Z Unify spark-runner EvaluationContext and StreamingEvaluationContext --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #941: [BEAM-626] Changes in AvroCoder serializat...
Github user aviemzur closed the pull request at: https://github.com/apache/incubator-beam/pull/941 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #941: [BEAM-626] Changes in AvroCoder serializat...
GitHub user aviemzur opened a pull request: https://github.com/apache/incubator-beam/pull/941 [BEAM-626] Changes in AvroCoder serialization so it can serialize in Kryo Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- Changes in AvroCoder serialization so it can serialize in Kryo. Changed writeReplace and readResolve to lazy initialized members. You can merge this pull request into a Git repository by running: $ git pull https://github.com/aviemzur/incubator-beam avro-coder-serialization-changes Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/941.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #941 commit e83658fd6a394b90e8b46b3e61d879e56ee7a767 Author: Aviem Zur <aviem...@gmail.com> Date: 2016-09-08T08:21:41Z Changes in AvroCoder serialization so it can serialize in Kryo --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #900: [BEAM-599] Return KafkaIO getWatermark log...
GitHub user aviemzur opened a pull request: https://github.com/apache/incubator-beam/pull/900 [BEAM-599] Return KafkaIO getWatermark log line in debug mode Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- KafkaIO getWatermark log line was removed in https://github.com/apache/incubator-beam/pull/859 I actually found this log line useful, instead of removing it completely, returned this log line but change the log level to 'debug'? You can merge this pull request into a Git repository by running: $ git pull https://github.com/aviemzur/incubator-beam return-kafka-io-getwatermark-log Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/900.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #900 commit 6742567a49e58abb44e9261f43baf73249a54647 Author: Aviem Zur <aviem...@gmail.com> Date: 2016-08-28T08:41:58Z Returned KafkaIO getWatermark log line in debug mode --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #897: [BEAM-584] Support Verifiers in TestSparkR...
GitHub user aviemzur opened a pull request: https://github.com/apache/incubator-beam/pull/897 [BEAM-584] Support Verifiers in TestSparkRunner You can merge this pull request into a Git repository by running: $ git pull https://github.com/aviemzur/incubator-beam test-spark-runner-verifiers Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/897.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #897 commit 84699ce15c315d4d61f515d8114fc6d3761534ab Author: Aviem Zur <aviem...@gmail.com> Date: 2016-08-25T14:23:07Z Support Verifiers in TestSparkRunner --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
