[jira] [Commented] (BEAM-570) Update AvroSource to support more compression types

[ https://issues.apache.org/jira/browse/BEAM-570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15547423#comment-15547423 ] ASF GitHub Bot commented on BEAM-570: - GitHub user katsiapis opened a pull request:

[GitHub] incubator-beam pull request #1053: [BEAM-570] Title of the pull request

GitHub user katsiapis opened a pull request: https://github.com/apache/incubator-beam/pull/1053 [BEAM-570] Title of the pull request - Getting rid of CompressionTypes.ZLIB and CompressionTypes.NO_COMPRESSION. - Introducing BZIP2 compression in analogy to Dataflow Java's BZIP2,

[GitHub] incubator-beam pull request #1052: Delete DatastoreWordCount

GitHub user dhalperi opened a pull request: https://github.com/apache/incubator-beam/pull/1052 Delete DatastoreWordCount Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like:

[GitHub] incubator-beam pull request #1051: Add RootTransformEvaluatorFactory

GitHub user tgroh opened a pull request: https://github.com/apache/incubator-beam/pull/1051 Add RootTransformEvaluatorFactory Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like:

[GitHub] incubator-beam pull request #1050: Makes FileBasedSink use a temporary direc...

GitHub user jkff opened a pull request: https://github.com/apache/incubator-beam/pull/1050 Makes FileBasedSink use a temporary directory When writing to `/path/to/foo`, temporary files would be written to `/path/too/foo-temp-$uid` (or something like that), i.e. as siblings of the

[jira] [Commented] (BEAM-498) Make DoFnWithContext the new DoFn

[ https://issues.apache.org/jira/browse/BEAM-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15546827#comment-15546827 ] ASF GitHub Bot commented on BEAM-498: - GitHub user kennknowles opened a pull request:

[GitHub] incubator-beam pull request #1049: [BEAM-498] Add DoFnInvoker for OldDoFn, f...

GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1049 [BEAM-498] Add DoFnInvoker for OldDoFn, for migration ease Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure

[GitHub] incubator-beam pull request #1046: Add equality methods to range source.

Github user robertwb closed the pull request at: https://github.com/apache/incubator-beam/pull/1046 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[1/2] incubator-beam git commit: Add equality methods to range source.

Repository: incubator-beam Updated Branches: refs/heads/python-sdk 3a69db0c5 -> 99a33ecdb Add equality methods to range source. Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/837c5aa3 Tree:

[2/2] incubator-beam git commit: Closes #1046

Closes #1046 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/99a33ecd Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/99a33ecd Diff:

[jira] [Commented] (BEAM-540) Dataflow streaming jobs running on windmill do not need data disks

[ https://issues.apache.org/jira/browse/BEAM-540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15546638#comment-15546638 ] Pei He commented on BEAM-540: - this can close? > Dataflow streaming jobs running on windmill do not need data

[jira] [Updated] (BEAM-702) Simple pattern for per-bundle and per-DoFn Closeable resources

[ https://issues.apache.org/jira/browse/BEAM-702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pei He updated BEAM-702: Component/s: beam-model > Simple pattern for per-bundle and per-DoFn Closeable resources >

[GitHub] incubator-beam pull request #1048: Converts KafkaIO builders to @AutoValue

GitHub user jkff opened a pull request: https://github.com/apache/incubator-beam/pull/1048 Converts KafkaIO builders to @AutoValue This is in the same spirit as https://github.com/apache/incubator-beam/pull/1031, https://github.com/jbonofre/incubator-beam/pull/1 and

[GitHub] incubator-beam pull request #1047: Little fixes to LatestFnTest

GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1047 Little fixes to LatestFnTest Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted

[jira] [Closed] (BEAM-545) Pipelines and their executions naming changes

[ https://issues.apache.org/jira/browse/BEAM-545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pei He closed BEAM-545. --- Resolution: Fixed Fix Version/s: 0.3.0-incubating > Pipelines and their executions naming changes >

[jira] [Updated] (BEAM-545) Pipelines and their executions naming changes

[ https://issues.apache.org/jira/browse/BEAM-545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pei He updated BEAM-545: Component/s: sdk-java-core > Pipelines and their executions naming changes >

[jira] [Issue Comment Deleted] (BEAM-696) Side-Inputs non-deterministic with merging main-input windows

[ https://issues.apache.org/jira/browse/BEAM-696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-696: --- Comment: was deleted (was: So it runs once, on the merged window ? That happens in the bundle level, correct ?

[jira] [Closed] (BEAM-433) Make Beam examples runners agnostic

[ https://issues.apache.org/jira/browse/BEAM-433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pei He closed BEAM-433. --- Resolution: Fixed Fix Version/s: 0.3.0-incubating > Make Beam examples runners agnostic >

[jira] [Commented] (BEAM-696) Side-Inputs non-deterministic with merging main-input windows

[ https://issues.apache.org/jira/browse/BEAM-696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15546539#comment-15546539 ] Amit Sela commented on BEAM-696: So it runs once, on the merged window ? That happens in the bundle level,

[jira] [Commented] (BEAM-696) Side-Inputs non-deterministic with merging main-input windows

[ https://issues.apache.org/jira/browse/BEAM-696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15546538#comment-15546538 ] Amit Sela commented on BEAM-696: So it runs once, on the merged window ? That happens in the bundle level,

[jira] [Closed] (BEAM-483) Generated job names easily collide

[ https://issues.apache.org/jira/browse/BEAM-483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pei He closed BEAM-483. --- Resolution: Fixed Fix Version/s: 0.3.0-incubating > Generated job names easily collide >

[jira] [Resolved] (BEAM-554) Dataflow runner to support bounded writes in streaming mode.

[ https://issues.apache.org/jira/browse/BEAM-554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pei He resolved BEAM-554. - Resolution: Fixed Fix Version/s: 0.3.0-incubating > Dataflow runner to support bounded writes in streaming

[jira] [Resolved] (BEAM-572) Remove Spark references in WordCount

[ https://issues.apache.org/jira/browse/BEAM-572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pei He resolved BEAM-572. - Resolution: Fixed Fix Version/s: 0.3.0-incubating > Remove Spark references in WordCount >

[GitHub] incubator-beam pull request #1046: Add equality methods to range source.

GitHub user robertwb opened a pull request: https://github.com/apache/incubator-beam/pull/1046 Add equality methods to range source. Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is

[jira] [Commented] (BEAM-586) CombineFns with no inputs should produce no outputs when used as a main input

[ https://issues.apache.org/jira/browse/BEAM-586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15546500#comment-15546500 ] Pei He commented on BEAM-586: - Is that fair to say triggering defines whether and when CombineFns output, and

[jira] [Commented] (BEAM-561) Add WindowedWordCountIT

[ https://issues.apache.org/jira/browse/BEAM-561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15546489#comment-15546489 ] ASF GitHub Bot commented on BEAM-561: - GitHub user markflyhigh opened a pull request:

[GitHub] incubator-beam pull request #1045: [BEAM-561] Add Streaming IT to Jenkins Pr...

GitHub user markflyhigh opened a pull request: https://github.com/apache/incubator-beam/pull/1045 [BEAM-561] Add Streaming IT to Jenkins Pre-commit Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR

[jira] [Commented] (BEAM-614) File compression/decompression should support auto detection

[ https://issues.apache.org/jira/browse/BEAM-614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15546481#comment-15546481 ] ASF GitHub Bot commented on BEAM-614: - Github user asfgit closed the pull request at:

[GitHub] incubator-beam pull request #1002: [BEAM-614] Updates FileBasedSource to sup...

Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1002 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[2/2] incubator-beam git commit: Closes #1002

Closes #1002 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3a69db0c Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3a69db0c Diff:

[1/2] incubator-beam git commit: Updates filebasedsource to support CompressionType.AUTO.

Repository: incubator-beam Updated Branches: refs/heads/python-sdk 731a77152 -> 3a69db0c5 Updates filebasedsource to support CompressionType.AUTO. Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/2126a34c

[jira] [Commented] (BEAM-696) Side-Inputs non-deterministic with merging main-input windows

[ https://issues.apache.org/jira/browse/BEAM-696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15546473#comment-15546473 ] Kenneth Knowles commented on BEAM-696: -- Ben points out (in person) that this latter spec is already the

[jira] [Updated] (BEAM-605) Create BigQuery Verifier

[ https://issues.apache.org/jira/browse/BEAM-605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pei He updated BEAM-605: Affects Version/s: Not applicable Component/s: testing > Create BigQuery Verifier >

[jira] [Issue Comment Deleted] (BEAM-702) Simple pattern for per-bundle and per-DoFn Closeable resources

[ https://issues.apache.org/jira/browse/BEAM-702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pei He updated BEAM-702: Comment: was deleted (was: 1. If there are failures in one bundle, we can easily call teardown()s of DoFns

[jira] [Commented] (BEAM-702) Simple pattern for per-bundle and per-DoFn Closeable resources

[ https://issues.apache.org/jira/browse/BEAM-702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15546412#comment-15546412 ] Pei He commented on BEAM-702: - 1. If there are failures in one bundle, we can easily call teardown()s of DoFns

[jira] [Commented] (BEAM-702) Simple pattern for per-bundle and per-DoFn Closeable resources

[ https://issues.apache.org/jira/browse/BEAM-702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15546411#comment-15546411 ] Pei He commented on BEAM-702: - 1. If there are failures in one bundle, we can easily call teardown()s of DoFns

[jira] [Resolved] (BEAM-703) SingletonViewFn might exhaust defaultValue if it's serialized after being used.

[ https://issues.apache.org/jira/browse/BEAM-703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles resolved BEAM-703. -- Resolution: Fixed Fix Version/s: 0.3.0-incubating > SingletonViewFn might exhaust

[GitHub] incubator-beam pull request #1040: [BEAM-703] SingletonViewFn might exhaust ...

Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1040 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[jira] [Commented] (BEAM-703) SingletonViewFn might exhaust defaultValue if it's serialized after being used.

[ https://issues.apache.org/jira/browse/BEAM-703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15546394#comment-15546394 ] ASF GitHub Bot commented on BEAM-703: - Github user asfgit closed the pull request at:

[1/2] incubator-beam git commit: Avoid losing the encoded defaultValue.

Repository: incubator-beam Updated Branches: refs/heads/master 8462acbcb -> 087dcef1e Avoid losing the encoded defaultValue. Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/0bd97efc Tree:

[2/2] incubator-beam git commit: This closes #1040

This closes #1040 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/087dcef1 Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/087dcef1 Diff:

[jira] [Commented] (BEAM-702) Simple pattern for per-bundle and per-DoFn Closeable resources

[ https://issues.apache.org/jira/browse/BEAM-702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15546300#comment-15546300 ] Eugene Kirpichov commented on BEAM-702: --- Hmm, I didn't realize that we don't call finishBundle and

[jira] [Commented] (BEAM-25) Add user-ready API for interacting with state

[ https://issues.apache.org/jira/browse/BEAM-25?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15546294#comment-15546294 ] ASF GitHub Bot commented on BEAM-25: GitHub user kennknowles opened a pull request:

[GitHub] incubator-beam pull request #1044: [BEAM-25] Refactor StateSpec out of State...

GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1044 [BEAM-25] Refactor StateSpec out of StateTag Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title

[jira] [Comment Edited] (BEAM-702) Simple pattern for per-bundle and per-DoFn Closeable resources

[ https://issues.apache.org/jira/browse/BEAM-702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15546277#comment-15546277 ] Pei He edited comment on BEAM-702 at 10/4/16 6:47 PM: -- The feature request is for

[jira] [Commented] (BEAM-702) Simple pattern for per-bundle and per-DoFn Closeable resources

[ https://issues.apache.org/jira/browse/BEAM-702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15546277#comment-15546277 ] Pei He commented on BEAM-702: - The feature request is for something like this? class MyDoFn extends DoFn

[jira] [Commented] (BEAM-696) Side-Inputs non-deterministic with merging main-input windows

[ https://issues.apache.org/jira/browse/BEAM-696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15546263#comment-15546263 ] Kenneth Knowles commented on BEAM-696: -- The SDK needs to lay out a spec, which I think is what Pei is

[jira] [Commented] (BEAM-703) SingletonViewFn might exhaust defaultValue if it's serialized after being used.

[ https://issues.apache.org/jira/browse/BEAM-703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15546196#comment-15546196 ] Kenneth Knowles commented on BEAM-703: -- I think that might have been an attempt to save memory by

[GitHub] incubator-beam pull request #1043: Datastore Connector prototype using googl...

GitHub user vikkyrk opened a pull request: https://github.com/apache/incubator-beam/pull/1043 Datastore Connector prototype using google-cloud python client Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure

[GitHub] incubator-beam pull request #1042: BigQueryIO: port trivial fixes from Dataf...

GitHub user peihe opened a pull request: https://github.com/apache/incubator-beam/pull/1042 BigQueryIO: port trivial fixes from Dataflow version. Will rebase once PR-1039 is merged You can merge this pull request into a Git repository by running: $ git pull

[GitHub] incubator-beam pull request #1041: Add a default bucket to Dataflow runner

GitHub user sammcveety opened a pull request: https://github.com/apache/incubator-beam/pull/1041 Add a default bucket to Dataflow runner There is a dependency failure that I'm not sure how to debug; all other tests pass. @tgroh could you please take a look? ---

[jira] [Updated] (BEAM-703) SingletonViewFn might exhaust defaultValue if it's serialized after being used.

[ https://issues.apache.org/jira/browse/BEAM-703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-703: --- Summary: SingletonViewFn might exhaust defaultValue if it's serialized after being used. (was:

[jira] [Commented] (BEAM-703) SingletonViewFn might exhaust defaultValue if it's serialized after being used.

[ https://issues.apache.org/jira/browse/BEAM-703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15545665#comment-15545665 ] ASF GitHub Bot commented on BEAM-703: - GitHub user amitsela opened a pull request:

[GitHub] incubator-beam pull request #1040: [BEAM-703] SingletonViewFn might exhaust ...

GitHub user amitsela opened a pull request: https://github.com/apache/incubator-beam/pull/1040 [BEAM-703] SingletonViewFn might exhaust defaultValue if it's serialized after being used. Be sure to do all of the following to help us incorporate your contribution quickly and

[jira] [Created] (BEAM-703) SingletonViewFn might exhaust defaultValue if it's serialized after used.

Amit Sela created BEAM-703: -- Summary: SingletonViewFn might exhaust defaultValue if it's serialized after used. Key: BEAM-703 URL: https://issues.apache.org/jira/browse/BEAM-703 Project: Beam

[jira] [Commented] (BEAM-696) Side-Inputs non-deterministic with merging main-input windows

[ https://issues.apache.org/jira/browse/BEAM-696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15544926#comment-15544926 ] Amit Sela commented on BEAM-696: [~pei...@gmail.com] shouldn't the SDK take care/enforce this behaviour and