[jira] [Updated] (BEAM-559) DoFnTester should handle Setup / TearDown

[ https://issues.apache.org/jira/browse/BEAM-559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikas Kedigehalli updated BEAM-559: --- Assignee: Daniel Halperin > DoFnTester should handle Setup / TearDown >

[jira] [Created] (BEAM-559) DoFnTester should handle Setup / TearDown

Vikas Kedigehalli created BEAM-559: -- Summary: DoFnTester should handle Setup / TearDown Key: BEAM-559 URL: https://issues.apache.org/jira/browse/BEAM-559 Project: Beam Issue Type: Improvemen

[jira] [Closed] (BEAM-534) README.md contains dead link

[ https://issues.apache.org/jira/browse/BEAM-534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Halperin closed BEAM-534. Resolution: Fixed Fix Version/s: Not applicable > README.md contains dead link > --

[jira] [Closed] (BEAM-326) A minor update to KafkaIO javadoc

[ https://issues.apache.org/jira/browse/BEAM-326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Halperin closed BEAM-326. Resolution: Fixed Fix Version/s: 0.2.0-incubating > A minor update to KafkaIO javadoc > ---

[jira] [Closed] (BEAM-306) Make java-only PubsubIO work in InProcessRunner

[ https://issues.apache.org/jira/browse/BEAM-306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Halperin closed BEAM-306. Resolution: Fixed Fix Version/s: 0.1.0-incubating > Make java-only PubsubIO work in InProcessRu

[jira] [Updated] (BEAM-79) Gearpump runner

[ https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Halperin updated BEAM-79: Component/s: (was: runner-ideas) runner-gearpump > Gearpump runner > -

[jira] [Closed] (BEAM-459) Add BigInteger to TypeDescriptors

[ https://issues.apache.org/jira/browse/BEAM-459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Halperin closed BEAM-459. Resolution: Fixed Fix Version/s: 0.2.0-incubating > Add BigInteger to TypeDescriptors > ---

[jira] [Closed] (BEAM-46) Unbounded sink for Google Cloud Bigtable

[ https://issues.apache.org/jira/browse/BEAM-46?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Halperin closed BEAM-46. --- Resolution: Fixed Fix Version/s: 0.3.0-incubating > Unbounded sink for Google Cloud Bigtable > ---

[jira] [Closed] (BEAM-38) Avoid repeated DoFn deserialization

[ https://issues.apache.org/jira/browse/BEAM-38?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Halperin closed BEAM-38. --- Resolution: Fixed Fix Version/s: 0.2.0-incubating > Avoid repeated DoFn deserialization >

[jira] [Updated] (BEAM-31) When triggers are changed via pipeline update, stale finished triggers data applied

[ https://issues.apache.org/jira/browse/BEAM-31?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Halperin updated BEAM-31: Component/s: (was: runner-core) runner-dataflow > When triggers are changed via pi

[jira] [Closed] (BEAM-122) GlobalWindow and allowedLateness can cause inconsistent timer interpretation

[ https://issues.apache.org/jira/browse/BEAM-122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Halperin closed BEAM-122. Resolution: Fixed Fix Version/s: 0.1.0-incubating > GlobalWindow and allowedLateness can cause

[jira] [Commented] (BEAM-153) Support timeout in runner API

[ https://issues.apache.org/jira/browse/BEAM-153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15422039#comment-15422039 ] Daniel Halperin commented on BEAM-153: -- Pei, this seems pretty well covered by some of

[jira] [Closed] (BEAM-86) CountingSource should expose PTransform> rather than sources directly

[ https://issues.apache.org/jira/browse/BEAM-86?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Halperin closed BEAM-86. --- Resolution: Fixed Fix Version/s: 0.1.0-incubating > CountingSource should expose PTransform> rathe

[GitHub] incubator-beam pull request #835: Fix NPE in BigQueryIO.TransformingReader w...

GitHub user peihe opened a pull request: https://github.com/apache/incubator-beam/pull/835 Fix NPE in BigQueryIO.TransformingReader when it is on top of an unsplittable reader. You can merge this pull request into a Git repository by running: $ git pull https://github.c

Jenkins build is back to stable : beam_PostCommit_RunnableOnService_GoogleCloudDataflow #961

See

[GitHub] incubator-beam pull request #832: [BEAM-557] Exclude guava-testlib from shad...

Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/832 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[jira] [Commented] (BEAM-557) Test-scoped dependencies should be excluded from shading package relocation

[ https://issues.apache.org/jira/browse/BEAM-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421926#comment-15421926 ] ASF GitHub Bot commented on BEAM-557: - Github user asfgit closed the pull request at:

[1/2] incubator-beam git commit: Exclude guava-testlib from shading relocation

Repository: incubator-beam Updated Branches: refs/heads/master ab104a16c -> f1935d5d1 Exclude guava-testlib from shading relocation Previously, guava-testlib guava-testlib was being relocated as part of the shading process, but test-scope dependencies aren't bundled in the uber-jar. As a resul

[2/2] incubator-beam git commit: [BEAM-557] Exclude guava-testlib from shading relocation

[BEAM-557] Exclude guava-testlib from shading relocation This closes #832 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f1935d5d Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f1935d5d Di

[jira] [Commented] (BEAM-558) DataflowRunner does not support setup/teardown

[ https://issues.apache.org/jira/browse/BEAM-558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421903#comment-15421903 ] Daniel Halperin commented on BEAM-558: -- RunnableOnService tests: ParDoTest.java and Par

[jira] [Created] (BEAM-558) DataflowRunner does not support setup/teardown

Daniel Halperin created BEAM-558: Summary: DataflowRunner does not support setup/teardown Key: BEAM-558 URL: https://issues.apache.org/jira/browse/BEAM-558 Project: Beam Issue Type: Bug

[GitHub] incubator-beam pull request #834: BigtableIO StreamingWrite Setup and Teardo...

GitHub user ianzhou1 opened a pull request: https://github.com/apache/incubator-beam/pull/834 BigtableIO StreamingWrite Setup and Teardown You can merge this pull request into a Git repository by running: $ git pull https://github.com/ianzhou1/incubator-beam BigtableStreaming

[1/2] incubator-beam git commit: Exclude ParDoTest from Dataflow @RunnableOnService

Repository: incubator-beam Updated Branches: refs/heads/master 7c680079b -> ab104a16c Exclude ParDoTest from Dataflow @RunnableOnService Until we implement it for Dataflow runner. Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/as

[GitHub] incubator-beam pull request #831: Exclude ParDoTest from Dataflow @RunnableO...

Github user dhalperi closed the pull request at: https://github.com/apache/incubator-beam/pull/831 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the featu

[2/2] incubator-beam git commit: Closes #831

Closes #831 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/ab104a16 Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/ab104a16 Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/

[jira] [Commented] (BEAM-550) Datastore should support writes for Unbounded PCollections

[ https://issues.apache.org/jira/browse/BEAM-550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421843#comment-15421843 ] ASF GitHub Bot commented on BEAM-550: - Github user vikkyrk closed the pull request at:

[GitHub] incubator-beam pull request #825: [BEAM-550] DatastoreIO Write PTranform for...

Github user vikkyrk closed the pull request at: https://github.com/apache/incubator-beam/pull/825 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the featur

[jira] [Commented] (BEAM-550) Datastore should support writes for Unbounded PCollections

[ https://issues.apache.org/jira/browse/BEAM-550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421823#comment-15421823 ] ASF GitHub Bot commented on BEAM-550: - GitHub user vikkyrk opened a pull request: h

[GitHub] incubator-beam pull request #833: [BEAM-550] DatastoreIO Sink as ParDo

GitHub user vikkyrk opened a pull request: https://github.com/apache/incubator-beam/pull/833 [BEAM-550] DatastoreIO Sink as ParDo Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted li

[jira] [Commented] (BEAM-557) Test-scoped dependencies should be excluded from shading package relocation

[ https://issues.apache.org/jira/browse/BEAM-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421818#comment-15421818 ] ASF GitHub Bot commented on BEAM-557: - GitHub user swegner opened a pull request: h

[GitHub] incubator-beam pull request #832: [BEAM-557] Exclude guava-testlib from shad...

GitHub user swegner opened a pull request: https://github.com/apache/incubator-beam/pull/832 [BEAM-557] Exclude guava-testlib from shading relocation Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR t

[2/2] incubator-beam git commit: Improve error handling in gcsio.py

Improve error handling in gcsio.py Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/4d6da9cf Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/4d6da9cf Diff: http://git-wip-us.apache.org/repos/a

[1/2] incubator-beam git commit: Closes #830

Repository: incubator-beam Updated Branches: refs/heads/python-sdk 1d53e28e9 -> 3b40945e9 Closes #830 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3b40945e Tree: http://git-wip-us.apache.org/repos/asf/

[1/2] incubator-beam git commit: Revert the changes to the accidentaly reverted files

Repository: incubator-beam Updated Branches: refs/heads/python-sdk b28e71979 -> 1d53e28e9 Revert the changes to the accidentaly reverted files Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a4760fe3 Tree

[GitHub] incubator-beam pull request #829: Revert the changes to the accidentaly reve...

Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/829 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[2/2] incubator-beam git commit: Closes #829

Closes #829 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/1d53e28e Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/1d53e28e Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/

[GitHub] incubator-beam pull request #831: Exclude ParDoTest from Dataflow @RunnableO...

GitHub user dhalperi opened a pull request: https://github.com/apache/incubator-beam/pull/831 Exclude ParDoTest from Dataflow @RunnableOnService Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title

Jenkins build became unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #960

See

[GitHub] incubator-beam pull request #830: Improve error handling in gcsio.py.

GitHub user charlesccychen opened a pull request: https://github.com/apache/incubator-beam/pull/830 Improve error handling in gcsio.py. You can merge this pull request into a Git repository by running: $ git pull https://github.com/charlesccychen/incubator-beam gcsio-fixes Al

[jira] [Created] (BEAM-557) Test-scoped dependencies should be excluded from shading package relocation

Scott Wegner created BEAM-557: - Summary: Test-scoped dependencies should be excluded from shading package relocation Key: BEAM-557 URL: https://issues.apache.org/jira/browse/BEAM-557 Project: Beam

[GitHub] incubator-beam pull request #829: Revert the changes to the accidentaly reve...

GitHub user aaltay opened a pull request: https://github.com/apache/incubator-beam/pull/829 Revert the changes to the accidentaly reverted files You can merge this pull request into a Git repository by running: $ git pull https://github.com/aaltay/incubator-beam und Alternat

incubator-beam git commit: Closes #537

Repository: incubator-beam Updated Branches: refs/heads/python-sdk 76de6109f -> b28e71979 Closes #537 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b28e7197 Tree: http://git-wip-us.apache.org/repos/asf/

incubator-beam git commit: [BEAM-378] integrate setuptools in Maven build

Repository: incubator-beam Updated Branches: refs/heads/python-sdk 18ce31c1d -> 76de6109f [BEAM-378] integrate setuptools in Maven build This PR provide an initial integration of the Python SDK in the Maven build, relying on the exec-maven-plugin to call setuptools (BEAM-378). Project: http:

[jira] [Commented] (BEAM-452) Implement DoFn per-instance setup and teardown methods

[ https://issues.apache.org/jira/browse/BEAM-452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421690#comment-15421690 ] ASF GitHub Bot commented on BEAM-452: - Github user asfgit closed the pull request at:

[1/5] incubator-beam git commit: Closes #690

Repository: incubator-beam Updated Branches: refs/heads/master 0b1f66421 -> 7c680079b Closes #690 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/7c680079 Tree: http://git-wip-us.apache.org/repos/asf/incu

[4/5] incubator-beam git commit: Move ParDo Lifecycle tests to their own file

Move ParDo Lifecycle tests to their own file These tests are not yet functional in all runners, and this makes them easier to ignore. Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/29cbdceb Tree: http://git

[2/5] incubator-beam git commit: Add DoFn @Setup and @Teardown

Add DoFn @Setup and @Teardown Methods annotated with these annotations are used to perform expensive setup work and clean up a DoFn after another method throws an exception or the DoFn is discarded. Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apa

[3/5] incubator-beam git commit: Add TransformEvaluatorFactory#cleanup

Add TransformEvaluatorFactory#cleanup This cleans up any state stored within the Transform Evaluator Factory. Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/12b19677 Tree: http://git-wip-us.apache.org/repos

[GitHub] incubator-beam pull request #690: [BEAM-452] Add DoFn setup and teardown met...

Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/690 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[5/5] incubator-beam git commit: Replace CloningThreadLocal with DoFnLifecycleManager

Replace CloningThreadLocal with DoFnLifecycleManager This is a more focused interface that interacts with a DoFn before it is available for use and after it has completed and the reference is lost. It is required to properly support setup and teardown, as the fields in a ThreadLocal cannot all be

[jira] [Created] (BEAM-556) typo in documentation

Frank Yellin created BEAM-556: - Summary: typo in documentation Key: BEAM-556 URL: https://issues.apache.org/jira/browse/BEAM-556 Project: Beam Issue Type: Bug Components: sdk-py

[jira] [Created] (BEAM-555) Documentation in BiqQueryIO.java has awkward cut-and-paste error.

Frank Yellin created BEAM-555: - Summary: Documentation in BiqQueryIO.java has awkward cut-and-paste error. Key: BEAM-555 URL: https://issues.apache.org/jira/browse/BEAM-555 Project: Beam Issue T

[jira] [Commented] (BEAM-554) Dataflow runner to support bounded writes in streaming mode.

[ https://issues.apache.org/jira/browse/BEAM-554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421523#comment-15421523 ] ASF GitHub Bot commented on BEAM-554: - GitHub user peihe opened a pull request: htt

[GitHub] incubator-beam pull request #828: [BEAM-554] Set Gcs upload buffer size to 1...

GitHub user peihe opened a pull request: https://github.com/apache/incubator-beam/pull/828 [BEAM-554] Set Gcs upload buffer size to 1M in DataflowRunner streaming mode This PR solves OOMs when DataflowRunner streaming mode writes hundreds files in parallel. You can merge this pull

[jira] [Updated] (BEAM-554) Dataflow runner to support bounded writes in streaming mode.

[ https://issues.apache.org/jira/browse/BEAM-554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pei He updated BEAM-554: Assignee: Pei He (was: Davor Bonaci) Summary: Dataflow runner to support bounded writes in streaming mode. (was

[jira] [Created] (BEAM-554) Dataflow runner to support bounded in streaming mode.

Pei He created BEAM-554: --- Summary: Dataflow runner to support bounded in streaming mode. Key: BEAM-554 URL: https://issues.apache.org/jira/browse/BEAM-554 Project: Beam Issue Type: Improvement

[jira] [Resolved] (BEAM-522) Update FileSink.finalize_write() to be idempotent

[ https://issues.apache.org/jira/browse/BEAM-522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Jayalath resolved BEAM-522. - Resolution: Fixed Fix Version/s: Not applicable > Update FileSink.finalize_write() to

[jira] [Created] (BEAM-553) Add a custom text source

Chamikara Jayalath created BEAM-553: --- Summary: Add a custom text source Key: BEAM-553 URL: https://issues.apache.org/jira/browse/BEAM-553 Project: Beam Issue Type: New Feature Com

[jira] [Closed] (BEAM-552) TimeZone.getTimeZone has performance issue.

[ https://issues.apache.org/jira/browse/BEAM-552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Satish Subhashrao Saley closed BEAM-552. Resolution: Invalid Fix Version/s: Not applicable Sorry, BEAM got auto selected

[jira] [Created] (BEAM-552) TimeZone.getTimeZone has performance issue.

Satish Subhashrao Saley created BEAM-552: Summary: TimeZone.getTimeZone has performance issue. Key: BEAM-552 URL: https://issues.apache.org/jira/browse/BEAM-552 Project: Beam Issue Ty

[jira] [Commented] (BEAM-435) DirectRunner GBK -- task per key?

[ https://issues.apache.org/jira/browse/BEAM-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421262#comment-15421262 ] Thomas Groh commented on BEAM-435: -- This is resolved w.r.t. Write with https://github.com/

[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions

[ https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421067#comment-15421067 ] ASF GitHub Bot commented on BEAM-551: - GitHub user sammcveety opened a pull request:

[GitHub] incubator-beam pull request #827: [BEAM-551] Add ValueProvider class

GitHub user sammcveety opened a pull request: https://github.com/apache/incubator-beam/pull/827 [BEAM-551] Add ValueProvider class Hi @lukecwik , can you please take a look? Currently implemented for ValueProvider only. Tests demonstrate intended functionality, once the qu

[jira] [Created] (BEAM-551) Support Dynamic PipelineOptions

Sam McVeety created BEAM-551: Summary: Support Dynamic PipelineOptions Key: BEAM-551 URL: https://issues.apache.org/jira/browse/BEAM-551 Project: Beam Issue Type: New Feature Components

[jira] [Closed] (BEAM-492) Spark runner should call Pipeline.run() instead of SparkRunner.run()

[ https://issues.apache.org/jira/browse/BEAM-492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela closed BEAM-492. -- Resolution: Fixed > Spark runner should call Pipeline.run() instead of SparkRunner.run() > ---

[jira] [Updated] (BEAM-492) Spark runner should call Pipeline.run() instead of SparkRunner.run()

[ https://issues.apache.org/jira/browse/BEAM-492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Sela updated BEAM-492: --- Fix Version/s: 0.3.0-incubating > Spark runner should call Pipeline.run() instead of SparkRunner.run() > ---