[jira] [Assigned] (BEAM-198) Spark runner batch translator to work with Datasets instead of RDDs

[ https://issues.apache.org/jira/browse/BEAM-198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Baptiste Onofré reassigned BEAM-198: - Assignee: Jean-Baptiste Onofré (was: Amit Sela) > Spark runner batch translator

svn commit: r23206 - in /dev/beam/2.2.0: apache-beam-2.2.0-source-release.zip apache-beam-2.2.0-source-release.zip.md5 apache-beam-2.2.0-source-release.zip.sha1

Author: reuvenlax Date: Mon Nov 20 08:13:47 2017 New Revision: 23206 Log: Remove generated files from source distribution Modified: dev/beam/2.2.0/apache-beam-2.2.0-source-release.zip dev/beam/2.2.0/apache-beam-2.2.0-source-release.zip.md5

[jira] [Assigned] (BEAM-3185) Build blocks on parsing long as int from github status json

[ https://issues.apache.org/jira/browse/BEAM-3185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía reassigned BEAM-3185: -- Assignee: Ismaël Mejía (was: Jason Kuster) > Build blocks on parsing long as int from github

[jira] [Commented] (BEAM-3215) Add a performance test for HBaseIO

[ https://issues.apache.org/jira/browse/BEAM-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258980#comment-16258980 ] Ismaël Mejía commented on BEAM-3215: Does this one supercede

Build failed in Jenkins: beam_PostCommit_Python_Verify #3579

See -- [...truncated 934.61 KB...] copying apache_beam/runners/direct/__init__.py -> apache-beam-2.3.0.dev0/apache_beam/runners/direct copying

[jira] [Created] (BEAM-3225) Non deterministic behaviour of AfterProcessingTime trigger with multiple group by transformations

Pawel Bartoszek created BEAM-3225: - Summary: Non deterministic behaviour of AfterProcessingTime trigger with multiple group by transformations Key: BEAM-3225 URL: https://issues.apache.org/jira/browse/BEAM-3225

[jira] [Comment Edited] (BEAM-3227) Consider sharing Udf/SkdFunctionSpec records via pointer

[ https://issues.apache.org/jira/browse/BEAM-3227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16259488#comment-16259488 ] Luke Cwik edited comment on BEAM-3227 at 11/20/17 5:05 PM: --- That makes a lot of

[jira] [Commented] (BEAM-3227) Consider sharing Udf/SkdFunctionSpec records via pointer

[ https://issues.apache.org/jira/browse/BEAM-3227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16259488#comment-16259488 ] Luke Cwik commented on BEAM-3227: - That makes a lot of sense. > Consider sharing Udf/SkdFunctionSpec

[2/6] beam git commit: Enable Java Gradle Precommit in Jenkins

Enable Java Gradle Precommit in Jenkins Project: http://git-wip-us.apache.org/repos/asf/beam/repo Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b190916d Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b190916d Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b190916d

[jira] [Commented] (BEAM-3223) PTransform spec should not reuse FunctionSpec

[ https://issues.apache.org/jira/browse/BEAM-3223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16259408#comment-16259408 ] Kenneth Knowles commented on BEAM-3223: --- Yea SGTM. I think that will make generated code more

[GitHub] beam pull request #4150: [BEAM-2885] Maintain In-flight preparation requests

GitHub user tgroh opened a pull request: https://github.com/apache/beam/pull/4150 [BEAM-2885] Maintain In-flight preparation requests Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Make sure there is a [JIRA

[jira] [Commented] (BEAM-2885) Support job+artifact APIs locally

[ https://issues.apache.org/jira/browse/BEAM-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16259552#comment-16259552 ] ASF GitHub Bot commented on BEAM-2885: -- GitHub user tgroh opened a pull request:

[jira] [Created] (BEAM-3227) Consider sharing Udf/SkdFunctionSpec records via pointer

Kenneth Knowles created BEAM-3227: - Summary: Consider sharing Udf/SkdFunctionSpec records via pointer Key: BEAM-3227 URL: https://issues.apache.org/jira/browse/BEAM-3227 Project: Beam Issue

[GitHub] beam pull request #4149: [BEAM-3060] Add Compressed TextIOIT

GitHub user lgajowy opened a pull request: https://github.com/apache/beam/pull/4149 [BEAM-3060] Add Compressed TextIOIT Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Make sure there is a [JIRA

[jira] [Commented] (BEAM-3060) Add performance tests for commonly used file-based I/O PTransforms

[ https://issues.apache.org/jira/browse/BEAM-3060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16259421#comment-16259421 ] ASF GitHub Bot commented on BEAM-3060: -- GitHub user lgajowy opened a pull request:

[3/6] beam git commit: Limit the commit status length to 255 characters to honor Github limit

Limit the commit status length to 255 characters to honor Github limit Project: http://git-wip-us.apache.org/repos/asf/beam/repo Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/9bd9c85c Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/9bd9c85c Diff:

[5/6] beam git commit: Sync with master and add new subproject for beam-runners-local-java-core

Sync with master and add new subproject for beam-runners-local-java-core Project: http://git-wip-us.apache.org/repos/asf/beam/repo Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b565925f Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b565925f Diff:

[1/6] beam git commit: Revert PR/3976, the Jenkins Gradle plugin expects the Gradle Wrapper to be in the root of the directory.

Repository: beam Updated Branches: refs/heads/master 022e74460 -> 6fe6bc7eb Revert PR/3976, the Jenkins Gradle plugin expects the Gradle Wrapper to be in the root of the directory. Project: http://git-wip-us.apache.org/repos/asf/beam/repo Commit:

[beam-site] 01/03: Add Etienne Chauchot as a committer

This is an automated email from the ASF dual-hosted git repository. echauchot pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam-site.git commit 0e97de7db9ebd55ce779faaeb5c4be02ee8ba617 Author: Etienne Chauchot AuthorDate: Mon Nov 13

[beam-site] 02/03: Regenerate website

This is an automated email from the ASF dual-hosted git repository. echauchot pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam-site.git commit a65055cc9915f2ba6daabd2635481ff824d2ab89 Author: Etienne Chauchot AuthorDate: Mon Nov 20

[beam-site] 03/03: This closes #348

This is an automated email from the ASF dual-hosted git repository. echauchot pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam-site.git commit 8b22ab9988349937b77c5245745bb9ee578f6d48 Merge: fbcb7fa a65055c Author: Etienne Chauchot

[jira] [Commented] (BEAM-3227) Consider sharing Udf/SkdFunctionSpec records via pointer

[ https://issues.apache.org/jira/browse/BEAM-3227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16259508#comment-16259508 ] Kenneth Knowles commented on BEAM-3227: --- At the moment, we don't really have any examples other than

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow #4380

See -- [...truncated 21.35 MB...] [INFO] 2017-11-20T22:41:54.647Z: (e280dbb3a5bcdf2f): Starting 1 workers in us-central1-f... [INFO]

[1/3] beam git commit: Update the location of the root build directory to coincide with the checkout directory.

Repository: beam Updated Branches: refs/heads/master ad3a5a9f5 -> ed40093d8 Update the location of the root build directory to coincide with the checkout directory. Project: http://git-wip-us.apache.org/repos/asf/beam/repo Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a7ba2b4c

[2/3] beam git commit: Re-apply PR/3976 and disable the Jenkins Gradle build to fix the other precommits.

Re-apply PR/3976 and disable the Jenkins Gradle build to fix the other precommits. Project: http://git-wip-us.apache.org/repos/asf/beam/repo Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/905facbb Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/905facbb Diff:

Build failed in Jenkins: beam_PerformanceTests_Python #580

See Changes: [lcwik] Enable Java Gradle Precommit in Jenkins [lcwik] Simplify maintainability by generating tasks and command line from list [lcwik] Limit the commit status length to 255 characters

Build failed in Jenkins: beam_PerformanceTests_Spark #1028

See Changes: [lcwik] Enable Java Gradle Precommit in Jenkins [lcwik] Simplify maintainability by generating tasks and command line from list [lcwik] Limit the commit status length to 255 characters

[jira] [Commented] (BEAM-3137) BigQueryIO.write() should better verify user schemas

[ https://issues.apache.org/jira/browse/BEAM-3137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16259925#comment-16259925 ] ASF GitHub Bot commented on BEAM-3137: -- Github user asfgit closed the pull request at:

[jira] [Closed] (BEAM-3137) BigQueryIO.write() should better verify user schemas

[ https://issues.apache.org/jira/browse/BEAM-3137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Kirpichov closed BEAM-3137. -- Resolution: Fixed Assignee: Eugene Kirpichov (was: Reuven Lax) > BigQueryIO.write()

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Flink #4310

See

Build failed in Jenkins: beam_PostCommit_Python_Verify #3581

See Changes: [kirpichov] Improved validation of DynamicDestinations results in BigQueryIO.write -- [...truncated 813.51 KB...] copying

svn commit: r23222 - /dev/beam/2.2.0/apache-beam-2.2.0-source-release.zip.asc

Author: reuvenlax Date: Mon Nov 20 23:46:25 2017 New Revision: 23222 Log: resign source release Modified: dev/beam/2.2.0/apache-beam-2.2.0-source-release.zip.asc Modified: dev/beam/2.2.0/apache-beam-2.2.0-source-release.zip.asc

Build failed in Jenkins: beam_PostCommit_Python_Verify #3582

See Changes: [chamikara] Dataflow and PerfKit profiles; hash for 100.000.000 lines -- [...truncated 814.78 KB...] copying

[jira] [Commented] (BEAM-3201) ElasticsearchIO should deal with documents id

[ https://issues.apache.org/jira/browse/BEAM-3201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16259885#comment-16259885 ] Nicholas Verbeck commented on BEAM-3201: [~chetaldrich] This might be a lib to use for this.

[GitHub] beam pull request #4154: Re-apply PR/3976 and disable the Jenkins Gradle bui...

GitHub user lukecwik opened a pull request: https://github.com/apache/beam/pull/4154 Re-apply PR/3976 and disable the Jenkins Gradle build to fix the other precommits. Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Make sure

[GitHub] beam pull request #4075: [BEAM-3137] Improved validation of DynamicDestinati...

Github user asfgit closed the pull request at: https://github.com/apache/beam/pull/4075 ---

[2/2] beam git commit: This closes #4075: [BEAM-3137] Improved validation of DynamicDestinations results in BigQueryIO.write

This closes #4075: [BEAM-3137] Improved validation of DynamicDestinations results in BigQueryIO.write Project: http://git-wip-us.apache.org/repos/asf/beam/repo Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/699e6d0f Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/699e6d0f

[2/2] beam git commit: This closes #4120

This closes #4120 Project: http://git-wip-us.apache.org/repos/asf/beam/repo Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ad3a5a9f Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ad3a5a9f Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ad3a5a9f Branch:

[1/2] beam git commit: Dataflow and PerfKit profiles; hash for 100.000.000 lines

Repository: beam Updated Branches: refs/heads/master 699e6d0f1 -> ad3a5a9f5 Dataflow and PerfKit profiles; hash for 100.000.000 lines Project: http://git-wip-us.apache.org/repos/asf/beam/repo Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/31a86fa4 Tree:

[GitHub] beam pull request #4120: [BEAM-3060] TextIOIT: DataFlow and PerfKit profiles...

Github user asfgit closed the pull request at: https://github.com/apache/beam/pull/4120 ---

[jira] [Commented] (BEAM-3060) Add performance tests for commonly used file-based I/O PTransforms

[ https://issues.apache.org/jira/browse/BEAM-3060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16260003#comment-16260003 ] ASF GitHub Bot commented on BEAM-3060: -- Github user asfgit closed the pull request at:

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Apex #2836

See

Build failed in Jenkins: beam_PostCommit_Python_Verify #3580

See Changes: [lcwik] Enable Java Gradle Precommit in Jenkins [lcwik] Simplify maintainability by generating tasks and command line from list [lcwik] Limit the commit status length to 255

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Apex #2835

See

[3/3] beam git commit: Re-apply PR/3976 and disable the Jenkins Gradle build to fix the other precommits.

Re-apply PR/3976 and disable the Jenkins Gradle build to fix the other precommits. This closes #4154 Project: http://git-wip-us.apache.org/repos/asf/beam/repo Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ed40093d Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ed40093d

[GitHub] beam pull request #4154: Re-apply PR/3976 and disable the Jenkins Gradle bui...

Github user asfgit closed the pull request at: https://github.com/apache/beam/pull/4154 ---

Jenkins build is back to normal : beam_PostCommit_Java_MavenInstall #5274

See

[jira] [Commented] (BEAM-3218) Change PubsubBoundedWriter to check total byte size

[ https://issues.apache.org/jira/browse/BEAM-3218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16260047#comment-16260047 ] Chamikara Jayalath commented on BEAM-3218: -- Assigning to Reuven for triage. > Change

[GitHub] beam pull request #4152: Test GitHub pr 4123

GitHub user robertwb opened a pull request: https://github.com/apache/beam/pull/4152 Test GitHub pr 4123 Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Make sure there is a [JIRA

[GitHub] beam pull request #4151: [BEAM-2899] Decompose Direct Execution Components

GitHub user tgroh opened a pull request: https://github.com/apache/beam/pull/4151 [BEAM-2899] Decompose Direct Execution Components Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Make sure there is a [JIRA

[jira] [Commented] (BEAM-2899) Universal Local Runner

[ https://issues.apache.org/jira/browse/BEAM-2899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16259675#comment-16259675 ] ASF GitHub Bot commented on BEAM-2899: -- GitHub user tgroh opened a pull request:

[jira] [Commented] (BEAM-3219) DataflowRunner: @Setup not called for batch stateful DoFn

[ https://issues.apache.org/jira/browse/BEAM-3219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16259821#comment-16259821 ] ASF GitHub Bot commented on BEAM-3219: -- GitHub user kennknowles opened a pull request:

[GitHub] beam pull request #4153: [BEAM-3219] DataflowRunner: delegate @Setup and @Te...

GitHub user kennknowles opened a pull request: https://github.com/apache/beam/pull/4153 [BEAM-3219] DataflowRunner: delegate @Setup and @Teardown in stateful ParDo Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Make sure there is

[jira] [Commented] (BEAM-3099) Implement HDFS FileSystem for Python SDK

[ https://issues.apache.org/jira/browse/BEAM-3099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16260132#comment-16260132 ] Udi Meiri commented on BEAM-3099: - Doc exploring Python HDFS library options:

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Dataflow #4381

See

[jira] [Commented] (BEAM-3200) Streaming Pipeline throws RuntimeException when using DynamicDestinations and Method.FILE_LOADS

[ https://issues.apache.org/jira/browse/BEAM-3200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16260166#comment-16260166 ] AJ commented on BEAM-3200: -- I am assuming you are asking about load size per trigger. It might be more than 1

[jira] [Closed] (BEAM-2718) Add bundle retry logic to the DirectRunner

[ https://issues.apache.org/jira/browse/BEAM-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] María GH closed BEAM-2718. -- Resolution: Fixed > Add bundle retry logic to the DirectRunner > -- > >

[jira] [Commented] (BEAM-3201) ElasticsearchIO should deal with documents id

[ https://issues.apache.org/jira/browse/BEAM-3201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16260193#comment-16260193 ] Nicholas Verbeck commented on BEAM-3201: >From what I dug into for the bull api, they are. Just 2

Build failed in Jenkins: beam_PostCommit_Python_Verify #3583

See Changes: [lcwik] Re-apply PR/3976 and disable the Jenkins Gradle build to fix the other [lcwik] Update the location of the root build directory to coincide with the

[jira] [Commented] (BEAM-3201) ElasticsearchIO should deal with documents id

[ https://issues.apache.org/jira/browse/BEAM-3201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16260187#comment-16260187 ] Chet Aldrich commented on BEAM-3201: [~nerdynick] Thanks for the lib reference, I'll look into it.

[jira] [Commented] (BEAM-3201) ElasticsearchIO should deal with documents id

[ https://issues.apache.org/jira/browse/BEAM-3201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16260188#comment-16260188 ] Chet Aldrich commented on BEAM-3201: Also [~echauchot] you can feel free to assign this to me when you

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow #4382

See Changes: [chamikara] Dataflow and PerfKit profiles; hash for 100.000.000 lines [lcwik] Re-apply PR/3976 and disable the Jenkins Gradle build to fix the other [lcwik] Update the

Build failed in Jenkins: beam_PostCommit_Python_Verify #3584

See -- [...truncated 816.07 KB...] copying apache_beam/runners/direct/__init__.py -> apache-beam-2.3.0.dev0/apache_beam/runners/direct copying

Build failed in Jenkins: beam_PerformanceTests_Spark #1029

See -- Started by timer [EnvInject] - Loading node environment variables. Building remotely on beam2 (beam) in workspace

Build failed in Jenkins: beam_PerformanceTests_Python #581

See -- Started by timer [EnvInject] - Loading node environment variables. Building remotely on beam7 (beam) in workspace

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Dataflow #4383

See

[jira] [Updated] (BEAM-3225) Non deterministic behaviour of AfterProcessingTime trigger with multiple group by transformations

[ https://issues.apache.org/jira/browse/BEAM-3225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pawel Bartoszek updated BEAM-3225: -- Description: *Context* I run my [test

Build failed in Jenkins: beam_PerformanceTests_Python #579

See -- Started by timer [EnvInject] - Loading node environment variables. Building remotely on beam7 (beam) in workspace

[jira] [Updated] (BEAM-3225) Non deterministic behaviour of AfterProcessingTime trigger with multiple group by transformations

[ https://issues.apache.org/jira/browse/BEAM-3225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pawel Bartoszek updated BEAM-3225: -- Description: *Context* I run my [test

[jira] [Created] (BEAM-3224) Add support for path with braces for Google Cloud Storage

Benjamin BENOIST created BEAM-3224: -- Summary: Add support for path with braces for Google Cloud Storage Key: BEAM-3224 URL: https://issues.apache.org/jira/browse/BEAM-3224 Project: Beam

[jira] [Updated] (BEAM-3224) Add support for path with braces for Google Cloud Storage

[ https://issues.apache.org/jira/browse/BEAM-3224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin BENOIST updated BEAM-3224: --- Description: At the moment we can not use braces in Google Cloud Storage paths, as explained

[jira] [Updated] (BEAM-3224) Add support for path with braces for Google Cloud Storage

[ https://issues.apache.org/jira/browse/BEAM-3224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin BENOIST updated BEAM-3224: --- Description: At the moment we can not use braces in Google Cloud Storage paths, as explained

[jira] [Updated] (BEAM-3224) Add support for path with braces for Google Cloud Storage

[ https://issues.apache.org/jira/browse/BEAM-3224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin BENOIST updated BEAM-3224: --- Description: At the moment we can not use braces in Google Cloud Storage paths, as explained

[jira] [Updated] (BEAM-3224) Add support for path with braces for Google Cloud Storage

[ https://issues.apache.org/jira/browse/BEAM-3224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin BENOIST updated BEAM-3224: --- Description: At the moment we can not use braces in Google Cloud Storage paths, as explained

[jira] [Updated] (BEAM-3224) Add support for path with braces for Google Cloud Storage

[ https://issues.apache.org/jira/browse/BEAM-3224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin BENOIST updated BEAM-3224: --- Description: At the moment we can not use braces in Google Cloud Storage paths, as explained

[jira] [Updated] (BEAM-3224) Add support for path with braces for Google Cloud Storage

[ https://issues.apache.org/jira/browse/BEAM-3224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin BENOIST updated BEAM-3224: --- Description: At the moment we can not use braces in Google Cloud Storage paths, as explained

[jira] [Created] (BEAM-3226) Gremlin IO

Utkarsh Sopan created BEAM-3226: --- Summary: Gremlin IO Key: BEAM-3226 URL: https://issues.apache.org/jira/browse/BEAM-3226 Project: Beam Issue Type: Wish Components: sdk-java-core