[jira] [Created] (BEAM-3206) Spark runner does not shut down after pipeline completion

2017-11-16 Thread Ben Sidhom (JIRA)
Ben Sidhom created BEAM-3206: Summary: Spark runner does not shut down after pipeline completion Key: BEAM-3206 URL: https://issues.apache.org/jira/browse/BEAM-3206 Project: Beam Issue Type: Bug

[jira] [Commented] (BEAM-2594) Python shim for submitting to the ULR

2018-05-08 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467995#comment-16467995 ] Ben Sidhom commented on BEAM-2594: -- I'm not sure exactly what this entails. It depends exactly what we

[jira] [Commented] (BEAM-2898) Flink supports chaining/fusion of single-SDK stages

2018-05-14 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474569#comment-16474569 ] Ben Sidhom commented on BEAM-2898: -- I'm going to mark this as fixed since we have fusion and executable

[jira] [Closed] (BEAM-2898) Flink supports chaining/fusion of single-SDK stages

2018-05-14 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom closed BEAM-2898. Resolution: Fixed Fix Version/s: Not applicable > Flink supports chaining/fusion of single-SDK stages

[jira] [Created] (BEAM-4285) Flink batch state request handler

2018-05-14 Thread Ben Sidhom (JIRA)
Ben Sidhom created BEAM-4285: Summary: Flink batch state request handler Key: BEAM-4285 URL: https://issues.apache.org/jira/browse/BEAM-4285 Project: Beam Issue Type: Bug Components:

[jira] [Created] (BEAM-4286) Pooled artifact source

2018-05-14 Thread Ben Sidhom (JIRA)
Ben Sidhom created BEAM-4286: Summary: Pooled artifact source Key: BEAM-4286 URL: https://issues.apache.org/jira/browse/BEAM-4286 Project: Beam Issue Type: Bug Components: runner-flink

[jira] [Closed] (BEAM-3972) Flink runner translates batch pipelines directly by proto

2018-05-07 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom closed BEAM-3972. Resolution: Fixed Fix Version/s: Not applicable > Flink runner translates batch pipelines directly by

[jira] [Created] (BEAM-4271) Executable stages allow side input coders to be set and/or queried

2018-05-10 Thread Ben Sidhom (JIRA)
Ben Sidhom created BEAM-4271: Summary: Executable stages allow side input coders to be set and/or queried Key: BEAM-4271 URL: https://issues.apache.org/jira/browse/BEAM-4271 Project: Beam Issue

[jira] [Commented] (BEAM-4267) Implement a reusable library that can run an ExecutableStage with a given Environment

2018-05-22 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16484482#comment-16484482 ] Ben Sidhom commented on BEAM-4267: -- Yep! Although in order to actually run stuff end-to-end, we'll also

[jira] [Created] (BEAM-4404) Fix docker run arguments

2018-05-24 Thread Ben Sidhom (JIRA)
Ben Sidhom created BEAM-4404: Summary: Fix docker run arguments Key: BEAM-4404 URL: https://issues.apache.org/jira/browse/BEAM-4404 Project: Beam Issue Type: Bug Components:

[jira] [Assigned] (BEAM-4285) Flink batch state request handler

2018-06-08 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom reassigned BEAM-4285: Assignee: Ben Sidhom > Flink batch state request handler > - > >

[jira] [Commented] (BEAM-4149) Java SDK Harness should populate worker id in control plane headers

2018-04-30 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459052#comment-16459052 ] Ben Sidhom commented on BEAM-4149: -- The workaround is technically fine for now, but it will break as soon

[jira] [Commented] (BEAM-4228) The FlinkRunner shouldn't require all of the values for a key to fit in memory

2018-05-02 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-4228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461769#comment-16461769 ] Ben Sidhom commented on BEAM-4228: -- For context, see

[jira] [Assigned] (BEAM-3327) Add abstractions to manage Environment Instance lifecycles.

2018-04-30 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom reassigned BEAM-3327: Assignee: Axel Magnuson (was: Ben Sidhom) > Add abstractions to manage Environment Instance

[jira] [Resolved] (BEAM-3914) 'Unzip' flattens before performing fusion

2018-04-30 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom resolved BEAM-3914. -- Resolution: Fixed Fix Version/s: Not applicable > 'Unzip' flattens before performing fusion >

[jira] [Created] (BEAM-4231) Runner utility for Coder instantiation

2018-05-03 Thread Ben Sidhom (JIRA)
Ben Sidhom created BEAM-4231: Summary: Runner utility for Coder instantiation Key: BEAM-4231 URL: https://issues.apache.org/jira/browse/BEAM-4231 Project: Beam Issue Type: Bug

[jira] [Commented] (BEAM-3587) User reports TextIO failure in FlinkRunner on master

2018-02-02 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350806#comment-16350806 ] Ben Sidhom commented on BEAM-3587: -- As Ken mentioned, this is strange since we have ValidatesRunner tests

[jira] [Commented] (BEAM-3587) User reports TextIO failure in FlinkRunner on master

2018-02-02 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351157#comment-16351157 ] Ben Sidhom commented on BEAM-3587: -- Just repeated it with literally just a read step. It still seems to

[jira] [Commented] (BEAM-3587) User reports TextIO failure in FlinkRunner on master

2018-02-02 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351150#comment-16351150 ] Ben Sidhom commented on BEAM-3587: -- I built at head and ran a simple program that reads in a text file and

[jira] [Created] (BEAM-3602) Add source set for generated Java gRPC code

2018-02-01 Thread Ben Sidhom (JIRA)
Ben Sidhom created BEAM-3602: Summary: Add source set for generated Java gRPC code Key: BEAM-3602 URL: https://issues.apache.org/jira/browse/BEAM-3602 Project: Beam Issue Type: Improvement

[jira] [Assigned] (BEAM-3602) Add source set for generated Java gRPC code

2018-02-05 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom reassigned BEAM-3602: Assignee: Kenneth Knowles (was: Ben Sidhom) > Add source set for generated Java gRPC code >

[jira] [Commented] (BEAM-3602) Add source set for generated Java gRPC code

2018-02-05 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352643#comment-16352643 ] Ben Sidhom commented on BEAM-3602: -- This should be fixed by https://github.com/apache/beam/pull/4583. >

[jira] [Commented] (BEAM-3587) User reports TextIO failure in FlinkRunner on master

2018-02-04 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351945#comment-16351945 ] Ben Sidhom commented on BEAM-3587: -- I think I've figured it out. I had been using SBT to build my own

[jira] [Commented] (BEAM-2591) Python shim for submitting to FlinkRunner

2018-02-08 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357726#comment-16357726 ] Ben Sidhom commented on BEAM-2591: -- [~kenn] Is this still the case? Do we want the Python SDK to spin up

[jira] [Created] (BEAM-3673) FlinkRunner: Harness manager for connecting operators to SDK Harnesses

2018-02-08 Thread Ben Sidhom (JIRA)
Ben Sidhom created BEAM-3673: Summary: FlinkRunner: Harness manager for connecting operators to SDK Harnesses Key: BEAM-3673 URL: https://issues.apache.org/jira/browse/BEAM-3673 Project: Beam

[jira] [Updated] (BEAM-3673) FlinkRunner: Harness manager for connecting operators to SDK Harnesses

2018-02-08 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom updated BEAM-3673: - Description: SDK harnesses require a common set of gRPC services to operate. The role of the harness

[jira] [Created] (BEAM-3672) FlinkRunner: Implement an Artifact service using the Flink DistributedCache

2018-02-08 Thread Ben Sidhom (JIRA)
Ben Sidhom created BEAM-3672: Summary: FlinkRunner: Implement an Artifact service using the Flink DistributedCache Key: BEAM-3672 URL: https://issues.apache.org/jira/browse/BEAM-3672 Project: Beam

[jira] [Created] (BEAM-3675) FlinkRunner: Logging server

2018-02-08 Thread Ben Sidhom (JIRA)
Ben Sidhom created BEAM-3675: Summary: FlinkRunner: Logging server Key: BEAM-3675 URL: https://issues.apache.org/jira/browse/BEAM-3675 Project: Beam Issue Type: Sub-task Components:

[jira] [Commented] (BEAM-3673) FlinkRunner: Harness manager for connecting operators to SDK Harnesses

2018-02-09 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358647#comment-16358647 ] Ben Sidhom commented on BEAM-3673: -- That was not intentional. I assume it's because the umbrella but was

[jira] [Comment Edited] (BEAM-3673) FlinkRunner: Harness manager for connecting operators to SDK Harnesses

2018-02-09 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358647#comment-16358647 ] Ben Sidhom edited comment on BEAM-3673 at 2/9/18 4:51 PM: -- That was not

[jira] [Assigned] (BEAM-2597) FlinkRunner Fn API based ParDo operator

2018-02-14 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom reassigned BEAM-2597: Assignee: Ben Sidhom (was: Aljoscha Krettek) > FlinkRunner Fn API based ParDo operator >

[jira] [Closed] (BEAM-4231) Runner utility for Coder instantiation

2018-06-21 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom closed BEAM-4231. Resolution: Fixed Fix Version/s: Not applicable > Runner utility for Coder instantiation >

[jira] [Closed] (BEAM-4523) Implement Flink batch ExecutableStage context

2018-06-21 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-4523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom closed BEAM-4523. Resolution: Fixed Fix Version/s: Not applicable > Implement Flink batch ExecutableStage context >

[jira] [Created] (BEAM-4615) Flink job server driver wrapper

2018-06-21 Thread Ben Sidhom (JIRA)
Ben Sidhom created BEAM-4615: Summary: Flink job server driver wrapper Key: BEAM-4615 URL: https://issues.apache.org/jira/browse/BEAM-4615 Project: Beam Issue Type: New Feature

[jira] [Commented] (BEAM-4148) Local server api descriptors contain urls that work on Mac and Linux

2018-08-03 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16568949#comment-16568949 ] Ben Sidhom commented on BEAM-4148: -- Basically, if the reference runner is manually creating its own

[jira] [Created] (BEAM-5114) Create example uber jars for supported runners

2018-08-08 Thread Ben Sidhom (JIRA)
Ben Sidhom created BEAM-5114: Summary: Create example uber jars for supported runners Key: BEAM-5114 URL: https://issues.apache.org/jira/browse/BEAM-5114 Project: Beam Issue Type: New Feature

[jira] [Commented] (BEAM-4148) Local server api descriptors contain urls that work on Mac and Linux

2018-08-03 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16568948#comment-16568948 ] Ben Sidhom commented on BEAM-4148: -- The linked PR should have actually solved this for the Docker

[jira] [Commented] (BEAM-5110) Reconile Flink JVM singleton management with deployment

2018-08-14 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16580545#comment-16580545 ] Ben Sidhom commented on BEAM-5110: -- I meant to mention in the previous comment: as of now, this is not a

[jira] [Commented] (BEAM-5110) Reconile Flink JVM singleton management with deployment

2018-08-14 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16580536#comment-16580536 ] Ben Sidhom commented on BEAM-5110: -- Sadly, after investigating this for a while, I think that the best

[jira] [Created] (BEAM-5110) Reconile Flink JVM singleton management with deployment

2018-08-08 Thread Ben Sidhom (JIRA)
Ben Sidhom created BEAM-5110: Summary: Reconile Flink JVM singleton management with deployment Key: BEAM-5110 URL: https://issues.apache.org/jira/browse/BEAM-5110 Project: Beam Issue Type: Bug

[jira] [Assigned] (BEAM-5110) Reconile Flink JVM singleton management with deployment

2018-08-08 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom reassigned BEAM-5110: Assignee: Ben Sidhom > Reconile Flink JVM singleton management with deployment >

[jira] [Commented] (BEAM-5110) Reconile Flink JVM singleton management with deployment

2018-08-08 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573681#comment-16573681 ] Ben Sidhom commented on BEAM-5110: -- Note that we can partially fix the issue by requiring that users who

[jira] [Created] (BEAM-3793) SDK harness should log bad gRPC responses

2018-03-06 Thread Ben Sidhom (JIRA)
Ben Sidhom created BEAM-3793: Summary: SDK harness should log bad gRPC responses Key: BEAM-3793 URL: https://issues.apache.org/jira/browse/BEAM-3793 Project: Beam Issue Type: Bug

[jira] [Assigned] (BEAM-2588) FlinkRunner shim for serving Job API

2018-03-07 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom reassigned BEAM-2588: Assignee: Axel Magnuson > FlinkRunner shim for serving Job API >

[jira] [Commented] (BEAM-3779) Enable deserialization of a non-java Pipeline

2018-04-11 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434685#comment-16434685 ] Ben Sidhom commented on BEAM-3779: -- This is no longer relevant as we have decided to use protos directly

[jira] [Closed] (BEAM-3779) Enable deserialization of a non-java Pipeline

2018-04-11 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom closed BEAM-3779. Resolution: Won't Do Fix Version/s: Not applicable > Enable deserialization of a non-java Pipeline >

[jira] [Assigned] (BEAM-2898) Flink supports chaining/fusion of single-SDK stages

2018-04-13 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom reassigned BEAM-2898: Assignee: Ben Sidhom > Flink supports chaining/fusion of single-SDK stages >

[jira] [Created] (BEAM-4071) Portable Runner Job API shim

2018-04-13 Thread Ben Sidhom (JIRA)
Ben Sidhom created BEAM-4071: Summary: Portable Runner Job API shim Key: BEAM-4071 URL: https://issues.apache.org/jira/browse/BEAM-4071 Project: Beam Issue Type: New Feature

[jira] [Commented] (BEAM-4068) Consistent option specification between SDKs and runners by URN

2018-04-13 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437686#comment-16437686 ] Ben Sidhom commented on BEAM-4068: -- See [https://github.com/axelmagn/beam/pull/5/files] for an example.

[jira] [Commented] (BEAM-2898) Flink supports chaining/fusion of single-SDK stages

2018-04-13 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437763#comment-16437763 ] Ben Sidhom commented on BEAM-2898: -- Pipeline fusion has been implemented in runner-core. With support for

[jira] [Commented] (BEAM-2588) Portable Flink Runner Job API

2018-04-13 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437781#comment-16437781 ] Ben Sidhom commented on BEAM-2588: -- I've renamed this bug to reflect the fact that the portable Flink

[jira] [Updated] (BEAM-2588) Portable Flink Runner Job API

2018-04-13 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom updated BEAM-2588: - Description: The portable Flink runner needs to be wired into a job server so that it can accept jobs the

[jira] [Created] (BEAM-4067) Add portable Flink test runner

2018-04-13 Thread Ben Sidhom (JIRA)
Ben Sidhom created BEAM-4067: Summary: Add portable Flink test runner Key: BEAM-4067 URL: https://issues.apache.org/jira/browse/BEAM-4067 Project: Beam Issue Type: New Feature

[jira] [Created] (BEAM-4068) Consistent option specification between SDKs and runners by URN

2018-04-13 Thread Ben Sidhom (JIRA)
Ben Sidhom created BEAM-4068: Summary: Consistent option specification between SDKs and runners by URN Key: BEAM-4068 URL: https://issues.apache.org/jira/browse/BEAM-4068 Project: Beam Issue

[jira] [Created] (BEAM-4069) Empty pipeline options can be gracefully serialized/deserialized

2018-04-13 Thread Ben Sidhom (JIRA)
Ben Sidhom created BEAM-4069: Summary: Empty pipeline options can be gracefully serialized/deserialized Key: BEAM-4069 URL: https://issues.apache.org/jira/browse/BEAM-4069 Project: Beam Issue

[jira] [Updated] (BEAM-2588) Portable Flink Runner Job API

2018-04-13 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom updated BEAM-2588: - Summary: Portable Flink Runner Job API (was: FlinkRunner shim for serving Job API) > Portable Flink

[jira] [Created] (BEAM-4063) Flink runner supports cluster-wide artifact deployments through the Distributed Cache

2018-04-12 Thread Ben Sidhom (JIRA)
Ben Sidhom created BEAM-4063: Summary: Flink runner supports cluster-wide artifact deployments through the Distributed Cache Key: BEAM-4063 URL: https://issues.apache.org/jira/browse/BEAM-4063 Project:

[jira] [Created] (BEAM-4056) Identify Side Inputs by PTransform ID and local name

2018-04-11 Thread Ben Sidhom (JIRA)
Ben Sidhom created BEAM-4056: Summary: Identify Side Inputs by PTransform ID and local name Key: BEAM-4056 URL: https://issues.apache.org/jira/browse/BEAM-4056 Project: Beam Issue Type: New

[jira] [Created] (BEAM-4116) Add a timeout to artifact stager

2018-04-18 Thread Ben Sidhom (JIRA)
Ben Sidhom created BEAM-4116: Summary: Add a timeout to artifact stager Key: BEAM-4116 URL: https://issues.apache.org/jira/browse/BEAM-4116 Project: Beam Issue Type: Improvement

[jira] [Updated] (BEAM-4109) Support arbitrary artifact names in local file artifact staging service

2018-04-18 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-4109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom updated BEAM-4109: - Issue Type: Bug (was: Improvement) > Support arbitrary artifact names in local file artifact staging

[jira] [Closed] (BEAM-4056) Identify Side Inputs by PTransform ID and local name

2018-04-18 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-4056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom closed BEAM-4056. Resolution: Fixed Fix Version/s: Not applicable > Identify Side Inputs by PTransform ID and local

[jira] [Closed] (BEAM-4069) Empty pipeline options can be gracefully serialized/deserialized

2018-04-18 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom closed BEAM-4069. Resolution: Fixed Fix Version/s: Not applicable > Empty pipeline options can be gracefully

[jira] [Closed] (BEAM-3994) Use typed sinks and sources for FnApiControlClientPoolService

2018-04-18 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom closed BEAM-3994. Resolution: Fixed Fix Version/s: Not applicable > Use typed sinks and sources for

[jira] [Updated] (BEAM-3886) Python SDK harness uses State API from ProcessBundleDescriptor

2018-04-20 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom updated BEAM-3886: - Summary: Python SDK harness uses State API from ProcessBundleDescriptor (was: Python SDK harness does not

[jira] [Updated] (BEAM-2897) Configurable container/process management

2018-04-20 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom updated BEAM-2897: - Issue Type: New Feature (was: Sub-task) Parent: (was: BEAM-2889) > Configurable

[jira] [Updated] (BEAM-3886) Python SDK harness uses State API from ProcessBundleDescriptor

2018-04-20 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom updated BEAM-3886: - Description: The Python harness should pull the state api descriptor from the current process bundle

[jira] [Updated] (BEAM-2588) Portable Flink Runner Job API

2018-04-20 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom updated BEAM-2588: - Issue Type: New Feature (was: Sub-task) Parent: (was: BEAM-2889) > Portable Flink Runner Job

[jira] [Updated] (BEAM-2795) FlinkRunner: translate using SDK-agnostic means

2018-04-20 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom updated BEAM-2795: - Issue Type: Improvement (was: Sub-task) Parent: (was: BEAM-2889) > FlinkRunner: translate

[jira] [Updated] (BEAM-3792) Python submits portable pipelines to the Flink-served endpoint.

2018-04-20 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom updated BEAM-3792: - Issue Type: Bug (was: Sub-task) Parent: (was: BEAM-2889) > Python submits portable pipelines

[jira] [Assigned] (BEAM-2889) Flink runs portable pipelines

2018-04-20 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom reassigned BEAM-2889: Assignee: Ben Sidhom (was: Aljoscha Krettek) > Flink runs portable pipelines >

[jira] [Created] (BEAM-4151) Prevent control client pool service from leaking stale client references

2018-04-20 Thread Ben Sidhom (JIRA)
Ben Sidhom created BEAM-4151: Summary: Prevent control client pool service from leaking stale client references Key: BEAM-4151 URL: https://issues.apache.org/jira/browse/BEAM-4151 Project: Beam

[jira] [Assigned] (BEAM-4151) Prevent control client pool service from leaking stale client references

2018-04-20 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-4151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom reassigned BEAM-4151: Assignee: Ben Sidhom (was: Kenneth Knowles) > Prevent control client pool service from leaking

[jira] [Assigned] (BEAM-4109) Support arbitrary artifact names in local file artifact staging service

2018-04-18 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-4109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom reassigned BEAM-4109: Assignee: (was: Kenneth Knowles) > Support arbitrary artifact names in local file artifact

[jira] [Updated] (BEAM-4106) Merge staging file options between Dataflow runner and portable runner

2018-04-20 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom updated BEAM-4106: - Description: Both runners (will) require staging file options. This should be refactored such that the

[jira] [Assigned] (BEAM-3327) Add abstractions to manage Environment Instance lifecycles.

2018-04-16 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom reassigned BEAM-3327: Assignee: Axel Magnuson (was: Ben Sidhom) > Add abstractions to manage Environment Instance

[jira] [Assigned] (BEAM-3327) Add abstractions to manage Environment Instance lifecycles.

2018-04-19 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom reassigned BEAM-3327: Assignee: Ben Sidhom (was: Axel Magnuson) > Add abstractions to manage Environment Instance

[jira] [Commented] (BEAM-2421) Migrate Apache Beam to use impulse primitive as the only root primitive

2018-04-23 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16448782#comment-16448782 ] Ben Sidhom commented on BEAM-2421: -- [https://github.com/apache/beam/pull/4783] adds support for the Flink

[jira] [Commented] (BEAM-4149) Java SDK Harness should populate worker id in control plane headers

2018-04-23 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16448638#comment-16448638 ] Ben Sidhom commented on BEAM-4149: -- As a workaround for now, we should explicitly set "worker_id" to the

[jira] [Created] (BEAM-4109) Support arbitrary artifact names in local file artifact staging service

2018-04-17 Thread Ben Sidhom (JIRA)
Ben Sidhom created BEAM-4109: Summary: Support arbitrary artifact names in local file artifact staging service Key: BEAM-4109 URL: https://issues.apache.org/jira/browse/BEAM-4109 Project: Beam

[jira] [Created] (BEAM-4106) Merge staging file options between Dataflow runner and portable runner

2018-04-17 Thread Ben Sidhom (JIRA)
Ben Sidhom created BEAM-4106: Summary: Merge staging file options between Dataflow runner and portable runner Key: BEAM-4106 URL: https://issues.apache.org/jira/browse/BEAM-4106 Project: Beam

[jira] [Assigned] (BEAM-4127) Flink portable runner translates streaming pipelines by proto

2018-04-18 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-4127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom reassigned BEAM-4127: Assignee: (was: Aljoscha Krettek) > Flink portable runner translates streaming pipelines by

[jira] [Resolved] (BEAM-2897) Configurable container/process management

2018-04-18 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom resolved BEAM-2897. -- Resolution: Duplicate Fix Version/s: Not applicable > Configurable container/process management >

[jira] [Created] (BEAM-4130) Portable Flink runner entry point

2018-04-18 Thread Ben Sidhom (JIRA)
Ben Sidhom created BEAM-4130: Summary: Portable Flink runner entry point Key: BEAM-4130 URL: https://issues.apache.org/jira/browse/BEAM-4130 Project: Beam Issue Type: New Feature

[jira] [Created] (BEAM-4131) Python SDK harness container image contains SDK and dependencies

2018-04-18 Thread Ben Sidhom (JIRA)
Ben Sidhom created BEAM-4131: Summary: Python SDK harness container image contains SDK and dependencies Key: BEAM-4131 URL: https://issues.apache.org/jira/browse/BEAM-4131 Project: Beam Issue

[jira] [Updated] (BEAM-3972) Flink runner translates batch pipelines directly by proto

2018-04-18 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom updated BEAM-3972: - Summary: Flink runner translates batch pipelines directly by proto (was: Flink runner translates

[jira] [Created] (BEAM-4127) Flink portable runner translates streaming pipelines by proto

2018-04-18 Thread Ben Sidhom (JIRA)
Ben Sidhom created BEAM-4127: Summary: Flink portable runner translates streaming pipelines by proto Key: BEAM-4127 URL: https://issues.apache.org/jira/browse/BEAM-4127 Project: Beam Issue

[jira] [Updated] (BEAM-2597) FlinkRunner ExecutableStage batch operator

2018-04-18 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom updated BEAM-2597: - Description: This operator will execute user code in the context of an SDK harness by constructing a

[jira] [Created] (BEAM-4128) Handle WindowInto transforms in portable Flink runner directly

2018-04-18 Thread Ben Sidhom (JIRA)
Ben Sidhom created BEAM-4128: Summary: Handle WindowInto transforms in portable Flink runner directly Key: BEAM-4128 URL: https://issues.apache.org/jira/browse/BEAM-4128 Project: Beam Issue

[jira] [Updated] (BEAM-2597) FlinkRunner ExecutableStage operator

2018-04-18 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom updated BEAM-2597: - Description:   To run non-Java SDK code is to put together an operator that manages a Fn API client

[jira] [Updated] (BEAM-2597) FlinkRunner ExecutableStage batch operator

2018-04-18 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom updated BEAM-2597: - Summary: FlinkRunner ExecutableStage batch operator (was: FlinkRunner ExecutableStage operator) >

[jira] [Updated] (BEAM-2597) FlinkRunner ExecutableStage operator

2018-04-18 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom updated BEAM-2597: - Summary: FlinkRunner ExecutableStage operator (was: FlinkRunner Fn API based ParDo operator) >

[jira] [Created] (BEAM-4148) Local server api descriptors contain urls that work on Mac and Linux

2018-04-19 Thread Ben Sidhom (JIRA)
Ben Sidhom created BEAM-4148: Summary: Local server api descriptors contain urls that work on Mac and Linux Key: BEAM-4148 URL: https://issues.apache.org/jira/browse/BEAM-4148 Project: Beam

[jira] [Created] (BEAM-4146) Python SDK sets environment in portable pipelines

2018-04-19 Thread Ben Sidhom (JIRA)
Ben Sidhom created BEAM-4146: Summary: Python SDK sets environment in portable pipelines Key: BEAM-4146 URL: https://issues.apache.org/jira/browse/BEAM-4146 Project: Beam Issue Type: Bug

[jira] [Created] (BEAM-4147) Portable runner Job API abstractions

2018-04-19 Thread Ben Sidhom (JIRA)
Ben Sidhom created BEAM-4147: Summary: Portable runner Job API abstractions Key: BEAM-4147 URL: https://issues.apache.org/jira/browse/BEAM-4147 Project: Beam Issue Type: New Feature

[jira] [Assigned] (BEAM-3673) FlinkRunner: Harness manager for connecting operators to SDK Harnesses

2018-04-19 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Sidhom reassigned BEAM-3673: Assignee: Axel Magnuson > FlinkRunner: Harness manager for connecting operators to SDK Harnesses >

[jira] [Created] (BEAM-4145) Java SDK Harness populates control request headers with worker id

2018-04-19 Thread Ben Sidhom (JIRA)
Ben Sidhom created BEAM-4145: Summary: Java SDK Harness populates control request headers with worker id Key: BEAM-4145 URL: https://issues.apache.org/jira/browse/BEAM-4145 Project: Beam Issue

[jira] [Created] (BEAM-4149) Java SDK Harness should populate worker id in control plane headers

2018-04-19 Thread Ben Sidhom (JIRA)
Ben Sidhom created BEAM-4149: Summary: Java SDK Harness should populate worker id in control plane headers Key: BEAM-4149 URL: https://issues.apache.org/jira/browse/BEAM-4149 Project: Beam

[jira] [Created] (BEAM-3976) SdkHarnessClient is thread-safe

2018-03-30 Thread Ben Sidhom (JIRA)
Ben Sidhom created BEAM-3976: Summary: SdkHarnessClient is thread-safe Key: BEAM-3976 URL: https://issues.apache.org/jira/browse/BEAM-3976 Project: Beam Issue Type: Bug Components:

[jira] [Created] (BEAM-3966) Move core utilities into a new top-level module

2018-03-28 Thread Ben Sidhom (JIRA)
Ben Sidhom created BEAM-3966: Summary: Move core utilities into a new top-level module Key: BEAM-3966 URL: https://issues.apache.org/jira/browse/BEAM-3966 Project: Beam Issue Type: Bug

[jira] [Commented] (BEAM-3970) Java SDK harness supports window_into

2018-03-29 Thread Ben Sidhom (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419739#comment-16419739 ] Ben Sidhom commented on BEAM-3970: -- It looks like

  1   2   >