[jira] [Created] (BEAM-5720) Default coder breaks with large ints on Python 3

2018-10-11 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-5720: - Summary: Default coder breaks with large ints on Python 3 Key: BEAM-5720 URL: https://issues.apache.org/jira/browse/BEAM-5720 Project: Beam Issue Type:

[jira] [Commented] (BEAM-5705) PRs 6557 and 6589 break internal Dataflow tests

2018-10-11 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646167#comment-16646167 ] Robert Bradshaw commented on BEAM-5705: --- I think this is fixed by 

[jira] [Resolved] (BEAM-4858) Clean up _BatchSizeEstimator in element-batching transform.

2018-10-10 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-4858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw resolved BEAM-4858. --- Resolution: Fixed Fix Version/s: 2.8.0 > Clean up _BatchSizeEstimator in

[jira] [Commented] (BEAM-4858) Clean up _BatchSizeEstimator in element-batching transform.

2018-10-10 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-4858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644983#comment-16644983 ] Robert Bradshaw commented on BEAM-4858: --- Submitted pr/6375. * Simplifies the aging out of old data.

[jira] [Created] (BEAM-5702) Avoid reshuffle for zero and one element creates

2018-10-10 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-5702: - Summary: Avoid reshuffle for zero and one element creates Key: BEAM-5702 URL: https://issues.apache.org/jira/browse/BEAM-5702 Project: Beam Issue Type:

[jira] [Created] (BEAM-5664) A canceled pipeline should not return a done status in the jobserver.

2018-10-05 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-5664: - Summary: A canceled pipeline should not return a done status in the jobserver. Key: BEAM-5664 URL: https://issues.apache.org/jira/browse/BEAM-5664 Project: Beam

[jira] [Updated] (BEAM-5664) A canceled pipeline should not return a done status in the jobserver.

2018-10-05 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw updated BEAM-5664: -- Labels: portability-flink (was: ) > A canceled pipeline should not return a done status in the

[jira] [Commented] (BEAM-5500) Portable python sdk worker leaks memory in streaming mode

2018-10-03 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636638#comment-16636638 ] Robert Bradshaw commented on BEAM-5500: --- In light of [https://github.com/apache/beam/pull/6517] can

[jira] [Created] (BEAM-5606) Automatically infer gcp project from environment.

2018-10-02 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-5606: - Summary: Automatically infer gcp project from environment. Key: BEAM-5606 URL: https://issues.apache.org/jira/browse/BEAM-5606 Project: Beam Issue Type:

[jira] [Created] (BEAM-5521) Cache execution trees in SDK worker

2018-09-28 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-5521: - Summary: Cache execution trees in SDK worker Key: BEAM-5521 URL: https://issues.apache.org/jira/browse/BEAM-5521 Project: Beam Issue Type: Improvement

[jira] [Updated] (BEAM-5521) Cache execution trees in SDK worker

2018-09-28 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw updated BEAM-5521: -- Labels: portability-flink (was: ) > Cache execution trees in SDK worker >

[jira] [Updated] (BEAM-5500) Portable python sdk worker leaks memory in streaming mode

2018-09-27 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw updated BEAM-5500: -- Labels: portability-flink (was: ) > Portable python sdk worker leaks memory in streaming mode

[jira] [Commented] (BEAM-5509) Python pipeline_options doesn't handle int type

2018-09-27 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630428#comment-16630428 ] Robert Bradshaw commented on BEAM-5509: --- I don't think we should be representing integral values as

[jira] [Updated] (BEAM-5509) Python pipeline_options doesn't handle int type

2018-09-27 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw updated BEAM-5509: -- Labels: portability-flink (was: ) > Python pipeline_options doesn't handle int type >

[jira] [Commented] (BEAM-5500) Portable python sdk worker leaks memory in streaming mode

2018-09-27 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630296#comment-16630296 ] Robert Bradshaw commented on BEAM-5500: --- I've tried running this repeatedly in the direct runner,

[jira] [Commented] (BEAM-5468) Allow runner to set worker log level in Python SDK harness.

2018-09-25 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16626973#comment-16626973 ] Robert Bradshaw commented on BEAM-5468: --- Namespaced logging in the Python SDK would be convenient

[jira] [Updated] (BEAM-3622) DirectRunner memory issue with Python SDK

2018-09-25 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw updated BEAM-3622: -- Component/s: (was: sdk-py-harness) > DirectRunner memory issue with Python SDK >

[jira] [Created] (BEAM-5468) Allow runner to set worker log level in Python SDK harness.

2018-09-24 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-5468: - Summary: Allow runner to set worker log level in Python SDK harness. Key: BEAM-5468 URL: https://issues.apache.org/jira/browse/BEAM-5468 Project: Beam

[jira] [Commented] (BEAM-5442) PortableRunner swallows custom options for Runner

2018-09-21 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623368#comment-16623368 ] Robert Bradshaw commented on BEAM-5442: --- It seems like we should validate the options we know about,

***UNCHECKED*** [jira] [Created] (BEAM-5428) Implement cross-bundle state caching.

2018-09-19 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-5428: - Summary: Implement cross-bundle state caching. Key: BEAM-5428 URL: https://issues.apache.org/jira/browse/BEAM-5428 Project: Beam Issue Type: Improvement

[jira] [Resolved] (BEAM-5395) BeamPython data plane streams data

2018-09-18 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw resolved BEAM-5395. --- Resolution: Fixed Fix Version/s: 2.8.0 > BeamPython data plane streams data >

[jira] [Created] (BEAM-5395) BeamPython data plane streams data

2018-09-17 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-5395: - Summary: BeamPython data plane streams data Key: BEAM-5395 URL: https://issues.apache.org/jira/browse/BEAM-5395 Project: Beam Issue Type: Bug

[jira] [Created] (BEAM-5394) BeamPython on Flink runs at scale

2018-09-17 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-5394: - Summary: BeamPython on Flink runs at scale Key: BEAM-5394 URL: https://issues.apache.org/jira/browse/BEAM-5394 Project: Beam Issue Type: Bug

[jira] [Updated] (BEAM-5392) GroupByKey on Spark: All values for a single key need to fit in-memory at once

2018-09-17 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw updated BEAM-5392: -- Summary: GroupByKey on Spark: All values for a single key need to fit in-memory at once (was:

[jira] [Commented] (BEAM-3106) Consider not pinning all python dependencies, or moving them to requirements.txt

2018-09-13 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614093#comment-16614093 ] Robert Bradshaw commented on BEAM-3106: --- Our main requirements now specify version ranges (generally

[jira] [Commented] (BEAM-5354) Side Inputs seems to be non-working in the sdk-go

2018-09-13 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613379#comment-16613379 ] Robert Bradshaw commented on BEAM-5354: --- There are some worker harness bugs in handling of side

[jira] [Commented] (BEAM-4858) Clean up _BatchSizeEstimator in element-batching transform.

2018-09-12 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-4858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611858#comment-16611858 ] Robert Bradshaw commented on BEAM-4858: --- I created [https://github.com/apache/beam/pull/6375] which

[jira] [Commented] (BEAM-5214) Update Java quickstart to use maven

2018-09-03 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16601920#comment-16601920 ] Robert Bradshaw commented on BEAM-5214: --- Yes, though the initial command to generate the example

[jira] [Created] (BEAM-5255) Fix over-aggressive division futurization.

2018-08-29 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-5255: - Summary: Fix over-aggressive division futurization. Key: BEAM-5255 URL: https://issues.apache.org/jira/browse/BEAM-5255 Project: Beam Issue Type: Bug

[jira] [Created] (BEAM-5214) Update Java quickstart to use maven

2018-08-24 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-5214: - Summary: Update Java quickstart to use maven Key: BEAM-5214 URL: https://issues.apache.org/jira/browse/BEAM-5214 Project: Beam Issue Type: Bug

[jira] [Created] (BEAM-4782) Enforce KV coders for MultiMap side inputs

2018-07-12 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-4782: - Summary: Enforce KV coders for MultiMap side inputs Key: BEAM-4782 URL: https://issues.apache.org/jira/browse/BEAM-4782 Project: Beam Issue Type: Task

[jira] [Created] (BEAM-4781) PTransforms that simply return their input cause portable Flink to crash.

2018-07-12 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-4781: - Summary: PTransforms that simply return their input cause portable Flink to crash. Key: BEAM-4781 URL: https://issues.apache.org/jira/browse/BEAM-4781 Project:

[jira] [Created] (BEAM-4729) Conditionally propagate local GCS credentials to locally spawned docker images.

2018-07-03 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-4729: - Summary: Conditionally propagate local GCS credentials to locally spawned docker images. Key: BEAM-4729 URL: https://issues.apache.org/jira/browse/BEAM-4729

[jira] [Reopened] (BEAM-3883) Python SDK stages artifacts when talking to job server

2018-06-21 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw reopened BEAM-3883: --- Staging code is written, but not yet hooked up. > Python SDK stages artifacts when talking to

[jira] [Commented] (BEAM-4605) Make GroupByKey the primitive in the Python SDK

2018-06-20 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16518749#comment-16518749 ] Robert Bradshaw commented on BEAM-4605: --- At least the old direct runner relies on the composite

[jira] [Created] (BEAM-4605) Make GroupByKey the primitive in the Python SDK

2018-06-20 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-4605: - Summary: Make GroupByKey the primitive in the Python SDK Key: BEAM-4605 URL: https://issues.apache.org/jira/browse/BEAM-4605 Project: Beam Issue Type:

[jira] [Assigned] (BEAM-3884) Python SDK supports Impulse as a primitive transform

2018-06-19 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw reassigned BEAM-3884: - Assignee: Robert Bradshaw (was: Ahmet Altay) > Python SDK supports Impulse as a

[jira] [Created] (BEAM-4588) Use wheels for numpy

2018-06-19 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-4588: - Summary: Use wheels for numpy Key: BEAM-4588 URL: https://issues.apache.org/jira/browse/BEAM-4588 Project: Beam Issue Type: Task Components:

[jira] [Created] (BEAM-4578) Cleanup paths due to obsolete worker/sdk upgrades.

2018-06-18 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-4578: - Summary: Cleanup paths due to obsolete worker/sdk upgrades. Key: BEAM-4578 URL: https://issues.apache.org/jira/browse/BEAM-4578 Project: Beam Issue Type:

[jira] [Created] (BEAM-4565) Hot key fanout should not distribute keys to all shards.

2018-06-14 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-4565: - Summary: Hot key fanout should not distribute keys to all shards. Key: BEAM-4565 URL: https://issues.apache.org/jira/browse/BEAM-4565 Project: Beam Issue

[jira] [Commented] (BEAM-4494) Migrate website source code to apache/beam [website-migration] branch

2018-06-14 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-4494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512692#comment-16512692 ] Robert Bradshaw commented on BEAM-4494: --- Yes, we could just ask people to make changes elsewhere if

[jira] [Assigned] (BEAM-4546) Implement with hot key fanout for combiners

2018-06-13 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-4546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw reassigned BEAM-4546: - Assignee: Robert Bradshaw > Implement with hot key fanout for combiners >

[jira] [Commented] (BEAM-4494) Migrate website source code to apache/beam [website-migration] branch

2018-06-12 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-4494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510373#comment-16510373 ] Robert Bradshaw commented on BEAM-4494: --- Is there any advantage to having a separate branch? There

[jira] [Created] (BEAM-4524) We should not be using md5 to validate artifact integrity.

2018-06-08 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-4524: - Summary: We should not be using md5 to validate artifact integrity. Key: BEAM-4524 URL: https://issues.apache.org/jira/browse/BEAM-4524 Project: Beam

[jira] [Commented] (BEAM-2980) BagState.isEmpty needs a tighter spec

2018-05-22 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16484480#comment-16484480 ] Robert Bradshaw commented on BEAM-2980: --- This seems a bit more subtle. Given  BagState myBag = //

[jira] [Created] (BEAM-4150) Standardize use of PCollection coder proto attribute

2018-04-20 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-4150: - Summary: Standardize use of PCollection coder proto attribute Key: BEAM-4150 URL: https://issues.apache.org/jira/browse/BEAM-4150 Project: Beam Issue

[jira] [Assigned] (BEAM-4097) Python SDK should set the environment in the job submission protos

2018-04-19 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-4097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw reassigned BEAM-4097: - Assignee: Robert Bradshaw (was: Ahmet Altay) > Python SDK should set the environment in

[jira] [Resolved] (BEAM-3792) Python submits portable pipelines to the Flink-served endpoint.

2018-04-19 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw resolved BEAM-3792. --- Resolution: Fixed Fix Version/s: 2.5.0 > Python submits portable pipelines to the

[jira] [Commented] (BEAM-3911) Do we want to encourage release pinning?

2018-04-17 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441725#comment-16441725 ] Robert Bradshaw commented on BEAM-3911: --- No, we don't want to encourage this.  > Do we want to

[jira] [Commented] (BEAM-4105) Python Postcommit suite does not start on Jenkins - proto generation failed

2018-04-17 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441669#comment-16441669 ] Robert Bradshaw commented on BEAM-4105: --- Perhaps the pip is too old?  > Python Postcommit suite does

[jira] [Created] (BEAM-4098) Handle WindowInto in the Java SDK Harness

2018-04-16 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-4098: - Summary: Handle WindowInto in the Java SDK Harness Key: BEAM-4098 URL: https://issues.apache.org/jira/browse/BEAM-4098 Project: Beam Issue Type: Task

[jira] [Created] (BEAM-4097) Python SDK should set the environment in the job submission protos

2018-04-16 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-4097: - Summary: Python SDK should set the environment in the job submission protos Key: BEAM-4097 URL: https://issues.apache.org/jira/browse/BEAM-4097 Project: Beam

[jira] [Created] (BEAM-4074) Cleanup metrics/counters code

2018-04-13 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-4074: - Summary: Cleanup metrics/counters code Key: BEAM-4074 URL: https://issues.apache.org/jira/browse/BEAM-4074 Project: Beam Issue Type: Task

[jira] [Commented] (BEAM-3407) Post-commit test fails with "Backing channel 'beam8' is disconnected"

2018-04-10 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433219#comment-16433219 ] Robert Bradshaw commented on BEAM-3407: --- This is still happening, e.g.

[jira] [Created] (BEAM-4030) Add CombineFn.compact, similar to Java

2018-04-06 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-4030: - Summary: Add CombineFn.compact, similar to Java Key: BEAM-4030 URL: https://issues.apache.org/jira/browse/BEAM-4030 Project: Beam Issue Type: Bug

[jira] [Commented] (BEAM-3737) Key-aware batching function

2018-04-04 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426269#comment-16426269 ] Robert Bradshaw commented on BEAM-3737: --- It seems that GroupByKey() would already give you values

[jira] [Updated] (BEAM-4012) Suppress tracebacks on failure-catching tests

2018-04-04 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-4012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw updated BEAM-4012: -- Labels: beginner (was: ) > Suppress tracebacks on failure-catching tests >

[jira] [Created] (BEAM-4012) Suppress tracebacks on failure-catching tests

2018-04-04 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-4012: - Summary: Suppress tracebacks on failure-catching tests Key: BEAM-4012 URL: https://issues.apache.org/jira/browse/BEAM-4012 Project: Beam Issue Type: Bug

[jira] [Created] (BEAM-3911) Do we want to encourage release pinning?

2018-03-22 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-3911: - Summary: Do we want to encourage release pinning? Key: BEAM-3911 URL: https://issues.apache.org/jira/browse/BEAM-3911 Project: Beam Issue Type: Bug

[jira] [Commented] (BEAM-1442) Performance improvement of the Python DirectRunner

2018-03-20 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407244#comment-16407244 ] Robert Bradshaw commented on BEAM-1442: --- As of today, you can get this from pip. Note for avro in

[jira] [Created] (BEAM-3890) Add withHotKeyFanout to Python

2018-03-20 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-3890: - Summary: Add withHotKeyFanout to Python Key: BEAM-3890 URL: https://issues.apache.org/jira/browse/BEAM-3890 Project: Beam Issue Type: Bug

[jira] [Resolved] (BEAM-1442) Performance improvement of the Python DirectRunner

2018-03-20 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw resolved BEAM-1442. --- Resolution: Fixed While one is rarely "done" with possible performance improvements, I'm going

[jira] [Updated] (BEAM-3851) Support element timestamps while publishing to Kafka.

2018-03-20 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw updated BEAM-3851: -- Fix Version/s: (was: 2.4.0) 2.5.0 > Support element timestamps while

[jira] [Resolved] (BEAM-3815) 2.4.0 RC2 uses java worker version beam-master-20180228, should be 2.4.0 or beam-2.4.0

2018-03-20 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw resolved BEAM-3815. --- Resolution: Fixed > 2.4.0 RC2 uses java worker version beam-master-20180228, should be 2.4.0

[jira] [Updated] (BEAM-3868) Move state and time logic out of the ParDoEvaluator

2018-03-20 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw updated BEAM-3868: -- Fix Version/s: (was: 2.4.0) > Move state and time logic out of the ParDoEvaluator >

[jira] [Created] (BEAM-3871) Ensure all non-root transforms have a Parent

2018-03-17 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-3871: - Summary: Ensure all non-root transforms have a Parent Key: BEAM-3871 URL: https://issues.apache.org/jira/browse/BEAM-3871 Project: Beam Issue Type: Bug

[jira] [Created] (BEAM-3865) Incorrect timestamp on merging window outputs.

2018-03-16 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-3865: - Summary: Incorrect timestamp on merging window outputs. Key: BEAM-3865 URL: https://issues.apache.org/jira/browse/BEAM-3865 Project: Beam Issue Type: Bug

[jira] [Updated] (BEAM-3815) 2.4.0 RC2 uses java worker version beam-master-20180228, should be 2.4.0 or beam-2.4.0

2018-03-08 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw updated BEAM-3815: -- Fix Version/s: 2.4.0 > 2.4.0 RC2 uses java worker version beam-master-20180228, should be 2.4.0

[jira] [Created] (BEAM-3812) Avoid pickling PTransforms in proto representation

2018-03-08 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-3812: - Summary: Avoid pickling PTransforms in proto representation Key: BEAM-3812 URL: https://issues.apache.org/jira/browse/BEAM-3812 Project: Beam Issue Type:

[jira] [Commented] (BEAM-3799) Nexmark Query 10 breaks with direct runner

2018-03-07 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389910#comment-16389910 ] Robert Bradshaw commented on BEAM-3799: --- The direct runner does extra checking. Looks like the

[jira] [Updated] (BEAM-3409) Unexpected behavior of DoFn teardown method running in unit tests

2018-03-07 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw updated BEAM-3409: -- Fix Version/s: (was: 2.4.0) 2.5.0 > Unexpected behavior of DoFn teardown

[jira] [Created] (BEAM-3792) Python submits portable pipelines to the Flink-served endpoint.

2018-03-06 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-3792: - Summary: Python submits portable pipelines to the Flink-served endpoint. Key: BEAM-3792 URL: https://issues.apache.org/jira/browse/BEAM-3792 Project: Beam

[jira] [Commented] (BEAM-3780) Add a utility to instantiate a partially unknown coder

2018-03-05 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386905#comment-16386905 ] Robert Bradshaw commented on BEAM-3780: --- Isn't this capability already present as part of

[jira] [Resolved] (BEAM-3409) Unexpected behavior of DoFn teardown method running in unit tests

2018-03-01 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw resolved BEAM-3409. --- Resolution: Fixed > Unexpected behavior of DoFn teardown method running in unit tests >

[jira] [Created] (BEAM-3768) Compile error for Flink translation

2018-03-01 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-3768: - Summary: Compile error for Flink translation Key: BEAM-3768 URL: https://issues.apache.org/jira/browse/BEAM-3768 Project: Beam Issue Type: Test

[jira] [Commented] (BEAM-3479) Add a regression test for the DoFn classloader selection

2018-03-01 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382865#comment-16382865 ] Robert Bradshaw commented on BEAM-3479: --- Bumping to the next version, as this is non-blocking (per

[jira] [Updated] (BEAM-3479) Add a regression test for the DoFn classloader selection

2018-03-01 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw updated BEAM-3479: -- Fix Version/s: (was: 2.4.0) 2.5.0 > Add a regression test for the DoFn

[jira] [Resolved] (BEAM-3611) Split KafkaIO.java into smaller files

2018-03-01 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw resolved BEAM-3611. --- Resolution: Fixed Closing this as resolved, based on an inspection of the PR.  > Split

[jira] [Updated] (BEAM-3700) PipelineOptionsFactory leaks memory

2018-02-28 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw updated BEAM-3700: -- Fix Version/s: (was: 2.4.0) 2.5.0 > PipelineOptionsFactory leaks memory >

[jira] [Closed] (BEAM-3689) Direct runner leak a reader for every 10 input records

2018-02-28 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw closed BEAM-3689. - Resolution: Fixed Based on my reading of the PR, this is fixed.  > Direct runner leak a reader

[jira] [Updated] (BEAM-3702) Support system properties source for pipeline options

2018-02-28 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw updated BEAM-3702: -- Fix Version/s: (was: 2.4.0) 2.5.0 > Support system properties source for

[jira] [Updated] (BEAM-3749) support customized trigger/accumulationMode in BeamSql

2018-02-28 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw updated BEAM-3749: -- Fix Version/s: (was: 2.4.0) 2.5.0 > support customized

[jira] [Commented] (BEAM-3749) support customized trigger/accumulationMode in BeamSql

2018-02-28 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381530#comment-16381530 ] Robert Bradshaw commented on BEAM-3749: --- I wonder if sink-based triggers would be a better fit here

[jira] [Created] (BEAM-3713) Consider moving away from nose to nose2 or pytest.

2018-02-15 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-3713: - Summary: Consider moving away from nose to nose2 or pytest. Key: BEAM-3713 URL: https://issues.apache.org/jira/browse/BEAM-3713 Project: Beam Issue Type:

[jira] [Closed] (BEAM-3205) Publicly document known coder wire formats and their URNs

2018-02-12 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw closed BEAM-3205. - Resolution: Fixed Fix Version/s: 2.4.0 > Publicly document known coder wire formats and

[jira] [Resolved] (BEAM-3074) Propagate pipeline protos through Dataflow API from Python

2018-02-12 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw resolved BEAM-3074. --- Resolution: Fixed Fix Version/s: 2.4.0 > Propagate pipeline protos through Dataflow API

[jira] [Updated] (BEAM-3644) Speed up Python DirectRunner execution by using the FnApiRunner when possible

2018-02-07 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw updated BEAM-3644: -- Description: Local execution of Beam pipelines on the current Python DirectRunner currently

[jira] [Created] (BEAM-3625) DoFn.XxxParam does not work for Map and FlatMap

2018-02-06 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-3625: - Summary: DoFn.XxxParam does not work for Map and FlatMap Key: BEAM-3625 URL: https://issues.apache.org/jira/browse/BEAM-3625 Project: Beam Issue Type: Bug

[jira] [Created] (BEAM-3595) Normalize URNs across SDKs and runners.

2018-02-01 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-3595: - Summary: Normalize URNs across SDKs and runners. Key: BEAM-3595 URL: https://issues.apache.org/jira/browse/BEAM-3595 Project: Beam Issue Type: Bug

[jira] [Resolved] (BEAM-2804) support TIMESTAMP in Sort

2018-01-23 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw resolved BEAM-2804. --- Resolution: Fixed Fix Version/s: 2.3.0 > support TIMESTAMP in Sort >

[jira] [Updated] (BEAM-3510) Python Runner API serialization for CombinePerKey drops arguments to CombineFn

2018-01-22 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw updated BEAM-3510: -- Fix Version/s: 2.3.0 > Python Runner API serialization for CombinePerKey drops arguments to

[jira] [Resolved] (BEAM-3510) Python Runner API serialization for CombinePerKey drops arguments to CombineFn

2018-01-22 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw resolved BEAM-3510. --- Resolution: Fixed > Python Runner API serialization for CombinePerKey drops arguments to

[jira] [Assigned] (BEAM-3510) Python Runner API serialization for CombinePerKey drops arguments to CombineFn

2018-01-22 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw reassigned BEAM-3510: - Assignee: Robert Bradshaw (was: Charles Chen) > Python Runner API serialization for

[jira] [Resolved] (BEAM-2906) Dataflow supports portable progress reporting

2018-01-22 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw resolved BEAM-2906. --- Resolution: Fixed Fix Version/s: 2.3.0 > Dataflow supports portable progress reporting

[jira] [Assigned] (BEAM-2906) Dataflow supports portable progress reporting

2018-01-22 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw reassigned BEAM-2906: - Assignee: Robert Bradshaw > Dataflow supports portable progress reporting >

[jira] [Created] (BEAM-3490) Reasonable Python direct runner batch performance.

2018-01-17 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-3490: - Summary: Reasonable Python direct runner batch performance. Key: BEAM-3490 URL: https://issues.apache.org/jira/browse/BEAM-3490 Project: Beam Issue Type:

[jira] [Created] (BEAM-3419) Enable iterable side input for beam runners.

2018-01-05 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-3419: - Summary: Enable iterable side input for beam runners. Key: BEAM-3419 URL: https://issues.apache.org/jira/browse/BEAM-3419 Project: Beam Issue Type: New

[jira] [Created] (BEAM-3393) Empty flattens cannot be used as side inputs.

2017-12-22 Thread Robert Bradshaw (JIRA)
Robert Bradshaw created BEAM-3393: - Summary: Empty flattens cannot be used as side inputs. Key: BEAM-3393 URL: https://issues.apache.org/jira/browse/BEAM-3393 Project: Beam Issue Type: Bug

[jira] [Resolved] (BEAM-3369) Python HEAD fails tests due to a ValueError

2017-12-19 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw resolved BEAM-3369. --- Resolution: Fixed Fix Version/s: 2.3.0 > Python HEAD fails tests due to a ValueError >

[jira] [Assigned] (BEAM-2937) Fn API combiner support w/ lifting to PGBK

2017-12-18 Thread Robert Bradshaw (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Bradshaw reassigned BEAM-2937: - Assignee: Robert Bradshaw > Fn API combiner support w/ lifting to PGBK >

  1   2   >