[jira] [Updated] (BEAM-5442) PortableRunner swallows custom options for Runner

2018-10-12 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated BEAM-5442: --- Fix Version/s: 2.9.0 > PortableRunner swallows custom options for Runner >

[jira] [Updated] (BEAM-5442) PortableRunner swallows custom options for Runner

2018-10-12 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated BEAM-5442: --- Fix Version/s: (was: 2.8.0) > PortableRunner swallows custom options for Runner >

[jira] [Reopened] (BEAM-5442) PortableRunner swallows custom options for Runner

2018-10-12 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise reopened BEAM-5442: > PortableRunner swallows custom options for Runner > -

[jira] [Commented] (BEAM-5713) Flink portable runner schedules all tasks of streaming job on same task manager

2018-10-12 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647978#comment-16647978 ] Thomas Weise commented on BEAM-5713: It isn't consistent / deterministic. Subsequent runs of same

[jira] [Updated] (BEAM-5724) Beam creates too many sdk_worker processes with --sdk-worker-parallelism=stage

2018-10-11 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated BEAM-5724: --- Labels: portability-flink (was: ) > Beam creates too many sdk_worker processes with

[jira] [Comment Edited] (BEAM-5713) Flink portable runner schedules all tasks of streaming job on same task manager

2018-10-11 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647169#comment-16647169 ] Thomas Weise edited comment on BEAM-5713 at 10/11/18 11:25 PM: --- The

[jira] [Comment Edited] (BEAM-5713) Flink portable runner schedules all tasks of streaming job on same task manager

2018-10-11 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647169#comment-16647169 ] Thomas Weise edited comment on BEAM-5713 at 10/11/18 11:25 PM: --- The

[jira] [Comment Edited] (BEAM-5713) Flink portable runner schedules all tasks of streaming job on same task manager

2018-10-11 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647169#comment-16647169 ] Thomas Weise edited comment on BEAM-5713 at 10/11/18 11:23 PM: --- With

[jira] [Comment Edited] (BEAM-5713) Flink portable runner schedules all tasks of streaming job on same task manager

2018-10-11 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647169#comment-16647169 ] Thomas Weise edited comment on BEAM-5713 at 10/11/18 11:23 PM: --- With

[jira] [Commented] (BEAM-5713) Flink portable runner schedules all tasks of streaming job on same task manager

2018-10-11 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647169#comment-16647169 ] Thomas Weise commented on BEAM-5713: With parallelism 4, *almost* same

[jira] [Updated] (BEAM-5713) Flink portable runner schedules all tasks of streaming job on same task manager

2018-10-11 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated BEAM-5713: --- Attachment: image-2018-10-11-16-20-45-221.png > Flink portable runner schedules all tasks of

[jira] [Comment Edited] (BEAM-5713) Flink portable runner schedules all tasks of streaming job on same task manager

2018-10-11 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646887#comment-16646887 ] Thomas Weise edited comment on BEAM-5713 at 10/11/18 6:48 PM: -- I added more

[jira] [Comment Edited] (BEAM-5713) Flink portable runner schedules all tasks of streaming job on same task manager

2018-10-11 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646887#comment-16646887 ] Thomas Weise edited comment on BEAM-5713 at 10/11/18 6:48 PM: -- I added more

[jira] [Commented] (BEAM-5713) Flink portable runner schedules all tasks of streaming job on same task manager

2018-10-11 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646887#comment-16646887 ] Thomas Weise commented on BEAM-5713: I added more tasks and they are all squeezed into the same slots

[jira] [Updated] (BEAM-5713) Flink portable runner schedules all tasks of streaming job on same task manager

2018-10-11 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated BEAM-5713: --- Attachment: image-2018-10-11-11-43-50-333.png > Flink portable runner schedules all tasks of

[jira] [Commented] (BEAM-5713) Flink portable runner schedules all tasks of streaming job on same task manager

2018-10-11 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646847#comment-16646847 ] Thomas Weise commented on BEAM-5713: Just to confirm my observation: We have 16 task slots per TM. The

[jira] [Commented] (BEAM-5467) Python Flink ValidatesRunner job fixes

2018-10-11 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646790#comment-16646790 ] Thomas Weise commented on BEAM-5467: [~swegner] here is an example. (It was necessary to download the

[jira] [Commented] (BEAM-5713) Flink portable runner schedules all tasks of streaming job on same task manager

2018-10-11 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646493#comment-16646493 ] Thomas Weise commented on BEAM-5713: That's something we should fix for the standard Impulse

[jira] [Resolved] (BEAM-5633) Python SDK harness logging client failure

2018-10-11 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise resolved BEAM-5633. Resolution: Fixed Fix Version/s: 2.8.0 > Python SDK harness logging client failure >

[jira] [Commented] (BEAM-4819) Make portable Flink runner JobBundleFactory configurable

2018-10-10 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-4819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16645975#comment-16645975 ] Thomas Weise commented on BEAM-4819: A good solution might be AutoService annotated Environment

[jira] [Commented] (BEAM-5713) Flink portable runner schedules all tasks of streaming job on same task manager

2018-10-10 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16645821#comment-16645821 ] Thomas Weise commented on BEAM-5713: Can be reproduced with branch: 

[jira] [Created] (BEAM-5713) Flink portable runner schedules all tasks of streaming job on same task manager

2018-10-10 Thread Thomas Weise (JIRA)
Thomas Weise created BEAM-5713: -- Summary: Flink portable runner schedules all tasks of streaming job on same task manager Key: BEAM-5713 URL: https://issues.apache.org/jira/browse/BEAM-5713 Project:

[jira] [Assigned] (BEAM-5713) Flink portable runner schedules all tasks of streaming job on same task manager

2018-10-10 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise reassigned BEAM-5713: -- Assignee: (was: Aljoscha Krettek) > Flink portable runner schedules all tasks of streaming

[jira] [Resolved] (BEAM-3727) Never shutdown sources in Flink Streaming execution mode

2018-10-10 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise resolved BEAM-3727. Resolution: Fixed Assignee: Maximilian Michels (was: Aljoscha Krettek) Fix

[jira] [Resolved] (BEAM-5687) Checkpointing in portable pipelines does not work

2018-10-09 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise resolved BEAM-5687. Resolution: Fixed Fix Version/s: (was: 2.9.0) 2.8.0 > Checkpointing

[jira] [Updated] (BEAM-3676) FlinkRunner: Portable state service

2018-10-08 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated BEAM-3676: --- Labels: portability (was: portability portability-flink) > FlinkRunner: Portable state service >

[jira] [Closed] (BEAM-3676) FlinkRunner: Portable state service

2018-10-08 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise closed BEAM-3676. -- Resolution: Done Fix Version/s: Not applicable > FlinkRunner: Portable state service >

[jira] [Resolved] (BEAM-2916) Python SDK support for portable user state

2018-10-08 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise resolved BEAM-2916. Resolution: Implemented Assignee: Robert Bradshaw Fix Version/s: 2.8.0 > Python SDK

[jira] [Commented] (BEAM-5442) PortableRunner swallows custom options for Runner

2018-10-07 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16641279#comment-16641279 ] Thomas Weise commented on BEAM-5442: [~mxm] I checked and see that an unknown option such as 

[jira] [Commented] (BEAM-5671) Website automation fit-and-finish

2018-10-05 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640524#comment-16640524 ] Thomas Weise commented on BEAM-5671: [~swegner] it looks like there are still empty commits being

[jira] [Comment Edited] (BEAM-5467) Python Flink ValidatesRunner job fixes

2018-10-05 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640413#comment-16640413 ] Thomas Weise edited comment on BEAM-5467 at 10/5/18 10:18 PM: -- Yes, and as

[jira] [Commented] (BEAM-5467) Python Flink ValidatesRunner job fixes

2018-10-05 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640413#comment-16640413 ] Thomas Weise commented on BEAM-5467: Yes, and as you had also noticed earlier, they fail in Jenkins

[jira] [Assigned] (BEAM-5633) Python SDK harness logging client failure

2018-10-05 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise reassigned BEAM-5633: -- Assignee: Micah Wylde > Python SDK harness logging client failure >

[jira] [Commented] (BEAM-5442) PortableRunner swallows custom options for Runner

2018-10-05 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640145#comment-16640145 ] Thomas Weise commented on BEAM-5442: [~udim] as stop gap we could probably skip --beam_plugins here:

[jira] [Commented] (BEAM-4819) Make portable Flink runner JobBundleFactory configurable

2018-10-04 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-4819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638445#comment-16638445 ] Thomas Weise commented on BEAM-4819: Partially. We still don't have support for a custom environment

[jira] [Resolved] (BEAM-5625) PortableRunner.PipelineResult.cancel not working

2018-10-03 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise resolved BEAM-5625. Resolution: Fixed > PortableRunner.PipelineResult.cancel not working >

[jira] [Commented] (BEAM-5442) PortableRunner swallows custom options for Runner

2018-10-03 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637627#comment-16637627 ] Thomas Weise commented on BEAM-5442: Yes, the PR is open. > PortableRunner swallows custom options

[jira] [Updated] (BEAM-5440) Add option to mount a directory inside SDK harness containers

2018-10-03 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated BEAM-5440: --- Fix Version/s: (was: 2.8.0) > Add option to mount a directory inside SDK harness containers >

[jira] [Resolved] (BEAM-5500) Portable python sdk worker leaks memory in streaming mode

2018-10-03 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise resolved BEAM-5500. Resolution: Fixed Fix Version/s: 2.8.0 > Portable python sdk worker leaks memory in

[jira] [Commented] (BEAM-5500) Portable python sdk worker leaks memory in streaming mode

2018-10-03 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637361#comment-16637361 ] Thomas Weise commented on BEAM-5500: I verified that this fixes the memory leak observed running:

[jira] [Updated] (BEAM-5633) Python SDK harness logging client failure

2018-10-03 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated BEAM-5633: --- Labels: portability-flink (was: ) > Python SDK harness logging client failure >

[jira] [Commented] (BEAM-5633) Python SDK harness logging client failure

2018-10-03 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637316#comment-16637316 ] Thomas Weise commented on BEAM-5633:   {code:java} [grpc-default-executor-1] WARN

[jira] [Created] (BEAM-5633) Python SDK harness logging client failure

2018-10-03 Thread Thomas Weise (JIRA)
Thomas Weise created BEAM-5633: -- Summary: Python SDK harness logging client failure Key: BEAM-5633 URL: https://issues.apache.org/jira/browse/BEAM-5633 Project: Beam Issue Type: Bug

[jira] [Commented] (BEAM-5187) Create a ProcessJobBundleFactory for non-dockerized SDK harness

2018-10-03 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637273#comment-16637273 ] Thomas Weise commented on BEAM-5187: We implemented a custom environment factory for Python that uses

[jira] [Created] (BEAM-5625) PortableRunner.PipelineResult.cancel not working

2018-10-02 Thread Thomas Weise (JIRA)
Thomas Weise created BEAM-5625: -- Summary: PortableRunner.PipelineResult.cancel not working Key: BEAM-5625 URL: https://issues.apache.org/jira/browse/BEAM-5625 Project: Beam Issue Type: Bug

[jira] [Commented] (BEAM-5520) Flink runner per operator SDK harness option

2018-10-02 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635934#comment-16635934 ] Thomas Weise commented on BEAM-5520: An new portable pipeline option was added to support this. The

[jira] [Commented] (BEAM-5443) Simplify Python pipeline options for portable runner

2018-10-02 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635920#comment-16635920 ] Thomas Weise commented on BEAM-5443: --job_endpoint has to stay because the default is already to

[jira] [Assigned] (BEAM-5527) Java PortableRunner produces wrongly shaded jar

2018-10-02 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise reassigned BEAM-5527: -- Assignee: (was: Maximilian Michels) > Java PortableRunner produces wrongly shaded jar >

[jira] [Resolved] (BEAM-5441) Portable Wordcount fails in GreedyPipelineFuser

2018-10-02 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise resolved BEAM-5441. Resolution: Fixed > Portable Wordcount fails in GreedyPipelineFuser >

[jira] [Resolved] (BEAM-5520) Flink runner per operator SDK harness option

2018-10-01 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise resolved BEAM-5520. Resolution: Fixed Fix Version/s: 2.8.0 > Flink runner per operator SDK harness option >

[jira] [Resolved] (BEAM-5516) Flink master option from Python SDK not honored by Flink runner

2018-10-01 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise resolved BEAM-5516. Resolution: Fixed Fix Version/s: 2.8.0 https://github.com/apache/beam/pull/6524 > Flink

[jira] [Assigned] (BEAM-5516) Flink master option from Python SDK not honored by Flink runner

2018-09-30 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise reassigned BEAM-5516: -- Assignee: Thomas Weise (was: Maximilian Michels) > Flink master option from Python SDK not

[jira] [Resolved] (BEAM-5509) Python pipeline_options doesn't handle int type

2018-09-29 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise resolved BEAM-5509. Resolution: Fixed Fix Version/s: 2.8.0 > Python pipeline_options doesn't handle int type >

[jira] [Resolved] (BEAM-5443) Simplify Python pipeline options for portable runner

2018-09-29 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise resolved BEAM-5443. Resolution: Fixed Fix Version/s: 2.8.0 > Simplify Python pipeline options for portable

[jira] [Updated] (BEAM-5525) Intuitive default behavior for sdk_location pipeline option

2018-09-29 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated BEAM-5525: --- Labels: portability (was: ) > Intuitive default behavior for sdk_location pipeline option >

[jira] [Assigned] (BEAM-5525) Intuitive default behavior for sdk_location pipeline option

2018-09-29 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise reassigned BEAM-5525: -- Assignee: (was: Robert Bradshaw) > Intuitive default behavior for sdk_location pipeline

[jira] [Created] (BEAM-5525) Intuitive default behavior for sdk_location pipeline option

2018-09-29 Thread Thomas Weise (JIRA)
Thomas Weise created BEAM-5525: -- Summary: Intuitive default behavior for sdk_location pipeline option Key: BEAM-5525 URL: https://issues.apache.org/jira/browse/BEAM-5525 Project: Beam Issue

[jira] [Commented] (BEAM-5509) Python pipeline_options doesn't handle int type

2018-09-28 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632172#comment-16632172 ] Thomas Weise commented on BEAM-5509: A bit more digging shows that the culprit is really the

[jira] [Created] (BEAM-5520) Flink runner per operator SDK harness option

2018-09-27 Thread Thomas Weise (JIRA)
Thomas Weise created BEAM-5520: -- Summary: Flink runner per operator SDK harness option Key: BEAM-5520 URL: https://issues.apache.org/jira/browse/BEAM-5520 Project: Beam Issue Type: Improvement

[jira] [Assigned] (BEAM-5509) Python pipeline_options doesn't handle int type

2018-09-27 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise reassigned BEAM-5509: -- Assignee: Thomas Weise (was: Robert Bradshaw) > Python pipeline_options doesn't handle int

[jira] [Commented] (BEAM-5509) Python pipeline_options doesn't handle int type

2018-09-27 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631152#comment-16631152 ] Thomas Weise commented on BEAM-5509: The unwanted conversion to floating point occurs in

[jira] [Assigned] (BEAM-5443) Simplify Python pipeline options for portable runner

2018-09-27 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise reassigned BEAM-5443: -- Assignee: Thomas Weise > Simplify Python pipeline options for portable runner >

[jira] [Commented] (BEAM-5467) Python Flink ValidatesRunner job fixes

2018-09-27 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631010#comment-16631010 ] Thomas Weise commented on BEAM-5467: [~angoenka] should we try to turn off the parallel execution? I

[jira] [Commented] (BEAM-5500) Portable python sdk worker leaks memory in streaming mode

2018-09-27 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630515#comment-16630515 ] Thomas Weise commented on BEAM-5500: The pipeline runs in streaming mode. With the Flink runner,

[jira] [Commented] (BEAM-5516) Flink master option from Python SDK not honored by Flink runner

2018-09-26 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629502#comment-16629502 ] Thomas Weise commented on BEAM-5516: Command: {code:java} python -m apache_beam.examples.wordcount  

[jira] [Created] (BEAM-5516) Flink master option from Python SDK not honored by Flink runner

2018-09-26 Thread Thomas Weise (JIRA)
Thomas Weise created BEAM-5516: -- Summary: Flink master option from Python SDK not honored by Flink runner Key: BEAM-5516 URL: https://issues.apache.org/jira/browse/BEAM-5516 Project: Beam

[jira] [Commented] (BEAM-5509) Python pipeline_options doesn't handle int type

2018-09-25 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628141#comment-16628141 ] Thomas Weise commented on BEAM-5509: Pass --parallelism=1 and then job_utils.dict_to_struct(options)

[jira] [Created] (BEAM-5509) Python pipeline_options doesn't handle int type

2018-09-25 Thread Thomas Weise (JIRA)
Thomas Weise created BEAM-5509: -- Summary: Python pipeline_options doesn't handle int type Key: BEAM-5509 URL: https://issues.apache.org/jira/browse/BEAM-5509 Project: Beam Issue Type: Bug

[jira] [Updated] (BEAM-5497) Provide support for Gevent

2018-09-25 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated BEAM-5497: --- Component/s: (was: runner-flink) sdk-py-harness > Provide support for Gevent >

[jira] [Assigned] (BEAM-5497) Provide support for Gevent

2018-09-25 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise reassigned BEAM-5497: -- Assignee: (was: Aljoscha Krettek) > Provide support for Gevent >

[jira] [Updated] (BEAM-4819) Make portable Flink runner JobBundleFactory configurable

2018-09-24 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-4819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated BEAM-4819: --- Labels: portability portability-flink (was: portability) > Make portable Flink runner

[jira] [Commented] (BEAM-5468) Allow runner to set worker log level in Python SDK harness.

2018-09-24 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16626687#comment-16626687 ] Thomas Weise commented on BEAM-5468: Looks good to but I think we also need to consider the runners

[jira] [Resolved] (BEAM-5455) Don't info log for every bundle in the python sdk

2018-09-24 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise resolved BEAM-5455. Resolution: Fixed Fix Version/s: 2.8.0 > Don't info log for every bundle in the python sdk >

[jira] [Commented] (BEAM-5468) Allow runner to set worker log level in Python SDK harness.

2018-09-24 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16626604#comment-16626604 ] Thomas Weise commented on BEAM-5468: I did not find a related option in the Java SDK. Logging in the

[jira] [Commented] (BEAM-5468) Allow runner to set worker log level in Python SDK harness.

2018-09-24 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625743#comment-16625743 ] Thomas Weise commented on BEAM-5468: I was also planning to look at this. My idea was to add another

[jira] [Created] (BEAM-5467) Python Flink ValidatesRunner job fixes

2018-09-22 Thread Thomas Weise (JIRA)
Thomas Weise created BEAM-5467: -- Summary: Python Flink ValidatesRunner job fixes Key: BEAM-5467 URL: https://issues.apache.org/jira/browse/BEAM-5467 Project: Beam Issue Type: Improvement

[jira] [Commented] (BEAM-5459) Publish javadocs/pydocs to an external source, i.e. buildbot

2018-09-21 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623923#comment-16623923 ] Thomas Weise commented on BEAM-5459: [~swegner] these are Python scripts where projects can add their

[jira] [Updated] (BEAM-5455) Don't info log for every bundle in the python sdk

2018-09-20 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated BEAM-5455: --- Labels: portability-flink (was: ) > Don't info log for every bundle in the python sdk >

[jira] [Assigned] (BEAM-5455) Don't info log for every bundle in the python sdk

2018-09-20 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise reassigned BEAM-5455: -- Assignee: Micah Wylde (was: Robert Bradshaw) > Don't info log for every bundle in the python

[jira] [Commented] (BEAM-5442) PortableRunner swallows custom options for Runner

2018-09-20 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622452#comment-16622452 ] Thomas Weise commented on BEAM-5442: Yeah, except that it would be nice to solve this generically (not

[jira] [Commented] (BEAM-5443) Simplify Python pipeline options for portable runner

2018-09-20 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622415#comment-16622415 ] Thomas Weise commented on BEAM-5443: The current minimum command line for wordcount: {code:java}

[jira] [Updated] (BEAM-5443) Simplify Python pipeline options for portable runner

2018-09-20 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated BEAM-5443: --- Labels: portability (was: ) > Simplify Python pipeline options for portable runner >

[jira] [Created] (BEAM-5443) Simplify Python pipeline options for portable runner

2018-09-20 Thread Thomas Weise (JIRA)
Thomas Weise created BEAM-5443: -- Summary: Simplify Python pipeline options for portable runner Key: BEAM-5443 URL: https://issues.apache.org/jira/browse/BEAM-5443 Project: Beam Issue Type:

[jira] [Commented] (BEAM-3676) FlinkRunner: Portable state service

2018-09-20 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622304#comment-16622304 ] Thomas Weise commented on BEAM-3676: Wiring for user state in the Flink runner isn't done yet. >

[jira] [Closed] (BEAM-3675) FlinkRunner: Logging server

2018-09-20 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise closed BEAM-3675. -- Resolution: Done Fix Version/s: Not applicable > FlinkRunner: Logging server >

[jira] [Commented] (BEAM-3675) FlinkRunner: Logging server

2018-09-20 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622302#comment-16622302 ] Thomas Weise commented on BEAM-3675: I don't think there is any work left to do in the Flink runner

[jira] [Updated] (BEAM-3676) FlinkRunner: Portable state service

2018-09-20 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-3676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated BEAM-3676: --- Labels: portability portability-flink (was: ) > FlinkRunner: Portable state service >

[jira] [Commented] (BEAM-4681) Integrate support for timers using the portability APIs into Flink

2018-09-20 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622287#comment-16622287 ] Thomas Weise commented on BEAM-4681: CC: [~mxm] > Integrate support for timers using the portability

[jira] [Closed] (BEAM-5110) Reconile Flink JVM singleton management with deployment

2018-09-20 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise closed BEAM-5110. -- Resolution: Fixed Fix Version/s: Not applicable > Reconile Flink JVM singleton management with

[jira] [Commented] (BEAM-5110) Reconile Flink JVM singleton management with deployment

2018-09-20 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622282#comment-16622282 ] Thomas Weise commented on BEAM-5110: Closing, current implementation is sufficient to run portable

[jira] [Closed] (BEAM-4285) Flink batch state request handler

2018-09-20 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise closed BEAM-4285. -- > Flink batch state request handler > - > > Key: BEAM-4285

[jira] [Updated] (BEAM-5396) Flink portable runner savepoint / upgrade support

2018-09-20 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated BEAM-5396: --- Labels: portability portability-flink (was: ) > Flink portable runner savepoint / upgrade support >

[jira] [Commented] (BEAM-4285) Flink batch state request handler

2018-09-20 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622277#comment-16622277 ] Thomas Weise commented on BEAM-4285: Closing the issue as initial implementation was done. Please open

[jira] [Resolved] (BEAM-4285) Flink batch state request handler

2018-09-20 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise resolved BEAM-4285. Resolution: Fixed Fix Version/s: 2.7.0 > Flink batch state request handler >

[jira] [Commented] (BEAM-5442) PortableRunner swallows custom options for Runner

2018-09-20 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622268#comment-16622268 ] Thomas Weise commented on BEAM-5442: Ideally all of the options would be passed through, I suppose

[jira] [Assigned] (BEAM-5442) PortableRunner swallows custom options for Runner

2018-09-20 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise reassigned BEAM-5442: -- Assignee: (was: Kenneth Knowles) > PortableRunner swallows custom options for Runner >

[jira] [Commented] (BEAM-5440) Add option to mount a directory inside SDK harness containers

2018-09-20 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622049#comment-16622049 ] Thomas Weise commented on BEAM-5440: I think that's a good idea that will help users during

[jira] [Commented] (BEAM-5396) Flink portable runner savepoint / upgrade support

2018-09-17 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16618418#comment-16618418 ] Thomas Weise commented on BEAM-5396: The operability features are table stakes for serious projects.

[jira] [Commented] (BEAM-5397) Flink portable runner GRPC cleanup failure after user class loader was removed

2018-09-17 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-5397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617416#comment-16617416 ] Thomas Weise commented on BEAM-5397: {code:java} Exception in thread "grpc-default-worker-ELG-3-2"

[jira] [Created] (BEAM-5397) Flink portable runner GRPC cleanup failure after user class loader was removed

2018-09-17 Thread Thomas Weise (JIRA)
Thomas Weise created BEAM-5397: -- Summary: Flink portable runner GRPC cleanup failure after user class loader was removed Key: BEAM-5397 URL: https://issues.apache.org/jira/browse/BEAM-5397 Project: Beam

  1   2   3   >