[jira] [Commented] (BEAM-1640) data file missing when submit a job on Flink

2017-03-09 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903038#comment-15903038 ] Aljoscha Krettek commented on BEAM-1640: Which Flink version are you using? > data file missing

[jira] [Commented] (BEAM-1641) Support synchronized processing time in Flink runner

2017-03-09 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902802#comment-15902802 ] Aljoscha Krettek commented on BEAM-1641: Wow, that's a tough one. I think we might first have to

[jira] [Commented] (BEAM-1631) Flink runner: submit job to a Flink-on-YARN cluster

2017-03-07 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15899797#comment-15899797 ] Aljoscha Krettek commented on BEAM-1631: [~davor] Yes, this analysis is correct! I'll look into how

[jira] [Commented] (BEAM-1612) Support real Bundle in Flink runner

2017-03-03 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894345#comment-15894345 ] Aljoscha Krettek commented on BEAM-1612: Yes, I think finding a solution for bundling is very

[jira] [Commented] (BEAM-1589) Add OnWindowExpiration method to Stateful DoFn

2017-03-03 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894162#comment-15894162 ] Aljoscha Krettek commented on BEAM-1589: This will probably take the form of a

[jira] [Updated] (BEAM-760) Validation needs to exist that @NeedsRunner / @RunnableOnService tests execute

2017-03-01 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek updated BEAM-760: -- Fix Version/s: First stable release > Validation needs to exist that @NeedsRunner /

[jira] [Updated] (BEAM-622) Add checkpointing tests for DoFnOperator and WindowDoFnOperator

2017-03-01 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek updated BEAM-622: -- Fix Version/s: First stable release > Add checkpointing tests for DoFnOperator and

[jira] [Updated] (BEAM-720) Run WindowedWordCount Integration Test in Flink

2017-03-01 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek updated BEAM-720: -- Fix Version/s: First stable release > Run WindowedWordCount Integration Test in Flink >

[jira] [Closed] (BEAM-1389) Remove or update Flink Runner README.md

2017-03-01 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek closed BEAM-1389. -- Resolution: Fixed Fix Version/s: 0.6.0 > Remove or update Flink Runner README.md >

[jira] [Closed] (BEAM-1517) Garbage collect user state in Flink Runner

2017-03-01 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek closed BEAM-1517. -- Resolution: Fixed Assignee: Jingsong Lee (was: Aljoscha Krettek) > Garbage collect user

[jira] [Updated] (BEAM-1517) Garbage collect user state in Flink Runner

2017-02-28 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek updated BEAM-1517: --- Fix Version/s: 0.6.0 > Garbage collect user state in Flink Runner >

[jira] [Updated] (BEAM-1517) Garbage collect user state in Flink Runner

2017-02-28 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek updated BEAM-1517: --- Priority: Blocker (was: Major) > Garbage collect user state in Flink Runner >

[jira] [Updated] (BEAM-1517) Garbage collect user state in Flink Runner

2017-02-28 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek updated BEAM-1517: --- Affects Version/s: 0.6.0 > Garbage collect user state in Flink Runner >

[jira] [Closed] (BEAM-1116) Support for new Timer API in Flink runner

2017-02-28 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek closed BEAM-1116. -- Resolution: Fixed Fix Version/s: 0.6.0 > Support for new Timer API in Flink runner >

[jira] [Commented] (BEAM-315) Flink Runner compares keys unencoded which may produce incorrect results

2017-02-28 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887826#comment-15887826 ] Aljoscha Krettek commented on BEAM-315: --- Strange... are you running in Streaming or Batch mode? >

[jira] [Closed] (BEAM-1036) Support for new State API in FlinkRunner

2017-02-28 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek closed BEAM-1036. -- Resolution: Fixed Fix Version/s: 0.6.0 > Support for new State API in FlinkRunner >

[jira] [Commented] (BEAM-1558) jetty-util 6.1.26.cloudera.4 absent from repository list

2017-02-27 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886382#comment-15886382 ] Aljoscha Krettek commented on BEAM-1558: On my machine, when I run your command I get this {code}

[jira] [Closed] (BEAM-1560) Use provided Function Runners in Flink Batch Runner

2017-02-27 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek closed BEAM-1560. -- Resolution: Fixed > Use provided Function Runners in Flink Batch Runner >

[jira] [Updated] (BEAM-1560) Use provided Function Runners in Flink Batch Runner

2017-02-27 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek updated BEAM-1560: --- Summary: Use provided Function Runners in Flink Batch Runner (was: Use DoFnRunner in Flink

[jira] [Updated] (BEAM-1560) Use DoFnRunner in Flink Batch Runner

2017-02-27 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek updated BEAM-1560: --- Description: This is a prerequisite for supporting the user-facing State API and Timer API in

[jira] [Updated] (BEAM-1560) Use DoFnRunner in Flink Batch Runner

2017-02-27 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek updated BEAM-1560: --- Summary: Use DoFnRunner in Flink Batch Runner (was: Use SimpleDoFnRunner to invoke and use

[jira] [Updated] (BEAM-1560) Use SimpleDoFnRunner to invoke and use new CombineFnRunner in batch mode of Flink runner

2017-02-27 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek updated BEAM-1560: --- Description: To support the user-facing State API and Timer API in Flink runner (batch) (was:

[jira] [Closed] (BEAM-1456) Make UnboundedSourceWrapper snapshot to rescalable operator state in Flink Runner

2017-02-24 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek closed BEAM-1456. -- Resolution: Fixed Fix Version/s: 0.6.0 > Make UnboundedSourceWrapper snapshot to

[jira] [Commented] (BEAM-1517) Garbage collect user state in Flink Runner

2017-02-22 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15877903#comment-15877903 ] Aljoscha Krettek commented on BEAM-1517: [~lzljs3620320] We cannot depend on users always cleaning

[jira] [Commented] (BEAM-1116) Support for new Timer API in Flink runner

2017-02-21 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15875726#comment-15875726 ] Aljoscha Krettek commented on BEAM-1116: This is resolved for the Flink Streaming Runner, I'm

[jira] [Commented] (BEAM-1517) Garbage collect user state in Flink Runner

2017-02-21 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15875634#comment-15875634 ] Aljoscha Krettek commented on BEAM-1517: [~tgroh] or [~kenn], do you have any ideas about this?

[jira] [Created] (BEAM-1517) Garbage collect user state in Flink Runner

2017-02-21 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created BEAM-1517: -- Summary: Garbage collect user state in Flink Runner Key: BEAM-1517 URL: https://issues.apache.org/jira/browse/BEAM-1517 Project: Beam Issue Type: Bug

[jira] [Commented] (BEAM-1498) Use Flink-native side outputs

2017-02-16 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869941#comment-15869941 ] Aljoscha Krettek commented on BEAM-1498: [~lzljs3620320] I thought you might find this interesting.

[jira] [Created] (BEAM-1498) Use Flink-native side outputs

2017-02-16 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created BEAM-1498: -- Summary: Use Flink-native side outputs Key: BEAM-1498 URL: https://issues.apache.org/jira/browse/BEAM-1498 Project: Beam Issue Type: Improvement

[jira] [Closed] (BEAM-1394) Use Flink InternalTimerService for TimerInternals

2017-02-16 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek closed BEAM-1394. -- Resolution: Fixed Fix Version/s: 0.6.0 > Use Flink InternalTimerService for

[jira] [Closed] (BEAM-1445) Use Flink broadcast state to store side-input data

2017-02-16 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek closed BEAM-1445. -- Resolution: Fixed Fix Version/s: 0.6.0 > Use Flink broadcast state to store side-input

[jira] [Closed] (BEAM-1393) Update Flink Runner to Flink 1.2.0

2017-02-16 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek closed BEAM-1393. -- Resolution: Fixed Fix Version/s: 0.6.0 > Update Flink Runner to Flink 1.2.0 >

[jira] [Commented] (BEAM-1346) Drop Late Data in ReduceFnRunner

2017-02-13 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865303#comment-15865303 ] Aljoscha Krettek commented on BEAM-1346: Ah dammit, you're right, I forgot about BEAM-696. In fact

[jira] [Commented] (BEAM-1461) duplication with StartBundle and prepareForProcessing in DoFn

2017-02-13 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863967#comment-15863967 ] Aljoscha Krettek commented on BEAM-1461: {{DoFn.prepareForProcessing()}} is only invoked once,

[jira] [Commented] (BEAM-1346) Drop Late Data in ReduceFnRunner

2017-02-13 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863721#comment-15863721 ] Aljoscha Krettek commented on BEAM-1346: [~kenn] another thing that crossed my mind is elements

[jira] [Commented] (BEAM-1458) Checkpoint support in Beam

2017-02-10 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15861511#comment-15861511 ] Aljoscha Krettek commented on BEAM-1458: Flink snapshots/checkpoints are different from how the

[jira] [Commented] (BEAM-1393) Update Flink Runner to Flink 1.2.0

2017-02-10 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15861079#comment-15861079 ] Aljoscha Krettek commented on BEAM-1393: Yes, your analysis is spot on! I think what we can do is

[jira] [Commented] (BEAM-1393) Update Flink Runner to Flink 1.2.0

2017-02-09 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859777#comment-15859777 ] Aljoscha Krettek commented on BEAM-1393: That last part is actually a bit more complicated. (I saw

[jira] [Assigned] (BEAM-1394) Use Flink InternalTimerService for TimerInternals

2017-02-05 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek reassigned BEAM-1394: -- Assignee: (was: Aljoscha Krettek) > Use Flink InternalTimerService for

[jira] [Updated] (BEAM-1394) Use Flink InternalTimerService for TimerInternals

2017-02-05 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek updated BEAM-1394: --- Description: When updating our Flink version to 1.2 we can use the new internal timer API for

[jira] [Created] (BEAM-1394) Use Flink InternalTimerService for TimerInternals

2017-02-05 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created BEAM-1394: -- Summary: Use Flink InternalTimerService for TimerInternals Key: BEAM-1394 URL: https://issues.apache.org/jira/browse/BEAM-1394 Project: Beam Issue Type:

[jira] [Created] (BEAM-1393) Update Flink Runner to Flink 1.2.0

2017-02-05 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created BEAM-1393: -- Summary: Update Flink Runner to Flink 1.2.0 Key: BEAM-1393 URL: https://issues.apache.org/jira/browse/BEAM-1393 Project: Beam Issue Type: Improvement

[jira] [Created] (BEAM-1389) Remove or update Flink Runner README.md

2017-02-04 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created BEAM-1389: -- Summary: Remove or update Flink Runner README.md Key: BEAM-1389 URL: https://issues.apache.org/jira/browse/BEAM-1389 Project: Beam Issue Type:

[jira] [Commented] (BEAM-1335) ValueState could use an initial/default value

2017-02-02 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850086#comment-15850086 ] Aljoscha Krettek commented on BEAM-1335: No, only the internal implementation is harder to get

[jira] (BEAM-1346) Drop Late Data in ReduceFnRunner

2017-01-31 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847474#comment-15847474 ] Aljoscha Krettek commented on BEAM-1346: Yep, that fact (the Apex runner never using the

[jira] (BEAM-1346) Drop Late Data in ReduceFnRunner

2017-01-31 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-1346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847134#comment-15847134 ] Aljoscha Krettek commented on BEAM-1346: Also, isn't dropping late data before giving it to the

[jira] (BEAM-1346) Drop Late Data in ReduceFnRunner

2017-01-31 Thread Aljoscha Krettek (JIRA)
Title: Message Title Aljoscha Krettek commented on BEAM-1346

[jira] (BEAM-1346) Drop Late Data in ReduceFnRunner

2017-01-30 Thread Aljoscha Krettek (JIRA)
Title: Message Title Aljoscha Krettek assigned an issue to Unassigned

[jira] (BEAM-1346) Drop Late Data in ReduceFnRunner

2017-01-30 Thread Aljoscha Krettek (JIRA)
Title: Message Title Aljoscha Krettek created an issue

[jira] (BEAM-1346) Drop Late Data in ReduceFnRunner

2017-01-30 Thread Aljoscha Krettek (JIRA)
Title: Message Title Aljoscha Krettek commented on BEAM-1346

[jira] (BEAM-843) Use New DoFn Directly in Flink Runner

2017-01-30 Thread Aljoscha Krettek (JIRA)
Title: Message Title Aljoscha Krettek commented on BEAM-843

[jira] [Commented] (BEAM-773) Implement Metrics support for Flink runner

2017-01-09 Thread Aljoscha Krettek (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15811610#comment-15811610 ] Aljoscha Krettek commented on BEAM-773: --- [~bchambers] is correct on both issues: AFAIK no one is

<    1   2   3