Unbearably slow Table API time-windowed stream join with RocksDBStateBackend

2019-08-14 Thread LIU Xiao
Example SQL: SELECT * FROM stream1 s1, stream2 s2 WHERE s1.id = s2.id AND s1.rowtime = s2.rowtime And we have lots of messages in stream1 and stream2 share a same rowtime. It runs fine when using heap as the state backend, but requires lots of heap memory sometimes (when upstream out of sync,

[jira] [Created] (FLINK-13708) transformations should be cleared because a table environment could execute multiple job

2019-08-14 Thread godfrey he (JIRA)
godfrey he created FLINK-13708: -- Summary: transformations should be cleared because a table environment could execute multiple job Key: FLINK-13708 URL: https://issues.apache.org/jira/browse/FLINK-13708

Re: Checkpointing under backpressure

2019-08-14 Thread Piotr Nowojski
Hi, Thomas: There are no Jira tickets yet (or maybe there is something very old somewhere). First we want to discuss it, next present FLIP and at last create tickets :) > if I understand correctly, then the proposal is to not block any > input channel at all, but only log data from the

Re: Checkpointing under backpressure

2019-08-14 Thread zhijiang
Hi Thomas, There are no Jira tickets or discussions at the moment. If there are any updates I would ping you. I agree that there is a benefit to firstly read other channels without barrier in high priority, otherwise it seems waste cpu resource to migrate blocked buffers to another cached

[jira] [Created] (FLINK-13709) Kafka09ITCase hangs when starting the KafkaServer on Travis

2019-08-14 Thread Till Rohrmann (JIRA)
Till Rohrmann created FLINK-13709: - Summary: Kafka09ITCase hangs when starting the KafkaServer on Travis Key: FLINK-13709 URL: https://issues.apache.org/jira/browse/FLINK-13709 Project: Flink

Re: Checkpointing under backpressure

2019-08-14 Thread Paris Carbone
Thanks for the responses. Starts getting a bit more clear for everyone now. @Zhuzhu overlapping unaligned snapshots should be aborted/avoided imho. @Piotr point II, it was a little too quickly written, sorry about that. Simply put the two following approaches are equivalent for a valid

[jira] [Created] (FLINK-13717) allow to set taskmanager.host and taskmanager.bind-host separately

2019-08-14 Thread Robert Fiser (JIRA)
Robert Fiser created FLINK-13717: Summary: allow to set taskmanager.host and taskmanager.bind-host separately Key: FLINK-13717 URL: https://issues.apache.org/jira/browse/FLINK-13717 Project: Flink

Re: [DISCUSS] Repository split

2019-08-14 Thread Chesnay Schepler
Let's recap a bit: Several people have raised the argument that build times can be kept in check via other means (mostly differential builds via some means, be it custom scripts or switching to gradle). I will start a separate discussion thread on this topic, since it is a useful discussion

Re: Checkpointing under backpressure

2019-08-14 Thread Stephan Ewen
Scaling with unaligned checkpoints might be a necessity. Let's assume the job failed due to a lost TaskManager, but no new TaskManager becomes available. In that case we need to scale down based on the latest complete checkpoint, because we cannot produce a new checkpoint. On Wed, Aug 14, 2019

Re: [DISCUSS] Drop stale class Program

2019-08-14 Thread Kostas Kloudas
Hi all, It is nice to have this discussion. I am totally up for removing the unused Program interface, as this will simplify a lot of other code paths in the ClusterClient and elsewhere. Also about the easier integration of Flink with other frameworks, there is another discussion in the mailing

Re: Unbearably slow Table API time-windowed stream join with RocksDBStateBackend

2019-08-14 Thread Jark Wu
Hi Xiao, Thanks for reporting this. You approach sounds good to me. But we have many similar problems in existing streaming sql operator implementations. So I think if State API / statebackend can provide a better state structure to handle this situation would be great. This is a similar problem

Re: [DISCUSS] Drop stale class Program

2019-08-14 Thread Stephan Ewen
+1 to drop it. It's one of the oldest pieces of legacy. On Wed, Aug 14, 2019 at 12:07 PM Aljoscha Krettek wrote: > Hi, > > I would be in favour of removing Program (and the code paths that support > it) for Flink 1.10. Most users of Flink don’t actually know it exists and > it is only making

Re: [DISCUSS] Drop stale class Program

2019-08-14 Thread Kostas Kloudas
I already opened a JIRA for the removal and I will also create a (short) FLIP, as it is a PublicEvolving interface and its removal should go through a FLIP. The JIRA can be found here https://issues.apache.org/jira/browse/FLINK-13713 Cheers, Kostas On Wed, Aug 14, 2019 at 12:13 PM Stephan Ewen

[jira] [Created] (FLINK-13716) Remove Package-related chinese documentation

2019-08-14 Thread Kostas Kloudas (JIRA)
Kostas Kloudas created FLINK-13716: -- Summary: Remove Package-related chinese documentation Key: FLINK-13716 URL: https://issues.apache.org/jira/browse/FLINK-13716 Project: Flink Issue Type:

Re: Checkpointing under backpressure

2019-08-14 Thread Stephan Ewen
Hi all! Yes, the first proposal of "unaligend checkpoints" (probably two years back now) drew a major inspiration from Chandy Lamport, as did actually the original checkpointing algorithm. "Logging data between first and last barrier" versus "barrier jumping over buffer and storing those

[DISCUSS] FLIP-52: Remove legacy Program interface.

2019-08-14 Thread Kostas Kloudas
Hi all, As discussed in [1] , the Program interface seems to be outdated and there seems to be no objection to remove it. Given that this interface is PublicEvolving, its removal should pass through a FLIP and this discussion and the associated FLIP are exactly for that purpose. Please let me

Re: why is not possible to handle a custom resource manager ?

2019-08-14 Thread Yang Wang
Hi Cristian, I think you could have a try on standalone cluster on X(underlying cluster framework). By now it works well for kubernetes[1]. If your custom distributed system support to start an application based on yaml/json configuration, then it will be very simple to start a session cluster.

Re: [DISCUSS] FLIP-52: Remove legacy Program interface.

2019-08-14 Thread Stephan Ewen
+1 the "main" method is the overwhelming default. getting rid of "two ways to do things" is a good idea. On Wed, Aug 14, 2019 at 1:42 PM Kostas Kloudas wrote: > Hi all, > > As discussed in [1] , the Program interface seems to be outdated and > there seems to be > no objection to remove it. > >

Re: Checkpointing under backpressure

2019-08-14 Thread Paris Carbone
+1 I think we are on the same page Stephan. Rescaling on unaligned checkpoint sounds challenging and a bit unnecessary. No? Why not sticking to aligned snapshots for live reconfiguration/rescaling? It’s a pretty rare operation and it would simplify things by a lot. Everything can be “staged”

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread Jark Wu
Congratulations Andrey! Cheers, Jark On Thu, 15 Aug 2019 at 00:57, jincheng sun wrote: > Congrats Andrey! Very happy to have you onboard :) > > Best, Jincheng > > Yu Li 于2019年8月15日周四 上午12:06写道: > > > Congratulations Andrey! Well deserved! > > > > Best Regards, > > Yu > > > > > > On Wed, 14

[jira] [Created] (FLINK-13730) Cache and share the downloaded external distribution (e.g. Kafka) in E2E tests

2019-08-14 Thread Jark Wu (JIRA)
Jark Wu created FLINK-13730: --- Summary: Cache and share the downloaded external distribution (e.g. Kafka) in E2E tests Key: FLINK-13730 URL: https://issues.apache.org/jira/browse/FLINK-13730 Project: Flink

[jira] [Created] (FLINK-13724) Remove unnecessary whitespace from the docs' sitenav

2019-08-14 Thread Nico Kruber (JIRA)
Nico Kruber created FLINK-13724: --- Summary: Remove unnecessary whitespace from the docs' sitenav Key: FLINK-13724 URL: https://issues.apache.org/jira/browse/FLINK-13724 Project: Flink Issue

[jira] [Created] (FLINK-13723) Use liquid-c for faster doc generation

2019-08-14 Thread Nico Kruber (JIRA)
Nico Kruber created FLINK-13723: --- Summary: Use liquid-c for faster doc generation Key: FLINK-13723 URL: https://issues.apache.org/jira/browse/FLINK-13723 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-13728) Fix wrong closing tag order in sidenav

2019-08-14 Thread Nico Kruber (JIRA)
Nico Kruber created FLINK-13728: --- Summary: Fix wrong closing tag order in sidenav Key: FLINK-13728 URL: https://issues.apache.org/jira/browse/FLINK-13728 Project: Flink Issue Type: Bug

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread Kurt Young
Congratulations Andery! Best, Kurt On Thu, Aug 15, 2019 at 10:09 AM Biao Liu wrote: > Congrats! > > Thanks, > Biao /'bɪ.aʊ/ > > > > On Thu, 15 Aug 2019 at 10:03, Jark Wu wrote: > > > Congratulations Andrey! > > > > > > Cheers, > > Jark > > > > On Thu, 15 Aug 2019 at 00:57, jincheng sun > >

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread leesf
Congratulations Andrey! Best, Leesf Zhu Zhu 于2019年8月15日周四 上午11:20写道: > Congratulations Andrey! > > Thanks, > Zhu Zhu > > vino yang 于2019年8月15日周四 上午11:05写道: > > > Congratulations Andrey! > > > > Best, > > Vino > > > > Yun Gao 于2019年8月15日周四 上午10:49写道: > > > > > Congratulations Andrey! > > > >

[jira] [Created] (FLINK-13722) Speed up documentation generation

2019-08-14 Thread Nico Kruber (JIRA)
Nico Kruber created FLINK-13722: --- Summary: Speed up documentation generation Key: FLINK-13722 URL: https://issues.apache.org/jira/browse/FLINK-13722 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-13729) Update website generation dependencies

2019-08-14 Thread Nico Kruber (JIRA)
Nico Kruber created FLINK-13729: --- Summary: Update website generation dependencies Key: FLINK-13729 URL: https://issues.apache.org/jira/browse/FLINK-13729 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-13725) Use sassc for faster doc generation

2019-08-14 Thread Nico Kruber (JIRA)
Nico Kruber created FLINK-13725: --- Summary: Use sassc for faster doc generation Key: FLINK-13725 URL: https://issues.apache.org/jira/browse/FLINK-13725 Project: Flink Issue Type: Sub-task

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread Biao Liu
Congrats! Thanks, Biao /'bɪ.aʊ/ On Thu, 15 Aug 2019 at 10:03, Jark Wu wrote: > Congratulations Andrey! > > > Cheers, > Jark > > On Thu, 15 Aug 2019 at 00:57, jincheng sun > wrote: > > > Congrats Andrey! Very happy to have you onboard :) > > > > Best, Jincheng > > > > Yu Li 于2019年8月15日周四

[jira] [Created] (FLINK-13731) flink sql support window with alignment

2019-08-14 Thread zhaoshijie (JIRA)
zhaoshijie created FLINK-13731: -- Summary: flink sql support window with alignment Key: FLINK-13731 URL: https://issues.apache.org/jira/browse/FLINK-13731 Project: Flink Issue Type: Improvement

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-14 Thread Kurt Young
Hi Robert, I will do it today. Best, Kurt On Wed, Aug 14, 2019 at 11:55 PM Robert Metzger wrote: > Has anybody verified the inclusion of all bundled dependencies into the > NOTICE files? > > I'm asking because we had some issues with that in the last release(s). > > On Wed, Aug 14, 2019 at

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread Yun Gao
Congratulations Andrey! Best, Yun -- From:Congxian Qiu Send Time:2019 Aug. 15 (Thu.) 10:28 To:dev@flink.apache.org Subject:Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer Congratulations Andery! Best, Congxian Kurt

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread vino yang
Congratulations Andrey! Best, Vino Yun Gao 于2019年8月15日周四 上午10:49写道: > Congratulations Andrey! > > Best, > Yun > > > -- > From:Congxian Qiu > Send Time:2019 Aug. 15 (Thu.) 10:28 > To:dev@flink.apache.org > Subject:Re: [ANNOUNCE]

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread Zhu Zhu
Congratulations Andrey! Thanks, Zhu Zhu vino yang 于2019年8月15日周四 上午11:05写道: > Congratulations Andrey! > > Best, > Vino > > Yun Gao 于2019年8月15日周四 上午10:49写道: > > > Congratulations Andrey! > > > > Best, > > Yun > > > > > > -- > >

[jira] [Created] (FLINK-13727) Build docs with jekyll 4.0.0 (final)

2019-08-14 Thread Nico Kruber (JIRA)
Nico Kruber created FLINK-13727: --- Summary: Build docs with jekyll 4.0.0 (final) Key: FLINK-13727 URL: https://issues.apache.org/jira/browse/FLINK-13727 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-13726) Build docs with jekyll 4.0.0.pre.beta1

2019-08-14 Thread Nico Kruber (JIRA)
Nico Kruber created FLINK-13726: --- Summary: Build docs with jekyll 4.0.0.pre.beta1 Key: FLINK-13726 URL: https://issues.apache.org/jira/browse/FLINK-13726 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-13732) Enhance JobManagerMetricGroup with FLIP-6 architecture

2019-08-14 Thread Biao Liu (JIRA)
Biao Liu created FLINK-13732: Summary: Enhance JobManagerMetricGroup with FLIP-6 architecture Key: FLINK-13732 URL: https://issues.apache.org/jira/browse/FLINK-13732 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-51: Rework of the Expression Design

2019-08-14 Thread Jark Wu
Thanks Jingsong for starting the discussion. The general design of the FLIP looks good to me. +1 for the FLIP. It's time to get rid of the old Expression! Regarding to the function behavior, shall we also include new functions from blink planner (e.g. LISTAGG, REGEXP, TO_DATE, etc..) ? Best,

Re: [DISCUSS] FLIP-52: Remove legacy Program interface.

2019-08-14 Thread SHI Xiaogang
+1 Glad that programming with flink becomes simpler and easier. Regards, Xiaogang Aljoscha Krettek 于2019年8月14日周三 下午11:31写道: > +1 (for the same reasons I posted on the other thread) > > > On 14. Aug 2019, at 15:03, Zili Chen wrote: > > > > +1 > > > > It could be regarded as part of Flink

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread Congxian Qiu
Congratulations Andery! Best, Congxian Kurt Young 于2019年8月15日周四 上午10:12写道: > Congratulations Andery! > > Best, > Kurt > > > On Thu, Aug 15, 2019 at 10:09 AM Biao Liu wrote: > > > Congrats! > > > > Thanks, > > Biao /'bɪ.aʊ/ > > > > > > > > On Thu, 15 Aug 2019 at 10:03, Jark Wu wrote: > > > >

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread Bowen Li
Congratulations Andrey! On Wed, Aug 14, 2019 at 10:18 PM Rong Rong wrote: > Congratulations Andrey! > > On Wed, Aug 14, 2019 at 10:14 PM chaojianok wrote: > > > Congratulations Andrey! > > At 2019-08-14 21:26:37, "Till Rohrmann" wrote: > > >Hi everyone, > > > > > >I'm very happy to announce

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread Rong Rong
Congratulations Andrey! On Wed, Aug 14, 2019 at 10:14 PM chaojianok wrote: > Congratulations Andrey! > At 2019-08-14 21:26:37, "Till Rohrmann" wrote: > >Hi everyone, > > > >I'm very happy to announce that Andrey Zagrebin accepted the offer of the > >Flink PMC to become a committer of the Flink

Re:[ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread chaojianok
Congratulations Andrey! At 2019-08-14 21:26:37, "Till Rohrmann" wrote: Hi everyone, I'm very happy to announce that Andrey Zagrebin accepted the offer of the Flink PMC to become a committer of the Flink project. Andrey has been an active community member for more than 15 months. He has

Re:[ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread chaojianok
Congratulations Andrey! At 2019-08-14 21:26:37, "Till Rohrmann" wrote: >Hi everyone, > >I'm very happy to announce that Andrey Zagrebin accepted the offer of the >Flink PMC to become a committer of the Flink project. > >Andrey has been an active community member for more than 15 months. He has

Re:Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread chaojianok
Congratulations Andrey! At 2019-08-15 10:02:49, "Jark Wu" wrote: >Congratulations Andrey! > > >Cheers, >Jark > >On Thu, 15 Aug 2019 at 00:57, jincheng sun wrote: > >> Congrats Andrey! Very happy to have you onboard :) >> >> Best, Jincheng >> >> Yu Li 于2019年8月15日周四 上午12:06写道: >> >> >

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread Gary Yao
Congratulations Andrey, well deserved! Best, Gary On Thu, Aug 15, 2019 at 7:50 AM Bowen Li wrote: > Congratulations Andrey! > > On Wed, Aug 14, 2019 at 10:18 PM Rong Rong wrote: > >> Congratulations Andrey! >> >> On Wed, Aug 14, 2019 at 10:14 PM chaojianok wrote: >> >> > Congratulations

Re: Launching Flink server from IntelliJ

2019-08-14 Thread Till Rohrmann
Hi Ramayan, you can start a random example [1] from the IDE. The local execution environment will start a MiniCluster to execute the Flink job in the process started by the IDE. [1] https://github.com/apache/flink/tree/master/flink-examples Cheers, Till On Wed, Aug 14, 2019 at 12:20 AM Ramayan

Re: [DISCUSS] FLIP-39: Flink ML pipeline and ML libs

2019-08-14 Thread Robert Metzger
It seems that this FLIP doesn't have a Wiki page yet [1], even though it is already partially implemented [2] We should try to stick more to the FLIP process to manage the project more efficiently. [1] https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals [2]

Re: [DISCUSS] Drop stale class Program

2019-08-14 Thread Aljoscha Krettek
Hi, I would be in favour of removing Program (and the code paths that support it) for Flink 1.10. Most users of Flink don’t actually know it exists and it is only making our code more complicated. Going forward with the new Client API discussions will be a lot easier without it as well. Best,

[jira] [Created] (FLINK-13711) Hive array values not properly displayed in SQL CLI

2019-08-14 Thread Rui Li (JIRA)
Rui Li created FLINK-13711: -- Summary: Hive array values not properly displayed in SQL CLI Key: FLINK-13711 URL: https://issues.apache.org/jira/browse/FLINK-13711 Project: Flink Issue Type: Bug

Re: Checkpointing under backpressure

2019-08-14 Thread Piotr Nowojski
Hi again, Zhu Zhu let me think about this more. Maybe as Paris is writing, we do not need to block any channels at all, at least assuming credit base flow control. Regarding what should happen with the following checkpoint is another question. Also, should we support concurrent checkpoints and

[jira] [Created] (FLINK-13713) Remove legacy Package interface.

2019-08-14 Thread Kostas Kloudas (JIRA)
Kostas Kloudas created FLINK-13713: -- Summary: Remove legacy Package interface. Key: FLINK-13713 URL: https://issues.apache.org/jira/browse/FLINK-13713 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-13715) Remove Package-related english documentation.

2019-08-14 Thread Kostas Kloudas (JIRA)
Kostas Kloudas created FLINK-13715: -- Summary: Remove Package-related english documentation. Key: FLINK-13715 URL: https://issues.apache.org/jira/browse/FLINK-13715 Project: Flink Issue

[jira] [Created] (FLINK-13714) Remove Package-related code.

2019-08-14 Thread Kostas Kloudas (JIRA)
Kostas Kloudas created FLINK-13714: -- Summary: Remove Package-related code. Key: FLINK-13714 URL: https://issues.apache.org/jira/browse/FLINK-13714 Project: Flink Issue Type: Sub-task

Re: Checkpointing under backpressure

2019-08-14 Thread Zhu Zhu
Thanks Piotr and Zhijiang for sharing the thoughts on unaligned checkpointing and the barrier overtaking. I have a question about 2.d) in Piotr's last mail that states "the Task first has to process the buffered data after that it can unblock the reads from the channels". Does this mean that we

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-14 Thread Jark Wu
Hi Gordon, I have verified the following things: - build the source release with Scala 2.12 and Scala 2.11 successfully - checked/verified signatures and hashes - checked that all POM files point to the same version - ran some flink table related end-to-end tests locally and succeeded (except

Re: Checkpointing under backpressure

2019-08-14 Thread Paris Carbone
Now I see a little more clearly what you have in mind. Thanks for the explanation! There are a few intermixed concepts here, some how to do with correctness some with performance. Before delving deeper I will just enumerate a few things to make myself a little more helpful if I can. I.

[jira] [Created] (FLINK-13710) JarListHandler always extract the jar package

2019-08-14 Thread ChengWei Ye (JIRA)
ChengWei Ye created FLINK-13710: --- Summary: JarListHandler always extract the jar package Key: FLINK-13710 URL: https://issues.apache.org/jira/browse/FLINK-13710 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-13712) Add 1.9.0 release notes to documentation

2019-08-14 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-13712: --- Summary: Add 1.9.0 release notes to documentation Key: FLINK-13712 URL: https://issues.apache.org/jira/browse/FLINK-13712 Project: Flink Issue

why is not possible to handle a custom resource manager ?

2019-08-14 Thread Cristian Lorenzetto
i want develop a project using flink stack in a project using a custom distributed system, so id like use my distrubuted system as resource manager instead to overload the project with many other additional sockets and code. Is there a way for embedding flink project in my server without using

Re: [DISCUSS] Best practice to run flink on kubernetes

2019-08-14 Thread Yang Wang
Hi till, Thanks for your reply. I agree with you that both option 1 and 3 need to be supported. Option 1 is reactive mode of resource management and flink is not aware of underlying cluster. If a user has limited resources to run flink jobs, this option will be very useful. On the other side,

Re: [DISCUSS] Drop stale class Program

2019-08-14 Thread Zili Chen
Thanks for your attentions! Thank Kostas for creating the JIRA and drafting the FLIP. I would volunteer to help review it :-) It's good to see that we make progress on this thread. Best, tison. Kostas Kloudas 于2019年8月14日周三 下午6:39写道: > I already opened a JIRA for the removal and I will also

Re: Checkpointing under backpressure

2019-08-14 Thread Yun Gao
Hi, Very thanks for sharing the thoughts on the unaligned checkpoint ! Another question regarding I 2.C (Performance) by Paris is that do we always snapshot and broadcast the marks once the task receives the first mark from JM o? If so, then we will always need to snapshot all the

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread Zili Chen
Congratulations Andrey! Best, tison. Till Rohrmann 于2019年8月14日周三 下午9:26写道: > Hi everyone, > > I'm very happy to announce that Andrey Zagrebin accepted the offer of the > Flink PMC to become a committer of the Flink project. > > Andrey has been an active community member for more than 15

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread zhijiang
Congratulations Andrey, great work and well deserved! Best, Zhijiang -- From:Till Rohrmann Send Time:2019年8月14日(星期三) 15:26 To:dev ; user Subject:[ANNOUNCE] Andrey Zagrebin becomes a Flink committer Hi everyone, I'm very happy to

[jira] [Created] (FLINK-13720) Include asm-commons in flink-shaded-asm

2019-08-14 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-13720: --- Summary: Include asm-commons in flink-shaded-asm Key: FLINK-13720 URL: https://issues.apache.org/jira/browse/FLINK-13720 Project: Flink Issue

Re: why is not possible to handle a custom resource manager ?

2019-08-14 Thread Xintong Song
Hi Cristian, For the latest off-the-shelf Flink releases, I agree with Yang that running a standalone cluster on top of your custom distributed system is AFAIK the only way. Customizing Flink and implement your own Flink ResourceManager is also an option. Flink ResourceManager is a component

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread Robert Metzger
Congratulations! Very happy to have you onboard :) On Wed, Aug 14, 2019 at 4:06 PM Kostas Kloudas wrote: > Congratulations Andrey! > Well deserved! > > Kostas > > On Wed, Aug 14, 2019 at 4:04 PM Yun Tang wrote: > > > > Congratulations Andrey. > > > > Best > > Yun Tang > >

Re: [DISCUSS] FLIP-52: Remove legacy Program interface.

2019-08-14 Thread Till Rohrmann
+1 Cheers, Till On Wed, Aug 14, 2019 at 1:50 PM Stephan Ewen wrote: > +1 > > the "main" method is the overwhelming default. getting rid of "two ways to > do things" is a good idea. > > On Wed, Aug 14, 2019 at 1:42 PM Kostas Kloudas wrote: > > > Hi all, > > > > As discussed in [1] , the

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-14 Thread Kurt Young
Hi, Thanks for preparing this release candidate. I have verified the following: - verified the checksums and GPG files match the corresponding release files - verified that the source archives do not contains any binaries - build the source release with Scala 2.11 successfully. - ran `mvn

[ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread Till Rohrmann
Hi everyone, I'm very happy to announce that Andrey Zagrebin accepted the offer of the Flink PMC to become a committer of the Flink project. Andrey has been an active community member for more than 15 months. He has helped shaping numerous features such as State TTL, FRocksDB release, Shuffle

[jira] [Created] (FLINK-13718) Disable HBase tests

2019-08-14 Thread Chesnay Schepler (JIRA)
Chesnay Schepler created FLINK-13718: Summary: Disable HBase tests Key: FLINK-13718 URL: https://issues.apache.org/jira/browse/FLINK-13718 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-13719) Update Yarn E2E test docker image to run on Java 11

2019-08-14 Thread Chesnay Schepler (JIRA)
Chesnay Schepler created FLINK-13719: Summary: Update Yarn E2E test docker image to run on Java 11 Key: FLINK-13719 URL: https://issues.apache.org/jira/browse/FLINK-13719 Project: Flink

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread Oytun Tez
Congratulations Andrey! I am glad the Flink committer team is growing at such a pace! --- Oytun Tez *M O T A W O R D* The World's Fastest Human Translation Platform. oy...@motaword.com — www.motaword.com On Wed, Aug 14, 2019 at 9:29 AM Zili Chen wrote: > Congratulations Andrey! > > Best, >

Re: Checkpointing under backpressure

2019-08-14 Thread Paris Carbone
Sure I see. In cases when no periodic aligned snapshots are employed this is the only option. Two things that were not highlighted enough so far on the proposed protocol (included my mails): - The Recovery/Reconfiguration strategy should strictly prioritise processing logged events

Re: [DISCUSS] FLIP-52: Remove legacy Program interface.

2019-08-14 Thread Zili Chen
+1 It could be regarded as part of Flink client api refactor. Removal of stale code paths helps reason refactor. There is one thing worth attention that in this thread[1] Thomas suggests an interface with a method return JobGraph based on the fact that REST API and in per job mode actually

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread Xintong Song
Congratulations Andery~! Thank you~ Xintong Song On Wed, Aug 14, 2019 at 3:31 PM Oytun Tez wrote: > Congratulations Andrey! > > I am glad the Flink committer team is growing at such a pace! > > --- > Oytun Tez > > *M O T A W O R D* > The World's Fastest Human Translation Platform. >

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread Yun Tang
Congratulations Andrey. Best Yun Tang From: Xintong Song Sent: Wednesday, August 14, 2019 21:40 To: Oytun Tez Cc: Zili Chen ; Till Rohrmann ; dev ; user Subject: Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer Congratulations Andery~! Thank you~

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread Kostas Kloudas
Congratulations Andrey! Well deserved! Kostas On Wed, Aug 14, 2019 at 4:04 PM Yun Tang wrote: > > Congratulations Andrey. > > Best > Yun Tang > > From: Xintong Song > Sent: Wednesday, August 14, 2019 21:40 > To: Oytun Tez > Cc: Zili Chen ; Till Rohrmann ; >

Re: Checkpointing under backpressure

2019-08-14 Thread Stephan Ewen
Quick note: The current implementation is Align -> Forward -> Sync Snapshot Part (-> Async Snapshot Part) On Wed, Aug 14, 2019 at 5:21 PM Piotr Nowojski wrote: > > Thanks for the great ideas so far. > > +1 > > Regarding other things raised, I mostly agree with Stephan. > > I like the idea of

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-14 Thread Robert Metzger
Has anybody verified the inclusion of all bundled dependencies into the NOTICE files? I'm asking because we had some issues with that in the last release(s). On Wed, Aug 14, 2019 at 5:31 PM Aljoscha Krettek wrote: > +1 > > I did some testing on a Google Cloud Dataproc cluster (it gives you a >

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread Aleksey Pak
Congratulations, Andrey! On Wed, Aug 14, 2019 at 4:53 PM Markos Sfikas wrote: > Congrats Andrey! > > On Wed, 14 Aug 2019 at 16:47, Becket Qin wrote: > > > Congratulations, Andrey! > > > > On Wed, Aug 14, 2019 at 4:35 PM Thomas Weise wrote: > > > > > Congrats! > > > > > > > > > On Wed, Aug 14,

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread Yu Li
Congratulations Andrey! Well deserved! Best Regards, Yu On Wed, 14 Aug 2019 at 17:55, Aleksey Pak wrote: > Congratulations, Andrey! > > On Wed, Aug 14, 2019 at 4:53 PM Markos Sfikas > wrote: > > > Congrats Andrey! > > > > On Wed, 14 Aug 2019 at 16:47, Becket Qin wrote: > > > > >

Re: Checkpointing under backpressure

2019-08-14 Thread zhijiang
Thanks for these great points and disccusions! 1. Considering the way of triggering checkpoint RPC calls to all the tasks from Chandy Lamport, it combines two different mechanisms together to make sure that the trigger could be fast in different scenarios. But in flink world it might be not

Re: Checkpointing under backpressure

2019-08-14 Thread Thomas Weise
--> On Wed, Aug 14, 2019 at 10:23 AM zhijiang wrote: > Thanks for these great points and disccusions! > > 1. Considering the way of triggering checkpoint RPC calls to all the tasks > from Chandy Lamport, it combines two different mechanisms together to make > sure that the trigger could be fast

Re: Checkpointing under backpressure

2019-08-14 Thread zhijiang
> For the checkpoint to complete, any buffer that > arrived prior to the barrier would be to be part of the checkpointed state. Yes, I agree. > So wouldn't it be important to finish persisting these buffers as fast as > possible by prioritizing respective inputs? The task won't be able to >

Watermarks not propagated to WebUI?

2019-08-14 Thread Jan Lukavský
Hi, is it possible, that watermarks are sometimes not propagated to WebUI, although they are internally moving as normal? I see in WebUI every operator showing "No Watermark", but outputs seem to be propagated to sink (and there are watermark sensitive operations involved - e.g. reductions

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread Becket Qin
Congratulations, Andrey! On Wed, Aug 14, 2019 at 4:35 PM Thomas Weise wrote: > Congrats! > > > On Wed, Aug 14, 2019, 7:12 AM Robert Metzger wrote: > > > Congratulations! Very happy to have you onboard :) > > > > On Wed, Aug 14, 2019 at 4:06 PM Kostas Kloudas > wrote: > > > > > Congratulations

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread Markos Sfikas
Congrats Andrey! On Wed, 14 Aug 2019 at 16:47, Becket Qin wrote: > Congratulations, Andrey! > > On Wed, Aug 14, 2019 at 4:35 PM Thomas Weise wrote: > > > Congrats! > > > > > > On Wed, Aug 14, 2019, 7:12 AM Robert Metzger > wrote: > > > > > Congratulations! Very happy to have you onboard :) >

Re: Checkpointing under backpressure

2019-08-14 Thread Piotr Nowojski
> Thanks for the great ideas so far. +1 Regarding other things raised, I mostly agree with Stephan. I like the idea of simultaneously starting the checkpoint everywhere via RPC call (especially in cases where Tasks are busy doing some synchronous operations for example for tens of

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-14 Thread Aljoscha Krettek
+1 I did some testing on a Google Cloud Dataproc cluster (it gives you a managed YARN and Google Cloud Storage (GCS)): - tried both YARN session mode and YARN per-job mode, also using bin/flink list/cancel/etc. against a YARN session cluster - ran examples that write to GCS, both with the

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread Piotr Nowojski
Congratulations! :) > On 14 Aug 2019, at 16:52, Markos Sfikas wrote: > > Congrats Andrey! > > On Wed, 14 Aug 2019 at 16:47, Becket Qin wrote: > >> Congratulations, Andrey! >> >> On Wed, Aug 14, 2019 at 4:35 PM Thomas Weise wrote: >> >>> Congrats! >>> >>> >>> On Wed, Aug 14, 2019, 7:12

Re: [DISCUSS] FLIP-51: Rework of the Expression Design

2019-08-14 Thread Timo Walther
Hi Jingsong, thanks for writing down this FLIP. Big +1 from my side to finally get rid of PlannerExpressions and have consistent and well-defined behavior for Table API and SQL updated to FLIP-37. We might need to discuss some of the behavior of particular functions but this should not

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread Thomas Weise
Congrats! On Wed, Aug 14, 2019, 7:12 AM Robert Metzger wrote: > Congratulations! Very happy to have you onboard :) > > On Wed, Aug 14, 2019 at 4:06 PM Kostas Kloudas wrote: > > > Congratulations Andrey! > > Well deserved! > > > > Kostas > > > > On Wed, Aug 14, 2019 at 4:04 PM Yun Tang wrote:

[jira] [Created] (FLINK-13721) BroadcastState should support StateTTL

2019-08-14 Thread JIRA
Kerem Ulutaş created FLINK-13721: Summary: BroadcastState should support StateTTL Key: FLINK-13721 URL: https://issues.apache.org/jira/browse/FLINK-13721 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-52: Remove legacy Program interface.

2019-08-14 Thread Aljoscha Krettek
+1 (for the same reasons I posted on the other thread) > On 14. Aug 2019, at 15:03, Zili Chen wrote: > > +1 > > It could be regarded as part of Flink client api refactor. > Removal of stale code paths helps reason refactor. > > There is one thing worth attention that in this thread[1] Thomas

Re: Watermarks not propagated to WebUI?

2019-08-14 Thread Thomas Weise
I have also noticed this issue (Flink 1.5, Flink 1.8), and it appears with higher parallelism. This can be confusing to the user when watermarks actually work and can be observed using the metrics. On Wed, Aug 14, 2019 at 7:36 AM Jan Lukavský wrote: > Hi, > > is it possible, that watermarks

Re: [DISCUSS] Flink Python User-Defined Function for Table API

2019-08-14 Thread jincheng sun
Hi Thomas, Thanks for your confirmation and the very important reminder about bundle processing. I have had add the description about how to perform bundle processing from the perspective of checkpoint and watermark. Feel free to leave comments if there are anything not describe clearly. Best,

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread jincheng sun
Congrats Andrey! Very happy to have you onboard :) Best, Jincheng Yu Li 于2019年8月15日周四 上午12:06写道: > Congratulations Andrey! Well deserved! > > Best Regards, > Yu > > > On Wed, 14 Aug 2019 at 17:55, Aleksey Pak wrote: > > > Congratulations, Andrey! > > > > On Wed, Aug 14, 2019 at 4:53 PM Markos

  1   2   >