Re: [VOTE] FLIP-58: Flink Python User-Defined Function for Table API

+1 It is extremely useful for ML users. On Mon, Sep 2, 2019 at 9:46 AM Shaoxuan Wang wrote: > +1 (binding) > > This will be a great feature for Flink users, especially for the data > science and AI engineers. > > Regards, > Shaoxuan > > > On Fri, Aug 30, 2019 at 1:35 PM Jeff Zhang wrote: > >

Re: [DISCUSS] Simplify Flink's cluster level RestartStrategy configuration

+1 for this proposal. IMO, it not only simplifies the cluster configuration, but also seems more fit logic to not rely on some low-level speicific parameters to judge the upper-level strategy. It is also resonable to push forward the restart strategy configuration step by step for batch later.

Re: [DISCUSS] FLIP-53: Fine Grained Resource Management

Updated the FLIP wiki page [1], with the following changes. - Remove the step of converting pipelined edges between different slot sharing groups into blocking edges. - Set `allSourcesInSamePipelinedRegion` to true by default. Thank you~ Xintong Song On Mon, Sep 2, 2019 at 11:50 AM

Re: [DISCUSS] FLIP-53: Fine Grained Resource Management

Regarding changing edge type, I think actually we don't need to do this for batch jobs neither, because we don't have public interfaces for users to explicitly set slot sharing groups in DataSet API and SQL/Table API. We have such interfaces in DataStream API only. Thank you~ Xintong Song On

Re: [DISCUSS] FLIP-49: Unified Memory Configuration for TaskExecutors

I just updated the FLIP wiki page [1], with the following changes: - Network memory uses JVM direct memory, and is accounted when setting JVM max direct memory size parameter. - Use dynamic configurations (`-Dkey=value`) to pass calculated memory configs into TaskExecutors, instead of

Re: [DISCUSS] FLIP-49: Unified Memory Configuration for TaskExecutors

Yes I'll address the memory reservation functionality in a separate FLIP to cooperate with FLIP-49 (sorry for being late for the discussion). Best Regards, Yu On Mon, 2 Sep 2019 at 11:14, Xintong Song wrote: > Sorry for the late response. > > - Regarding the `TaskExecutorSpecifics` naming,

Re: [DISCUSS] Simplify Flink's cluster level RestartStrategy configuration

+1 to simplify the RestartStrategy configuration One thing to confirm is whether the default delay should be "0 s" in the case of "If the config option `restart-strategy` is not configured" and "If checkpointing is enabled". I see a related discussion([SURVEY] Is the default restart delay of 0s

Re: [DISCUSS] FLIP-49: Unified Memory Configuration for TaskExecutors

Sorry for the late response. - Regarding the `TaskExecutorSpecifics` naming, let's discuss the detail in PR. - Regarding passing parameters into the `TaskExecutor`, +1 for using dynamic configuration at the moment, given that there are more questions to be discussed to have a general framework

Re: [DISCUSS] Releasing Flink 1.8.2

+1 for a 1.8.2 release, thanks for bringing this up Jincheng! Best Regards, Yu On Mon, 2 Sep 2019 at 09:19, Thomas Weise wrote: > +1 for the 1.8.2 release > > I marked https://issues.apache.org/jira/browse/FLINK-13586 for this > release. It would be good to compensate for the backward

Re: [DISCUSS] Releasing Flink 1.8.2

Thanks Jincheng, I will look into the release guidelines. Hi @Thomas Weise , should we mark FLINK-13586 as a blocker? And how long do you think this issue will take? I summarized the current status of issues we need to track: [Bloker]: [FLINK-13897] OSS FS NOTICE file is placed in wrong

Re: How to handle Flink Job with 400MB+ Uberjar with 800+ containers ?

Hi Elkhan, >>Regarding "One optimization that we take is letting yarn to reuse the flink-dist jar which was localized when running previous jobs." >>We are intending to use Flink Real-time pipeline for Replay from Hive/HDFS (from offline source), to have 1 single pipeline for both batch and

Re: [PROPOSAL] Force rebase on master before merge

Hi all, Thanks for your replies. For Till's question, as Chesnay said if we cannot attach travis checks via CIBot workflow the mechanism provided by GitHub doesn't work at all, which states "This setting will not take effect unless at least one status check is enabled". Technically we can

Re: [VOTE] FLIP-58: Flink Python User-Defined Function for Table API

+1 (binding) This will be a great feature for Flink users, especially for the data science and AI engineers. Regards, Shaoxuan On Fri, Aug 30, 2019 at 1:35 PM Jeff Zhang wrote: > +1， very looking forward this feature in flink 1.10 > > > Yu Li 于2019年8月30日周五 上午11:08写道： > > > +1 (non-binding)

Re: [DISCUSS] Releasing Flink 1.8.2

+1 for the 1.8.2 release I marked https://issues.apache.org/jira/browse/FLINK-13586 for this release. It would be good to compensate for the backward incompatible change to ClosureCleaner that was introduced in 1.8.1, which affects downstream dependencies. Thanks, Thomas On Sun, Sep 1, 2019 at

Re: [DISCUSS] Simplify Flink's cluster level RestartStrategy configuration

+1. The new behavior makes sense to me. BTW, we need a FLIP for this :) On Fri, Aug 30, 2019 at 10:17 PM Till Rohrmann wrote: > After an offline discussion with Stephan, we concluded that changing the > default restart strategy for batch jobs is not that easy because the > cluster level

Re: [DISCUSS] Releasing Flink 1.8.2

Hi Jark, Glad to hear that you want to be the Release Manager of flink 1.8.2. I believe that you will be a great RM, and I am very willing to help you with the final release in the final stages. :) The release of Apache Flink involves a number of tasks. For details, you can consult the

Re: [ANNOUNCE] Kinesis connector becomes part of Flink releases

Great to know, thanks for the efforts Bowen! And I believe it worth a release note in the original JIRA, wdyt? Thanks. Best Regards, Yu On Sat, 31 Aug 2019 at 11:01, Bowen Li wrote: > Hi all, > > I'm glad to announce that, as #9494 > was merged

Re: [SURVEY] Is the default restart delay of 0s causing problems?

-1 on increasing the default delay to none zero, with below reasons: a) I could see some concerns about setting the delay to zero in the very original JIRA (FLINK-2993 ) but later on in FLINK-9158

Re: State of FLIPs

Thanks for the reminder Chesnay. I've just moved FLIP-50 into accepted list since it has already passed the vote and is under development. Best Regards, Yu On Fri, 30 Aug 2019 at 22:29, Dian Fu wrote: > Hi Chesnay, > > Thanks a lot for the remind. FLIP-38 has been released in 1.9 and I have >

Re: Potential block size issue with S3 binary files

Sounds reasonable. I am adding Arvid to the thread - IIRC he authored that tool in his Stratosphere days. And my a stroke of luck, he is now working on Flink again. @Arvid - what are your thoughts on Ken's suggestions? On Fri, Aug 30, 2019 at 8:56 PM Ken Krugler wrote: > Hi Stephan (switching

Re: CiBot Update

Thanks for @Chesnay Schepler for this really helpful command! I agreed with @Dian Fu that we should include this in the "Bot commands". I just wanted to find the exact command but found nothing in the template and come here for the

[jira] [Created] (FLINK-13936) NOTICE-binary is outdated

Chesnay Schepler created FLINK-13936: Summary: NOTICE-binary is outdated Key: FLINK-13936 URL: https://issues.apache.org/jira/browse/FLINK-13936 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-13935) YarnPrioritySchedulingITCase fails on hadoop 2.4.1

Chesnay Schepler created FLINK-13935: Summary: YarnPrioritySchedulingITCase fails on hadoop 2.4.1 Key: FLINK-13935 URL: https://issues.apache.org/jira/browse/FLINK-13935 Project: Flink

[jira] [Created] (FLINK-13934) HistoryServerStaticFileServerHandlerTest failed on Travis

Chesnay Schepler created FLINK-13934: Summary: HistoryServerStaticFileServerHandlerTest failed on Travis Key: FLINK-13934 URL: https://issues.apache.org/jira/browse/FLINK-13934 Project: Flink