Re: [DISCUSS] Expose or setup a secur...@flink.apache.org mailing list for security report and discussion

Hi all, There are no new feedbacks and it seems that we have received enough feedback about setup a secur...@flink.apache.org mailing list[1] for security report and discussion. It shows that it's optional as we can use either secur...@flink.apache.org or

Re: [DISCUSS] FLIP-83: Flink End-to-end Performance Testing Framework

Thanks Yu for bringing up this discussion. The e2e perf tests can be really helpful and the overall design looks good to me. Sorry it's late but I have 2 questions about the result check. 1. How do we measure the job throughput? By measuring the job execution time on a finite input data set, or

Re: [VOTE] FLIP-83: Flink End-to-end Performance Testing Framework

Thanks Yu for this proposal. +1(non-binding) for the overall design. I have 2 questions about the result check though. Posted in the discussion ML thread. Thanks, Zhu Zhu Becket Qin 于2019年11月21日周四 上午11:57写道： > Hi Yu, > > Thanks for updating the FLIP wiki. +1 from me. I do not have further >

Re: Cron style for checkpoint

Hi Currently, Flink does not support such feature, from what you describe, does set an appropriate timeout for checkpoint can solve your problem? Best, Congxian shuwen zhou 于2019年11月21日周四 下午12:06写道： > Hi Jiayi, > It would be great if Flink could have a user defined interface for user to >

Re: Cron style for checkpoint

Hi Jiayi, It would be great if Flink could have a user defined interface for user to implement to control checkpoint behavior, at least for time related behavior. I brought up a wish on JIRA [1], perhaps it described clearly enough. [1] https://issues.apache.org/jira/browse/FLINK-14884 On Thu,

[jira] [Created] (FLINK-14884) User defined function for checkpoint trigger time and interval behavior

Shuwen Zhou created FLINK-14884: --- Summary: User defined function for checkpoint trigger time and interval behavior Key: FLINK-14884 URL: https://issues.apache.org/jira/browse/FLINK-14884 Project: Flink

Re: [VOTE] FLIP-83: Flink End-to-end Performance Testing Framework

Hi Yu, Thanks for updating the FLIP wiki. +1 from me. I do not have further questions. Jiangjie (Becket) Qin On Wed, Nov 20, 2019 at 8:58 PM Yu Li wrote: > Thanks all for the vote and further discussions. > > I have updated the FLIP document according to aihua's reply (with some > minor

[jira] [Created] (FLINK-14883) Resource management on state backends

Yu Li created FLINK-14883: - Summary: Resource management on state backends Key: FLINK-14883 URL: https://issues.apache.org/jira/browse/FLINK-14883 Project: Flink Issue Type: New Feature

Re:Cron style for checkpoint

Hi Shuwen, As far as I know, Flink can only support checkpoint with a fixed interval. However I think the flexible mechanism of triggering checkpoint is worth working on, at least from my perspective. And it may not only be a cron style. In our business scenario, the data traffic

Cron style for checkpoint

Hi Community, I would like to know if there is a existing function to support cron style checkpoint? The case is, our data traffic is huge on HH:30 for each hour. We don't wont checkpoint to fall in that range of time. A cron like 15,45 * * * * to set for checkpoint would be nice. If a checkpoint

[jira] [Created] (FLINK-14882) Wrong null order when change flink default null order from NullCollation.LOW to NullCollation.FIRST

Leonard Xu created FLINK-14882: -- Summary: Wrong null order when change flink default null order from NullCollation.LOW to NullCollation.FIRST Key: FLINK-14882 URL: https://issues.apache.org/jira/browse/FLINK-14882

Re: [DISCUSS] remove the default in-memory catalog from Flink table module

Hi Bowen, I agree with Jingsong. I doubt whether it works if we remove the default catalog. Because all the temporary objects requires the current catalog and current database to qualify. That means even the simplest Table API programs will need to specify a catalog first, if it is something have

Re: [DISCUSS] Primary keys in Table API

Hi Dawid, Actually I'm not 100% sure about both choices: always enforce primary key by Flink and not enforce at all. As you said, always enforce primary key is hard to achieve since Flink doesn't own data most of the time. But with some certain cases, it's actually doable, like writing a new

Re: [VOTE] FLIP-79: Flink Function DDL Support (1.10 Release Feature Only)

Hi Dawid, In general, I agree we should keep separate FLIPs when possible, if they each have some good scope and are decoupled. This one might hardly qualify for a FLIP because the scope is quite small and straight forward, and can be an overkill for FLIP itself. It's also a bit special as the

[jira] [Created] (FLINK-14881) Upgrade AWS SDK to support "IAM Roles for Service Accounts" in AWS EKS

Vincent Chenal created FLINK-14881: -- Summary: Upgrade AWS SDK to support "IAM Roles for Service Accounts" in AWS EKS Key: FLINK-14881 URL: https://issues.apache.org/jira/browse/FLINK-14881 Project:

[jira] [Created] (FLINK-14880) package org.apache.flink.sql.parser.impl does not exist

Muideen A. Lawal created FLINK-14880: Summary: package org.apache.flink.sql.parser.impl does not exist Key: FLINK-14880 URL: https://issues.apache.org/jira/browse/FLINK-14880 Project: Flink

Flink Forward North America 2020 - Call for Presentations open until January 12th, 2020

Hi all, Flink Forward North America returns to San Francisco on March 23-25, 2020. For the first time in North America, the conference will feature two days of talks and one day of training. We are happy to announce that the Call for Presentations is open! If you'd like to give a talk and share

Re: [DISCUSSION] Kafka Metrics Reporter

@Becket , Yun: Regarding the core/ecosystem project: I don't completely agree with your arguments regarding why this should be an external ecosystem project instead of part of the Flink repo. A metric connector is relevant for the Flink users, not the metric store. Metric storage systems don't

[jira] [Created] (FLINK-14879) Support database related DDLs that needs return value in TableEnvironment

Terry Wang created FLINK-14879: -- Summary: Support database related DDLs that needs return value in TableEnvironment Key: FLINK-14879 URL: https://issues.apache.org/jira/browse/FLINK-14879 Project: Flink

[jira] [Created] (FLINK-14878) Support `use catalog` through sqlUpdate() method in TableEnvironment

Terry Wang created FLINK-14878: -- Summary: Support `use catalog` through sqlUpdate() method in TableEnvironment Key: FLINK-14878 URL: https://issues.apache.org/jira/browse/FLINK-14878 Project: Flink

[jira] [Created] (FLINK-14877) Enable FRocksDB to optimize performance via options when bounding total memory

Yun Tang created FLINK-14877: Summary: Enable FRocksDB to optimize performance via options when bounding total memory Key: FLINK-14877 URL: https://issues.apache.org/jira/browse/FLINK-14877 Project:

Re: [DISCUSS] Primary keys in Table API

Hi Kurt, You are right that this is included in the SQL standard. The problem is we are not a database and we do not own the data. There is no way for us to truly enforce the uniqueness constraint. We are always writing to some external systems that, moreover we are not an exclusive

Re: [VOTE] FLIP-83: Flink End-to-end Performance Testing Framework

Thanks all for the vote and further discussions. I have updated the FLIP document according to aihua's reply (with some minor refinement/supplements). For your convenience, below are the contents I added: ** *There're also other dimensions other than Flink

Re: [DISCUSS] Primary keys in Table API

Hi all, Thanks Dawid for bringing this up, this is very important property of SQL and I'm big +1 to have this. But before the discussion going deeply, I want to first mention that according to SQL standard, any unique key constraint which including primary key, should be always be enforced.

Re: [VOTE] Release flink-shaded 9.0, release candidate #1

+1 (non-binding) Best, tison. Aljoscha Krettek 于2019年11月20日周三 下午6:58写道： > +1 (binding) > > Best, > Aljoscha > > > On 19. Nov 2019, at 23:13, Chesnay Schepler wrote: > > > > Hi everyone, > > Please review and vote on the release candidate #1 for the version 9.0, > as follows: > > [ ] +1,

Re: [VOTE] FLIP-86: Improve Connector Properties

+1 (binding) Best, Kurt On Wed, Nov 20, 2019 at 6:56 PM Aljoscha Krettek wrote: > +1 (binding) > > Best, > Aljoscha > > > On 20. Nov 2019, at 11:36, Jingsong Li wrote: > > > > +1 (non-binding) > > Thanks Jark for driving this. > > > > Best, > > Jingsong Lee > > > > On Wed, Nov 20, 2019 at

[jira] [Created] (FLINK-14876) Putting xercesImpl related classes into alwaysParentFirstLoaderPatterns

liupengcheng created FLINK-14876: Summary: Putting xercesImpl related classes into alwaysParentFirstLoaderPatterns Key: FLINK-14876 URL: https://issues.apache.org/jira/browse/FLINK-14876 Project:

Re: [VOTE] Release flink-shaded 9.0, release candidate #1

+1 (binding) Best, Aljoscha > On 19. Nov 2019, at 23:13, Chesnay Schepler wrote: > > Hi everyone, > Please review and vote on the release candidate #1 for the version 9.0, as > follows: > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide specific comments) > >

Re: [VOTE] FLIP-86: Improve Connector Properties

+1 (binding) Best, Aljoscha > On 20. Nov 2019, at 11:36, Jingsong Li wrote: > > +1 (non-binding) > Thanks Jark for driving this. > > Best, > Jingsong Lee > > On Wed, Nov 20, 2019 at 6:25 PM Jark Wu wrote: > >> +1 (binding) >> >> Best, >> Jark >> >> On Wed, 20 Nov 2019 at 18:11, Terry

[jira] [Created] (FLINK-14875) Set Unmodifiable collection serializer of Kryo in Flink code

Jing Zhang created FLINK-14875: -- Summary: Set Unmodifiable collection serializer of Kryo in Flink code Key: FLINK-14875 URL: https://issues.apache.org/jira/browse/FLINK-14875 Project: Flink

[jira] [Created] (FLINK-14874) add local aggregate to solve data skew for ROLLUP/CUBE case

godfrey he created FLINK-14874: -- Summary: add local aggregate to solve data skew for ROLLUP/CUBE case Key: FLINK-14874 URL: https://issues.apache.org/jira/browse/FLINK-14874 Project: Flink

Re: [VOTE] FLIP-86: Improve Connector Properties

+1 (non-binding) Thanks Jark for driving this. Best, Jingsong Lee On Wed, Nov 20, 2019 at 6:25 PM Jark Wu wrote: > +1 (binding) > > Best, > Jark > > On Wed, 20 Nov 2019 at 18:11, Terry Wang wrote: > > > +1 (non-binding) > > > > Best, > > Terry Wang > > > > > > > > > 2019年11月20日 17:47，Dawid

Re: [VOTE] FLIP-86: Improve Connector Properties

+1 (binding) Best, Jark On Wed, 20 Nov 2019 at 18:11, Terry Wang wrote: > +1 (non-binding) > > Best, > Terry Wang > > > > > 2019年11月20日 17:47，Dawid Wysakowicz 写道： > > > > +1 from my side > > > > Best, > > > > Dawid > > > > On 20/11/2019 10:36, Jark Wu wrote: > >> Hi everyone, > >> > >> I

Re: [VOTE] FLIP-86: Improve Connector Properties

+1 (non-binding) Best, Terry Wang > 2019年11月20日 17:47，Dawid Wysakowicz 写道： > > +1 from my side > > Best, > > Dawid > > On 20/11/2019 10:36, Jark Wu wrote: >> Hi everyone, >> >> I would like to start a vote on FLIP-86. The discussion seems to have >> reached an agreement. >> >> Please

Re: [VOTE] FLIP-86: Improve Connector Properties

+1 from my side Best, Dawid On 20/11/2019 10:36, Jark Wu wrote: > Hi everyone, > > I would like to start a vote on FLIP-86. The discussion seems to have > reached an agreement. > > Please vote for the following design document: > >

Re: [DISCUSS] Primary keys in Table API

Hey, After an offline discussion I updated the FLIP slightly. In the initial FLIP I put the OUT_OF_LINE definition at a wrong position. I moved it into the statement paranthesis. I also moved the primary key to the TableSchema class. I also extended sections about UNIQUE key support and how do

[VOTE] FLIP-86: Improve Connector Properties

Hi everyone, I would like to start a vote on FLIP-86. The discussion seems to have reached an agreement. Please vote for the following design document: https://cwiki.apache.org/confluence/display/FLINK/FLIP-86%3A+Improve+Connector+Properties The discussion can be found at:

Re: [DISCUSS] FLIP-86: Improve Connector Properties

Hi everyone, Thanks for the comments in the doc. I have updated the doc into wiki page. As there are no further concerns, I will start a voting process soon. Thanks, Jark On Fri, 15 Nov 2019 at 23:36, Jark Wu wrote: > Hi, > > Thank you all for the positive feedbacks so far. > > Timo

Re: [DISCUSSION] Kafka Metrics Reporter

@Bowen I can see where you're coming from, but I don't think this would work too well. Your "stream" would have to contain events for added/removed metrics, but metrics are inherently not Serializable. I think this would end up being a weird special case. (Periodically emitting the values of

Re: [DISCUSS] Primary keys in Table API

Hi Jark, Thank you for your comments. Ad. 1 My intention was to throw an exception if a user tries to use the ENFORCE mode. SQL standard states this is the default mode, so if we ever want to support the ENFORCE/NOT ENFORCE mode we need to make sure users now always provide the NOT ENFORCE flag.

[jira] [Created] (FLINK-14873) Make PackagedProgram#savepointSettings final

Zili Chen created FLINK-14873: - Summary: Make PackagedProgram#savepointSettings final Key: FLINK-14873 URL: https://issues.apache.org/jira/browse/FLINK-14873 Project: Flink Issue Type:

Re: How to estimate the memory size of flink state

Thank you. Your suggestion is good and I benefit a lot. For my case, I want to know the state memory size for other reasons. When the the gc pressure is bigger, I need to limit the source or discard some data from the source to ensure job’s running. If the state size is bigger, I

[jira] [Created] (FLINK-14872) Potential deadlock for task reading from blocking ResultPartition.

Yingjie Cao created FLINK-14872: --- Summary: Potential deadlock for task reading from blocking ResultPartition. Key: FLINK-14872 URL: https://issues.apache.org/jira/browse/FLINK-14872 Project: Flink