[jira] [Created] (FLINK-19793) KafkaTableITCase.testKafkaSourceSinkWithMetadata fails on AZP

2020-10-23 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-19793: - Summary: KafkaTableITCase.testKafkaSourceSinkWithMetadata fails on AZP Key: FLINK-19793 URL: https://issues.apache.org/jira/browse/FLINK-19793 Project: Flink

Re: [DISCUSS] Releasing Apache Flink 1.11.3

2020-10-23 Thread Till Rohrmann
Thanks for starting this discussion Gordon. There are over 100 issues which are fixed for 1.11.3. Hence +1 for a soonish 1.11.3 release. Thanks for volunteering as our release managers Gordon and Xintong! Cheers, Till On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon) Tai wrote: > Hi, > > Xintong

[DISCUSS] Releasing Apache Flink 1.11.3

2020-10-23 Thread Tzu-Li (Gordon) Tai
Hi, Xintong and I would like to start a discussion for releasing Flink 1.11.3 soon. It seems like we already have a few pressing issues that needs to be included in a new hotfix release: - Heap-based timers’ restore behaviour is causing a critical recovery issue for StateFun [1] [2] [3].

[jira] [Created] (FLINK-19792) Interval join with equal time attributes is not recognized

2020-10-23 Thread Timo Walther (Jira)
Timo Walther created FLINK-19792: Summary: Interval join with equal time attributes is not recognized Key: FLINK-19792 URL: https://issues.apache.org/jira/browse/FLINK-19792 Project: Flink

Re: [SURVEY] Remove Mesos support

2020-10-23 Thread Piyush Narang
Thanks Kostas. If there's items we can help with, I'm sure we'd be able to find folks who would be excited to contribute / help in any way. -- Piyush On 10/23/20, 10:25 AM, "Kostas Kloudas" wrote: Thanks Piyush for the message. After this, I revoke my +1. I agree with the previous

Re: [SURVEY] Remove Mesos support

2020-10-23 Thread Kostas Kloudas
Thanks Piyush for the message. After this, I revoke my +1. I agree with the previous opinions that we cannot drop code that is actively used by users, especially if it something that deep in the stack as support for cluster management framework. Cheers, Kostas On Fri, Oct 23, 2020 at 4:15 PM

Re: [SURVEY] Remove Mesos support

2020-10-23 Thread Piyush Narang
Hi folks, We at Criteo are active users of the Flink on Mesos resource management component. We are pretty heavy users of Mesos for scheduling workloads on our edge datacenters and we do want to continue to be able to run some of our Flink topologies (to compute machine learning short term

[jira] [Created] (FLINK-19791) PartitionRequestClientFactoryTest.testInterruptsNotCached fails with NullPointerException

2020-10-23 Thread Robert Metzger (Jira)
Robert Metzger created FLINK-19791: -- Summary: PartitionRequestClientFactoryTest.testInterruptsNotCached fails with NullPointerException Key: FLINK-19791 URL: https://issues.apache.org/jira/browse/FLINK-19791

[jira] [Created] (FLINK-19790) Writing MAP to Kafka with JSON format produces incorrect data.

2020-10-23 Thread Fabian Hueske (Jira)
Fabian Hueske created FLINK-19790: - Summary: Writing MAP to Kafka with JSON format produces incorrect data. Key: FLINK-19790 URL: https://issues.apache.org/jira/browse/FLINK-19790 Project: Flink

Re: [SURVEY] Remove Mesos support

2020-10-23 Thread Till Rohrmann
Thanks for starting this survey Robert! I second Konstantin and Xintong in the sense that our Mesos user's opinions should matter most here. If our community is no longer using the Mesos integration, then I would be +1 for removing it in order to decrease the maintenance burden. Cheers, Till On

Re: [SURVEY] Remove Mesos support

2020-10-23 Thread Kostas Kloudas
+1 for adding a warning about the removal of Mesos support and I would also propose to state explicitly in the warning the version that we are planning to actually remove it (e.g. 1.13 or even 1.14 if we feel it is too aggressive). This will help as a reminder to users and devs about the upcoming

[jira] [Created] (FLINK-19789) Migrate Hive connector to new table source sink interface

2020-10-23 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-19789: Summary: Migrate Hive connector to new table source sink interface Key: FLINK-19789 URL: https://issues.apache.org/jira/browse/FLINK-19789 Project: Flink

Re: [SURVEY] Remove Mesos support

2020-10-23 Thread Xintong Song
+1 for adding a warning in 1.12 about planning to remove Mesos support. With my developer hat on, removing the Mesos support would definitely reduce the maintaining overhead for the deployment and resource management related components. On the other hand, the Flink on Mesos users' voices

Re: [VOTE] FLIP-135: Approximate Task-Local Recovery

2020-10-23 Thread Yuan Mei
Hey All, The voting time for FLIP-135 has passed. I'm closing the vote now. There are 6 +1 votes, 4 of which are binding: - Piotr (binding) - Zhijiang (binding) - Steven (non-binding) - Roman (non-binding) - Yu (binding) - Becket (binding) There were no disapproving votes. Thus, FLIP-135 has

Re: [SURVEY] Remove Mesos support

2020-10-23 Thread Konstantin Knauf
Hi Robert, +1 to the plan you outlined. If we were to drop support in Flink 1.13+, we would still support it in Flink 1.12- with bug fixes for some time so that users have time to move on. It would certainly be very interesting to hear from current Flink on Mesos users, on how they see the

Re: [DISCUSS] Release 1.12 Feature Freeze

2020-10-23 Thread Yang Wang
+1 for Sunday night. This will help a lot for FLIP-144(Native Kubernetes HA for Flink). Best, Yang Zhu Zhu 于2020年10月23日周五 下午2:15写道: > +1 for November 2nd > > Thanks, > Zhu > > Xintong Song 于2020年10月20日周二 下午11:26写道: > > > +1 for Nov. 2nd. > > > > Thank you~ > > > > Xintong Song > > > > > > > >

Re: [VOTE] FLIP-135: Approximate Task-Local Recovery

2020-10-23 Thread Becket Qin
+1 (binding) Thanks for the proposal and well written wiki, Yuan. It is a very useful feature. Thanks, Jiangjie (Becket) Qin On Wed, Oct 21, 2020 at 11:31 AM Yu Li wrote: > +1 (binding) > > Thanks for driving this Yuan! It definitely helps in relative scenarios, > especially in the machine

[SURVEY] Remove Mesos support

2020-10-23 Thread Robert Metzger
Hi all, I wanted to discuss if it makes sense to remove support for Mesos in Flink. It seems that nobody is actively maintaining that component (except for necessary refactorings because of interfaces we are changing), and there are almost no users reporting issues or asking for features. The

Re: [VOTE] NEW FLIP-102: Add More Metrics to TaskManager

2020-10-23 Thread Xintong Song
I'm aware that many steps are already done. But I believe not all of them are done, as the FLIP doc currently shows, unless the FLIP is already completed. Taking a closer look at the issues, I believe step 5 should point to FLINK-19764, rather than FLINK-15328 which is closed for duplication.

Re: [VOTE] NEW FLIP-102: Add More Metrics to TaskManager

2020-10-23 Thread Andrey Zagrebin
Thanks a lot for this nice UI guys! +1 and for closed issues that is just because many steps have been already done. Best, Andrey On Fri, Oct 23, 2020 at 11:12 AM Till Rohrmann wrote: > Thanks for reviving this Flip Yadong! The changes look good to me and the > new memory UI looks awesome :-)

[jira] [Created] (FLINK-19788) Flaky test ShuffleCompressionITCase.testDataCompressionForBlockingShuffle?

2020-10-23 Thread Matthias (Jira)
Matthias created FLINK-19788: Summary: Flaky test ShuffleCompressionITCase.testDataCompressionForBlockingShuffle? Key: FLINK-19788 URL: https://issues.apache.org/jira/browse/FLINK-19788 Project: Flink

[jira] [Created] (FLINK-19787) Migrate Filesystem source to new table source sink interface

2020-10-23 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-19787: Summary: Migrate Filesystem source to new table source sink interface Key: FLINK-19787 URL: https://issues.apache.org/jira/browse/FLINK-19787 Project: Flink

[jira] [Created] (FLINK-19786) Flink doesn't set proper nullability for Logical types for Confluent Avro Serialization

2020-10-23 Thread Jira
Maciej Bryński created FLINK-19786: -- Summary: Flink doesn't set proper nullability for Logical types for Confluent Avro Serialization Key: FLINK-19786 URL: https://issues.apache.org/jira/browse/FLINK-19786

Re: [VOTE]FLIP-149: Introduce the upsert-kafka connector

2020-10-23 Thread Timo Walther
+1 Thanks, Timo On 23.10.20 10:21, Jingsong Li wrote: +1 On Fri, Oct 23, 2020 at 3:52 PM Konstantin Knauf wrote: +1 On Fri, Oct 23, 2020 at 9:36 AM Jark Wu wrote: +1 On Fri, 23 Oct 2020 at 15:25, Shengkai Fang wrote: Hi, all, I would like to start the vote for FLIP-149[1], which

[jira] [Created] (FLINK-19785) Upgrade commons-io to 2.7 or newer

2020-10-23 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-19785: - Summary: Upgrade commons-io to 2.7 or newer Key: FLINK-19785 URL: https://issues.apache.org/jira/browse/FLINK-19785 Project: Flink Issue Type: Task

[jira] [Created] (FLINK-19784) Upgrade okhttp to 3.13.0 or newer

2020-10-23 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-19784: - Summary: Upgrade okhttp to 3.13.0 or newer Key: FLINK-19784 URL: https://issues.apache.org/jira/browse/FLINK-19784 Project: Flink Issue Type: Task

[jira] [Created] (FLINK-19783) Upgrade mesos to 1.7 or newer

2020-10-23 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-19783: - Summary: Upgrade mesos to 1.7 or newer Key: FLINK-19783 URL: https://issues.apache.org/jira/browse/FLINK-19783 Project: Flink Issue Type: Task

[jira] [Created] (FLINK-19782) Upgrade antlr to 4.7.1 or newer

2020-10-23 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-19782: - Summary: Upgrade antlr to 4.7.1 or newer Key: FLINK-19782 URL: https://issues.apache.org/jira/browse/FLINK-19782 Project: Flink Issue Type: Task

[jira] [Created] (FLINK-19781) Upgrade commons_codec to 1.13 or newer

2020-10-23 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-19781: - Summary: Upgrade commons_codec to 1.13 or newer Key: FLINK-19781 URL: https://issues.apache.org/jira/browse/FLINK-19781 Project: Flink Issue Type: Task

Re: [VOTE]FLIP-149: Introduce the upsert-kafka connector

2020-10-23 Thread Jingsong Li
+1 On Fri, Oct 23, 2020 at 3:52 PM Konstantin Knauf wrote: > +1 > > On Fri, Oct 23, 2020 at 9:36 AM Jark Wu wrote: > > > +1 > > > > On Fri, 23 Oct 2020 at 15:25, Shengkai Fang wrote: > > > > > Hi, all, > > > > > > I would like to start the vote for FLIP-149[1], which is discussed and > > >

Re: [DISCUSS] FLIP-149: Introduce the KTable Connector

2020-10-23 Thread Jingsong Li
I see, I understand what you mean is avoiding the loss of historical data Logically, another option is never clean up, so don't have to turn on compact I am OK with the implementation, It's that feeling shouldn't be a logical limitation Best, Jingsong On Fri, Oct 23, 2020 at 4:09 PM Kurt Young

Re: [VOTE] NEW FLIP-102: Add More Metrics to TaskManager

2020-10-23 Thread Till Rohrmann
Thanks for reviving this Flip Yadong! The changes look good to me and the new memory UI looks awesome :-) I think the reason why the REST issues are closed is because they are already done. In that sense some of the work already finished. +1 for adopting this FLIP and moving forward with

Re: [DISCUSS] FLIP-149: Introduce the KTable Connector

2020-10-23 Thread Kurt Young
To be precise, it means the Kakfa topic should set the configuration "cleanup.policy" to "compact" not "delete". Best, Kurt On Fri, Oct 23, 2020 at 4:01 PM Jingsong Li wrote: > I just notice there is a limitation in the FLIP: > > > Generally speaking, the underlying topic of the upsert-kafka

[jira] [Created] (FLINK-19780) FlinkRelMdDistinctRowCount#getDistinctRowCount(Calc) will always return 0 when number of rows are large

2020-10-23 Thread Caizhi Weng (Jira)
Caizhi Weng created FLINK-19780: --- Summary: FlinkRelMdDistinctRowCount#getDistinctRowCount(Calc) will always return 0 when number of rows are large Key: FLINK-19780 URL:

Re: Resuming Savepoint issue with upgraded Flink version 1.11.2

2020-10-23 Thread Till Rohrmann
Hi Partha, have you set explicit operator uids [1]? This is usually recommended in order to ensure that Flink can match operator state across re-submissions in particular when doing job upgrades. If you haven't set the uids explicitly, then Flink will generate them automatically. It could be the

Re: [DISCUSS] FLIP-149: Introduce the KTable Connector

2020-10-23 Thread Jingsong Li
I just notice there is a limitation in the FLIP: > Generally speaking, the underlying topic of the upsert-kafka source must be compacted. Besides, the underlying topic must have all the data with the same key in the same partition, otherwise, the result will be wrong. According to my

Re: [VOTE]FLIP-149: Introduce the upsert-kafka connector

2020-10-23 Thread Konstantin Knauf
+1 On Fri, Oct 23, 2020 at 9:36 AM Jark Wu wrote: > +1 > > On Fri, 23 Oct 2020 at 15:25, Shengkai Fang wrote: > > > Hi, all, > > > > I would like to start the vote for FLIP-149[1], which is discussed and > > reached a consensus in the discussion thread[2]. The vote will be open > > until

Re: [VOTE]FLIP-149: Introduce the upsert-kafka connector

2020-10-23 Thread Jark Wu
+1 On Fri, 23 Oct 2020 at 15:25, Shengkai Fang wrote: > Hi, all, > > I would like to start the vote for FLIP-149[1], which is discussed and > reached a consensus in the discussion thread[2]. The vote will be open > until 16:00(UTC+8) 28th Oct. (72h, exclude weekends), unless there is an >

[VOTE]FLIP-149: Introduce the upsert-kafka connector

2020-10-23 Thread Shengkai Fang
Hi, all, I would like to start the vote for FLIP-149[1], which is discussed and reached a consensus in the discussion thread[2]. The vote will be open until 16:00(UTC+8) 28th Oct. (72h, exclude weekends), unless there is an objection or not enough votes. [1]

Re: [DISCUSS] FLIP-149: Introduce the KTable Connector

2020-10-23 Thread Timo Walther
+1 for voting Regards, Timo On 23.10.20 09:07, Jark Wu wrote: Thanks Shengkai! +1 to start voting. Best, Jark On Fri, 23 Oct 2020 at 15:02, Shengkai Fang wrote: Add one more message, I have already updated the FLIP[1]. [1]

Re: [DISCUSS] FLIP-149: Introduce the KTable Connector

2020-10-23 Thread Jark Wu
Thanks Shengkai! +1 to start voting. Best, Jark On Fri, 23 Oct 2020 at 15:02, Shengkai Fang wrote: > Add one more message, I have already updated the FLIP[1]. > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-149%3A+Introduce+the+upsert-kafka+Connector > > Shengkai Fang

[jira] [Created] (FLINK-19779) Remove the "record_" field name prefix for Confluent Avro format deserialization

2020-10-23 Thread Danny Chen (Jira)
Danny Chen created FLINK-19779: -- Summary: Remove the "record_" field name prefix for Confluent Avro format deserialization Key: FLINK-19779 URL: https://issues.apache.org/jira/browse/FLINK-19779

Re: [DISCUSS] FLIP-149: Introduce the KTable Connector

2020-10-23 Thread Shengkai Fang
Add one more message, I have already updated the FLIP[1]. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-149%3A+Introduce+the+upsert-kafka+Connector Shengkai Fang 于2020年10月23日周五 下午2:55写道: > Hi, all. > It seems we have reached a consensus on the FLIP. If no one has other >

Re: [VOTE] NEW FLIP-102: Add More Metrics to TaskManager

2020-10-23 Thread Jark Wu
+1 Thanks for the work. Best, Jark On Fri, 23 Oct 2020 at 10:13, Xintong Song wrote: > Thanks Yadong, Mattias and Lining for reviving this FLIP. > > I've seen so many users confused by the current webui page of task manager > metrics. This FLIP should definitely help them understand the

Re: [DISCUSS] FLIP-149: Introduce the KTable Connector

2020-10-23 Thread Shengkai Fang
Hi, all. It seems we have reached a consensus on the FLIP. If no one has other objections, I would like to start the vote for FLIP-149. Best, Shengkai Jingsong Li 于2020年10月23日周五 下午2:25写道: > Thanks for explanation, > > I am OK for `upsert`. Yes, Its concept has been accepted by many systems. >

[jira] [Created] (FLINK-19778) Failed job reinitiated with wrong checkpoint after a ZK reconnection

2020-10-23 Thread Paul Lin (Jira)
Paul Lin created FLINK-19778: Summary: Failed job reinitiated with wrong checkpoint after a ZK reconnection Key: FLINK-19778 URL: https://issues.apache.org/jira/browse/FLINK-19778 Project: Flink

Re: [DISCUSS] FLIP-149: Introduce the KTable Connector

2020-10-23 Thread Jingsong Li
Thanks for explanation, I am OK for `upsert`. Yes, Its concept has been accepted by many systems. Best, Jingsong On Fri, Oct 23, 2020 at 12:38 PM Jark Wu wrote: > Hi Timo, > > I have some concerns about `kafka-cdc`, > 1) cdc is an abbreviation of Change Data Capture which is commonly used for

Re: [DISCUSS] Release 1.12 Feature Freeze

2020-10-23 Thread Zhu Zhu
+1 for November 2nd Thanks, Zhu Xintong Song 于2020年10月20日周二 下午11:26写道: > +1 for Nov. 2nd. > > Thank you~ > > Xintong Song > > > > On Tue, Oct 20, 2020 at 8:56 PM Leonard Xu wrote: > > > +1 for **Sunday night**. > > > > > > Best > > Leonard > > > > > 在 2020年10月20日,19:03,Jingsong Li 写道: > > >

Re: [DISCUSS] FLIP-148: Introduce Sort-Merge Based Blocking Shuffle to Flink

2020-10-23 Thread Yingjie Cao
Hi devs, This discussion thread has been opened for over a week. If there is no other concerns, I'd like to open a voting thread soon. Best, Yingjie Yingjie Cao 于2020年10月23日周五 上午11:56写道: > Hi Zhijiang, > > Thanks for your reply and suggestions. > > 1. For >

[jira] [Created] (FLINK-19777) java.lang.NullPointerException

2020-10-23 Thread frank wang (Jira)
frank wang created FLINK-19777: -- Summary: java.lang.NullPointerException Key: FLINK-19777 URL: https://issues.apache.org/jira/browse/FLINK-19777 Project: Flink Issue Type: Bug