Re: [REMINDER] Please only merge pull requests when CI is green and help keep our CI stable

2020-10-28 Thread Robert Metzger
Quick update: The issue has been resolved. It was actually not as bad as I thought, because flinkbot was showing the build as "FAILED", so people probably saw that something is wrong. On Tue, Oct 27, 2020 at 3:51 PM Robert Metzger wrote: > Hi all, > > Quick update on our CI system during the

Re: [VOTE] NEW FLIP-104: Add More Metrics to JobManager

2020-10-28 Thread Robert Metzger
Thank you for your UI work Yadong! +1 On Tue, Oct 27, 2020 at 6:33 PM Matthias Pohl wrote: > Thanks for restarting the vote, Yadong. I really like your UI proposals. > +1 for adding the changes of FLIP-104. > > Matthias > > On Tue, Oct 27, 2020 at 10:29 AM Xintong Song > wrote: > > > Thanks

[jira] [Created] (FLINK-19846) Grammar mistakes in annotations and log

2020-10-28 Thread zhouchao (Jira)
zhouchao created FLINK-19846: Summary: Grammar mistakes in annotations and log Key: FLINK-19846 URL: https://issues.apache.org/jira/browse/FLINK-19846 Project: Flink Issue Type: Wish Affects

Re: [DISCUSS] Remove flink-connector-filesystem module.

2020-10-28 Thread Kostas Kloudas
Hi all, I am bringing the up again to see if there are any users actively using the BucketingSink. So far, if I am not mistaken (and really sorry if I forgot anything), it is only a discussion between devs about the potential problems of removing it. I totally understand Chesnay's concern about

[jira] [Created] (FLINK-19851) flink sql client connector type jdbc exception

2020-10-28 Thread hulingchan (Jira)
hulingchan created FLINK-19851: -- Summary: flink sql client connector type jdbc exception Key: FLINK-19851 URL: https://issues.apache.org/jira/browse/FLINK-19851 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-19852) Managed memory released check can block IterativeTask

2020-10-28 Thread shaomeng.wang (Jira)
shaomeng.wang created FLINK-19852: - Summary: Managed memory released check can block IterativeTask Key: FLINK-19852 URL: https://issues.apache.org/jira/browse/FLINK-19852 Project: Flink

[jira] [Created] (FLINK-19854) TableScanTest.testTemporalJoinOnUpsertSource fails

2020-10-28 Thread Robert Metzger (Jira)
Robert Metzger created FLINK-19854: -- Summary: TableScanTest.testTemporalJoinOnUpsertSource fails Key: FLINK-19854 URL: https://issues.apache.org/jira/browse/FLINK-19854 Project: Flink Issue

Re: [DISCUSS] Remove flink-connector-filesystem module.

2020-10-28 Thread Kostas Kloudas
No, I do not think that "we are fine with removing it at the cost of friction for some users". I believe that this can be another discussion that we should have as soon as we establish that someone is actually using it. The point I am trying to make is that if no user is using it, we should

[jira] [Created] (FLINK-19856) Add EndOfChannelRecovery rescaling epoch

2020-10-28 Thread Roman Khachatryan (Jira)
Roman Khachatryan created FLINK-19856: - Summary: Add EndOfChannelRecovery rescaling epoch Key: FLINK-19856 URL: https://issues.apache.org/jira/browse/FLINK-19856 Project: Flink Issue

Re: [DISCUSS] Remove flink-connector-filesystem module.

2020-10-28 Thread Kostas Kloudas
I think that the mailing lists is the best we can do and I would say that they seem to be working pretty well (e.g. the recent Mesos discussion). Of course they are not perfect but the alternative would be to never remove anything user facing until the next major release, which I find pretty

[jira] [Created] (FLINK-19853) BulkSlotProviderImplTest.testFailedBulkSlotAllocationReleasesAllocatedSlot failed in azure

2020-10-28 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-19853: Summary: BulkSlotProviderImplTest.testFailedBulkSlotAllocationReleasesAllocatedSlot failed in azure Key: FLINK-19853 URL: https://issues.apache.org/jira/browse/FLINK-19853

[jira] [Created] (FLINK-19845) Migrate all FileSystemFormatFactory implementations

2020-10-28 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-19845: Summary: Migrate all FileSystemFormatFactory implementations Key: FLINK-19845 URL: https://issues.apache.org/jira/browse/FLINK-19845 Project: Flink Issue

[jira] [Created] (FLINK-19850) Add E2E tests for the new streaming case of the new FileSink

2020-10-28 Thread Yun Gao (Jira)
Yun Gao created FLINK-19850: --- Summary: Add E2E tests for the new streaming case of the new FileSink Key: FLINK-19850 URL: https://issues.apache.org/jira/browse/FLINK-19850 Project: Flink Issue

Re: [DISCUSS] Remove flink-connector-filesystem module.

2020-10-28 Thread Chesnay Schepler
The alternative could also be to use a different argument than "no one uses it", e.g., we are fine with removing it at the cost of friction for some users because there are better alternatives. On 10/28/2020 10:46 AM, Kostas Kloudas wrote: I think that the mailing lists is the best we can do

[jira] [Created] (FLINK-19855) Incompatible semantics of channelIndex in UnionInputGate.resumeConsumption and its clients

2020-10-28 Thread Roman Khachatryan (Jira)
Roman Khachatryan created FLINK-19855: - Summary: Incompatible semantics of channelIndex in UnionInputGate.resumeConsumption and its clients Key: FLINK-19855 URL:

Re: [DISCUSS] Remove flink-connector-filesystem module.

2020-10-28 Thread Chesnay Schepler
Then we can't remove it, because there is no way for us to ascertain whether anyone is still using it. Sure, the user ML is the best we got, but you can't argue that we don't want any users to be affected and then use an imperfect mean to find users. If you are fine with relying on the user

Re: [VOTE]FLIP-149: Introduce the upsert-kafka connector

2020-10-28 Thread Shengkai Fang
Hi all, The vote has been opened for more than 72 hours. We have already received 5 votes (4 binding) and no -1 votes. I am happy to announce the FLIP-149 is approved. * Jark (binding) * Konstantin (binding) * Jingsong (binding) * Timo (binding) * Danny (non-binding) Thanks everyone for

[jira] [Created] (FLINK-19858) Add the new table factory for upsert-kafka connector

2020-10-28 Thread Jark Wu (Jira)
Jark Wu created FLINK-19858: --- Summary: Add the new table factory for upsert-kafka connector Key: FLINK-19858 URL: https://issues.apache.org/jira/browse/FLINK-19858 Project: Flink Issue Type:

[jira] [Created] (FLINK-19860) Consider skipping restart and traverse regions which are already being restarted in RestartPipelinedRegionFailoverStrategy

2020-10-28 Thread Andrey Zagrebin (Jira)
Andrey Zagrebin created FLINK-19860: --- Summary: Consider skipping restart and traverse regions which are already being restarted in RestartPipelinedRegionFailoverStrategy Key: FLINK-19860 URL:

[jira] [Created] (FLINK-19857) FLIP-149: Introduce the upsert-kafka Connector

2020-10-28 Thread Jark Wu (Jira)
Jark Wu created FLINK-19857: --- Summary: FLIP-149: Introduce the upsert-kafka Connector Key: FLINK-19857 URL: https://issues.apache.org/jira/browse/FLINK-19857 Project: Flink Issue Type: New Feature

[jira] [Created] (FLINK-19859) Add documentation for the upsert-kafka connector

2020-10-28 Thread Jark Wu (Jira)
Jark Wu created FLINK-19859: --- Summary: Add documentation for the upsert-kafka connector Key: FLINK-19859 URL: https://issues.apache.org/jira/browse/FLINK-19859 Project: Flink Issue Type: Sub-task

Re: [VOTE] NEW FLIP-104: Add More Metrics to JobManager

2020-10-28 Thread Till Rohrmann
+1 Cheers, Till On Wed, Oct 28, 2020 at 9:07 AM Kurt Young wrote: > +1 > > Best, > Kurt > > > On Wed, Oct 28, 2020 at 2:44 PM Robert Metzger > wrote: > > > Thank you for your UI work Yadong! > > > > +1 > > > > On Tue, Oct 27, 2020 at 6:33 PM Matthias Pohl > > wrote: > > > > > Thanks for

Re: [DISCUSS] Remove flink-connector-filesystem module.

2020-10-28 Thread Chesnay Schepler
If the conclusion is that we shouldn't remove it if _anyone_ is using it, then we cannot remove it because the user ML obviously does not reach all users. On 10/28/2020 9:28 AM, Kostas Kloudas wrote: Hi all, I am bringing the up again to see if there are any users actively using the

Re: [DISCUSS] Releasing Apache Flink 1.11.3

2020-10-28 Thread Tzu-Li (Gordon) Tai
Thanks for the replies so far! Just to provide a brief update on the status of blockers for 1.11.3 so far: *PR opened, pending reviewer*- [FLINK-19717] SourceReaderBase.pollNext may return END_OF_INPUT if SplitReader.fetch throws ( https://github.com/apache/flink/pull/13776) *PR opened,

[jira] [Created] (FLINK-19848) flink docs of Building Flink from Source bug

2020-10-28 Thread jackylau (Jira)
jackylau created FLINK-19848: Summary: flink docs of Building Flink from Source bug Key: FLINK-19848 URL: https://issues.apache.org/jira/browse/FLINK-19848 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-19847) Can we create a fast support on the Nested table join?

2020-10-28 Thread xiaogang zhou (Jira)
xiaogang zhou created FLINK-19847: - Summary: Can we create a fast support on the Nested table join? Key: FLINK-19847 URL: https://issues.apache.org/jira/browse/FLINK-19847 Project: Flink

Re: [BULK]Re: [SURVEY] Remove Mesos support

2020-10-28 Thread Till Rohrmann
Hi Oleksandr, yes you are right. The biggest problem is at the moment the lack of test coverage and thereby confidence to make changes. We have some e2e tests which you can find here [1]. These tests are, however, quite coarse grained and are missing a lot of cases. One idea would be to add a

Re: [VOTE] NEW FLIP-104: Add More Metrics to JobManager

2020-10-28 Thread Kurt Young
+1 Best, Kurt On Wed, Oct 28, 2020 at 2:44 PM Robert Metzger wrote: > Thank you for your UI work Yadong! > > +1 > > On Tue, Oct 27, 2020 at 6:33 PM Matthias Pohl > wrote: > > > Thanks for restarting the vote, Yadong. I really like your UI proposals. > > +1 for adding the changes of FLIP-104.

[jira] [Created] (FLINK-19849) Check NOTICE files for 1.12 release

2020-10-28 Thread Robert Metzger (Jira)
Robert Metzger created FLINK-19849: -- Summary: Check NOTICE files for 1.12 release Key: FLINK-19849 URL: https://issues.apache.org/jira/browse/FLINK-19849 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-19861) Improve the document of now() function

2020-10-28 Thread hailong wang (Jira)
hailong wang created FLINK-19861: Summary: Improve the document of now() function Key: FLINK-19861 URL: https://issues.apache.org/jira/browse/FLINK-19861 Project: Flink Issue Type:

Re: [DISCUSS] Releasing Apache Flink 1.11.3

2020-10-28 Thread Stephan Ewen
Thanks for starting this. +1 form my side to backport the FLIP-27 adjustments to 1.11.x. There were quite a few changes, and I think we need to cherry-pick them all to not get some inconsistent mix of changes and many merge conflicts. I made a list below of what we need to add to "release-1.11".

Re: [DISCUSS] Releasing Apache Flink 1.11.3

2020-10-28 Thread Steven Wu
I would love to see this FLIP-27 source interface improvement [1] made to 1.11.3. [1] https://issues.apache.org/jira/browse/FLINK-19698 On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon) Tai wrote: > Thanks for the replies so far! > > Just to provide a brief update on the status of blockers for

[jira] [Created] (FLINK-19866) FunctionsStateBootstrapOperator.createStateAccessor fails due to uninitialized runtimeContext

2020-10-28 Thread wang (Jira)
wang created FLINK-19866: Summary: FunctionsStateBootstrapOperator.createStateAccessor fails due to uninitialized runtimeContext Key: FLINK-19866 URL: https://issues.apache.org/jira/browse/FLINK-19866

[jira] [Created] (FLINK-19863) SQLClientHBaseITCase.testHBase failed with "java.io.IOException: Process failed due to timeout"

2020-10-28 Thread Dian Fu (Jira)
Dian Fu created FLINK-19863: --- Summary: SQLClientHBaseITCase.testHBase failed with "java.io.IOException: Process failed due to timeout" Key: FLINK-19863 URL: https://issues.apache.org/jira/browse/FLINK-19863

[jira] [Created] (FLINK-19868) Csv Serialization schema contains line delimiter

2020-10-28 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-19868: Summary: Csv Serialization schema contains line delimiter Key: FLINK-19868 URL: https://issues.apache.org/jira/browse/FLINK-19868 Project: Flink Issue Type:

[jira] [Created] (FLINK-19864) TwoInputStreamTaskTest.testWatermarkMetrics failed with "expected:<1> but was:<-9223372036854775808>"

2020-10-28 Thread Dian Fu (Jira)
Dian Fu created FLINK-19864: --- Summary: TwoInputStreamTaskTest.testWatermarkMetrics failed with "expected:<1> but was:<-9223372036854775808>" Key: FLINK-19864 URL: https://issues.apache.org/jira/browse/FLINK-19864

Re: [DISCUSS] Releasing Apache Flink 1.11.3

2020-10-28 Thread Jingsong Li
+1 to backport the FLIP-27 adjustments to 1.11.x. If possible, that would be great. Many people are looking forward to the FLIP-27 interface, but they don't want to take the risk to upgrade to 1.12 (And wait 1.12). After all, 1.11 is a relatively stable version. Best, Jingsong On Thu, Oct 29,

[jira] [Created] (FLINK-19870) Fix special case when the reuse of exchange causes the deadlock

2020-10-28 Thread Caizhi Weng (Jira)
Caizhi Weng created FLINK-19870: --- Summary: Fix special case when the reuse of exchange causes the deadlock Key: FLINK-19870 URL: https://issues.apache.org/jira/browse/FLINK-19870 Project: Flink

[jira] [Created] (FLINK-19865) YARN tests failed with "java.lang.NumberFormatException: For input string: "${env:MAX_LOG_FILE_NUMBER}" java.lang.NumberFormatException: For input string: "${env:MAX_LOG

2020-10-28 Thread Dian Fu (Jira)
Dian Fu created FLINK-19865: --- Summary: YARN tests failed with "java.lang.NumberFormatException: For input string: "${env:MAX_LOG_FILE_NUMBER}" java.lang.NumberFormatException: For input string: "${env:MAX_LOG_FILE_NUMBER}"" Key:

Re: [DISCUSS] Remove flink-connector-filesystem module.

2020-10-28 Thread Jingsong Li
+1 to remove the Bucketing Sink. Thanks for the effort on ORC and `HadoopPathBasedBulkFormatBuilder`, I think it's safe to get rid of the old Bucketing API with them. Best, Jingsong On Thu, Oct 29, 2020 at 3:06 AM Kostas Kloudas wrote: > Thanks for the discussion! > > From this thread I do

Re: [DISCUSS] 1.12 feature freeze date += one week ?

2020-10-28 Thread Yuan Mei
Hey Stephan, Thanks for bringing this up! +1 to one week extension for 1.12 feature freeze. Things I am working on are under review and one week would definitely ease my (and reviewers') life a lot. Best, Yuan On Thu, Oct 29, 2020 at 2:13 AM Stephan Ewen wrote: > Hi all! > > We are

[jira] [Created] (FLINK-19867) Validation fails for UDF that accepts var-args

2020-10-28 Thread Rui Li (Jira)
Rui Li created FLINK-19867: -- Summary: Validation fails for UDF that accepts var-args Key: FLINK-19867 URL: https://issues.apache.org/jira/browse/FLINK-19867 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-19869) Support java.util.UUID as RAW type in PostgresCatalog and handle it in PostgresRowConverter

2020-10-28 Thread Jingwei Zhang (Jira)
Jingwei Zhang created FLINK-19869: - Summary: Support java.util.UUID as RAW type in PostgresCatalog and handle it in PostgresRowConverter Key: FLINK-19869 URL: https://issues.apache.org/jira/browse/FLINK-19869

[jira] [Created] (FLINK-19862) DeclarativeSlotManager#suspend can fail with NPE

2020-10-28 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-19862: Summary: DeclarativeSlotManager#suspend can fail with NPE Key: FLINK-19862 URL: https://issues.apache.org/jira/browse/FLINK-19862 Project: Flink

Re: [DISCUSS] Remove flink-connector-filesystem module.

2020-10-28 Thread Kostas Kloudas
Thanks for the discussion! >From this thread I do not see any objection with moving forward with removing the sink. Given this I will open a voting thread tomorrow. Cheers, Kostas On Wed, Oct 28, 2020 at 6:50 PM Stephan Ewen wrote: > > +1 to remove the Bucketing Sink. > > It has been very

Is there any way to change operator level parallelism dynamically?

2020-10-28 Thread DEEP NARAYAN Singh
Hi Team, I just want quick help here. How to achieve the dynamic nature of operator level parallelism for the flink job running in AWS EMR cluster during runtime to avoid downtime and backpressure based on the incoming load. As I am very new to flink and currently using flink 1.9 version.Is there

[DISCUSS] 1.12 feature freeze date += one week ?

2020-10-28 Thread Stephan Ewen
Hi all! We are approaching the feature freeze date for the 1.12 release that was discussed earlier. >From my side and the developments I am involved with, we are close and in good shape, but could really use one more week to round things off. It would help both the code quality and our mental

Re: [DISCUSS] Remove flink-connector-filesystem module.

2020-10-28 Thread Stephan Ewen
+1 to remove the Bucketing Sink. It has been very common in the past to remove code that was deprecated for multiple releases in favor of reducing baggage. Also in cases that had no perfect drop-in replacement, but needed users to forward fit the code. I am not sure I understand why this case is