Re: [VOTE] FLIP-199: Change some default config values of blocking shuffle for better usability

+1 Thanks Yingjie for driving. Best, Jingsong Lee On Wed, Jan 12, 2022 at 3:16 PM 刘建刚 wrote: > > +1 for the proposal. In fact, we have used these params in our inner flink > version for good performance. > > Yun Gao 于2022年1月12日周三 10:42写道： > > > +1 since it would highly improve the open-box

[jira] [Created] (FLINK-25625) Introduce FileFormat for table-store

Jingsong Lee created FLINK-25625: Summary: Introduce FileFormat for table-store Key: FLINK-25625 URL: https://issues.apache.org/jira/browse/FLINK-25625 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-25624) KafkaSinkITCase.testRecoveryWithExactlyOnceGuarantee blocked on azure pipeline

Yun Gao created FLINK-25624: --- Summary: KafkaSinkITCase.testRecoveryWithExactlyOnceGuarantee blocked on azure pipeline Key: FLINK-25624 URL: https://issues.apache.org/jira/browse/FLINK-25624 Project: Flink

Re: [DISCUSS] FLIP-200: Support Multiple Rule and Dynamic Rule Changing (Flink CEP)

Hi folks, Till and I had an offline discussion about this FLIP. And it looks that currently OC is still coupled with the Source operator in some cases. So it is not really ready for the CEP use case at this point. In order to provide the CEP dynamic pattern update feature to the users, at the

[jira] [Created] (FLINK-25623) TPC-DS end-to-end test (Blink planner) failed on azure due to download file tpcds.idx failed.

Yun Gao created FLINK-25623: --- Summary: TPC-DS end-to-end test (Blink planner) failed on azure due to download file tpcds.idx failed. Key: FLINK-25623 URL: https://issues.apache.org/jira/browse/FLINK-25623

[jira] [Created] (FLINK-25622) Throws NPE in Python UDTF

Huang Xingbo created FLINK-25622: Summary: Throws NPE in Python UDTF Key: FLINK-25622 URL: https://issues.apache.org/jira/browse/FLINK-25622 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-25621) LegacyStatefulJobSavepointMigrationITCase failed on azure with exit code 127

Yun Gao created FLINK-25621: --- Summary: LegacyStatefulJobSavepointMigrationITCase failed on azure with exit code 127 Key: FLINK-25621 URL: https://issues.apache.org/jira/browse/FLINK-25621 Project: Flink

Re: [DISCUSS] Seek help for making JIRA links clickable in github

Currently it seems the issues of flink-statefun and flink-ml are also managed in the issues.apache.org ? Best, Yun --Original Mail -- Sender:Jingsong Li Send Date:Wed Jan 12 13:54:03 2022 Recipients:dev Subject:[DISCUSS] Seek help for making JIRA links

Re: [VOTE] FLIP-199: Change some default config values of blocking shuffle for better usability

+1 for the proposal. In fact, we have used these params in our inner flink version for good performance. Yun Gao 于2022年1月12日周三 10:42写道： > +1 since it would highly improve the open-box experience for batch jobs. > > Thanks Yingjie for drafting the PR and initiating the discussion. > > Best, >

[jira] [Created] (FLINK-25620) Upload artifacts to S3 failed on azure pipeline

Yun Gao created FLINK-25620: --- Summary: Upload artifacts to S3 failed on azure pipeline Key: FLINK-25620 URL: https://issues.apache.org/jira/browse/FLINK-25620 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-25619) Init flink-table-store repository

Jingsong Lee created FLINK-25619: Summary: Init flink-table-store repository Key: FLINK-25619 URL: https://issues.apache.org/jira/browse/FLINK-25619 Project: Flink Issue Type: Sub-task

[DISCUSS] Seek help for making JIRA links clickable in github

Hi everyone, We are creating flink-table-store[1] and we also find that flink-ml[2] does not have clickable JIRA links, while flink-statefun[3] and flink[4] do. So I'm asking for PMC's help on how to make JIRA links clickable in github. [1] https://github.com/apache/flink-table-store [2]

[jira] [Created] (FLINK-25618) Data quality by apache flink

tanjialiang created FLINK-25618: --- Summary: Data quality by apache flink Key: FLINK-25618 URL: https://issues.apache.org/jira/browse/FLINK-25618 Project: Flink Issue Type: New Feature

Re: [VOTE] FLIP-199: Change some default config values of blocking shuffle for better usability

+1 since it would highly improve the open-box experience for batch jobs. Thanks Yingjie for drafting the PR and initiating the discussion. Best, Yun --Original Mail -- Sender:Yingjie Cao Send Date:Tue Jan 11 15:15:01 2022 Recipients:dev Subject:[VOTE]

[jira] [Created] (FLINK-25617) Support VectorAssembler in FlinkML

weibo zhao created FLINK-25617: -- Summary: Support VectorAssembler in FlinkML Key: FLINK-25617 URL: https://issues.apache.org/jira/browse/FLINK-25617 Project: Flink Issue Type: New Feature

[jira] [Created] (FLINK-25616) Support VectorAssembler in FlinkML

weibo zhao created FLINK-25616: -- Summary: Support VectorAssembler in FlinkML Key: FLINK-25616 URL: https://issues.apache.org/jira/browse/FLINK-25616 Project: Flink Issue Type: New Feature

[RESULT][VOTE] Create a separate sub project for FLIP-188

I am happy to announce that creating a separate sub project for FLIP-188[1][2][3]. There are 12 approving votes, 10 of which are binding, 7 of which are voting for flink-table-store: * Timo Walther (binding) * Till Rohrmann (binding) * Konstantin Knauf(binding) * David Moravek (binding) * Yun

Re: [DISCUSS] FLIP-208: Update KafkaSource to detect EOF based on de-serialized record

Hi Fabian, Thanks for the comments. Please see my reply inline. On Tue, Jan 11, 2022 at 11:46 PM Fabian Paul wrote: > Hi Dong, > > I wouldn't change the org.apache.flink.api.connector.source.Source > interface because it either breaks existing sinks or we introduce it > as some kind of

Re: [DISCUSS] FLIP-208: Update KafkaSource to detect EOF based on de-serialized record

Hi Martijn, Thank you for the comments. Please find my reply inline. On Wed, Jan 12, 2022 at 3:07 AM Martijn Visser wrote: > Hi Dong, > > Thanks for updating the FLIP and including Pulsar. I was indeed referring > that we should have a generic interface that allows connector maintainers > to

Re: [DISCUSS] FLIP-211: Kerberos delegation token framework

Hi G, Thanks for taking this challenge on. Scalable Kerberos authentication support is important for Flink, delegation tokens is a great mechanism to future-proof this. I second your assessment that the existing implementation could use some improvement too and like the approach you have

Re: [VOTE] Create a separate sub project for FLIP-188: flink-store

+1 (non-binding) for `flink-table-store` On Mon, Jan 10, 2022 at 11:20 PM Jingsong Li wrote: > Thanks everyone for your voting. > > If there are no objections, I'll close this vote and send a vote result > mail: > - create a sub project named `flink-table-store`. > > Best, > Jingsong > > On

Use of JIRA fixVersion

Hi, As part of preparing the 1.14.3 release, I observed that there were around 200 JIRA issues with fixVersion 1.14.3 that were unresolved (after blocking issues had been dealt with). Further cleanup resulted in removing fixVersion 1.14.3 from most of these and we are left with [1] - these are

Re: [DISCUSS] Creating an external connector repository

Good question: we want to use the same setup as we currently have for Flink, so using the existing CI infrastructure. On Mon, 10 Jan 2022 at 11:19, Chesnay Schepler wrote: > What CI resources do you actually intend use? Asking since the ASF GHA > resources are afaik quite overloaded. > > On

Re: [DISCUSS] FLIP-208: Update KafkaSource to detect EOF based on de-serialized record

Hi Dong, Thanks for updating the FLIP and including Pulsar. I was indeed referring that we should have a generic interface that allows connector maintainers to implement this capability if they think it should be supported. Could you see a feature like this also be useful for a connector like

Re: [DISCUSS] FLIP-206: Support PyFlink Runtime Execution in Thread Mode

Hi Xingbo, +1 from my side Thanks for the clarification. For your use case the parameter size and therefore serialization overhead was the limiting factor. I have seen use cases where that is not the concern, because the Python logic itself is heavy and dwarfs the protocol overhead (for example

Re: Need help with finding inner workings of watermark stream idleness

Hi Jeff, I think this happens in the WatermarksWithIdleness [1]. [1] https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/eventtime/WatermarksWithIdleness.java#L73 Cheers, Till On Tue, Jan 11, 2022 at 6:05 PM Jeff Carter wrote: > I'm looking into

Need help with finding inner workings of watermark stream idleness

I'm looking into making a feature for flink related to watermarks and am digging into the inner watermark mechanisms, specifically with idleness. I'm familiar with idleness, but digging into the root code I can only get to where idlenessTimeout gets set in WatermarkStrategyWithIdleness.java. But

[VOTE] Release 1.14.3, release candidate #1

Hi everyone, Please review and vote on the release candidate #1 for the version 1.14.3, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1],

[jira] [Created] (FLINK-25615) FlinkKafkaProducer fail to correctly migrate pre Flink 1.9 state

Matthias Schwalbe created FLINK-25615: - Summary: FlinkKafkaProducer fail to correctly migrate pre Flink 1.9 state Key: FLINK-25615 URL: https://issues.apache.org/jira/browse/FLINK-25615 Project:

Re: [DISCUSS] FLIP-208: Update KafkaSource to detect EOF based on de-serialized record

Hi Dong, I wouldn't change the org.apache.flink.api.connector.source.Source interface because it either breaks existing sinks or we introduce it as some kind of optional. I deem both options as not great. My idea is to introduce a new interface that extends the Source. This way users who want to

Re: [DISCUSS] Releasing Flink 1.14.3

Hi Thomas, Thanks! I'll prepare the website PR and send out the VOTE in a couple of hours. Best regards, Martijn On Tue, 11 Jan 2022 at 05:53, Thomas Weise wrote: > Thank you Xingbo. I meanwhile also got my Azure pipeline working and > was able to build the artifacts. Although in general it

[jira] [Created] (FLINK-25614) Let LocalWindowAggregate be chained with upstream

Q Kang created FLINK-25614: -- Summary: Let LocalWindowAggregate be chained with upstream Key: FLINK-25614 URL: https://issues.apache.org/jira/browse/FLINK-25614 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-210: Change logging level dynamically at runtime

Hi all, Yes, indeed. After I did some investigation on similar features provided by the Cloud platforms, I actually found several popular Clouds have already offered this. - AWS Kinesis: Setting the Application Logging Level [1], which is implemented by UpdateApplication API [2]. - Ververica:

[DISCUSS] FLIP-211: Kerberos delegation token framework

Hi All, Hope all of you have enjoyed the holiday season. I would like to start the discussion on FLIP-211 which aims to provide a Kerberos delegation token framework that

Could not find any factory for identifier 'jdbc'

Correcting subject -> Could not find any factory for identifier 'jdbc' From: Ronak Beejawat (rbeejawa) Sent: Tuesday, January 11, 2022 6:43 PM To: 'dev@flink.apache.org' ; 'commun...@flink.apache.org' ; 'u...@flink.apache.org' Cc: 'Hang Ruan' ; Shrinath Shenoy K (sshenoyk) ; Karthikeyan

what is efficient way to write Left join in flink

Hi Team, Getting below exception while using jdbc connector : Caused by: org.apache.flink.table.api.ValidationException: Could not find any factory for identifier 'jdbc' that implements 'org.apache.flink.table.factories.DynamicTableFactory' in the classpath. Available factory identifiers are:

[jira] [Created] (FLINK-25613) Remove excessive surefire-plugin versions

Chesnay Schepler created FLINK-25613: Summary: Remove excessive surefire-plugin versions Key: FLINK-25613 URL: https://issues.apache.org/jira/browse/FLINK-25613 Project: Flink Issue

RE: what is efficient way to write Left join in flink

Can please someone help / reply on below Question ? From: Ronak Beejawat (rbeejawa) Sent: Monday, January 10, 2022 7:40 PM To: dev@flink.apache.org; commun...@flink.apache.org; u...@flink.apache.org Cc: Hang Ruan ; Shrinath Shenoy K (sshenoyk) Subject: what is efficient way to write Left join

Re: [DISCUSS] Releasing Flink 1.14.3

The assumption is that everyone that works enough on Flink to volunteer as a RM already has a working azure setup, in which case there isn't any additional setup overhead. We may improve this in the future though. On 11/01/2022 05:53, Thomas Weise wrote: Thank you Xingbo. I meanwhile also

Re: [DISCUSS] FLIP-210: Change logging level dynamically at runtime

Hi all, I agree with Konstantin, this feels like a problem that shouldn't be solved via Apache Flink but via the logging ecosystem itself. Best regards, Martijn On Tue, 11 Jan 2022 at 13:11, Konstantin Knauf wrote: > I've now read over the discussion on the ticket, and I am personally not in

Re: [VOTE][CANCELED] Release flink-shaded 15.0, release candidate #1

FLINK-25588 has been put onto the agenda for the next flink-shaded release, so we might as well cancel this vote to cover everything at once. On 14/12/2021 09:57, Chesnay Schepler wrote: Hi everyone, Please review and vote on the release candidate #1 for the version 15.0, as follows: [ ] +1,

Re: [DISCUSS] FLIP-210: Change logging level dynamically at runtime

I've now read over the discussion on the ticket, and I am personally not in favor of adding this functionality to Flink via the REST API or Web UI. I believe that changing the logging configuration via the existing configuration files (log4j or logback) is good enough, to justify not increasing

[jira] [Created] (FLINK-25612) Update the outdated illustration of ExecutionState in the documentation

Zhilong Hong created FLINK-25612: Summary: Update the outdated illustration of ExecutionState in the documentation Key: FLINK-25612 URL: https://issues.apache.org/jira/browse/FLINK-25612 Project:

[jira] [Created] (FLINK-25611) Remove CoordinatorExecutorThreadFactory thread creation guards

Chesnay Schepler created FLINK-25611: Summary: Remove CoordinatorExecutorThreadFactory thread creation guards Key: FLINK-25611 URL: https://issues.apache.org/jira/browse/FLINK-25611 Project:

Re: [DISCUSS] FLIP-206: Support PyFlink Runtime Execution in Thread Mode

Hi everyone, Thanks to all of you for the discussion. If there are no objections, I would like to start a vote thread tomorrow. Best, Xingbo Xingbo Huang 于2022年1月7日周五 16:18写道： > Hi Till, > > I have written a more complicated PyFlink job. Compared with the previous > single python udf job,

[jira] [Created] (FLINK-25610) [FLIP-171] Kinesis Firehose implementation of Async Sink Table API

Ahmed Hamdy created FLINK-25610: --- Summary: [FLIP-171] Kinesis Firehose implementation of Async Sink Table API Key: FLINK-25610 URL: https://issues.apache.org/jira/browse/FLINK-25610 Project: Flink

[jira] [Created] (FLINK-25609) Avoid creating temporary tables for inline tables

Timo Walther created FLINK-25609: Summary: Avoid creating temporary tables for inline tables Key: FLINK-25609 URL: https://issues.apache.org/jira/browse/FLINK-25609 Project: Flink Issue

what is efficient way to write Left join in flink

Hi Team, We want a clarification on one real time processing scenario for below mentioned use case. Use case : 1. We have topic one (testtopic1) which will get half a million data every minute. 2. We have topic two (testtopic2) which will get one million data every minute. So we are doing

Re: [DISCUSS] Deprecate MapR FS

+1 for dropping the MapR FS. Thanks for driving. Best, Jingsong On Tue, Jan 11, 2022 at 5:22 PM Chang Li wrote: > > +1 for dropping the MapR FS. > > Till Rohrmann 于2022年1月5日周三 18:33写道： > > > +1 for dropping the MapR FS. > > > > Cheers, > > Till > > > > On Wed, Jan 5, 2022 at 10:11 AM Martijn

Re: [DISCUSS] FLIP-210: Change logging level dynamically at runtime

Reloading the config from the filesystem  is already enabled by default; that was one of the things that made us switch to Log4j 2. The core point of contention w.r.t. this topic is whether having the admin ssh into the machine is too inconvenient. Personally I still think that the the

[jira] [Created] (FLINK-25608) Mark metrics as Public(Evolving)

Fabian Paul created FLINK-25608: --- Summary: Mark metrics as Public(Evolving) Key: FLINK-25608 URL: https://issues.apache.org/jira/browse/FLINK-25608 Project: Flink Issue Type: Improvement

Re: [DISCUSS] Deprecate MapR FS

+1 for dropping the MapR FS. Till Rohrmann 于2022年1月5日周三 18:33写道： > +1 for dropping the MapR FS. > > Cheers, > Till > > On Wed, Jan 5, 2022 at 10:11 AM Martijn Visser > wrote: > > > Hi everyone, > > > > Thanks for your input. I've checked the MapR implementation and it has no > > annotation at

[jira] [Created] (FLINK-25607) Sorting by duration on Flink Web UI does not work correctly

Yingjie Cao created FLINK-25607: --- Summary: Sorting by duration on Flink Web UI does not work correctly Key: FLINK-25607 URL: https://issues.apache.org/jira/browse/FLINK-25607 Project: Flink

Re: [DISCUSS] FLIP-203: Incremental savepoints

Hi Piotr, would it be possible to provide a table that shows the compatibility guarantees provided by the different snapshots going forward? Like type of change (Topology. State Schema, Parallelism, ..) in one dimension, and type of snapshot as the other dimension. Based on that, it would be

[jira] [Created] (FLINK-25606) Requesting exclusive buffers timeout when recovering from unaligned checkpoint under fine-grained resource mode

Yingjie Cao created FLINK-25606: --- Summary: Requesting exclusive buffers timeout when recovering from unaligned checkpoint under fine-grained resource mode Key: FLINK-25606 URL:

Re: [DISCUSS] FLIP-210: Change logging level dynamically at runtime

Thank you for proposing this improvement, Wenhao. Changing the logging level dynamically at runtime is very useful when users are trying to debug their jobs. They can set the logging level to DEBUG and find out more details in the logs. 1. I'm wondering if we could add a REST API to query the

[jira] [Created] (FLINK-25605) Batch get statistics of multiple partitions instead of get one by one

Jing Zhang created FLINK-25605: -- Summary: Batch get statistics of multiple partitions instead of get one by one Key: FLINK-25605 URL: https://issues.apache.org/jira/browse/FLINK-25605 Project: Flink