Re: [DISCUSS] FLIP-221 Abstraction for lookup source cache and metric

Hi Qingsheng, Thank you for considering my comments. > there might be custom logic before making retry, such as re-establish the > connection Yes, I understand that. I meant that such logic can be placed in a separate function, that can be implemented by connectors. Just moving the retry

Re: [VOTE] FLIP-229: Introduces Join Hint for Flink SQL Batch Job

Thanks Xuyang for driving this, +1(binding) Best, Godfrey Xuyang 于2022年5月17日周二 10:21写道： > > Hi, everyone. > Thanks for your feedback for FLIP-229: Introduces Join Hint for Flink SQL > Batch Job[1] on the discussion thread[2]. > I'd like to start a vote for it. The vote will be open for at

[jira] [Created] (FLINK-27675) Improve manual savepoint tracking

Gyula Fora created FLINK-27675: -- Summary: Improve manual savepoint tracking Key: FLINK-27675 URL: https://issues.apache.org/jira/browse/FLINK-27675 Project: Flink Issue Type: Improvement

Re: Re: [DISCUSS] FLIP-218: Support SELECT clause in CREATE TABLE(CTAS)

Hi Mang, Thanks for driving this FLIP. Please follow the FLIP template[1] style, and the `Syntax ` is part of the `Public API Changes` section. ‘Program research’ and 'Implementation Plan' are part of the `Proposed Changes` section, or move ‘Program research’ to the appendix. > Providing

Re:Metrics in Flink UI

Hi Zain Do you have only one operator or all of the operators are chained together? Maybe you can break the operator chain and check it out. Best JasonLee Replied Message | From | Zain Haider Nemati | | Date | 05/18/2022 03:28 | | To | user , | | Subject | Metrics in Flink UI |

[jira] [Created] (FLINK-27674) Elasticsearch6SinkE2ECase test hangs

Huang Xingbo created FLINK-27674: Summary: Elasticsearch6SinkE2ECase test hangs Key: FLINK-27674 URL: https://issues.apache.org/jira/browse/FLINK-27674 Project: Flink Issue Type: Bug

Re: [DISCUSS] FLIP-224: Blacklist Mechanism

Hi Weihua, thanks for feedback. 1. Yes, only *Manually* is supported in this FLIP, but it's the first step towards auto-detection. 2. We wii print the blocked nodes in logs. Maybe also put it into the exception of insufficient resources. 3. No. This FLIP won't change the WebUI. The blocklist

Memory configuration for Queue

Hi, I am using a kafka source with a kinesis sink and the speed of data coming in is not the same as data flowing out hence the need to configure a relatively larger queue to hold the data before backpressuring. Which memory configuration corresponds to this that I'll need to configure?

Metrics in Flink UI

Hi, I'm running a job on a local flink cluster but metrics are showing as Bytes received,records received,bytes sent,backpressure all 0 in the flink UI even though I'm receiving data in the sink. Do I need to additionally configure something to see these metrics work in real time?

[jira] [Created] (FLINK-27673) [JUnit5 Migration] Module: flink-table-api-scala

Sergey Nuyanzin created FLINK-27673: --- Summary: [JUnit5 Migration] Module: flink-table-api-scala Key: FLINK-27673 URL: https://issues.apache.org/jira/browse/FLINK-27673 Project: Flink Issue

Data Engineering Track at ApacheCon (October 3-6, New Orleans) - CFP ends 23/05

Hello, ApacheCon North America is back in person this year in October. https://apachecon.com/acna2022/ Together with Jarek Potiuk, we are organizing for the first time a Data Engineering Track as part of ApacheCon. You might be wondering why a different track if we already have the Big Data

[jira] [Created] (FLINK-27672) [JUnit5 Migration] Module: flink-table-common

Sergey Nuyanzin created FLINK-27672: --- Summary: [JUnit5 Migration] Module: flink-table-common Key: FLINK-27672 URL: https://issues.apache.org/jira/browse/FLINK-27672 Project: Flink Issue

Re: [VOTE] Creating an Apache Flink slack workspace

+1 (binding) On Tue, 17 May 2022 at 19:52, Yufei Zhang wrote: > +1 (nonbinding) > > On Tue, May 17, 2022 at 5:29 PM Márton Balassi > wrote: > > > +1 (binding) > > > > On Tue, May 17, 2022 at 11:00 AM Jingsong Li > > wrote: > > > > > Thank Xintong for driving this work. > > > > > > +1 > > > >

Re: [VOTE] Creating an Apache Flink slack workspace

+1 (nonbinding) On Tue, May 17, 2022 at 5:29 PM Márton Balassi wrote: > +1 (binding) > > On Tue, May 17, 2022 at 11:00 AM Jingsong Li > wrote: > > > Thank Xintong for driving this work. > > > > +1 > > > > Best, > > Jingsong > > > > On Tue, May 17, 2022 at 4:49 PM Martijn Visser > > > wrote: >

Re: [ANNOUNCE] Kubernetes Operator release-1.0 branch cut

Thanks Gyula and Yang. Awesome! On Tue, May 17, 2022 at 4:46 PM Gyula Fóra wrote: > Hi Flink devs! > > The release-1.0 branch has been forked from main and version numbers have > been upgraded accordingly. > > https://github.com/apache/flink-kubernetes-operator/tree/release-1.0 > > The version

[jira] [Created] (FLINK-27671) Publish SNAPSHOT docker images

Alexander Fedulov created FLINK-27671: - Summary: Publish SNAPSHOT docker images Key: FLINK-27671 URL: https://issues.apache.org/jira/browse/FLINK-27671 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-233: Introduce HTTP Connector

Hey Jeremy, Thanks for kicking off this discussion. As a Flink user, I too struggled with the lack of HTTP support and rolled my own with AsyncIO. Reading through the FLIP, I just have a few general questions and comments. * It is not clear to me if multiple HTTP methods are supported or not?

[ANNOUNCE] Kubernetes Operator release-1.0 branch cut

Hi Flink devs! The release-1.0 branch has been forked from main and version numbers have been upgraded accordingly. https://github.com/apache/flink-kubernetes-operator/tree/release-1.0 The version on the main branch has been updated to 1.1-SNAPSHOT. >From now on, for PRs that should be

[jira] [Created] (FLINK-27670) Python wrappers for Kinesis Sinks

Zichen Liu created FLINK-27670: -- Summary: Python wrappers for Kinesis Sinks Key: FLINK-27670 URL: https://issues.apache.org/jira/browse/FLINK-27670 Project: Flink Issue Type: Improvement

[DISCUSS] FLIP-233: Introduce HTTP Connector

Hi there, We would like to start a discussion thread on FLIP-233: Introduce HTTP Connector where we propose to create a connector for delivering arbitrary data packets from Apache Flink to an HTTP Endpoint.

[jira] [Created] (FLINK-27669) [JUnit5 Migration] Migrate flink-file-sink-common

Sergey Nuyanzin created FLINK-27669: --- Summary: [JUnit5 Migration] Migrate flink-file-sink-common Key: FLINK-27669 URL: https://issues.apache.org/jira/browse/FLINK-27669 Project: Flink

Re: [DISCUSS] FLIP-224: Blacklist Mechanism

Hi, Thanks for creating this FLIP. We have implemented an automatic blocklist detection mechanism internally, which is indeed very effective for handling node failures. Due to the large number of nodes, although SREs already support automatic offline failure nodes, the detection is not 100%

[jira] [Created] (FLINK-27668) Document dynamic operator configuration

Gyula Fora created FLINK-27668: -- Summary: Document dynamic operator configuration Key: FLINK-27668 URL: https://issues.apache.org/jira/browse/FLINK-27668 Project: Flink Issue Type: Improvement

Re: [DISCUSS] FLIP-221 Abstraction for lookup source cache and metric

Thank Qingsheng for updating the design. I just have a minor comment. Changing RescanRuntimeProvider into the builder pattern seems not necessary. The builder pattern is usually used when there are several optional parameters. However, ScanRuntimeProvider and rescan interval are both required

Re: Channel became inactive while submitting job

Hi, Which version of Flink are you using? And what is the start cmd? Best, Weihua > 2022年5月17日 下午6:33，Zain Haider Nemati 写道： > > main method caused an error: Failed to execute job 'Tracer Processor'. > at >

Re: [DISCUSS] Release first version of Elasticsearch connector from external repository

Thanks Chesnay for the notes. The 1.16-SNAPSHOT dependency is only a temporary solution. Once all related commits w.r.t. architecture tests have been cp from the master to 1.15, the dependency will be changed to 1.15. It is on the todo list. On Tue, May 17, 2022 at 10:04 AM Chesnay Schepler

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

Hi Paul, Thanks for driving this, LGTM overall. I have a few minor comments: >SHOW QUERIES I want to clear the scope the command, does the command show the queries submitted via SqlClient, or all queries in current cluster (submitted via other CLI)? History queries are included? What's the

Re: [DISCUSS] FLIP-221 Abstraction for lookup source cache and metric

Hi devs, I just updated FLIP-221 [1] according to discussions addressed above. This version made minor changes to make interfaces clearer: 1. The argument type of LookupFunction#lookup and AsyncLookupFunction#asyncLookup are changed from Object... to RowData, in order to be symmetric with

Re:Re: [DISCUSS] FLIP-221 Abstraction for lookup source cache and metric

Hi Qingsheng, Alexander, Thanks for your reply. > Can you give an example of such upper - level Cache usage? It's not clear for > me currently. I think it's unnecessary to have such high level abstraction, > if nowhere in the code we won't operate with objects as instances of Cache. > But

[jira] [Created] (FLINK-27667) YARNHighAvailabilityITCase fails with "Failed to delete temp directory /tmp/junit1681"

Martijn Visser created FLINK-27667: -- Summary: YARNHighAvailabilityITCase fails with "Failed to delete temp directory /tmp/junit1681" Key: FLINK-27667 URL: https://issues.apache.org/jira/browse/FLINK-27667

Channel became inactive while submitting job

Hi, I am trying to run a job in my local cluster and facing this issue. org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Failed to execute job 'Tracer Processor'. at

[jira] [Created] (FLINK-27666) Cover manual savepoint triggering in E2E tests

Gyula Fora created FLINK-27666: -- Summary: Cover manual savepoint triggering in E2E tests Key: FLINK-27666 URL: https://issues.apache.org/jira/browse/FLINK-27666 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-221 Abstraction for lookup source cache and metric

Hi Yuan, Thanks for the review! Basically I’m with Alexander opinion. We’d like to limit the scope in lookup scenario so we didn’t extend the cache to a generic one. And as for the metric I think the existing metric definitions are also applicable for all-cache case. Cheers, Qingsheng >

[jira] [Created] (FLINK-27665) Optimise event triggering on DeploymentFailedExceptions

Matyas Orhidi created FLINK-27665: - Summary: Optimise event triggering on DeploymentFailedExceptions Key: FLINK-27665 URL: https://issues.apache.org/jira/browse/FLINK-27665 Project: Flink

Re: [DISCUSS] FLIP-221 Abstraction for lookup source cache and metric

Hi Alexander, Thanks for the review and glad to see we are on the same page! I think you forgot to cc the dev mailing list so I’m also quoting your reply under this email. > We can add 'maxRetryTimes' option into this class In my opinion the retry logic should be implemented in lookup()

[jira] [Created] (FLINK-27664) cron_snapshot_deployment_maven fails due to JavaDoc building error

Martijn Visser created FLINK-27664: -- Summary: cron_snapshot_deployment_maven fails due to JavaDoc building error Key: FLINK-27664 URL: https://issues.apache.org/jira/browse/FLINK-27664 Project:

Re: [VOTE] Creating an Apache Flink slack workspace

+1 (binding) On Tue, May 17, 2022 at 11:00 AM Jingsong Li wrote: > Thank Xintong for driving this work. > > +1 > > Best, > Jingsong > > On Tue, May 17, 2022 at 4:49 PM Martijn Visser > wrote: > > > +1 (binding) > > > > On Tue, 17 May 2022 at 10:38, Yu Li wrote: > > > > > +1 (binding) > > > >

Re: [VOTE] Creating an Apache Flink slack workspace

+1 (binding) And thanks Xintong for driving this work. Best, Yang Jingsong Li 于2022年5月17日周二 17:00写道： > Thank Xintong for driving this work. > > +1 > > Best, > Jingsong > > On Tue, May 17, 2022 at 4:49 PM Martijn Visser > wrote: > > > +1 (binding) > > > > On Tue, 17 May 2022 at 10:38, Yu Li

[jira] [Created] (FLINK-27663) upsert-kafka can't process delete message from upsert-kafka sink

Zhiwen Sun created FLINK-27663: -- Summary: upsert-kafka can't process delete message from upsert-kafka sink Key: FLINK-27663 URL: https://issues.apache.org/jira/browse/FLINK-27663 Project: Flink

Re: [VOTE] Creating an Apache Flink slack workspace

Thank Xintong for driving this work. +1 Best, Jingsong On Tue, May 17, 2022 at 4:49 PM Martijn Visser wrote: > +1 (binding) > > On Tue, 17 May 2022 at 10:38, Yu Li wrote: > > > +1 (binding) > > > > Thanks Xintong for driving this! > > > > Best Regards, > > Yu > > > > > > On Tue, 17 May 2022

Re: [VOTE] Creating an Apache Flink slack workspace

+1 (binding) On Tue, 17 May 2022 at 10:38, Yu Li wrote: > +1 (binding) > > Thanks Xintong for driving this! > > Best Regards, > Yu > > > On Tue, 17 May 2022 at 16:32, Robert Metzger wrote: > > > Thanks for starting the VOTE! > > > > +1 (binding) > > > > > > > > On Tue, May 17, 2022 at 10:29 AM

Re: [VOTE] Creating an Apache Flink slack workspace

+1 (binding) Thanks Xintong for driving this! Best Regards, Yu On Tue, 17 May 2022 at 16:32, Robert Metzger wrote: > Thanks for starting the VOTE! > > +1 (binding) > > > > On Tue, May 17, 2022 at 10:29 AM Jark Wu wrote: > > > Thank Xintong for driving this work. > > > > +1 from my side

Re: [VOTE] Creating an Apache Flink slack workspace

Thanks for starting the VOTE! +1 (binding) On Tue, May 17, 2022 at 10:29 AM Jark Wu wrote: > Thank Xintong for driving this work. > > +1 from my side (binding) > > Best, > Jark > > On Tue, 17 May 2022 at 16:24, Xintong Song wrote: > > > Hi everyone, > > > > As previously discussed in [1], I

Re: [VOTE] Creating an Apache Flink slack workspace

Sorry, I forgot to attach the link to the discussion thread. [1] Best, Xintong [1] https://lists.apache.org/thread/n43r4qmwprhdmzrj494dbbwr9w7bbdcv On Tue, May 17, 2022 at 4:28 PM Jark Wu wrote: > Thank Xintong for driving this work. > > +1 from my side (binding) > > Best, > Jark > > On

Re: [VOTE] Creating an Apache Flink slack workspace

Thank Xintong for driving this work. +1 from my side (binding) Best, Jark On Tue, 17 May 2022 at 16:24, Xintong Song wrote: > Hi everyone, > > As previously discussed in [1], I would like to open a vote on creating an > Apache Flink slack workspace channel. > > The proposed actions include: >

[VOTE] Creating an Apache Flink slack workspace

Hi everyone, As previously discussed in [1], I would like to open a vote on creating an Apache Flink slack workspace channel. The proposed actions include: - Creating a dedicated slack workspace with the name Apache Flink that is controlled and maintained by the Apache Flink PMC - Updating the

Re: [Discuss] Creating an Apache Flink slack workspace

Sorry for the delay in response. > Nice, cool to hear Kyle! How do you all approach moderation? Moderation has generally not been a problem so far. So unfortunately I can’t answer that from experience. But if there’s an existing code of conduct amongst Apache or within Flink that would probably

Re: [Discuss] Creating an Apache Flink slack workspace

Thanks everyone for the opinions. And thanks Robert for the summary. I'll start a voting thread. Best, Xintong On Tue, May 17, 2022 at 3:33 PM Robert Metzger wrote: > Thanks a lot Kyle! > > What do you think of concluding this discussion and starting a VOTE about: > 1. Setting up a PMC

Re: [DISCUSS] Release first version of Elasticsearch connector from external repository

some notes: - the ES repo currently depends on 1.16-SNAPSHOT; so we'd have to go back to 1.15 to do such a release as Konstantin proposed. This may have repercussions on the architecture tests. - the version scheme in the repo does not yet match what was discussed in the original proposal

[jira] [Created] (FLINK-27662) [JUnit5 Migration] Migrate TypeInformationTestBase to Junit5

Sergey Nuyanzin created FLINK-27662: --- Summary: [JUnit5 Migration] Migrate TypeInformationTestBase to Junit5 Key: FLINK-27662 URL: https://issues.apache.org/jira/browse/FLINK-27662 Project: Flink

Re: [Discuss] Creating an Apache Flink slack workspace

Thanks a lot Kyle! What do you think of concluding this discussion and starting a VOTE about: 1. Setting up a PMC controlled Slack instance for the Flink community 2. Updating the Flink website about the various communication channels 3. Setting up an Archive for our Slack instance 4. Revisiting

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

Hi, all. After discussing with Becket Qin offline, I modified the FLIP a little. The change is as follow: 1. Add /api_versions in the REST API. The api is used by the client to know the current REST Endpoint supports which version. The client is able to choose the version for later

[jira] [Created] (FLINK-27661) [Metric]Flink-Metrics pushgateway support authentication

jiangchunyang created FLINK-27661: - Summary: [Metric]Flink-Metrics pushgateway support authentication Key: FLINK-27661 URL: https://issues.apache.org/jira/browse/FLINK-27661 Project: Flink

[jira] [Created] (FLINK-27660) Table API support create function using customed jar

dalongliu created FLINK-27660: - Summary: Table API support create function using customed jar Key: FLINK-27660 URL: https://issues.apache.org/jira/browse/FLINK-27660 Project: Flink Issue Type:

[jira] [Created] (FLINK-27659) Planner support to use jar which is registered by "USING JAR" syntax

dalongliu created FLINK-27659: - Summary: Planner support to use jar which is registered by "USING JAR" syntax Key: FLINK-27659 URL: https://issues.apache.org/jira/browse/FLINK-27659 Project: Flink

[jira] [Created] (FLINK-27658) Introduce MutableURLClassLoader allow to register and remove user jar dynamically

dalongliu created FLINK-27658: - Summary: Introduce MutableURLClassLoader allow to register and remove user jar dynamically Key: FLINK-27658 URL: https://issues.apache.org/jira/browse/FLINK-27658 Project:

Re: Re: [DISCUSS] FLIP-221 Abstraction for lookup source cache and metric

Hi Yuan! > How about abtract the LookupCache to a higher level with a common Cache? Can you give an example of such upper - level Cache usage? It's not clear for me currently. I think it's unnecessary to have such high level abstraction, if nowhere in the code we won't operate with objects as

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

Hi, Jark, Timo. Nice to have an agreement! Thanks for Jark's inputs about the multiple version Flink. I have already updated the FLIP in the rejected alternatives about details. 1. We should definitely just use LogicalTypeJsonSerializer and not a second JSON representation. Our concern is

[jira] [Created] (FLINK-27657) Implement remote operator state backend in PyFlink

Juntao Hu created FLINK-27657: - Summary: Implement remote operator state backend in PyFlink Key: FLINK-27657 URL: https://issues.apache.org/jira/browse/FLINK-27657 Project: Flink Issue Type: