Re: [DISCUSS] FLIP-316: Introduce SQL Driver

2023-05-31 Thread Weihua Hu
Thanks Paul for your reply. SQLDriver looks good to me. 2. Do you mean a pass the SQL string a configuration or a program argument? I brought this up because we were unable to pass the SQL file to Flink using Kubernetes mode. For DataStream/Python users, they need to prepare their images for

Re: [DISCUSS] FLIP-311: Support Call Stored Procedure

2023-05-31 Thread Benchao Li
Thanks Yuxia for opening this discussion, The general idea looks good to me, I only have one question about the `ProcedureContext#getExecutionEnvironment`. Why are you proposing to return a `StreamExecutionEnvironment` instead of `TableEnvironment`, could you elaborate a little more on this?

[jira] [Created] (FLINK-32234) Support execute truncate table statement

2023-05-31 Thread luoyuxia (Jira)
luoyuxia created FLINK-32234: Summary: Support execute truncate table statement Key: FLINK-32234 URL: https://issues.apache.org/jira/browse/FLINK-32234 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-32233) Introduce SupportsTruncate interface

2023-05-31 Thread luoyuxia (Jira)
luoyuxia created FLINK-32233: Summary: Introduce SupportsTruncate interface Key: FLINK-32233 URL: https://issues.apache.org/jira/browse/FLINK-32233 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-32232) Supports parse support truncate table statement

2023-05-31 Thread luoyuxia (Jira)
luoyuxia created FLINK-32232: Summary: Supports parse support truncate table statement Key: FLINK-32232 URL: https://issues.apache.org/jira/browse/FLINK-32232 Project: Flink Issue Type: Sub-task

Re: [DISCUSSION] Improved USER/SYSTEM exception wrapping in Flink code base

2023-05-31 Thread Paul Lam
Hi Hong, Thanks for starting the discussion! I believe the exception classification between user exceptions and system exceptions has been long-awaited. It's worth mentioning that years ago there was a related discussion [1], FYI. I’m in favor of the heuristic approach to classify the

Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

2023-05-31 Thread yuxia
Thanks for explanation. Make sense to me. Best regards, Yuxia - 原始邮件 - 发件人: "Shammon FY" 收件人: "dev" 发送时间: 星期四, 2023年 6 月 01日 上午 10:45:12 主题: Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener Thanks yuxia, you're right and I'll add the new database to

Re: [DISCUSS] Hive dialect shouldn't fall back to Flink's default dialect

2023-05-31 Thread yuxia
Hi, Jingsong. It's hard to provide an option regarding to the fact that we also want to decouple Hive with flink planner. If we still need this fall back behavior, we will still depend on `ParserImpl` provided by flink-table-planner on HiveParser. But to try best to minimize the impact to

Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration

2023-05-31 Thread Feng Jin
Hi , thanks all for reviewing the flip. @Ron > Regarding the CatalogStoreFactory#createCatalogStore method, do we need to provide a default implementation? Yes, we will provide a default InMemoryCatalogStoreFactory to create an InMemoryCatalogStore. > If we get a Catalog from CatalogStore,

Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

2023-05-31 Thread Shammon FY
Thanks yuxia, you're right and I'll add the new database to AlterDatabaseEvent. I added `ignoreIfNotExists` for AlterDatabaseEvent because it is a parameter in the `Catalog.alterDatabase` method. Although this value is currently always false in `AlterDatabaseOperation`, I think it's better to

Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

2023-05-31 Thread yuxia
Hi, Shammon. I mean do we need to contain the new database after alter in AlterDatabaseEvent? So that the listener can know what has been modified for the database. Or the listener don't need to care about the actual modification. Also, I'm wondering whether AlterDatabaseEven need to include

Re: [VOTE] Release flink-connector-pulsar 3.0.1, release candidate #1

2023-05-31 Thread Jing Ge
+1(non-binding) - verified sign - verified hash - checked repos - checked tag. NIT: the tag link should be: https://github.com/apache/flink-connector-pulsar/releases/tag/v3.0.1-rc1 - reviewed PR. NIT: left a comment. Best regards, Jing On Wed, May 31, 2023 at 11:16 PM Neng Lu wrote: > +1 > >

Re: [DISCUSS] FLIP-308: Support Time Travel In Batch Mode

2023-05-31 Thread Jing Ge
Hi Feng, Thanks for the proposal! Very interesting feature. Would you like to update your thoughts described in your previous email about why SupportsTimeTravel has been rejected into the FLIP? This will help readers understand the context (in the future). Since we always directly add overload

Re: [VOTE] Release flink-connector-pulsar 3.0.1, release candidate #1

2023-05-31 Thread Neng Lu
+1 I verified + the release now can communicate with Pulsar using OAuth2 auth plugin + build from source and run unit tests with JDK 17 on macOS M1Max On Wed, May 31, 2023 at 4:24 AM Zili Chen wrote: > +1 > > I verified > > + LICENSE and NOTICE present > + Checksum and GPG sign matches > +

Re: [VOTE] Release flink-connector-pulsar 3.0.1, release candidate #1

2023-05-31 Thread Neng Lu
+1 I verified this release can communicate with pulsar using Oauth2 authentication. On 2023/05/25 16:18:51 Leonard Xu wrote: > Hey all, > > Please review and vote on the release candidate #1 for the version 3.0.1 of > the > Apache Flink Pulsar Connector as follows: > > [ ] +1, Approve the

[jira] [Created] (FLINK-32231) libssl not found when running CI

2023-05-31 Thread Sergey Nuyanzin (Jira)
Sergey Nuyanzin created FLINK-32231: --- Summary: libssl not found when running CI Key: FLINK-32231 URL: https://issues.apache.org/jira/browse/FLINK-32231 Project: Flink Issue Type: Bug

A few questions regarding Docker

2023-05-31 Thread Amir Hossein Sharifzadeh
Hi Flink Dev Team, I am trying to create a Docker image. Before asking my question, I will explain to you about my application. My main (command) jar file: 1) Has dependencies on other jar files (they are all in the same directory) 2) It Needs to read some arguments from a config file. 3)

Re: [DISCUSS] FLIP-316: Introduce SQL Driver

2023-05-31 Thread Paul Lam
Hi Biao, Thanks for your comments! > 1. Scope: is this FLIP only targeted for non-interactive Flink SQL jobs in > Application mode? More specifically, if we use SQL client/gateway to > execute some interactive SQLs like a SELECT query, can we ask flink to use > Application mode to execute those

[jira] [Created] (FLINK-32230) Deadlock in AWS Kinesis Data Streams connector

2023-05-31 Thread Antonio Vespoli (Jira)
Antonio Vespoli created FLINK-32230: --- Summary: Deadlock in AWS Kinesis Data Streams connector Key: FLINK-32230 URL: https://issues.apache.org/jira/browse/FLINK-32230 Project: Flink Issue

[jira] [Created] (FLINK-32229) Implement metrics and logging for Initial implementation

2023-05-31 Thread Hong Liang Teoh (Jira)
Hong Liang Teoh created FLINK-32229: --- Summary: Implement metrics and logging for Initial implementation Key: FLINK-32229 URL: https://issues.apache.org/jira/browse/FLINK-32229 Project: Flink

[jira] [Created] (FLINK-32228) Bump testcontainers

2023-05-31 Thread Jira
João Boto created FLINK-32228: - Summary: Bump testcontainers Key: FLINK-32228 URL: https://issues.apache.org/jira/browse/FLINK-32228 Project: Flink Issue Type: Improvement Components:

[jira] [Created] (FLINK-32227) Correct some grammar errors and optimize some code styles

2023-05-31 Thread zhihao song (Jira)
zhihao song created FLINK-32227: --- Summary: Correct some grammar errors and optimize some code styles Key: FLINK-32227 URL: https://issues.apache.org/jira/browse/FLINK-32227 Project: Flink

[jira] [Created] (FLINK-32226) RestClusterClient leaks jobgraph file if submission fails

2023-05-31 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-32226: Summary: RestClusterClient leaks jobgraph file if submission fails Key: FLINK-32226 URL: https://issues.apache.org/jira/browse/FLINK-32226 Project: Flink

Re: [VOTE] Release flink-connector-pulsar 3.0.1, release candidate #1

2023-05-31 Thread Zili Chen
+1 I verified + LICENSE and NOTICE present + Checksum and GPG sign matches + No unexpected binaries in the source release + Build from source and run unit tests with JDK 17 on macOS M1 On 2023/05/25 16:18:51 Leonard Xu wrote: > Hey all, > > Please review and vote on the release candidate #1

[jira] [Created] (FLINK-32225) merge task deployment related fields into a new configuration

2023-05-31 Thread Weihua Hu (Jira)
Weihua Hu created FLINK-32225: - Summary: merge task deployment related fields into a new configuration Key: FLINK-32225 URL: https://issues.apache.org/jira/browse/FLINK-32225 Project: Flink

Re: [DISCUSS] FLIP-307: Flink connector Redshift

2023-05-31 Thread Samrat Deb
Hi Liu Ron, > 1. Regarding the `read.mode` and `write.mode`, you say here provides two modes, respectively, jdbc and `unload or copy`, What is the default value for `read.mode` and `write.mode? I have made an effort to make the configuration options `read.mode` and `write.mode` mandatory for

Re: [DISCUSS] FLIP-308: Support Time Travel In Batch Mode

2023-05-31 Thread Krzysztof Chmielewski
Ok after second though I'm retracting my previous statement about Catalog changes you proposed. I do see a benefit for Delta connector actually with this change and see why this could be coupled with Catalog. Delta Connector SQL support, also ships a Delta Catalog implementation for Flink. For

Re: [DISCUSS] FLIP-316: Introduce SQL Driver

2023-05-31 Thread Paul Lam
Hi Shengkai, Thanks a lot for your comments! Please see my comments inline. > 1. The FLIP does not specify the kind of SQL that will be submitted with > the application mode. I believe only a portion of the SQL will be delegated > to the SqlRunner. You’re right. For SQL Gateway, only DMLs need

[jira] [Created] (FLINK-32224) Unable to connect multiple host using RabbitMQ Connector

2023-05-31 Thread someshwar (Jira)
someshwar created FLINK-32224: - Summary: Unable to connect multiple host using RabbitMQ Connector Key: FLINK-32224 URL: https://issues.apache.org/jira/browse/FLINK-32224 Project: Flink Issue

[jira] [Created] (FLINK-32223) Add Hive delegation token support

2023-05-31 Thread qingbo jiao (Jira)
qingbo jiao created FLINK-32223: --- Summary: Add Hive delegation token support Key: FLINK-32223 URL: https://issues.apache.org/jira/browse/FLINK-32223 Project: Flink Issue Type: Improvement

Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration

2023-05-31 Thread Shammon FY
Hi feng, Thanks for updating, I have some minor comments 1. The modification of `CatalogManager` should not be in `Public Interfaces`, it is not a public interface. 2. `@PublicEvolving` should be added for `CatalogStore` and `CatalogStoreFactory` 3. The code `Optional optionalDescriptor =

[jira] [Created] (FLINK-32222) Cassandra Source uses DataInputDeserializer and DataOutputSerializer non public apis

2023-05-31 Thread Etienne Chauchot (Jira)
Etienne Chauchot created FLINK-3: Summary: Cassandra Source uses DataInputDeserializer and DataOutputSerializer non public apis Key: FLINK-3 URL: https://issues.apache.org/jira/browse/FLINK-3

Re: [DISCUSS] FLIP-308: Support Time Travel In Batch Mode

2023-05-31 Thread Krzysztof Chmielewski
Hi, happy to see such a feature. Small note from my end regarding Catalog changes. TL;DR I don't think it is necessary to delegate this feature to the catalog. I think that since "timetravel" is per job/query property, its should not be coupled with the Catalog or table definition. In my opinion

Re: [VOTE] Release flink-connector-jdbc v3.1.1, release candidate #1

2023-05-31 Thread Leonard Xu
+1 (binding) - built from source code succeeded - verified signatures - verified hashsums - checked Github release tag, left one minor comment about the release commit - checked release notes - checked the contents contains jar and pom files in apache repo - reviewed the web PR Best, Leonard

Release 2.0 Status Updates - 05/31/2023

2023-05-31 Thread Xintong Song
Hi devs, The release managers had some online discussions recently, and I'd like to post the summary here for transparency. We'd like to have 3 separate discussion tracks, so that each track can be more focused. - Jark will help drive a discussion on the project's long-term roadmap.

Re: [DISCUSS] FLIP-287: Extend Sink#InitContext to expose ExecutionConfig and JobID

2023-05-31 Thread João Boto
If there is no more questions, I will start the voting thread tomorrow On 2023/01/13 14:15:04 Joao Boto wrote: > Hi flink devs, > > I'd like to start a discussion thread for FLIP-287[1]. > This comes from an offline discussion with @Lijie Wang, from FLIP-239[2] > specially for the sink[3]. > >

[jira] [Created] (FLINK-32221) Attacker can achieve Remote Code Execution when they can control Flink SQL script content

2023-05-31 Thread gaoyu shi (Jira)
gaoyu shi created FLINK-32221: - Summary: Attacker can achieve Remote Code Execution when they can control Flink SQL script content Key: FLINK-32221 URL: https://issues.apache.org/jira/browse/FLINK-32221

Re: [DISCUSS] FLIP-309: Enable operators to trigger checkpoints dynamically

2023-05-31 Thread Dong Lin
Hi Piotr, Thanks for the reply. Please see my comments inline. On Wed, May 31, 2023 at 12:58 AM Piotr Nowojski wrote: > Hi Dong, > > First of all we don't need to send any extra signal from source (or non > source) operators. All of the operators are already reporting backpressured > metrics

Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

2023-05-31 Thread Shammon FY
Hi yuxia Thanks for your input. The `AlterDatabaseEvent` extends `DatabaseModificationEvent` which has the original database. Best, Shammon FY On Wed, May 31, 2023 at 2:24 PM yuxia wrote: > Thanks Shammon for driving it. > The FLIP generally looks good to me. I only have one question. > WRT

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

2023-05-31 Thread Aitozi
Hi Jing, What do you think about it? Can we move forward this feature? Thanks, Aitozi. Aitozi 于2023年5月29日周一 09:56写道: > Hi Jing, > > "Do you mean to support the AyncTableFunction beyond the > LookupTableSource?" > Yes, I mean to support the AyncTableFunction beyond the

Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

2023-05-31 Thread Shammon FY
Hi ron, Thanks for your feedback. > 1. Regarding CatalogModificationListenerFactory#createListener method, I think it would be better to pass Context as its parameter instead of two specific Object. In this way, we can easily extend it in the future and there will be no compatibility problems.

[jira] [Created] (FLINK-32220) Improving the adaptive local hash agg code to avoid get value from RowData repeatedly

2023-05-31 Thread dalongliu (Jira)
dalongliu created FLINK-32220: - Summary: Improving the adaptive local hash agg code to avoid get value from RowData repeatedly Key: FLINK-32220 URL: https://issues.apache.org/jira/browse/FLINK-32220

Re: [DISCUSS] FLIP-315: Support Operator Fusion Codegen for Flink SQL

2023-05-31 Thread liu ron
Hi, Jinsong Thanks for your valuable suggestions. Best, Ron Jingsong Li 于2023年5月30日周二 13:22写道: > Thanks Ron for your information. > > I suggest that it can be written in the Motivation of FLIP. > > Best, > Jingsong > > On Tue, May 30, 2023 at 9:57 AM liu ron wrote: > > > > Hi, Jingsong > > >

Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

2023-05-31 Thread yuxia
Thanks Shammon for driving it. The FLIP generally looks good to me. I only have one question. WRT AlterDatabaseEvent, IIUC, it'll contain the origin database name and the new CatalogDatabase after modified. Is it enough only pass the origin database name? Will it be better to contain the origin

Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration

2023-05-31 Thread liu ron
Hi, Feng Thanks for driving this FLIP, this proposal is very useful for catalog management. I have some small questions: 1. Regarding the CatalogStoreFactory#createCatalogStore method, do we need to provide a default implementation? 2. If we get Catalog from CatalogStore, after initializing it,