Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

2022-11-30 Thread Martijn Visser
Hi all, A couple of first comments on this: 1. I'm missing the problem statement in the overall introduction. It immediately goes into proposal mode, I would like to first read what is the actual problem, before diving into solutions. 2. "Each ETL job creates snapshots with checkpoint info on

Re: [VOTE] FLIP-273: Improve Catalog API to Support ALTER TABLE syntax

2022-11-30 Thread Jingsong Li
+1 (binding) Thanks Shengkai for driving this FLIP. Best, Jingsong On Thu, Dec 1, 2022 at 12:20 PM Shengkai Fang wrote: > > Hi All, > > Thanks for all the feedback so far. Based on the discussion[1] we seem > to have a consensus, so I would like to start a vote on FLIP-273. > > The vote will

[jira] [Created] (FLINK-30256) LogalWindowAgg can set the chaining Strategy to always

2022-11-30 Thread xiaogang zhou (Jira)
xiaogang zhou created FLINK-30256: - Summary: LogalWindowAgg can set the chaining Strategy to always Key: FLINK-30256 URL: https://issues.apache.org/jira/browse/FLINK-30256 Project: Flink

[VOTE] FLIP-273: Improve Catalog API to Support ALTER TABLE syntax

2022-11-30 Thread Shengkai Fang
Hi All, Thanks for all the feedback so far. Based on the discussion[1] we seem to have a consensus, so I would like to start a vote on FLIP-273. The vote will last for at least 72 hours (Dec 5th at 13:00 GMT, excluding weekend days) unless there is an objection or insufficient votes. Best,

[DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

2022-11-30 Thread Shammon FY
Hi devs: I'd like to start a discussion about FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store[1]. In the whole data stream processing, there are consistency problems such as how to manage the dependencies of multiple jobs and tables, how to define and handle E2E

[jira] [Created] (FLINK-30255) Throw exception for upper case fields is used in hive metastore

2022-11-30 Thread Shammon (Jira)
Shammon created FLINK-30255: --- Summary: Throw exception for upper case fields is used in hive metastore Key: FLINK-30255 URL: https://issues.apache.org/jira/browse/FLINK-30255 Project: Flink Issue

[DISCUSS] OLM Bundles for Flink Kubernetes Operator

2022-11-30 Thread Hao t Chang
Hi Gyula, This is a good model. I will prepare a PR. Jim and I will help the verification and release process. -- Best, Ted Chang | Software Engineer | htch...@us.ibm.com

Re: [DISCUSS] OLM Bundles for Flink Kubernetes Operator

2022-11-30 Thread Gyula Fóra
Hi All! I think for this first version let's go with the following simple model: 1. We include the OLM bundle generation script in the operator git repo before the release 2. We add one extra release verification step to the wiki with detailed instructions on generating and verifying the bundle

Re: [DISCUSS] FLIP-275: Support Remote SQL Client Based on SQL Gateway

2022-11-30 Thread Jim Hughes
Hi Yu, Thanks for moving my comments to this thread! Also, thank you for answering my questions; it is helping me understand the SQL Gateway better. 5. > Our idea is to introduce a new session option (like 'sql-client.result.fetch-interval') to control the fetching requests sending frequency.

[jira] [Created] (FLINK-30254) Sync Pulsar updates to external Pulsar connector repository

2022-11-30 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-30254: -- Summary: Sync Pulsar updates to external Pulsar connector repository Key: FLINK-30254 URL: https://issues.apache.org/jira/browse/FLINK-30254 Project: Flink

Re: SQL Gateway and SQL Client

2022-11-30 Thread yu zelin
Hi, Jim, Thanks for your feedback, your suggestions are very good. I have replied in the discussion mail, please take a look. > 2022年11月29日 03:18,Jim Hughes 写道: > > Hi Shengkai, Yu, > > Thanks for the FLIP! I have had a chance to read it, and it looks good. I > do have some questions: > >

[RESULT][VOTE] FLIP-271: Autoscaling

2022-11-30 Thread Maximilian Michels
The vote has been closed. Thank you for your votes. Here is the result: Binding votes (9 in total): - Gyula Fóra - Őrhidi Mátyás - Yang Wang - Rui Fan - Dong Lin - Thomas Weise - Marton Balassi - Yun Tang - Maximilian Michels Non-binding votes (6 in total): - Rui Fan - Zheng Yu Chen -

Re: [DISCUSS] FLIP-275: Support Remote SQL Client Based on SQL Gateway

2022-11-30 Thread yu zelin
Hi, all Thanks Jim’s questions below. Here I’d like to reply to them. > 1. For the Client Parser, is it going to work with the extended syntax > from the Flink Table Store? > > 2. Relatedly, what will happen if an older Client tries to handle syntax > that a newer service supports?

Re: [VOTE] FLIP-271: Autoscaling

2022-11-30 Thread Maximilian Michels
+1 (binding) I hereby close the vote. I'll send the result in a separate email with the subject "[RESULT][VOTE] FLIP-271: Autoscaling" Thanks, Max On Wed, Nov 30, 2022 at 2:24 PM Yun Tang wrote: > + 1 (binding) > > Best > Yun Tang > > From: Mason Chen > Sent:

Re: [jira] [Created] (FLINK-30238) Unified Sink committer does not clean up state on final savepoint

2022-11-30 Thread Galen Warren
Thanks Fabian. Do you know if the legacy StreamingFileSink has the same issues? Not asking you to do research but just wondering if you happen to know. On Tue, Nov 29, 2022 at 8:46 AM Fabian Paul wrote: > Hi folks, > > I did some initial investigation, and the problem seems twofold. > > If no

Re: [DISCUSS]Introduce a time-segment based restart strategy

2022-11-30 Thread weijie guo
Hi, all, Thank you very much for this interesting discussion. TBH, Dong's proposal made me very excited. Our users don't need to be tortured by choosing the right one among many strategies. However, as Gen said, it may need to make some changes to the RestartBackoffTimeStrategy interface as it

Re: [VOTE] FLIP-271: Autoscaling

2022-11-30 Thread Yun Tang
+ 1 (binding) Best Yun Tang From: Mason Chen Sent: Wednesday, November 30, 2022 12:52 To: dev@flink.apache.org Subject: Re: [VOTE] FLIP-271: Autoscaling +1 (non-binding) On Tue, Nov 29, 2022 at 11:55 AM Ferenc Csaky wrote: > +1 (non-binding) > > > > >

[jira] [Created] (FLINK-30253) Introduce io executor for standalone checkpoint store

2022-11-30 Thread Yun Tang (Jira)
Yun Tang created FLINK-30253: Summary: Introduce io executor for standalone checkpoint store Key: FLINK-30253 URL: https://issues.apache.org/jira/browse/FLINK-30253 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-273: Improve Catalog API to Support ALTER TABLE syntax

2022-11-30 Thread Shengkai Fang
Hi, Considering there has been no response for a long time, I want to start the vote on Thursday. Best, Shengkai Jingsong Li 于2022年11月24日周四 15:20写道: > Thanks for the update! > > Looks good to me. > > Best, > Jingsong > > On Thu, Nov 24, 2022 at 3:07 PM Shengkai Fang wrote: > > > > Hi,

[jira] [Created] (FLINK-30252) Publish flink-shaded pom

2022-11-30 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-30252: Summary: Publish flink-shaded pom Key: FLINK-30252 URL: https://issues.apache.org/jira/browse/FLINK-30252 Project: Flink Issue Type: Technical Debt

[VOTE] Release flink-connector-jdbc v3.0.0, release candidate #1

2022-11-30 Thread Martijn Visser
Hi everyone, Please review and vote on the release candidate #1 for the flink-connector-jdbc version v3.0.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) Note: This is the first externalized version of the JDBC connector. The

Re: [DISCUSS] FLIP-269: Properly Handling the Processing Timers on Job Termination

2022-11-30 Thread Yun Gao
Hi Dawid, Piotr Very thanks for the discussion! As a whole I think we are already consistent with the callback option, and I don't think I opposed that we could modify the current internal implementation. But from my side it is still not clear what the actual interfaces are proposing. Let me

[jira] [Created] (FLINK-30251) Move the IO with DFS during abort checkpoint to an asynchronous thread.

2022-11-30 Thread ming li (Jira)
ming li created FLINK-30251: --- Summary: Move the IO with DFS during abort checkpoint to an asynchronous thread. Key: FLINK-30251 URL: https://issues.apache.org/jira/browse/FLINK-30251 Project: Flink

[jira] [Created] (FLINK-30250) The flame graph type is wrong

2022-11-30 Thread Rui Fan (Jira)
Rui Fan created FLINK-30250: --- Summary: The flame graph type is wrong Key: FLINK-30250 URL: https://issues.apache.org/jira/browse/FLINK-30250 Project: Flink Issue Type: Bug Components:

[VOTE] Release flink-connector-pulsar v3.0.0, release candidate #1

2022-11-30 Thread Martijn Visser
Hi everyone, Please review and vote on the release candidate #1 for the flink-connector-pulsar version v3.0.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) Note: this is equivalent to the Pulsar connector that was released with

[jira] [Created] (FLINK-30249) TableUtils.getRowTypeInfo() creating wrong TypeInformation

2022-11-30 Thread Zhipeng Zhang (Jira)
Zhipeng Zhang created FLINK-30249: - Summary: TableUtils.getRowTypeInfo() creating wrong TypeInformation Key: FLINK-30249 URL: https://issues.apache.org/jira/browse/FLINK-30249 Project: Flink

Re: [DISCUSS] FLIP-269: Properly Handling the Processing Timers on Job Termination

2022-11-30 Thread Piotr Nowojski
Hi, I have a couple of remarks. First a general one. For me the important part in the design of this API is how to expose this to Flink users in public interfaces. Namely ProcessFunction and StreamOperator. InternalTimerService is an internal class, so we can change it and break it as needed in

[jira] [Created] (FLINK-30248) Spark writer supports insert overwrite

2022-11-30 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-30248: Summary: Spark writer supports insert overwrite Key: FLINK-30248 URL: https://issues.apache.org/jira/browse/FLINK-30248 Project: Flink Issue Type: New

Re: [VOTE] Release flink-connector-dynamodb v3.0.0, release candidate #0

2022-11-30 Thread Danny Cranmer
+1 (binding) - Validated hashes/signature - Verified that no binaries exist in the source archive - Build the source with Maven - Verified NOTICE file - Verified versions in pom files are correct - Verified SinkIntoDynamoDb sample application writes to DynamoDB Thanks, Danny On Tue, Nov 29,

Re: [DISCUSS] FLIP-269: Properly Handling the Processing Timers on Job Termination

2022-11-30 Thread Dawid Wysakowicz
WindowOperator is not implemented by users. I can see that for InternalTimerService we'll need interface PendingTimerProcessor { void onTimer(InternalTimer timer) { doHandleTimer(timer); } I don't see a problem with that. As you said ProcessingTimeService is a user facing interface and

[jira] [Created] (FLINK-30247) Introduce Time Travel reading for table store

2022-11-30 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-30247: Summary: Introduce Time Travel reading for table store Key: FLINK-30247 URL: https://issues.apache.org/jira/browse/FLINK-30247 Project: Flink Issue Type: