Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration

2023-06-01 Thread Jingsong Li
Thanks Feng, Just naming, maybe `createCatalog` in TableEnv, I can see many functions are converted to createxxx from registerxxx. On Fri, Jun 2, 2023 at 11:04 AM Feng Jin wrote: > > Hi jark, thanks for your suggestion. > > > 1. How to register the CatalogStore for Table API? I think the >

Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration

2023-06-01 Thread Feng Jin
Hi jark, thanks for your suggestion. > 1. How to register the CatalogStore for Table API? I think the CatalogStore should be immutable once TableEnv is created. Otherwise, there might be some data inconsistencies when CatalogStore is changed. Yes, We should initialize the CatalogStore when

[jira] [Created] (FLINK-32245) NonDeterministicTests #testTemporalFunctionsInBatchMode failure masked due to incorrect test initialization

2023-06-01 Thread Jane Chan (Jira)
Jane Chan created FLINK-32245: - Summary: NonDeterministicTests #testTemporalFunctionsInBatchMode failure masked due to incorrect test initialization Key: FLINK-32245 URL:

[jira] [Created] (FLINK-32244) flink-master-benchmarks-regression-check always fails since 2023.05.30

2023-06-01 Thread Lijie Wang (Jira)
Lijie Wang created FLINK-32244: -- Summary: flink-master-benchmarks-regression-check always fails since 2023.05.30 Key: FLINK-32244 URL: https://issues.apache.org/jira/browse/FLINK-32244 Project: Flink

[jira] [Created] (FLINK-32243) Bump okhttp version to 4.11.0

2023-06-01 Thread ConradJam (Jira)
ConradJam created FLINK-32243: - Summary: Bump okhttp version to 4.11.0 Key: FLINK-32243 URL: https://issues.apache.org/jira/browse/FLINK-32243 Project: Flink Issue Type: Improvement

Re: [DISCUSS] FLIP-311: Support Call Stored Procedure

2023-06-01 Thread yuxia
Hi, Paul Lam Thanks for your attention. 1: Yes, you need to have a catalog to provide procedure 2: Actually, we don't need to manage stored procedures' life cycles. The whole calling for stored procedure is a synchronize process. Its procedure implementation to decide to wait the job finish

Re: [DISCUSS] FLIP-309: Enable operators to trigger checkpoints dynamically

2023-06-01 Thread Dong Lin
Hi Ahmed, Thanks for the comments. I agree with you and Piotr that it would be useful to provide a more generic approach to address more use-case in one proposal. On the other hand, I also think it is important to make sure that the alternative (more generic) approach can indeed address the

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

2023-06-01 Thread Aitozi
Hi Jing, I have updated the proposed changes to the FLIP. IMO, lookup has its clear async call requirement is due to its IO heavy operator. In our usage, sql users have logic to do some RPC call or query the third-party service which is also IO intensive. In these case, we'd like to leverage

Re: [VOTE] Release flink-connector-jdbc v3.1.1, release candidate #1

2023-06-01 Thread Sergey Nuyanzin
+1 (non-binding) * built from source * verified signatures and hashsums * checked release notes * reviewed web PR thanks for driving this On Thu, Jun 1, 2023 at 8:15 PM Ahmed Hamdy wrote: > Hi Martijn, > > +1 (non-binding) > > - built from source code succeeded > - verified signatures match

Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

2023-06-01 Thread Martijn Visser
Hi Shammon, Just wanted to chip-in that I like the overall FLIP. Will be interesting to see the follow-up discussion on FLIP-314. Best regards, Martijn On Thu, Jun 1, 2023 at 5:45 AM yuxia wrote: > Thanks for explanation. Make sense to me. > > Best regards, > Yuxia > > - 原始邮件 - >

Re: [DISCUSS] FLIP-307: Flink connector Redshift

2023-06-01 Thread Ahmed Hamdy
Hi Samrat Thanks for putting up this FLIP. I agree regarding the importance of the use case. please let me know If you need any collaboration regarding integration with AWS connectors credential providers or regarding FLIP-171 I would be more than happy to assist. I also like Leonard's proposal

Re: [VOTE] Release flink-connector-jdbc v3.1.1, release candidate #1

2023-06-01 Thread Ahmed Hamdy
Hi Martijn, +1 (non-binding) - built from source code succeeded - verified signatures match yours - verified hashsums - checked release notes - reviewed the web PR Best regards, Ahmed Hamdy On Wed, 31 May 2023 at 09:33, Leonard Xu wrote: > +1 (binding) > > - built from source code succeeded

Re: [DISCUSS] FLIP-309: Enable operators to trigger checkpoints dynamically

2023-06-01 Thread Ahmed Hamdy
Hi Dong, Thanks for the great proposal. The thread is very intuitive along with suggestions from Jing and Piotr. As much as I like the simplicity of the proposed approach I think a much wider benefit is achieved by taking a more generic approach similar to Piotr's suggestion of having a

Re: [DISCUSS] Update Flink Roadmap

2023-06-01 Thread Jing Ge
Hi Jark, Fair enough. Let's do it like you suggested. Thanks! Best regards, Jing On Thu, Jun 1, 2023 at 6:00 PM Jark Wu wrote: > Hi Jing, > > This thread is for discussing the roadmap for versions 1.18, 2.0, and even > more. > One of the outcomes of this discussion will be an updated version

Re: [DISCUSS] FLIP-312: Add Yarn ACLs to Flink Containers

2023-06-01 Thread Archit Goyal
Hi All, If there are no further concerns about this FLIP, I will start a vote thread next Monday. Thanks, Archit Goyal From: Yang Wang Date: Thursday, June 1, 2023 at 12:16 AM To: dev@flink.apache.org Subject: Re: [DISCUSS] FLIP-312: Add Yarn ACLs to Flink Containers Thanks Archit Goyal for

[jira] [Created] (FLINK-32242) Datadog HTTP Reporter produces a huge outgoing traffic and CPU overhead

2023-06-01 Thread Mathieu DESPRIEE (Jira)
Mathieu DESPRIEE created FLINK-32242: Summary: Datadog HTTP Reporter produces a huge outgoing traffic and CPU overhead Key: FLINK-32242 URL: https://issues.apache.org/jira/browse/FLINK-32242

Re: [DISCUSS] Update Flink Roadmap

2023-06-01 Thread Jark Wu
Hi Jing, This thread is for discussing the roadmap for versions 1.18, 2.0, and even more. One of the outcomes of this discussion will be an updated version of the current roadmap. Let's work together on refining the roadmap in this thread. Best, Jark On Thu, 1 Jun 2023 at 23:25, Jing Ge wrote:

Re: [DISCUSS] FLIP-317: Upgrade Kryo from 2.24.0 to 5.5.0

2023-06-01 Thread Chesnay Schepler
The version in the state is the serializer version, and applies to the entire state, independent of what it contains. If you use Kryo2 for reading and Kryo5 for writing (which also implies writing the new serializer version into state), then I'd assume that a migration is an all-or-nothing kind

Re: [DISCUSS] Update Flink Roadmap

2023-06-01 Thread Jing Ge
Hi Jark, Thanks for driving it! For point 2, since we are developing 1.18 now, does it make sense to update the roadmap this time while we are releasing 1.18? This discussion thread will be focusing on the Flink 2.0 roadmap, as you mentioned previously. WDYT? Best regards, Jing On Thu, Jun 1,

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

2023-06-01 Thread Jing Ge
Hi Aitozi, Sorry for the late reply. Would you like to update the proposed changes with more details into the FLIP too? I got your point. It looks like a rational idea. However, since lookup has its clear async call requirement, are there any real use cases that need this change? This will help

[jira] [Created] (FLINK-32241) UnsupportedFileSystemException when using the ABFS Hadoop driver for checkpointing in Flink 1.17

2023-06-01 Thread Anton Ippolitov (Jira)
Anton Ippolitov created FLINK-32241: --- Summary: UnsupportedFileSystemException when using the ABFS Hadoop driver for checkpointing in Flink 1.17 Key: FLINK-32241 URL:

Re: [DISCUSS] Hive dialect shouldn't fall back to Flink's default dialect

2023-06-01 Thread Lincoln Lee
+1, thanks yuxia for driving the hive decoupling work! Since the 1.16 release, the compatibility of Hive queries has reached a relatively high level, so it is time to abandon the internal fallback, which will make the behavior of the Hive dialect clearer. Best, Lincoln Lee Jark Wu 于2023年6月1日周四

[DISCUSS] Update Flink Roadmap

2023-06-01 Thread Jark Wu
Hi all, Martijn and I would like to initiate a discussion on the Flink roadmap, which should cover the project's long-term roadmap and the regular update mechanism. Xintong has already started a discussion about Flink 2.0 planning. One of the points raised in that discussion is that we should

Re: [DISCUSS] Hive dialect shouldn't fall back to Flink's default dialect

2023-06-01 Thread Jark Wu
+1, I think this can make the grammar more clear. Please remember to add a release note once the issue is finished. Best, Jark On Thu, 1 Jun 2023 at 11:28, yuxia wrote: > Hi, Jingsong. It's hard to provide an option regarding to the fact that we > also want to decouple Hive with flink planner.

Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration

2023-06-01 Thread Jark Wu
Hi Feng, This is a useful FLIP. Thanks for starting this discussion. The current design looks pretty good to me. I just have some minor comments. 1. How to register the CatalogStore for Table API? I think the CatalogStore should be immutable once TableEnv is created. Otherwise, there might be

[jira] [Created] (FLINK-32240) Update Flink client to 1.17.1

2023-06-01 Thread Gyula Fora (Jira)
Gyula Fora created FLINK-32240: -- Summary: Update Flink client to 1.17.1 Key: FLINK-32240 URL: https://issues.apache.org/jira/browse/FLINK-32240 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-32239) Unify TestJvmProcess and TestProcessBuilder

2023-06-01 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-32239: Summary: Unify TestJvmProcess and TestProcessBuilder Key: FLINK-32239 URL: https://issues.apache.org/jira/browse/FLINK-32239 Project: Flink Issue

Re: [DISCUSS] FLIP-315: Support Operator Fusion Codegen for Flink SQL

2023-06-01 Thread Jark Wu
Hi Ron, Thanks a lot for the great proposal. The FLIP looks good to me in general. It looks like not an easy work but the performance sounds promising. So I think it's worth doing. Besides, if there is a complete test graph with all TPC-DS queries, the effect of this FLIP will be more intuitive.

Re: [DISCUSS] FLIP-316: Introduce SQL Driver

2023-06-01 Thread Jark Wu
Hi Paul, Thanks for starting this discussion. I like the proposal! This is a frequently requested feature! I agree with Shengkai that ExecNodeGraph as the submission object is a better idea than SQL file. To be more specific, it should be JsonPlanGraph or CompiledPlan which is the serializable

[jira] [Created] (FLINK-32238) Stable approach for installing libssl

2023-06-01 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-32238: Summary: Stable approach for installing libssl Key: FLINK-32238 URL: https://issues.apache.org/jira/browse/FLINK-32238 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-311: Support Call Stored Procedure

2023-06-01 Thread Paul Lam
Hi Yuxia, +1 for the stored procedure, which would make Flink SQL much more extendable. I have some concerns regarding the FLIP: 1. IIUC, stored procedures are bound to a catalog (I found no registration or discovery interfaces), does that mean a connector has to provide a dedicated

Re: [DISCUSS] FLIP-308: Support Time Travel In Batch Mode

2023-06-01 Thread Shammon FY
Hi Feng, I have one minor comment about the public interface `Optional getSnapshot()` in the `CatalogTable`. As we can get tables from the new method `Catalog.getTable(ObjectPath tablePath, long timestamp)`, I think the returned `CatalogBaseTable` will have the information of timestamp. Flink or

[jira] [Created] (FLINK-32237) add jackson-module-kotlin to Jackson packages that get shaded

2023-06-01 Thread Samael Bate (Jira)
Samael Bate created FLINK-32237: --- Summary: add jackson-module-kotlin to Jackson packages that get shaded Key: FLINK-32237 URL: https://issues.apache.org/jira/browse/FLINK-32237 Project: Flink

[jira] [Created] (FLINK-32236) Ease YarnTestBase allowlist address regex

2023-06-01 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-32236: Summary: Ease YarnTestBase allowlist address regex Key: FLINK-32236 URL: https://issues.apache.org/jira/browse/FLINK-32236 Project: Flink Issue

Re: [DISCUSS] FLIP-311: Support Call Stored Procedure

2023-06-01 Thread yuxia
Sorry for missing the importart part, repeat it agin: "But we can't get the `StreamExecutionEnvironment` which is the entrypoint to build datastream from `TableEnvironment`; Best regards, Yuxia - 原始邮件 - 发件人: "yuxia" 收件人: "dev" 发送时间: 星期四, 2023年 6 月 01日 下午 4:56:15 主题: Re: [DISCUSS]

Re: [DISCUSS] FLIP-311: Support Call Stored Procedure

2023-06-01 Thread yuxia
Hi, Benchao. Thanks for your attention. Initially, I also want to pass `TableEnvironment` to procedure. But according my investegation and offline discussion with Jingson, the real important thing for procedure devs is the ability to build Flink datastream. But we can't get the

Re: [DISCUSS] FLIP-316: Introduce SQL Driver

2023-06-01 Thread Paul Lam
Hi Weihua, You’re right. Distributing the SQLs to the TMs is one of the challenging parts of this FLIP. Web submission is not enabled in application mode currently as you said, but it could be changed if we have good reasons. What do you think about introducing a distributed storage for SQL

[jira] [Created] (FLINK-32235) Translate CrateDB Docs to chinese

2023-06-01 Thread Marios Trivyzas (Jira)
Marios Trivyzas created FLINK-32235: --- Summary: Translate CrateDB Docs to chinese Key: FLINK-32235 URL: https://issues.apache.org/jira/browse/FLINK-32235 Project: Flink Issue Type: Sub-task

Re: [DISCUSS] FLIP-312: Add Yarn ACLs to Flink Containers

2023-06-01 Thread Yang Wang
Thanks Archit Goyal for the explanation and updating the FLIP. No more concerns from my part. +1 Best, Yang Archit Goyal 于2023年5月27日周六 05:19写道: > Thanks Yang for review. > > > 1. FLIP-312 relies on Hadoop version 2.6.0 or later. > 2. I have updated the FLIP and made it more