[jira] [Created] (FLINK-34994) JobIDLoggingITCase fails because of "checkpoint confirmation for unknown task"

2024-04-03 Thread Roman Khachatryan (Jira)
Roman Khachatryan created FLINK-34994: - Summary: JobIDLoggingITCase fails because of "checkpoint confirmation for unknown task" Key: FLINK-34994 URL: https://issues.apache.org/jira/browse/FLINK-34994

[jira] [Created] (FLINK-34998) Wordcount on Docker test failed on azure

2024-04-03 Thread Weijie Guo (Jira)
Weijie Guo created FLINK-34998: -- Summary: Wordcount on Docker test failed on azure Key: FLINK-34998 URL: https://issues.apache.org/jira/browse/FLINK-34998 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-35002) GitHub action/upload-artifact@v4 can timeout

2024-04-03 Thread Ryan Skraba (Jira)
Ryan Skraba created FLINK-35002: --- Summary: GitHub action/upload-artifact@v4 can timeout Key: FLINK-35002 URL: https://issues.apache.org/jira/browse/FLINK-35002 Project: Flink Issue Type: Bug

Re: Re: [ANNOUNCE] Apache Paimon is graduated to Top Level Project

2024-04-03 Thread lorenzo . affetti
Congratulations! Big milestone reached :) Best, Lorenzo On Apr 2, 2024 at 03:50 +0200, Ron liu , wrote: > Congratulations! > > Best, > Ron > > Jeyhun Karimov 于2024年4月1日周一 18:12写道: > > > Congratulations! > > > > Regards, > > Jeyhun > > > > On Mon, Apr 1, 2024 at 7:43 AM Guowei Ma wrote: > > > >

Re: [DISCUSS] FLIP-435: Introduce a New Dynamic Table for Simplifying Data Pipelines

2024-04-03 Thread lorenzo . affetti
Hello everybody! Thanks for the FLIP as it looks amazing (and I think the prove is this deep discussion it is provoking :)) I have a couple of comments to add to this: Even though I get the reason why you rejected MATERIALIZED VIEW, I still like it a lot, and I would like to provide pointers

[jira] [Created] (FLINK-35003) Update zookeeper to 3.8.4 to address CVE-2024-23944

2024-04-03 Thread Shilun Fan (Jira)
Shilun Fan created FLINK-35003: -- Summary: Update zookeeper to 3.8.4 to address CVE-2024-23944 Key: FLINK-35003 URL: https://issues.apache.org/jira/browse/FLINK-35003 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-434: Support optimizations for pre-partitioned data sources

2024-04-03 Thread Lincoln Lee
Hi Jeyhun, Thanks for your quick response! In streaming scenario, shuffle commonly occurs before the stateful operator, and there's a sanity check[1] when the stateful operator accesses the state. This implies the consistency requirement of the partitioner used for data shuffling and state key

[jira] [Created] (FLINK-35001) Avoid scientific notation for DOUBLE to STRING

2024-04-03 Thread Timo Walther (Jira)
Timo Walther created FLINK-35001: Summary: Avoid scientific notation for DOUBLE to STRING Key: FLINK-35001 URL: https://issues.apache.org/jira/browse/FLINK-35001 Project: Flink Issue Type:

Re: [DISCUSS] Externalized Google Cloud Connectors

2024-04-03 Thread lorenzo . affetti
@Leonard @Martijn Following up on @Claire question, what is the role of Bahir (https://bahir.apache.org/) in this scenario? I am also trying to understand how connectors fir in the Flink project scenario :) Thank you, Lorenzo On Apr 2, 2024 at 06:13 +0200, Leonard Xu , wrote: > Hey, Claire > >

Participate in the ASF 25th Anniversary Campaign

2024-04-03 Thread Brian Proffitt
Hi everyone, As part of The ASF’s 25th anniversary campaign[1], we will be celebrating projects and communities in multiple ways. We invite all projects and contributors to participate in the following ways: * Individuals - submit your first contribution:

Parallelism is not working as expected for Apache Beam Code Running on a Flink Kubernetes Cluster

2024-04-03 Thread Dipak Tandel
Hi Everyone I have deployed a Flink cluster using a Flink Kubernetes operator and then submitted an Apache Beam Pipeline using a FlinkRunner. I submitted two jobs. One with *parallelism=20* and another with *parallelism=1* but both jobs took almost the same time to complete the task (A

[jira] [Created] (FLINK-35005) SqlClientITCase Failed to build JobManager image

2024-04-03 Thread Ryan Skraba (Jira)
Ryan Skraba created FLINK-35005: --- Summary: SqlClientITCase Failed to build JobManager image Key: FLINK-35005 URL: https://issues.apache.org/jira/browse/FLINK-35005 Project: Flink Issue Type:

Re: [VOTE] FLIP-437: Support ML Models in Flink SQL

2024-04-03 Thread David Morávek
+1 (binding) My only suggestion would be to move Catalog changes into a separate interface to allow us to begin with lower stability guarantees. Existing Catalogs would be able to opt-in by implementing it. It's a minor thing though, overall the FLIP is solid and the direction is pretty exciting.

Re: [DISCUSS] FLIP-438: Make Flink's Hadoop and YARN configuration probing consistent

2024-04-03 Thread Ferenc Csaky
Hi Venkata, Thank you for opening the discussion about this! After taking a look at the YARN and Hadoop configurations, the reason why it was implemented this way is that, in case of YARN, every YARN-specific property is prefixed with "yarn.", so to get the final, YARN-side property it is enough

[jira] [Created] (FLINK-35000) PullRequest template doesn't use the correct format to refer to the testing code convention

2024-04-03 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-35000: - Summary: PullRequest template doesn't use the correct format to refer to the testing code convention Key: FLINK-35000 URL: https://issues.apache.org/jira/browse/FLINK-35000

Re: [DISCUSS] FLIP-435: Introduce a New Dynamic Table for Simplifying Data Pipelines

2024-04-03 Thread Martijn Visser
Hi all, Thanks for the proposal. While the FLIP talks extensively on how Snowflake has Dynamic Tables and Databricks has Delta Live Tables, my understanding is that Databricks has CREATE STREAMING TABLE [1] which relates with this proposal. I do have concerns about using CREATE DYNAMIC TABLE,

[jira] [Created] (FLINK-35004) SqlGatewayE2ECase could not start container

2024-04-03 Thread Ryan Skraba (Jira)
Ryan Skraba created FLINK-35004: --- Summary: SqlGatewayE2ECase could not start container Key: FLINK-35004 URL: https://issues.apache.org/jira/browse/FLINK-35004 Project: Flink Issue Type: Bug

Re: [VOTE] FLIP-437: Support ML Models in Flink SQL

2024-04-03 Thread Leonard Xu
+1(binding) Best, Leonard > 2024年4月3日 下午3:37,Piotr Nowojski 写道: > > +1 (binding) > > Best, > Piotrek > > śr., 3 kwi 2024 o 04:29 Yu Chen napisał(a): > >> +1 (non-binding) >> >> Looking forward to this future. >> >> Thanks, >> Yu Chen >> >>> 2024年4月3日 10:23,Jark Wu 写道: >>> >>> +1

Inquiry Regarding Azure Pipelines

2024-04-03 Thread Yisha Zhou
Hi devs, I hope this email finds you well. I am writing to seek clarification regarding the status of Azure Pipelines within the Apache community and seek assistance with a specific issue I encountered. Today, I made some new commits to a pull request in one of the Apache repositories.

[jira] [Created] (FLINK-34997) PyFlink YARN per-job on Docker test failed on azure

2024-04-03 Thread Weijie Guo (Jira)
Weijie Guo created FLINK-34997: -- Summary: PyFlink YARN per-job on Docker test failed on azure Key: FLINK-34997 URL: https://issues.apache.org/jira/browse/FLINK-34997 Project: Flink Issue Type:

[jira] [Created] (FLINK-34995) flink kafka connector source stuck when partition leader invalid

2024-04-03 Thread yansuopeng (Jira)
yansuopeng created FLINK-34995: -- Summary: flink kafka connector source stuck when partition leader invalid Key: FLINK-34995 URL: https://issues.apache.org/jira/browse/FLINK-34995 Project: Flink

Re: [DISCUSS] FLIP-434: Support optimizations for pre-partitioned data sources

2024-04-03 Thread Leonard Xu
Hey, Jeyhun Thanks for kicking off this discussion. I have two questions about streaming sources: (1)The FLIP motivation section says Kafka broker is already partitioned w.r.t. some key[s] , Is this the main use case in Kafka world? Partitioning by key fields is not the default partitioner

Re: [VOTE] FLIP-437: Support ML Models in Flink SQL

2024-04-03 Thread Martijn Visser
+1 (binding) On Wed, Apr 3, 2024 at 9:52 AM Leonard Xu wrote: > +1(binding) > > Best, > Leonard > > > 2024年4月3日 下午3:37,Piotr Nowojski 写道: > > > > +1 (binding) > > > > Best, > > Piotrek > > > > śr., 3 kwi 2024 o 04:29 Yu Chen napisał(a): > > > >> +1 (non-binding) > >> > >> Looking forward to

Re: [VOTE] FLIP-437: Support ML Models in Flink SQL

2024-04-03 Thread David Radley
Hi Hao, I don’t think this counts as an objection, I have some comments. I should have put this on the discussion thread earlier but have just got to this. - I suggest we can put a model version in the model resource. Versions are notoriously difficult to add later; I don’t think we want to

Re: [DISCUSS] FLIP-XXX: Introduce Flink SQL variables

2024-04-03 Thread Ferenc Csaky
Hi Jeyhun, Thank you for your questions, please see my answers below. > What is its impact on query optimization because resolving > variables at the parsing stage might affect query optimization. The approach I mentioned in the FLIP would not affect query optimization, as it restricts

Re: [VOTE] FLIP-437: Support ML Models in Flink SQL

2024-04-03 Thread Piotr Nowojski
+1 (binding) Best, Piotrek śr., 3 kwi 2024 o 04:29 Yu Chen napisał(a): > +1 (non-binding) > > Looking forward to this future. > > Thanks, > Yu Chen > > > 2024年4月3日 10:23,Jark Wu 写道: > > > > +1 (binding) > > > > Best, > > Jark > > > > On Tue, 2 Apr 2024 at 15:12, Timo Walther wrote: > > > >>

[jira] [Created] (FLINK-34996) Deserializer can't be instantiated when connector-kafka installed into Flink Libs

2024-04-03 Thread Hugo Gu (Jira)
Hugo Gu created FLINK-34996: --- Summary: Deserializer can't be instantiated when connector-kafka installed into Flink Libs Key: FLINK-34996 URL: https://issues.apache.org/jira/browse/FLINK-34996 Project:

[jira] [Created] (FLINK-34999) PR CI stopped operating

2024-04-03 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-34999: - Summary: PR CI stopped operating Key: FLINK-34999 URL: https://issues.apache.org/jira/browse/FLINK-34999 Project: Flink Issue Type: Bug

Re: [VOTE] FLIP-437: Support ML Models in Flink SQL

2024-04-03 Thread Hao Li
Thanks David Radley and David Moravek for the comments. I'll reply in the discussion thread. Hao On Wed, Apr 3, 2024 at 5:45 AM David Morávek wrote: > +1 (binding) > > My only suggestion would be to move Catalog changes into a separate > interface to allow us to begin with lower stability

Re: Inquiry Regarding Azure Pipelines

2024-04-03 Thread Robert Metzger
Hi Yisha, flinkbot is currently not active, so new PRs are not triggering any AZP builds. We hope to restore the service soon. AZP is still the source of truth for CI builds. On Wed, Apr 3, 2024 at 11:34 AM Yisha Zhou wrote: > Hi devs, > > I hope this email finds you well. I am writing to

Community over Code EU 2024: Start planning your trip!

2024-04-03 Thread Ryan Skraba
[Note: You're receiving this email because you are subscribed to one or more project dev@ mailing lists at the Apache Software Foundation.] Dear community, We hope you are doing great, are you ready for Community Over Code EU? Check out the featured sessions, get your tickets with special

Re: [DISCUSS] FLIP-437: Support ML Models in Flink SQL

2024-04-03 Thread Hao Li
Cross post David Radley's comments here from voting thread: > I don’t think this counts as an objection, I have some comments. I should have put this on the discussion thread earlier but have just got to this. > - I suggest we can put a model version in the model resource. Versions are

Re: [DISCUSS] FLIP-434: Support optimizations for pre-partitioned data sources

2024-04-03 Thread Jeyhun Karimov
Hi Leonard, Thanks a lot for your comments. Please find my answers below: (1)The FLIP motivation section says Kafka broker is already partitioned > w.r.t. some key[s] , Is this the main use case in Kafka world? Partitioning > by key fields is not the default partitioner of Kafka default >

[IMPROVEMENT] Using ServiceLoader to load ExtendedParseStrategy

2024-04-03 Thread Naveen Kumar
Hi All, [*DISCLAIMER]* Please ignore if it's a duplicate request. I was looking into supported grammars for flink-sql. We do have two dialects* DEFAULT & HIVE. *With this we do have ExtendedParser

Re: [DISCUSS] FLIP-434: Support optimizations for pre-partitioned data sources

2024-04-03 Thread Jeyhun Karimov
Hi Lincoln, Thanks for your reply. My idea was to utilize MapBundleFunction as it was already used in a similar context - MiniBatchLocalGroupAggFunction. I can also extend my PoC for streaming sources and get back to continue our discussion. Regards, Jeyhun On Wed, Apr 3, 2024 at 4:33 PM