[jira] [Created] (FLINK-34037) FLIP-398: Improve Serialization Configuration And Usage In Flink

Fang Yong created FLINK-34037: - Summary: FLIP-398: Improve Serialization Configuration And Usage In Flink Key: FLINK-34037 URL: https://issues.apache.org/jira/browse/FLINK-34037 Project: Flink

[jira] [Created] (FLINK-34036) Various HiveDialectQueryITCase tests fail in GitHub Actions workflow with Hadoop 3.1.3 enabled

Matthias Pohl created FLINK-34036: - Summary: Various HiveDialectQueryITCase tests fail in GitHub Actions workflow with Hadoop 3.1.3 enabled Key: FLINK-34036 URL: https://issues.apache.org/jira/browse/FLINK-34036

Re: [VOTE] Accept Flink CDC into Apache Flink

+1 (binding) Best, Jark On Tue, 9 Jan 2024 at 15:31, Benchao Li wrote: > +1 (non-binding) > > Feng Wang 于2024年1月9日周二 15:29写道： > > > > +1 non-binding > > Regards, > > Feng > > > > On Tue, Jan 9, 2024 at 3:05 PM Leonard Xu wrote: > > > > > Hello all, > > > > > > This is the official vote

Re: [VOTE] Accept Flink CDC into Apache Flink

+1 (binding) Best, Xintong On Tue, Jan 9, 2024 at 3:31 PM Benchao Li wrote: > +1 (non-binding) > > Feng Wang 于2024年1月9日周二 15:29写道： > > > > +1 non-binding > > Regards, > > Feng > > > > On Tue, Jan 9, 2024 at 3:05 PM Leonard Xu wrote: > > > > > Hello all, > > > > > > This is the official

Re: [VOTE] Accept Flink CDC into Apache Flink

+1 non-binding Regards, Feng On Tue, Jan 9, 2024 at 3:05 PM Leonard Xu wrote: > Hello all, > > This is the official vote whether to accept the Flink CDC code contribution > to Apache Flink. > > The current Flink CDC code, documentation, and website can be > found here: > code:

Re: [VOTE] Accept Flink CDC into Apache Flink

+1 (non-binding) Feng Wang 于2024年1月9日周二 15:29写道： > > +1 non-binding > Regards, > Feng > > On Tue, Jan 9, 2024 at 3:05 PM Leonard Xu wrote: > > > Hello all, > > > > This is the official vote whether to accept the Flink CDC code contribution > > to Apache Flink. > > > > The current Flink CDC

Re: [VOTE] Accept Flink CDC into Apache Flink

+1 On Tue, Jan 9, 2024 at 3:23 PM Yuan Mei wrote: > > +1 binding > > > > On Tue, Jan 9, 2024 at 3:21 PM Yuan Mei wrote: > > > +1 > > > > Best, > > Yuan > > > > On Tue, Jan 9, 2024 at 3:06 PM tison wrote: > > > >> +1 non-binding > >> > >> Best, > >> tison. > >> > >> Leonard Xu 于2024年1月9日周二

[jira] [Created] (FLINK-34035) when flinksql with group by partition some errors field occured in jobmanager.log

hansonhe created FLINK-34035: Summary: when flinksql with group by partition some errors field occured in jobmanager.log Key: FLINK-34035 URL: https://issues.apache.org/jira/browse/FLINK-34035 Project:

Re: [VOTE] Accept Flink CDC into Apache Flink

+1 binding On Tue, Jan 9, 2024 at 3:21 PM Yuan Mei wrote: > +1 > > Best, > Yuan > > On Tue, Jan 9, 2024 at 3:06 PM tison wrote: > >> +1 non-binding >> >> Best, >> tison. >> >> Leonard Xu 于2024年1月9日周二 15:05写道： >> > >> > Hello all, >> > >> > This is the official vote whether to accept the

Re: [VOTE] Accept Flink CDC into Apache Flink

+1 Best, Yuan On Tue, Jan 9, 2024 at 3:06 PM tison wrote: > +1 non-binding > > Best, > tison. > > Leonard Xu 于2024年1月9日周二 15:05写道： > > > > Hello all, > > > > This is the official vote whether to accept the Flink CDC code > contribution > > to Apache Flink. > > > > The current Flink CDC code,

Re: [VOTE] Accept Flink CDC into Apache Flink

+1 (binding) Thanks for driving this, Leonard! Best, Qingsheng On Tue, Jan 9, 2024 at 3:06 PM tison wrote: > +1 non-binding > > Best, > tison. > > Leonard Xu 于2024年1月9日周二 15:05写道： > > > > Hello all, > > > > This is the official vote whether to accept the Flink CDC code > contribution > > to

Re: [VOTE] Accept Flink CDC into Apache Flink

+1 non-binding Best, tison. Leonard Xu 于2024年1月9日周二 15:05写道： > > Hello all, > > This is the official vote whether to accept the Flink CDC code contribution > to Apache Flink. > > The current Flink CDC code, documentation, and website can be > found here: > code:

[VOTE] Accept Flink CDC into Apache Flink

Hello all, This is the official vote whether to accept the Flink CDC code contribution to Apache Flink. The current Flink CDC code, documentation, and website can be found here: code: https://github.com/ververica/flink-cdc-connectors docs:

[jira] [Created] (FLINK-34034) When kv hint and list hint handle duplicate query hints, the results are different.

xuyang created FLINK-34034: -- Summary: When kv hint and list hint handle duplicate query hints, the results are different. Key: FLINK-34034 URL: https://issues.apache.org/jira/browse/FLINK-34034 Project:

Re: [Discuss] FLIP-407: Improve Flink Client performance in interactive scenarios

Thanks for proposing this FLIP. Experiments have shown that it significantly enhances the real-time query experience. +1 for this. Best, Weihua On Mon, Jan 8, 2024 at 5:19 PM Rui Fan <1996fan...@gmail.com> wrote: > Thanks Xiangyu for the quick update! > > LGTM > > Best, > Rui > > On Mon, Jan

[RESULT] [VOTE] FLIP-398: Improve Serialization Configuration And Usage In Flink

Hi devs, I'm happy to announce that FLIP-398: Improve Serialization Configuration And Usage In Flink[1] has been accepted with 4 approving votes (3 binding) [2]: - Xintong Song (binding) - Zhanghao Chen (non-binding) - Zhu Zhu (binding) - weijie guo (binding) There're no disapproving votes.

Re: [DISCUSS] Externalized Python Connector Release/Dependency Process

+1 Thanks Danny for driving this. Best, Leonard > 2024年1月9日 上午2:01，Márton Balassi 写道： > > +1 > > Thanks, Danny - I really appreciate you taking the time for the in-depth > investigation. Please proceed, looking forward to your experience. > > On Mon, Jan 8, 2024 at 6:04 PM Martijn Visser

[jira] [Created] (FLINK-34033) flink json supports raw type

Jacky Lau created FLINK-34033: - Summary: flink json supports raw type Key: FLINK-34033 URL: https://issues.apache.org/jira/browse/FLINK-34033 Project: Flink Issue Type: Bug

Re: [VOTE] FLIP-405: Migrate string configuration key to ConfigOption

+1 (binding) Best, Xintong On Mon, Jan 8, 2024 at 1:48 PM Hang Ruan wrote: > +1(non-binding) > > Best, > Hang > > Rui Fan <1996fan...@gmail.com> 于2024年1月8日周一 13:04写道： > > > +1(binding) > > > > Best, > > Rui > > > > On Mon, Jan 8, 2024 at 1:00 PM Xuannan Su wrote: > > > > > Hi everyone, > >

[jira] [Created] (FLINK-34032) Cleanup local=recovery dir when switching local-recovery from enabled to disabled

Hangxiang Yu created FLINK-34032: Summary: Cleanup local=recovery dir when switching local-recovery from enabled to disabled Key: FLINK-34032 URL: https://issues.apache.org/jira/browse/FLINK-34032

[jira] [Created] (FLINK-34031) Hive table sources report statistics in various formats

hanjie created FLINK-34031: -- Summary: Hive table sources report statistics in various formats Key: FLINK-34031 URL: https://issues.apache.org/jira/browse/FLINK-34031 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

Hi, Zakelly. Thanks for driving this. Overall LGTM as we discussed offline. Some comments/suggestions just came to mind: 1. Could execution.recovery also contain some other behaviors about recovery ? e.g. restart-strategy. 2. Could we also remove some legacy configuration value ? e.g. LEGACY Mode

Re: [DISCUSS] FLIP-403: High Availability Services for OLAP Scenarios

Thank you for your comments, Zhu! 1. I would treat refactoring as a technical debt and a side effect of this FLIP. The idea is inspired by Matthias' comments in [1]. It suggests having a single implementation of HighAvailabilityServices that requires a factory method for persistence services and

[jira] [Created] (FLINK-34030) Avoid using negative value for periodic-materialize.interval

Hangxiang Yu created FLINK-34030: Summary: Avoid using negative value for periodic-materialize.interval Key: FLINK-34030 URL: https://issues.apache.org/jira/browse/FLINK-34030 Project: Flink

[jira] [Created] (FLINK-34029) Support different profiling mode on Flink WEB

Yu Chen created FLINK-34029: --- Summary: Support different profiling mode on Flink WEB Key: FLINK-34029 URL: https://issues.apache.org/jira/browse/FLINK-34029 Project: Flink Issue Type: Sub-task

Re: Re: FLIP-413: Enable unaligned checkpoints by default

Hi Piotr, I also agree with Zhanghao's assessment on the limitations of unaligned checkpoints. Some of them are already handled properly by Flink, but in the case of the "Interplay with watermarks" limitation, it is quite confusing for a new user to find that their code doesn't generate

[RESULT][VOTE] Release flink-shaded 18.0, release candidate #1

I'm happy to announce that we have unanimously approved this release. There are 7 approving votes, 4 of which are binding: * Rui Fan * Matthias Pohl (binding) * Sergey Nuyanzin * Jing Ge * Martijn Visser (binding) * Piotr Nowojski (binding) * Dawid Wysakowicz (binding) There are no disapproving

Re: [VOTE] Release flink-shaded 18.0, release candidate #1

Thanks all. This vote is now closed, I'll announce the results in a separate thread. On Mon, Jan 8, 2024 at 11:25 AM Dawid Wysakowicz wrote: > +1 (binding) > > - Validated hashes > - Verified signature > - Verified that no binaries exist in the source archive > > Best, > Dawid > > On Tue, 28

Re: [DISCUSS] FLIP-329: Add operator attribute to specify support for object-reuse

sounds good. Is the requirement to send an email thread about the voting? What else is needed? What's the passing criteria? Best Lu On Sun, Jan 7, 2024 at 5:41 PM Xuannan Su wrote: > Hi Liu, > > The voting thread has been open for a long time. We may want to start > a new voting thread. WDYT?

[jira] [Created] (FLINK-34027) Create initial PR for FLIP-400

Alan Sheinberg created FLINK-34027: -- Summary: Create initial PR for FLIP-400 Key: FLINK-34027 URL: https://issues.apache.org/jira/browse/FLINK-34027 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-34028) Allow multiple parallel async calls in a single operator

Alan Sheinberg created FLINK-34028: -- Summary: Allow multiple parallel async calls in a single operator Key: FLINK-34028 URL: https://issues.apache.org/jira/browse/FLINK-34028 Project: Flink

Re: [DISCUSS] Externalized Python Connector Release/Dependency Process

+1 Thanks, Danny - I really appreciate you taking the time for the in-depth investigation. Please proceed, looking forward to your experience. On Mon, Jan 8, 2024 at 6:04 PM Martijn Visser wrote: > Thanks for investigating Danny. It looks like the best direction to go to > :) > > On Mon, Jan

Re: [DISCUSS] Externalized Python Connector Release/Dependency Process

Thanks for investigating Danny. It looks like the best direction to go to :) On Mon, Jan 8, 2024 at 5:56 PM Péter Váry wrote: > > Thanks Danny for working on this! > > It would be good to do this in a way that the different connectors could > reuse as much code as possible, so if possible put

Re: [DISCUSS] Externalized Python Connector Release/Dependency Process

Thanks Danny for working on this! It would be good to do this in a way that the different connectors could reuse as much code as possible, so if possible put most of the code to the flink connector shared utils repo [1] +1 from for the general direction (non-binding) Thanks, Peter [1]

[jira] [Created] (FLINK-34026) Azure Pipelines not running for master and the release branches since 4a852fe

Matthias Pohl created FLINK-34026: - Summary: Azure Pipelines not running for master and the release branches since 4a852fe Key: FLINK-34026 URL: https://issues.apache.org/jira/browse/FLINK-34026

[DISCUSS] Externalized Python Connector Release/Dependency Process

Hello all, I have been working with Péter and Marton on externalizing python connectors [1] from the main repo to the connector repositories. We have the code moved and the CI running tests for Kafka and AWS Connectors. I am now looking into the release process. When we undertake a Flink release

Flink pending record metric weired after autoscaler rescaling

Dear Flink Community, I've encountered an issue during the testing of my Flink autoscaler. It appears that Flink is losing track of specific Kafka partitions, leading to a persistent increase in lag on these partitions. The logs indicate a 'kafka connector metricGroup name collision exception.'

[jira] [Created] (FLINK-34025) Show data skew score on Flink Dashboard

Emre Kartoglu created FLINK-34025: - Summary: Show data skew score on Flink Dashboard Key: FLINK-34025 URL: https://issues.apache.org/jira/browse/FLINK-34025 Project: Flink Issue Type: New

[jira] [Created] (FLINK-34024) Update connector release process for Python connectors

Danny Cranmer created FLINK-34024: - Summary: Update connector release process for Python connectors Key: FLINK-34024 URL: https://issues.apache.org/jira/browse/FLINK-34024 Project: Flink

[RESULT][VOTE] FLIP-397: Add config options for administrator JVM options

Hi everyone, I'm happy to announce that FLIP-397: Add config options for administrator JVM options, has been accepted with 4 approving votes (3 binding) [1]: - Benchao Li (binding) - Rui Fan (binding) - Xiangyu Feng (non-binding) - Yong Fang (binding) There're no disapproving votes.

[jira] [Created] (FLINK-34023) Expose Kinesis client retry config in sink

Brad Atcheson created FLINK-34023: - Summary: Expose Kinesis client retry config in sink Key: FLINK-34023 URL: https://issues.apache.org/jira/browse/FLINK-34023 Project: Flink Issue Type:

Re: [VOTE] FLIP-397: Add config options for administrator JVM options

Thank you all! Closing the vote. The result will be sent in a separate email. Best, Zhanghao Chen From: Zhanghao Chen Sent: Thursday, January 4, 2024 10:29 To: dev Subject: [VOTE] FLIP-397: Add config options for administrator JVM options Hi everyone, Thanks

Re: [VOTE] Release 1.18.1, release candidate #2

Thanks Jing for driving this. +1 (binding) - Build and compile the source code locally: *OK* - Verified signatures and hashes: *OK* - Checked no missing artifacts in the staging area: *OK* - Reviewed the website release PR: *OK* - Went through the quick start: *OK* * Started a cluster and ran

Re: [DISCUSS] FLIP-403: High Availability Services for OLAP Scenarios

Thanks for creating the FLIP and starting the discussion, Yangze. It makes sense to me to improve the job submission performance in OLAP scenarios. I have a few questions regarding the proposed changes: 1. How about skipping the job graph persistence if the proposed config

[jira] [Created] (FLINK-34022) A Duration that cannot be expressed as long in nanoseconds causes an ArithmeticException in TimeUtils.formatWithHighestUnit

Matthias Pohl created FLINK-34022: - Summary: A Duration that cannot be expressed as long in nanoseconds causes an ArithmeticException in TimeUtils.formatWithHighestUnit Key: FLINK-34022 URL:

[jira] [Created] (FLINK-34021) Print jobKey in the Autoscaler standalone log

Rui Fan created FLINK-34021: --- Summary: Print jobKey in the Autoscaler standalone log Key: FLINK-34021 URL: https://issues.apache.org/jira/browse/FLINK-34021 Project: Flink Issue Type: Sub-task

Re: [VOTE] Release flink-shaded 18.0, release candidate #1

+1 (binding) - Validated hashes - Verified signature - Verified that no binaries exist in the source archive Best, Dawid On Tue, 28 Nov 2023 at 23:10, Sergey Nuyanzin wrote: > Hi everyone, > Please review and vote on the release candidate #1 for the version 18.0, as > follows: > [ ] +1,

[jira] [Created] (FLINK-34020) Bump CI flink version on flink-connector-rabbitmq

Martijn Visser created FLINK-34020: -- Summary: Bump CI flink version on flink-connector-rabbitmq Key: FLINK-34020 URL: https://issues.apache.org/jira/browse/FLINK-34020 Project: Flink Issue

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

Hi Yun, Thanks for your comments! 1. We shall not describe the configuration with its implementation for > 'execution.checkpointing.local-copy.*' options, for hashmap state-backend, > it would write two streams and for Rocksdb state-backend, it would use > hard-link for backup. Thus, I think >

[jira] [Created] (FLINK-34019) Bump com.rabbitmq:amqp-client from 5.13.1 to 5.20.0

Martijn Visser created FLINK-34019: -- Summary: Bump com.rabbitmq:amqp-client from 5.13.1 to 5.20.0 Key: FLINK-34019 URL: https://issues.apache.org/jira/browse/FLINK-34019 Project: Flink

Re:Re: FLIP-413: Enable unaligned checkpoints by default

退订 At 2024-01-08 17:45:01, "Piotr Nowojski" wrote: >Hi thanks for the responses, > >And thanks for pointing out the jobs upgrade issue. Indeed that has >slipped my mind. I was mistakenly >thinking that we are supporting all upgrades only via savepoint. Anyway, >maybe in that case we

Re:[jira] [Created] (FLINK-34018) Add flink 1.19 verison in ci for flink-connector-aws

退订 At 2024-01-08 17:49:00, "Zhongqiang Gong (Jira)" wrote: >Zhongqiang Gong created FLINK-34018: >--- > > Summary: Add flink 1.19 verison in ci for flink-connector-aws > Key: FLINK-34018 > URL:

[jira] [Created] (FLINK-34018) Add flink 1.19 verison in ci for flink-connector-aws

Zhongqiang Gong created FLINK-34018: --- Summary: Add flink 1.19 verison in ci for flink-connector-aws Key: FLINK-34018 URL: https://issues.apache.org/jira/browse/FLINK-34018 Project: Flink

Re: FLIP-413: Enable unaligned checkpoints by default

Hi thanks for the responses, And thanks for pointing out the jobs upgrade issue. Indeed that has slipped my mind. I was mistakenly thinking that we are supporting all upgrades only via savepoint. Anyway, maybe in that case we should guide users towards that? Using savepoints for upgrades? That

Re: FLIP-413: Enable unaligned checkpoints by default

Hi Piotr, Thanks for driving this! Generally I support enabling the alignment timeout for aligned checkpoint. And I second Rui's opinion, 30s seems a reasonable value. However I'm worried if there are some operators that do not support the unaligned CP, which may cause data accuracy problems (as

Re: [VOTE] Release flink-shaded 18.0, release candidate #1

Hi! +1 (binding) Best, Piotrek pt., 5 sty 2024 o 00:58 Sergey Nuyanzin napisał(a): > Bubble up, we need more votes, especially from PMC members. > > On Thu, Dec 28, 2023 at 1:29 PM Martijn Visser > wrote: > > > Hi, > > > > +1 (binding) > > > > - Validated hashes > > - Verified signature > >

[jira] [Created] (FLINK-34017) Connectors docs with 404 link in dynamodb.md

Zhongqiang Gong created FLINK-34017: --- Summary: Connectors docs with 404 link in dynamodb.md Key: FLINK-34017 URL: https://issues.apache.org/jira/browse/FLINK-34017 Project: Flink Issue

Re: [Discuss] FLIP-407: Improve Flink Client performance in interactive scenarios

Thanks Xiangyu for the quick update! LGTM Best, Rui On Mon, Jan 8, 2024 at 4:27 PM xiangyu feng wrote: > Hi Rui and Yong, > > Thx for ur reply. > > My initial attention here is that for short-lived jobs under high QPS: a > fixed delay retry strategy will cause extra resource waste and not

Re: [DISCUSS] FLIP-403: High Availability Services for OLAP Scenarios

Thanks for the pointer, Rui! I have reviewed FLIP-383, and based on my understanding, this feature should be enabled by default for batch jobs in the future. Therefore, +1 for checking the parameters and issuing log warnings when the user explicitly configures execution.batch.job-recovery.enabled

Re: [DISCUSS] FLIP-402: Extend ZooKeeper Curator configurations

Thanks Zhanghao Chen for the feedback. In general right now there is no way to authorise Flink via the Curator framework, which is probably an important feature which is missing. Since public configuration change requires a Flip, current Flip is proposing to add several more potentially important

Re: [Discuss] FLIP-407: Improve Flink Client performance in interactive scenarios

Hi Rui and Yong, Thx for ur reply. My initial attention here is that for short-lived jobs under high QPS: a fixed delay retry strategy will cause extra resource waste and not flexible enough, an exponential-backoff strategy might significantly increase the query latency since the interval time