SQL / Table input format for Kafka that can access full record

Hey All! I have been browsing through the codebase but don't really have a good answer. Is there any way to access the entire Kafka consumer record (key, value, headers) when deserializing into a dynamic table? I see that we can specify, key format , value format and metadata columns

[jira] [Created] (FLINK-33581) FLIP-381: Deprecate configuration getters/setters that return/set complex Java objects

Junrui Li created FLINK-33581: - Summary: FLIP-381: Deprecate configuration getters/setters that return/set complex Java objects Key: FLINK-33581 URL: https://issues.apache.org/jira/browse/FLINK-33581

[RESULT] [VOTE] flink-shaded 16.2, release candidate #1

I'm happy to announce that we have unanimously approved this release. There are 4 approving votes, 3 of which are binding: * Yuxin Tan(non-binding) * Xintong Song(binding) * Zhu Zhu(binding) * Martijn Visser(binding) There are no disapproving votes. Thanks everyone!

[jira] [Created] (FLINK-33580) composite actions do not work in release branchs where GHA work wasn't backported

Matthias Pohl created FLINK-33580: - Summary: composite actions do not work in release branchs where GHA work wasn't backported Key: FLINK-33580 URL: https://issues.apache.org/jira/browse/FLINK-33580

[jira] [Created] (FLINK-33579) Join sql error

waywtdcc created FLINK-33579: Summary: Join sql error Key: FLINK-33579 URL: https://issues.apache.org/jira/browse/FLINK-33579 Project: Flink Issue Type: Bug Affects Versions: 1.17.1

[jira] [Created] (FLINK-33578) Remove scala-2.12 system variable

Matthias Pohl created FLINK-33578: - Summary: Remove scala-2.12 system variable Key: FLINK-33578 URL: https://issues.apache.org/jira/browse/FLINK-33578 Project: Flink Issue Type: Technical

Re: [VOTE] FLIP-378: Support Avro timestamp with local timezone

+1(binding) Thanks, Matyas On Thu, Nov 16, 2023 at 2:11 AM Jing Ge wrote: > +1(binding) > Thanks! > > Best regards, > Jing > > On Thu, Nov 16, 2023 at 8:46 AM Jark Wu wrote: > > > +1 (binding) > > > > Best, > > Jark > > > > On Thu, 16 Nov 2023 at 12:41, Leonard Xu wrote: > > > > > +1(binding)

[jira] [Created] (FLINK-33577) Make "conf.yaml" as the default Flink configuration file

Junrui Li created FLINK-33577: - Summary: Make "conf.yaml" as the default Flink configuration file Key: FLINK-33577 URL: https://issues.apache.org/jira/browse/FLINK-33577 Project: Flink Issue

[jira] [Created] (FLINK-33576) Introduce new Flink conf file config.yaml supporting standard YAML syntax

Junrui Li created FLINK-33576: - Summary: Introduce new Flink conf file config.yaml supporting standard YAML syntax Key: FLINK-33576 URL: https://issues.apache.org/jira/browse/FLINK-33576 Project: Flink

Re: [DISCUSS] FLIP-364: Improve the restart-strategy

Thank you Rui. It makes sense to me now. On Thu, Nov 16, 2023 at 2:57 AM Rui Fan <1996fan...@gmail.com> wrote: > Hi all, > > Zhu and I had an offline discussion today. We prefer this FLIP > focuses on improving exponential-delay and uses exponential-delay > as the default strategy. It means this

Re: [VOTE] Release 1.16.3, release candidate #1

+1 (non-binding) * Verified signatures * Build from source code, and it looks good * Verified that jar packages are built with maven-3.2.5 and JDK8 * Reviewed the flink-web PR * Start a local standalone cluster and submit examples Best Yun Tang

Re: [DISCUSS] FLIP-389: Annotate SingleThreadFetcherManager and FutureCompletingBlockingQueue as PublicEvolving

Hi Hongshun, Thanks for updating the FLIP. I think that makes sense. A few comments below: 1. If SplitFetcherManager becomes PublicEvolving, that also means SplitFetcher needs to be PublicEvolving, because it is returned by the protected method SplitFetcherManager.createSplitFetcher(). 2. When

Re: [DISCUSS] FLIP-390: Support System out and err to be redirected to LOG or discarded

Hi all, Unless there are some objections or new suggestions, I will open a voting thread next Monday, thanks~ Best, Rui On Wed, Nov 15, 2023 at 4:11 PM Rui Fan <1996fan...@gmail.com> wrote: > Thanks to Hangxiang and Feng's feedback! > > Best, > Rui > > On Wed, Nov 15, 2023 at 3:37 PM Feng Jin

Re: [DISCUSS] FLIP-394: Add Metrics for Connector Agnostic Autoscaling

Hi Mason, Thank you for driving this proposal! Currently, Autoscaler only supports the maximum source parallelism of KafkaSource. Introducing the generic metric to support it is good to me, +1 for this proposal. I have a question: You added the metric in the flink repo, and Autoscaler will

[DISCUSS] FLIP-394: Add Metrics for Connector Agnostic Autoscaling

Hi all, I would like to start a discussion on FLIP-394: Add Metrics for Connector Agnostic Autoscaling [1]. This FLIP recommends adding two metrics to make autoscaling work for bounded split source implementations like IcebergSource. These metrics are required by the Flink Kubernetes Operator

[jira] [Created] (FLINK-33575) FLIP-394: Add Metrics for Connector Agnostic Autoscaling

Mason Chen created FLINK-33575: -- Summary: FLIP-394: Add Metrics for Connector Agnostic Autoscaling Key: FLINK-33575 URL: https://issues.apache.org/jira/browse/FLINK-33575 Project: Flink Issue

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.7.0, release candidate #1

+1 (non-binding) - Checked signatures, checksums - No binaries found in the source release - Verified all source files contain the license header - All pom files point to the correct version - Verified Helm chart version and appVersion - Verified Docker image tag - Ran flink-autoscaler-standalone

Re: [VOTE] Release flink-shaded 16.2, release candidate #1

+1 (binding) - Validated hashes - Verified signature - Verified that no binaries exist in the source archive - Build the source with Maven - Verified licenses - Verified web PRs On Thu, Nov 16, 2023 at 11:40 AM Zhu Zhu wrote: > > +1 (binding) > > - verified checksum and signature > - verified

Re: [DISCUSS] FLIP-386: Support adding custom metrics in Recovery Spans

Thanks for the proposal, Can you please explain: 1. why the existing MetricGroup interface can't be used? It already had methods to add metrics and spans ... 2. IIUC, based on these numbers, we're going to report span(s). Shouldn't the backend report them as spans? 3. How is the implementation

Re: [DISCUSS] FLIP-385: Add OpenTelemetryTraceReporter and OpenTelemetryMetricReporter

Thanks Piotr, the proposal totally makes sense to me. Does it depend on FLIP-384 for voting? Otherwise, we could probably start the vote already as there're no counter proposals or objections. Regards, Roman On Tue, Nov 7, 2023, 1:19 PM Piotr Nowojski wrote: > Hey, sorry for the misclick.

Re: [DISCUSS] FLIP-384: Introduce TraceReporter and use it to create checkpointing and recovery traces

Thanks for the proposal, Starting with the minimal functionality and expanding if necessary as the FLIP describes makes a lot of sense to me. Regards, Roman On Wed, Nov 15, 2023, 9:31 PM Jing Ge wrote: > Hi Piotr, > > Sorry for the late reply and thanks for the proposal, it looks awesome! > >

[jira] [Created] (FLINK-33574) testRecoverAfterMultiplePersistsStateWithMultiPart andtestRecoverAfterMultiplePersistsStateWithMultiPart run into timeouts

Matthias Pohl created FLINK-33574: - Summary: testRecoverAfterMultiplePersistsStateWithMultiPart andtestRecoverAfterMultiplePersistsStateWithMultiPart run into timeouts Key: FLINK-33574 URL:

Re: [DISCUSS] FLIP-393: Make QueryOperations SQL serializable

Yes, the idea is to convert the QueryOperation tree into a proper/compilable query. To be honest I didn't think it could be done differently, sorry if I wasn't clear enough. Yes, it is very much like SqlNode#toSqlString you mentioned. I'll add an example of a single QueryOperation tree to the

Re: [DISCUSS] FLIP-389: Annotate SingleThreadFetcherManager and FutureCompletingBlockingQueue as PublicEvolving

Hi Devs, I have just modified the content of FLIP-389: Annotate SingleThreadFetcherManager as PublicEvolving[1]. Now this Flip mainly do two thing: 1. Annotate SingleThreadFetcherManager as PublicEvolving 2. Remove all public constructors which use FutureCompletingBlockingQueue. This

Re: [DISCUSS] FLIP-393: Make QueryOperations SQL serializable

Sorry that I wasn't expressing it clearly. Since the FLIP talks about two things: ResolvedExpression and QueryOperation, and you have illustrated how to serialize ResolvedExpression into SQL string. I'm wondering how you'll gonna to convert QueryOperation into SQL string. I was thinking that you

Re: [DISCUSS] FLIP-364: Improve the restart-strategy

Hi all, Zhu and I had an offline discussion today. We prefer this FLIP focuses on improving exponential-delay and uses exponential-delay as the default strategy. It means this FLIP doesn't include improvements related to fixed-delay and failover-delay, and the second part of FLIP(Improve

Re: [VOTE] Release flink-shaded 16.2, release candidate #1

+1 (binding) - verified checksum and signature - verified the source archive does not contain any binary - built from source code successfully - checked the web pr Thanks, Zhu Xintong Song 于2023年11月15日周三 10:42写道： > +1 (binding) > > - verified checksum and signature > - verified license and

[jira] [Created] (FLINK-33573) Flink GCP Connector implement retries to handle 429's and 503's

Jayadeep Jayaraman created FLINK-33573: -- Summary: Flink GCP Connector implement retries to handle 429's and 503's Key: FLINK-33573 URL: https://issues.apache.org/jira/browse/FLINK-33573 Project:

[jira] [Created] (FLINK-33572) Minimize ConfigMap API operations for autoscaler state

Maximilian Michels created FLINK-33572: -- Summary: Minimize ConfigMap API operations for autoscaler state Key: FLINK-33572 URL: https://issues.apache.org/jira/browse/FLINK-33572 Project: Flink

Re: [VOTE] FLIP-378: Support Avro timestamp with local timezone

+1(binding) Thanks! Best regards, Jing On Thu, Nov 16, 2023 at 8:46 AM Jark Wu wrote: > +1 (binding) > > Best, > Jark > > On Thu, 16 Nov 2023 at 12:41, Leonard Xu wrote: > > > +1(binding) > > > > Best, > > Leonard > > > > > 2023年11月16日 下午12:13，Mingliang Liu 写道： > > > > > > +1 (non-binding) >

Re: [DISCUSS] FLIP-377: Support configuration to disable filter push down for Table/SQL Sources

Hi Jiabao, Arguments like "because Spark has it so Flink should also have it" does not make sense. Different projects have different API flavors and styles. What is really important is the rationale and the design principle behind the API. They should conform to the convention of the project.

[jira] [Created] (FLINK-33571) Bump json-path from 2.7.0 to 2.8.0

Yubin Li created FLINK-33571: Summary: Bump json-path from 2.7.0 to 2.8.0 Key: FLINK-33571 URL: https://issues.apache.org/jira/browse/FLINK-33571 Project: Flink Issue Type: Bug Affects

[jira] [Created] (FLINK-33570) ChangelogRecoverySwitchStateBackendITCase failed fataly with exit code 127

Matthias Pohl created FLINK-33570: - Summary: ChangelogRecoverySwitchStateBackendITCase failed fataly with exit code 127 Key: FLINK-33570 URL: https://issues.apache.org/jira/browse/FLINK-33570

[jira] [Created] (FLINK-33569) Could not deploy yarn-application when using yarn over s3a filesystem.

Bodong Liu created FLINK-33569: -- Summary: Could not deploy yarn-application when using yarn over s3a filesystem. Key: FLINK-33569 URL: https://issues.apache.org/jira/browse/FLINK-33569 Project: Flink

[jira] [Created] (FLINK-33568) SinkV2MetricsITCase.testCommitterMetrics fails with NullPointerException

Matthias Pohl created FLINK-33568: - Summary: SinkV2MetricsITCase.testCommitterMetrics fails with NullPointerException Key: FLINK-33568 URL: https://issues.apache.org/jira/browse/FLINK-33568 Project:

Re: [DISCUSS] Release 1.17.2

Thanks Rui Fan. > On 16. Nov 2023, at 09:16, Rui Fan <1996fan...@gmail.com> wrote: > > They are included, it can be checked out by Fix Version/s. > > Best, > Rui > > On Thu, Nov 16, 2023 at 4:11 PM Jun Qin > wrote: >> Thanks Yun Tang for driving the release. >>

Re: [DISCUSS] Release Flink 1.16.3

Thanks Rui Fan. > On 16. Nov 2023, at 09:14, Rui Fan <1996fan...@gmail.com> wrote: > > They are included. > > Best, > Rui > > On Thu, Nov 16, 2023 at 4:12 PM Jun Qin wrote: > >> Thanks Rui Fan for driving the release. >> >> Just wanted to double check whether the following two fixes will be

[jira] [Created] (FLINK-33567) Flink documentation should only display connector downloads links when a connector is available

Martijn Visser created FLINK-33567: -- Summary: Flink documentation should only display connector downloads links when a connector is available Key: FLINK-33567 URL:

Re: [DISCUSS] FLIP-393: Make QueryOperations SQL serializable

I think the FLIP covers all public contracts that are necessary to be discussed at that level. If you meant you could not find a method that would be called to trigger the translation then it is already there. It's just not implemented yet: QueryOperation#asSerializableString[1]. As I mentioned

Re: [DISCUSS] Release 1.17.2

They are included, it can be checked out by Fix Version/s. Best, Rui On Thu, Nov 16, 2023 at 4:11 PM Jun Qin wrote: > Thanks Yun Tang for driving the release. > > Just wanted to double check whether the following two fixes will be > included in 1.17.2 release: > >

Re: [DISCUSS] Release Flink 1.16.3

They are included. Best, Rui On Thu, Nov 16, 2023 at 4:12 PM Jun Qin wrote: > Thanks Rui Fan for driving the release. > > Just wanted to double check whether the following two fixes will be > included in 1.16.3 release: > > https://issues.apache.org/jira/browse/FLINK-30559 >

Re: [DISCUSS] Release Flink 1.16.3

Thanks Rui Fan for driving the release. Just wanted to double check whether the following two fixes will be included in 1.16.3 release: https://issues.apache.org/jira/browse/FLINK-30559 https://issues.apache.org/jira/browse/FLINK-30966 Thanks Jun > On 9. Nov 2023, at 05:17, Rui Fan

Re: [DISCUSS] Release 1.17.2

Thanks Yun Tang for driving the release. Just wanted to double check whether the following two fixes will be included in 1.17.2 release: https://issues.apache.org/jira/browse/FLINK-30559 https://issues.apache.org/jira/browse/FLINK-30966 Thanks Jun > On 9. Nov 2023, at 04:48, Yun Tang wrote: