Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

Hi David, Thanks for the comments from you and @Piotr. I'd like to explain the details about the FLIP first. 1) Global Checkpoint Commit: a) "rolling fashion" or b) altogether This mainly depends on the needs of users. Users can decide the data version of tables in their queries according to

[jira] [Created] (FLINK-30395) Refactor module name and documentation for filesystems

Jingsong Lee created FLINK-30395: Summary: Refactor module name and documentation for filesystems Key: FLINK-30395 URL: https://issues.apache.org/jira/browse/FLINK-30395 Project: Flink Issue

[jira] [Created] (FLINK-30394) [umbrella] Refactor filesystem support in table store

Jingsong Lee created FLINK-30394: Summary: [umbrella] Refactor filesystem support in table store Key: FLINK-30394 URL: https://issues.apache.org/jira/browse/FLINK-30394 Project: Flink Issue

[jira] [Created] (FLINK-30393) PulsarSourceUnorderedE2ECase.testScaleDown failed with TimeoutException

Qingsheng Ren created FLINK-30393: - Summary: PulsarSourceUnorderedE2ECase.testScaleDown failed with TimeoutException Key: FLINK-30393 URL: https://issues.apache.org/jira/browse/FLINK-30393 Project:

[jira] [Created] (FLINK-30392) Increase the default value of cluster.thread-dump.stacktrace-max-depth

Yun Tang created FLINK-30392: Summary: Increase the default value of cluster.thread-dump.stacktrace-max-depth Key: FLINK-30392 URL: https://issues.apache.org/jira/browse/FLINK-30392 Project: Flink

[jira] [Created] (FLINK-30390) Ensure that no compaction is in progress before closing the writer

Jingsong Lee created FLINK-30390: Summary: Ensure that no compaction is in progress before closing the writer Key: FLINK-30390 URL: https://issues.apache.org/jira/browse/FLINK-30390 Project: Flink

[jira] [Created] (FLINK-30389) Add retry to read hints

Jingsong Lee created FLINK-30389: Summary: Add retry to read hints Key: FLINK-30389 URL: https://issues.apache.org/jira/browse/FLINK-30389 Project: Flink Issue Type: Improvement

[VOTE] Release flink-connector-pulsar v3.0.0, release candidate #2

Hi everyone, Please review and vote on the release candidate #2 for the flink-connector-pulsar version v3.0.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) Note: this is equivalent to the Pulsar connector that was released with

Edit permission

I didn’t provide my cwiki username☹. It’s tedhtchang.

[jira] [Created] (FLINK-30388) Add support for ElementConverted open() method for KDS/KDF/DDB

Danny Cranmer created FLINK-30388: - Summary: Add support for ElementConverted open() method for KDS/KDF/DDB Key: FLINK-30388 URL: https://issues.apache.org/jira/browse/FLINK-30388 Project: Flink

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.3.0, release candidate #1

+1 (binding) Source release looks good. 1. Downloaded, compiled, and verified the signature of the source release staged at https://dist.apache.org/repos/dist/dev/flink/flink-kubernetes-operator-1.3.0-rc1/ 2.

[jira] [Created] (FLINK-30387) support job singleton within TM

James Z.M. Gao created FLINK-30387: -- Summary: support job singleton within TM Key: FLINK-30387 URL: https://issues.apache.org/jira/browse/FLINK-30387 Project: Flink Issue Type: New Feature

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

Hi Shammon, I'm starting to see what you're trying to achieve, and it's really exciting. I share Piotr's concerns about e2e latency and disability to use unaligned checkpoints. I have a couple of questions that are not clear to me from going over the FLIP: 1) Global Checkpoint Commit Are you

[jira] [Created] (FLINK-30386) Column primary key constraint lacks enforced check

Jane Chan created FLINK-30386: - Summary: Column primary key constraint lacks enforced check Key: FLINK-30386 URL: https://issues.apache.org/jira/browse/FLINK-30386 Project: Flink Issue Type: Bug

Re: [DISCUSS] Release new FRocksDB

Hi Yanfei, Makes sense, arm architecture support would be great. Thanks for looking into this. On Mon, Dec 12, 2022 at 3:03 PM Yanfei Lei wrote: > Hi devs, > > I'd like to bring up a discussion about releasing the new frocksdbjni > version, we are planning to adapt frocksdbjni to Apple M1

[jira] [Created] (FLINK-30385) Remove KDS/KDF/AWS Base connectors from master branch

Danny Cranmer created FLINK-30385: - Summary: Remove KDS/KDF/AWS Base connectors from master branch Key: FLINK-30385 URL: https://issues.apache.org/jira/browse/FLINK-30385 Project: Flink

[jira] [Created] (FLINK-30383) UseLogicalIdentifier makes datadog consider metric as custom

Chesnay Schepler created FLINK-30383: Summary: UseLogicalIdentifier makes datadog consider metric as custom Key: FLINK-30383 URL: https://issues.apache.org/jira/browse/FLINK-30383 Project: Flink

[jira] [Created] (FLINK-30384) Sync master KDS/KDF connector changes to flink-connector-aws

Danny Cranmer created FLINK-30384: - Summary: Sync master KDS/KDF connector changes to flink-connector-aws Key: FLINK-30384 URL: https://issues.apache.org/jira/browse/FLINK-30384 Project: Flink

[jira] [Created] (FLINK-30382) Flink 1.16 to integrate KDS/KDF docs from flink-connector-aws

Danny Cranmer created FLINK-30382: - Summary: Flink 1.16 to integrate KDS/KDF docs from flink-connector-aws Key: FLINK-30382 URL: https://issues.apache.org/jira/browse/FLINK-30382 Project: Flink

[DISCUSS] Release new FRocksDB

Hi devs, I'd like to bring up a discussion about releasing the new frocksdbjni version, we are planning to adapt frocksdbjni to Apple M1 machines in Flink 1.17[1], a new frocksdbjni jar needs to be released. Thanks to the efforts of the community, FLINK-24932[1] has been merged into

[jira] [Created] (FLINK-30381) Bump minikube kubernetes version for operator

Gabor Somogyi created FLINK-30381: - Summary: Bump minikube kubernetes version for operator Key: FLINK-30381 URL: https://issues.apache.org/jira/browse/FLINK-30381 Project: Flink Issue Type:

[jira] [Created] (FLINK-30380) Look into enable parallel compilation

Chesnay Schepler created FLINK-30380: Summary: Look into enable parallel compilation Key: FLINK-30380 URL: https://issues.apache.org/jira/browse/FLINK-30380 Project: Flink Issue Type:

[jira] [Created] (FLINK-30379) Decouple connector docs integration from connector releases

Chesnay Schepler created FLINK-30379: Summary: Decouple connector docs integration from connector releases Key: FLINK-30379 URL: https://issues.apache.org/jira/browse/FLINK-30379 Project: Flink

Re: [DISCUSS] FLIP-277: Native GlueCatalog Support in Flink

Hi Konstantin Knauf, Can you explain how users are expected to authenticate with AWS Glue? I > don't see any catalog options regardng authx. So I assume the credentials > are taken from the environment? We are planning to put GlueCatalog in flink-connector-aws[1]. flink-connector-aws already

[jira] [Created] (FLINK-30378) Rework sql_download_table shortcode

Chesnay Schepler created FLINK-30378: Summary: Rework sql_download_table shortcode Key: FLINK-30378 URL: https://issues.apache.org/jira/browse/FLINK-30378 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-277: Native GlueCatalog Support in Flink

Hi Jark, Apologies for late reply. Thank you for your valuable input. Besides, I have a question about Glue Namespace. Could you share the > documentation of the Glue > Namespaces? (Sorry, I didn't find it.) According to the "Flink Glue > Metaspace Mapping" section, > if there is a database

[jira] [Created] (FLINK-30377) CommonExecSink does not use ClientWrapperClassLoader while extracting Type from KeyedStream

Prabhu Joseph created FLINK-30377: - Summary: CommonExecSink does not use ClientWrapperClassLoader while extracting Type from KeyedStream Key: FLINK-30377 URL: https://issues.apache.org/jira/browse/FLINK-30377

[jira] [Created] (FLINK-30376) Introduce a new flink busy join reorder rule which based on greedy algorithm

Yunhong Zheng created FLINK-30376: - Summary: Introduce a new flink busy join reorder rule which based on greedy algorithm Key: FLINK-30376 URL: https://issues.apache.org/jira/browse/FLINK-30376

[jira] [Created] (FLINK-30375) SqlClient leaks flink-table-planner jar under /tmp

Prabhu Joseph created FLINK-30375: - Summary: SqlClient leaks flink-table-planner jar under /tmp Key: FLINK-30375 URL: https://issues.apache.org/jira/browse/FLINK-30375 Project: Flink Issue

[ANNOUNCE] Apache flink-connector-aws v4.0.0 released

The Apache Flink community is very happy to announce the release of Apache flink-connector-aws 4.0.0. This release contains the Amazon Kinesis Data Streams/Firehose and Amazon DynamoDB connectors with support for Flink 1.16. Apache Flink® is an open-source stream processing framework for

Re: [DISCUSS] Hybrid Source Connector

Hi Ran, Thanks for proposing a FLIP. Btw according to the process, the subject of this email should be `[DISCUSS] FLIP-278: Hybrid Source Connector` so that people can identify this discussion as a FLIP discussion. Supporting the hybrid source for SQL was a long-standing issue on our

[jira] [Created] (FLINK-30374) DefaultExecutionGraphCacheTest fails with 239 exit code

Leonard Xu created FLINK-30374: -- Summary: DefaultExecutionGraphCacheTest fails with 239 exit code Key: FLINK-30374 URL: https://issues.apache.org/jira/browse/FLINK-30374 Project: Flink Issue

Re: [DISCUSS] FLIP-275: Support Remote SQL Client Based on SQL Gateway

Hi everyone, sorry to jump into this discussion so late. > So we decided to revert the RowFormat related changes and let the client to resolve the print format. Could you elaborate a bit on this topic in the FLIP? I still believe that we need 2 types of output formats. Format A: for the

Re: [DISCUSS] Externalizing Kafka connector

Hi Mason, great, thanks a lot for working on this. Will greatly speed up CI of the core repository once this is finalized. Cheers Konstantin Am Mi., 7. Dez. 2022 um 07:23 Uhr schrieb Mason Chen : > Hi all, > > I've finished the first pass on externalizing the Kafka connector under the >

Re: [DISCUSS] FLIP-277: Native GlueCatalog Support in Flink

Hi Samrat, +1 to the effort and +1 to adding it to flink-connector-aws. Can you explain how users are expected to authenticate with AWS Glue? I don't see any catalog options regardng authx. So I assume the credentials are taken from the environment? Best, Konstantin Am Fr., 9. Dez. 2022 um

[jira] [Created] (FLINK-30373) Flink-table-runtime free for flink-table-store-codegen

Jingsong Lee created FLINK-30373: Summary: Flink-table-runtime free for flink-table-store-codegen Key: FLINK-30373 URL: https://issues.apache.org/jira/browse/FLINK-30373 Project: Flink Issue

[jira] [Created] (FLINK-30372) dependencyManagement bom for flink-dist

James Z.M. Gao created FLINK-30372: -- Summary: dependencyManagement bom for flink-dist Key: FLINK-30372 URL: https://issues.apache.org/jira/browse/FLINK-30372 Project: Flink Issue Type: New

[jira] [Created] (FLINK-30371) JdbcOutputFormat is at risk of database connection leaks

Echo Lee created FLINK-30371: Summary: JdbcOutputFormat is at risk of database connection leaks Key: FLINK-30371 URL: https://issues.apache.org/jira/browse/FLINK-30371 Project: Flink Issue Type:

[jira] [Created] (FLINK-30370) Move existing delegation token framework authentication to providers

Gabor Somogyi created FLINK-30370: - Summary: Move existing delegation token framework authentication to providers Key: FLINK-30370 URL: https://issues.apache.org/jira/browse/FLINK-30370 Project:

Re: [VOTE] Release flink-connector-jdbc v3.0.0, release candidate #1

On 09/12/2022 16:12, Martijn Visser wrote: That was because the number of changes between 1.16.0 and master were limited. Doesn't the filter pushdown in FLINK-16024 affect topologies and thus potentially savepoint compatibility?

Re: [DISCUSSION] Scan mode in Table Store for Flink Stream and Batch job

Hi jinsong, caizhi Thank you for your reply. I found that what really confused me was the starting position of the streaming job to read data. As @caizhi mentioned in kafka, streaming jobs only read incremental data and its definition is very clear. In the Table Store, streaming jobs also have