Re: Re: Re: [Discuss]- Donate Iceberg Flink Connector

2022-10-11 Thread Péter Váry
Thanks Abid, Count me in, and drop a note, if I can help in any way. Thanks, Peter On Tue, Oct 11, 2022, 20:13 wrote: > Hi Martijn, > > Yes catalog integration exists and catalogs can be created using Flink > SQL. > >

Re: State of the Rescale API

2022-10-11 Thread Jiangang Liu
Thanks for the attention to the rescale api. Dynamic resource adjust is useful for streaming jobs since the throughput can change in different time. The rescale api is a lightweight way to change the job's parallelism. This is importance for some jobs, for example, the jobs are in activities or

[jira] [Created] (FLINK-29590) Fix constant fold issue for HiveDialect

2022-10-11 Thread luoyuxia (Jira)
luoyuxia created FLINK-29590: Summary: Fix constant fold issue for HiveDialect Key: FLINK-29590 URL: https://issues.apache.org/jira/browse/FLINK-29590 Project: Flink Issue Type: Bug

Re: [ANNOUNCE] New Apache Flink PMC Member - Danny Cranmer

2022-10-11 Thread yuxia
Congratulations Danny! Best regards, Yuxia - 原始邮件 - 发件人: "Xingbo Huang" 收件人: "dev" 发送时间: 星期三, 2022年 10 月 12日 上午 9:44:22 主题: Re: [ANNOUNCE] New Apache Flink PMC Member - Danny Cranmer Congratulations Danny! Best, Xingbo Sergey Nuyanzin 于2022年10月12日周三 01:26写道: > Congratulations,

Re: [ANNOUNCE] New Apache Flink PMC Member - Danny Cranmer

2022-10-11 Thread Xingbo Huang
Congratulations Danny! Best, Xingbo Sergey Nuyanzin 于2022年10月12日周三 01:26写道: > Congratulations, Danny > > On Tue, Oct 11, 2022, 15:18 Lincoln Lee wrote: > > > Congratulations Danny! > > > > Best, > > Lincoln Lee > > > > > > Congxian Qiu 于2022年10月11日周二 19:42写道: > > > > > Congratulations Danny!

Re: [DISCUSS] Updating Flink HBase Connectors

2022-10-11 Thread Ferenc Csaky
Hi Martijn, Thank you for your comment. About HBase 2.x, correct, that is my thought process, but it has to be tested and verified. +1 from my side about merging these updates with the connector externalization. Best, F --- Original Message --- On Tuesday, October 11th, 2022 at

[jira] [Created] (FLINK-29589) Data Loss in Sink GlobalCommitter during Task Manager recovery

2022-10-11 Thread Krzysztof Chmielewski (Jira)
Krzysztof Chmielewski created FLINK-29589: - Summary: Data Loss in Sink GlobalCommitter during Task Manager recovery Key: FLINK-29589 URL: https://issues.apache.org/jira/browse/FLINK-29589

Re: [DISCUSS] Updating Flink HBase Connectors

2022-10-11 Thread Martijn Visser
Hi Ferenc, Thanks for opening the discussion on this topic! +1 for dropping HBase 1.x. Regarding HBase 2.x, if I understand correctly it should be possible to connect to any 2.x cluster if you're using the 2.x client. Wouldn't it make more sense to always support the latest available version,

RE: Re: Re: [Discuss]- Donate Iceberg Flink Connector

2022-10-11 Thread abmo . work
Hi Martijn, Yes catalog integration exists and catalogs can be created using Flink SQL. https://iceberg.apache.org/docs/latest/flink/#creating-catalogs-and-using-catalogs has more details. We may need some discussion within Iceberg community but based on the current iceberg-flink code

[jira] [Created] (FLINK-29588) Add Flink Version and Application Version to Savepoint properties

2022-10-11 Thread Clara Xiong (Jira)
Clara Xiong created FLINK-29588: --- Summary: Add Flink Version and Application Version to Savepoint properties Key: FLINK-29588 URL: https://issues.apache.org/jira/browse/FLINK-29588 Project: Flink

Re: [ANNOUNCE] New Apache Flink PMC Member - Danny Cranmer

2022-10-11 Thread Sergey Nuyanzin
Congratulations, Danny On Tue, Oct 11, 2022, 15:18 Lincoln Lee wrote: > Congratulations Danny! > > Best, > Lincoln Lee > > > Congxian Qiu 于2022年10月11日周二 19:42写道: > > > Congratulations Danny! > > > > Best, > > Congxian > > > > > > Leonard Xu 于2022年10月11日周二 18:03写道: > > > > > Congratulations

Re: [ANNOUNCE] New Apache Flink PMC Member - Danny Cranmer

2022-10-11 Thread Lincoln Lee
Congratulations Danny! Best, Lincoln Lee Congxian Qiu 于2022年10月11日周二 19:42写道: > Congratulations Danny! > > Best, > Congxian > > > Leonard Xu 于2022年10月11日周二 18:03写道: > > > Congratulations Danny! > > > > > > Best, > > Leonard > > > > >

Re: State of the Rescale API

2022-10-11 Thread Chesnay Schepler
The AdaptiveScheduler is not limited to reactive mode. There are no deployment limitations for the scheduler itself. In a nutshell, all that reactive mode does is crank the target parallelism to infinity, when usually it is the parallelism the user has set in the job/configuration. I think it

[jira] [Created] (FLINK-29587) Fail to generate code for SearchOperator

2022-10-11 Thread luoyuxia (Jira)
luoyuxia created FLINK-29587: Summary: Fail to generate code for SearchOperator Key: FLINK-29587 URL: https://issues.apache.org/jira/browse/FLINK-29587 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-29586) Let write buffer spillable

2022-10-11 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-29586: Summary: Let write buffer spillable Key: FLINK-29586 URL: https://issues.apache.org/jira/browse/FLINK-29586 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-29585) Migrate TableSchema to Schema for Hive connector

2022-10-11 Thread luoyuxia (Jira)
luoyuxia created FLINK-29585: Summary: Migrate TableSchema to Schema for Hive connector Key: FLINK-29585 URL: https://issues.apache.org/jira/browse/FLINK-29585 Project: Flink Issue Type:

[jira] [Created] (FLINK-29584) Upgrade java 11 version on the microbenchmark worker

2022-10-11 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-29584: -- Summary: Upgrade java 11 version on the microbenchmark worker Key: FLINK-29584 URL: https://issues.apache.org/jira/browse/FLINK-29584 Project: Flink

[jira] [Created] (FLINK-29583) Ensure correct subtaskId and checkpointId is set during committer state migration

2022-10-11 Thread Fabian Paul (Jira)
Fabian Paul created FLINK-29583: --- Summary: Ensure correct subtaskId and checkpointId is set during committer state migration Key: FLINK-29583 URL: https://issues.apache.org/jira/browse/FLINK-29583

Re: [ANNOUNCE] New Apache Flink PMC Member - Danny Cranmer

2022-10-11 Thread Congxian Qiu
Congratulations Danny! Best, Congxian Leonard Xu 于2022年10月11日周二 18:03写道: > Congratulations Danny! > > > Best, > Leonard > >

[jira] [Created] (FLINK-29582) SavepointWriter should be usable without any transformation

2022-10-11 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-29582: Summary: SavepointWriter should be usable without any transformation Key: FLINK-29582 URL: https://issues.apache.org/jira/browse/FLINK-29582 Project: Flink

[jira] [Created] (FLINK-29581) Emit warning events for session job reconciliation exception

2022-10-11 Thread Xin Hao (Jira)
Xin Hao created FLINK-29581: --- Summary: Emit warning events for session job reconciliation exception Key: FLINK-29581 URL: https://issues.apache.org/jira/browse/FLINK-29581 Project: Flink Issue

[jira] [Created] (FLINK-29580) pulsar.consumer.autoUpdatePartitionsIntervalSeconds doesn't works and should be removed

2022-10-11 Thread Yufan Sheng (Jira)
Yufan Sheng created FLINK-29580: --- Summary: pulsar.consumer.autoUpdatePartitionsIntervalSeconds doesn't works and should be removed Key: FLINK-29580 URL: https://issues.apache.org/jira/browse/FLINK-29580

Re: [DISCUSS] Reverting sink metric name changes made in 1.15

2022-10-11 Thread Chesnay Schepler
Currently I think that would be a mistake. Ultimately what we have here is the culmination of us never really considering how the numRecordsOut metric should behave for operators that emit data to other operators _and_ external systems. This goes beyond sinks. This even applies to

Re: [ANNOUNCE] New Apache Flink PMC Member - Danny Cranmer

2022-10-11 Thread Leonard Xu
Congratulations Danny! Best, Leonard

[jira] [Created] (FLINK-29579) Flink parquet reader cannot read fully optional elements in a repeated list

2022-10-11 Thread Tiansu Yu (Jira)
Tiansu Yu created FLINK-29579: - Summary: Flink parquet reader cannot read fully optional elements in a repeated list Key: FLINK-29579 URL: https://issues.apache.org/jira/browse/FLINK-29579 Project: Flink

[jira] [Created] (FLINK-29578) Multiple executions of `runVertexCentricIteration` cause memory errors (flink gelly)

2022-10-11 Thread Shunyang Li (Jira)
Shunyang Li created FLINK-29578: --- Summary: Multiple executions of `runVertexCentricIteration` cause memory errors (flink gelly) Key: FLINK-29578 URL: https://issues.apache.org/jira/browse/FLINK-29578

[jira] [Created] (FLINK-29577) Disable rocksdb wal when restore from full snapshot

2022-10-11 Thread Cai Liuyang (Jira)
Cai Liuyang created FLINK-29577: --- Summary: Disable rocksdb wal when restore from full snapshot Key: FLINK-29577 URL: https://issues.apache.org/jira/browse/FLINK-29577 Project: Flink Issue

Re: Re: [Discuss]- Donate Iceberg Flink Connector

2022-10-11 Thread Martijn Visser
Hi Abid, Thanks for the FLIP. I have a question about Iceberg's Catalog: has that integration between Flink and Iceberg been created already and are you looking to externalize that as well? Thanks, Martijn On Tue, Oct 11, 2022 at 12:14 AM wrote: > Hi Marton, > > Yes, we are initiating this

Re: [VOTE] Apache Flink Table Store 0.2.1, release candidate #2

2022-10-11 Thread Yu Li
+1 (binding) - Checked release notes: *Action Required* * The fix version of FLINK-29554 is 0.2.1 but still open, please confirm whether this should be included or we should move it out of 0.2.1 - Checked sums and signatures: *OK* - Checked the jars in the staging repo: *OK* - Checked source

[SUMMARY] Flink 1.16 release sync of 2022-10-11

2022-10-11 Thread Xingbo Huang
I would like to give you a brief update of the Flink 1.16 release sync meating of 2022-10-11. *rc1 is canceled because we found a metric compatibility change, which is expected to be fixed before Thursday https://issues.apache.org/jira/browse/FLINK-29567. We will prepare rc2 once this blocker

[jira] [Created] (FLINK-29576) SourceNAryInputChainingITCase.testDirectSourcesOnlyExecution failed

2022-10-11 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-29576: - Summary: SourceNAryInputChainingITCase.testDirectSourcesOnlyExecution failed Key: FLINK-29576 URL: https://issues.apache.org/jira/browse/FLINK-29576 Project: Flink

Flink falls back on to kryo serializer for GenericTypes

2022-10-11 Thread Sucheth S
Hello, How to avoid flink's kryo serializer for GenericTypes ? Kryo is having some performance issues. Tried below but no luck. env.getConfig().disableForceKryo(); env.getConfig().enableForceAvro(); Tried this - env.getConfig().disableGenericTypes(); getting - Generic types have been disabled