[RESULT][VOTE] FLIP-309: Support using larger checkpointing interval when source is processing backlog

Hi all, Thanks everyone for your review and the votes! I am happy to announce that FLIP-309: Support using larger checkpointing interval when source is processing backlog [1] has been accepted. There are 11 binding votes and 1 non-binding vote [2]: - Rui Fan (binding) - Jing Ge (binding) -

Re: [DISCUSS][2.0] FLIP-347: Remove IOReadableWritable serialization in Path

+1 Best, Xintong On Fri, Jul 21, 2023 at 10:54 AM Wencong Liu wrote: > Hi devs, > > I would like to start a discussion on FLIP-347: Remove IOReadableWritable > serialization in Path [1]. > > > The Path class is currently mutable to support IOReadableWritable > serialization. However, many

Re: [DISCUSS][2.0] FLIP-344: Remove parameter in RichFunction#open

+1 Best, Xintong On Fri, Jul 21, 2023 at 10:52 AM Wencong Liu wrote: > Hi devs, > > I would like to start a discussion on FLIP-344: Remove parameter in > RichFunction#open [1]. > > The open() method in RichFunction requires a Configuration instance as an > argument, > which is always passed

Re: [DISCUSS][2.0] FLIP-343: Remove parameter in WindowAssigner#getDefaultTrigger()

+1 Best, Xintong On Fri, Jul 21, 2023 at 10:53 AM Wencong Liu wrote: > Hi devs, > > I would like to start a discussion on FLIP-343: Remove parameter in > WindowAssigner#getDefaultTrigger() [1]. > > > The method getDefaultTrigger() in WindowAssigner takes a > StreamExecutionEnvironment >

Re: Re: [DISCUSS] Release 2.0 Work Items

Thanks for the inputs, Matthias, - FLINK-4503: Yes, this should be subsumed by "Deprecated methods/fields/classes in DataStream", which doesn't really need any action in 1.18. Sorry for overlooking it. - FLINK-5875: Based on the JIRA descriptions, it seems this only makes sense if we want to

[jira] [Created] (FLINK-32640) Hard backpressure time should also be measured in hybrid shuffle

Weijie Guo created FLINK-32640: -- Summary: Hard backpressure time should also be measured in hybrid shuffle Key: FLINK-32640 URL: https://issues.apache.org/jira/browse/FLINK-32640 Project: Flink

Re: Rocksdb info

> This is the case. Yanfei is correct that the uploading happens during the checkpoint stage. Before checkpointing, there is a snapshotting stage that synchronously takes RocksDB snapshot by forcing all memtables into sstables on disk, and hard-linked in a local temporary directory. In the

Re: [DISUCSS] Deprecate multiple APIs in 1.18

Hi, Jing, I share the same opinion with Xintong. The base class `TimestampExtractor` for these two classes was deprecated in release-1.12[1][2], so deprecating these two subclasses is relatively straightforward. [1] https://issues.apache.org/jira/browse/FLINK-19453 [2]

Re: Kubernetes Operator 1.6.0 release planning

Thanks, Gyula. +1 (non-binding) Leonard Xu 于2023年7月20日周四 19:50写道： > +1 for the release and Gyula as release manager. > > Best, > Leonard > > > On Jul 20, 2023, at 7:45 PM, Paul Lam wrote: > > > > Thanks, Gyula. > > > > +1 (non-binding) > > > > Best, > > Paul Lam > > > >> 2023年7月20日

Re: [DISCUSS][2.0] FLIP-337: Remove JarRequestBody#programArgs

+1 @Chesnay ScheplerI'd like to be assigned the task to try to do it ~ Jing Ge 于2023年7月18日周二 17:54写道： > got it thanks! > > For @Deprecated, I meant to force using like: @Deprecated(since = "1.18", > forRemoval = true) > > Best regards, > Jing > > On Tue, Jul 18, 2023 at 11:06 AM Hong Teoh

Re: Rocksdb info

Hi Taher, > is flink using rocks-cloud too? Or flink is moving rocks files after checkpoints? Flink isn't using rocks-cloud. flink would upload local rocks files to s3/DFS during checkpointing. Here is the related code: - RocksNativeFullSnapshotStrategy:

Re: Rocksdb info

> flink is moving rocks > files after checkpoints This is the case. You can read the code and docs around EmbeddedRocksDBStateBackend and FileSystemCheckpointStorage. > Please share N.B. An imperative tone can read uncomfortable. Maybe it's not your intention but this read like a command.

[DISCUSS][2.0] FLIP-347: Remove IOReadableWritable serialization in Path

Hi devs, I would like to start a discussion on FLIP-347: Remove IOReadableWritable serialization in Path [1]. The Path class is currently mutable to support IOReadableWritable serialization. However, many parts of the code assume that the Path is immutable. By making the Path class

[DISCUSS][2.0] FLIP-343: Remove parameter in WindowAssigner#getDefaultTrigger()

Hi devs, I would like to start a discussion on FLIP-343: Remove parameter in WindowAssigner#getDefaultTrigger() [1]. The method getDefaultTrigger() in WindowAssigner takes a StreamExecutionEnvironment parameter, but this parameter is not actually used for any subclasses of WindowAssigner.

[DISCUSS][2.0] FLIP-344: Remove parameter in RichFunction#open

Hi devs, I would like to start a discussion on FLIP-344: Remove parameter in RichFunction#open [1]. The open() method in RichFunction requires a Configuration instance as an argument, which is always passed as a new instance without any configuration parameters in

Re: [VOTE] Release 2.0 must-have work items

Hi Alexander > I see your concerns regarding the complexity of migration, but we still > have one year to address them. Not only the complexity of migration, but also we lack migration path for now. We have to deprecate SourceFunction/SinkFunction in 1.18 which feature freeze date is

Rocksdb info

Hi all, I know that Flink uses rocksdb to write checkpoints. While researching rocksdb i found that there is something called rocks-cloud that writes data over to s3. I know flink also does checkpoints over rocksdb to s3. Question is, is flink using rocks-cloud too? Or flink is moving

Re: [DISCUSS] FLIP-346: Deprecate ManagedTable related APIs

Hi Leonard, I got your point and adjusted the FLIP to cover this. Thanks so much. Best, Jane On Fri, Jul 21, 2023 at 10:01 AM Leonard Xu wrote: > > Hi, Jane > > I mean these pages, > https://nightlies.apache.org/flink/flink-table-store-docs-stable/ < >

Re: [DISUCSS] Deprecate multiple APIs in 1.18

I personally don't think it's necessary. IIUC, we have already decided to deprecate TableSource / TableSink, and not marking StreamRecordTimestamp and ExistingField deprecated is just an oversight in carrying out that decision. Best, Xintong On Fri, Jul 21, 2023 at 7:48 AM Jing Ge wrote: >

Re: [VOTE] Release 2.0 must-have work items

Hi Alex, As I replied in the FLINK-28046, I don't think it's fair to resolve the migration complexity issues after marking them deprecated, because that would practically shorten the migration period. But that can be discussed. Given that eliminating the removal of SourceFunction was proposed 10

Re: [DISCUSS] FLIP-346: Deprecate ManagedTable related APIs

Hi, Jane I mean these pages, https://nightlies.apache.org/flink/flink-table-store-docs-stable/ Best, Leonard > On Jul 20, 2023, at 8:03 PM, Jane Chan wrote: > > Hi, all, > > Thanks for your opinions. > > @Leonard >> We

Re: FLIP-342: Remove brackets around keys returned by MetricGroup#getAllVariables

Hi all, I agree getScopeVariables is in line with the existing terminology but I’ve seen that the existing terminology is a bit confusing with regards to how users end up querying these metrics and building alerts/dashboards. I often get the question how do I add a tag or label to my Flink

Re: [DISUCSS] Deprecate multiple APIs in 1.18

Hi Jane, Hi Xintong, The specific discussion of deprecating `StreamRecordTimestamp` & `ExistingField` was kind of hiding in the discussion of the umbrella topic that people might miss out. Does it make sense to start an individual Discussion thread for it like we deprecate any other APIs? For

Re: [DISCUSS] Connectors, Formats, and even User Code should also be pluggable.

Hi Zhiqiang, Thanks for your proposal. The idea is very interesting! Since almost all connectors are or will be externalized, the pluggable design you suggested could help reduce the development effort. As you mentioned, the attached doc contains only your preliminary idea. I would suggest that

Re: [VOTE] FLIP-309: Support using larger checkpointing interval when source is processing backlog

+1 (binding) Thanks, Stefan > On 20. Jul 2023, at 05:24, Zhu Zhu wrote: > > +1 (binding) > > Thanks, > Zhu > > yuxia mailto:luoyu...@alumni.sjtu.edu.cn>> > 于2023年7月20日周四 09:23写道： >> >> +1（binding) >> >> Best regards, >> Yuxia >> >> - 原始邮件 - >> 发件人: "Guowei Ma" >> 收件人: "dev" >>

Re: [VOTE] Release 2.0 must-have work items

@Leonard I see your concerns regarding the complexity of migration, but we still have one year to address them. @Xintong I believe it makes sense to keep it in the list and mark it as a stretch goal if you have concerns with the "must have" label. Best, Alexander On Thu, 20 Jul 2023 at 11:25,

[DISCUSS] Add missing visibility annotation for Table APIs

Hi, Devs, I would like to start a discussion on adding missing visibility annotation (PublicEvolving / Internal etc.) for Table APIs. The motivation for starting this discussion was during the cleanup of which Table API to be deprecated for version 2.0, I noticed that some of the APIs lack

Re: [DISCUSS] FLIP-327: Support stream-batch unified operator to improve job throughput when processing backlog data

Hi Piotr, Thank you for the very detailed comments! Please see my reply inline. On Thu, Jul 20, 2023 at 12:24 AM Piotr Nowojski wrote: > Hi Dong, > > I have a couple of follow up questions about switching back and forth > between streaming and batching mode. > Especially around

[jira] [Created] (FLINK-32639) Filter and Limit exist at the same time, limit cannot take effect

Dove created FLINK-32639: Summary: Filter and Limit exist at the same time, limit cannot take effect Key: FLINK-32639 URL: https://issues.apache.org/jira/browse/FLINK-32639 Project: Flink Issue

[jira] [Created] (FLINK-32638) CI build failed because e2e_1_ci throw error Bash exited with code '1'

Yunhong Zheng created FLINK-32638: - Summary: CI build failed because e2e_1_ci throw error Bash exited with code '1' Key: FLINK-32638 URL: https://issues.apache.org/jira/browse/FLINK-32638 Project:

Re: Re: [DISCUSS] Release 2.0 Work Items

Sorry for the late reply in that matter. I was off the last few days. I should have made this clear in the ML. Anyway, I went over the issues as well. Xintong's summary matches more or less my findings aside from the following items: - FLINK-4503 (remove deprecated methods from CoGroupedStreams

Re: [DISCUSS] FLIP-346: Deprecate ManagedTable related APIs

Hi, all, Thanks for your opinions. @Leonard > We can remove related document as well, Keeping a external link to Apache Paimon should be enough. Thanks Leonard, for the kind reminder. I checked the JIRA[1] related to FLIP-188, and there is no issue related to documentation. Meanwhile, the

Re: Kubernetes Operator 1.6.0 release planning

+1 for the release and Gyula as release manager. Best, Leonard > On Jul 20, 2023, at 7:45 PM, Paul Lam wrote: > > Thanks, Gyula. > > +1 (non-binding) > > Best, > Paul Lam > >> 2023年7月20日 19:18，Márton Balassi 写道： >> >> Thanks, Gyula. >> >> +1 (binding) >> >> On Thu, Jul 20, 2023 at 5:01 

Re: Kubernetes Operator 1.6.0 release planning

Thanks, Gyula. +1 (non-binding) Best, Paul Lam > 2023年7月20日 19:18，Márton Balassi 写道： > > Thanks, Gyula. > > +1 (binding) > > On Thu, Jul 20, 2023 at 5:01 AM Samrat Deb wrote: > >> thank you gyula , >> for driving it. >> +1(non binding) >> >> >> Bests, >> Samrat >> >> On Thu, 20 Jul

Re: Kubernetes Operator 1.6.0 release planning

Thanks, Gyula. +1 (binding) On Thu, Jul 20, 2023 at 5:01 AM Samrat Deb wrote: > thank you gyula , > for driving it. > +1(non binding) > > > Bests, > Samrat > > On Thu, 20 Jul 2023 at 8:02 AM, Rui Fan <1996fan...@gmail.com> wrote: > > > Thanks Gyula for driving this release. > > > > +1 for the

Flink suspected BUG

Hi all, I encountered a suspected bug in Flink. A flink job associates with 30+ HBase2.2 asynchronous dimension tables（Temporal Join AND lookup.async = 'true'), When I submitted this job, it took a very long time to submit to the YARN cluster. During the generation of the job graph and

Re: [VOTE] Release 2.0 must-have work items

Thanks all for the feedback. I've updated the wiki page as discussed, and started another round of VOTE [1]. Best, Xintong [1] https://lists.apache.org/thread/odftvr5ozyrrl9nl2p3gv4d9fbmt2wvz On Tue, Jul 18, 2023 at 2:26 PM Xintong Song wrote: > Let me try to summarize the proposed changes

[VOTE] Release 2.0 must-have work items - Round 2

Hi all, I'd like to start another round of VOTE for the must-have work items for release 2.0 [1]. The corresponding discussion thread is [2], and the previous voting thread is [3]. All comments from the previous voting thread have been addressed. Please note that once the vote is approved, any

[jira] [Created] (FLINK-32637) The UNNEST function will cause the Delayed Retry Strategy For Lookup doesn't work

jianhao.Li created FLINK-32637: -- Summary: The UNNEST function will cause the Delayed Retry Strategy For Lookup doesn't work Key: FLINK-32637 URL: https://issues.apache.org/jira/browse/FLINK-32637

[jira] [Created] (FLINK-32636) Promethues metric exporter expose stopped job metric

chenyuzhi created FLINK-32636: - Summary: Promethues metric exporter expose stopped job metric Key: FLINK-32636 URL: https://issues.apache.org/jira/browse/FLINK-32636 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-346: Deprecate ManagedTable related APIs

+1 On Thu, Jul 20, 2023 at 10:37 AM Lincoln Lee wrote: > +1 > > > Best, > Lincoln Lee > > > Leonard Xu 于2023年7月20日周四 16:26写道： > > > Thanks Jane for driving this FLIP. > > > > > > +1, > > > > Minor: We can remove related document as well, Keeping a external link to > > Apache Paimon should be

Re: [DISCUSS][2.0] FLIP-338: Remove terminationMode query parameter from job cancellation REST endpoint

We can introduce a V2 API at any point while supporting V1. Given that this has been the case for years I'm not entirely convinced that the pain points are large enough to push anyone to pursue that. ( in particular, providing an _entire_ API and not just a few special endpoints (which would

[jira] [Created] (FLINK-32635) Add Lag function to Table API

padavan created FLINK-32635: --- Summary: Add Lag function to Table API Key: FLINK-32635 URL: https://issues.apache.org/jira/browse/FLINK-32635 Project: Flink Issue Type: Improvement

Re: FLIP-342: Remove brackets around keys returned by MetricGroup#getAllVariables

environment variables is a very different though though, that'd just confuse users. It's also not a term we've used in the documentation. getScopeVariables would I guess be most in line with existing terminology. On 19/07/2023 18:10, Matthias Pohl wrote: We don't have a well-defined process

Re: [DISCUSS] FLIP-346: Deprecate ManagedTable related APIs

+1 Best, Lincoln Lee Leonard Xu 于2023年7月20日周四 16:26写道： > Thanks Jane for driving this FLIP. > > > +1, > > Minor: We can remove related document as well, Keeping a external link to > Apache Paimon should be enough. > > Best, > Leonard > > > > > On Jul 20, 2023, at 3:07 PM, Martijn Visser >

Re: [DISCUSS] FLIP-346: Deprecate ManagedTable related APIs

Thanks Jane for driving this FLIP. +1, Minor: We can remove related document as well, Keeping a external link to Apache Paimon should be enough. Best, Leonard > On Jul 20, 2023, at 3:07 PM, Martijn Visser wrote: > > +1 > > On Thu, Jul 20, 2023 at 7:53 AM Xintong Song wrote: > >> +1

Re: [DISCUSS][2.0] FLIP-338: Remove terminationMode query parameter from job cancellation REST endpoint

+1 nothing to add @Austin It sounds reasonable. But this topic might deserve a dedicated discussion thread, I think. On Thu, Jul 20, 2023 at 12:52 AM Austin Cawley-Edwards < austin.caw...@gmail.com> wrote: > It doesn't need to be part of the Flink 2.0 release perse, but starting to > wonder if

Re: [DISCUSS] FLIP 333 - Redesign Apache Flink website

I think Martijn and Markos brought up a few good points: - We shouldn't degrade the accessibility but ideally improve it as part of the redesign. The current proposal doesn't look like we're doing changes in a way that it lowers the accessibility (considering that the menu structure stays the

Re: [DISCUSS] FLIP-346: Deprecate ManagedTable related APIs

+1 On Thu, Jul 20, 2023 at 7:53 AM Xintong Song wrote: > +1 > > Best, > > Xintong > > > > On Thu, Jul 20, 2023 at 1:25 PM Jingsong Li > wrote: > > > +1 > > > > On Thu, Jul 20, 2023 at 12:31 PM Jane Chan > wrote: > > > > > > Hi, devs, > > > > > > I would like to start a discussion on FLIP-346:

[jira] [Created] (FLINK-32634) Deprecate StreamRecordTimestamp and ExistingField

Jane Chan created FLINK-32634: - Summary: Deprecate StreamRecordTimestamp and ExistingField Key: FLINK-32634 URL: https://issues.apache.org/jira/browse/FLINK-32634 Project: Flink Issue Type: