Re: [VOTE] Release 2.0 must-have work items

[Xintong Song]

Thanks for the support, Yu.

We will have the guideline before removing DataSet. We are currently
prioritizing works that need to be done before the 1.18 feature freeze, and
will soon get back to working on the guidelines. We expect to get the
guideline ready before or soon after the 1.18 release, which will
definitely be before removing DataSet in 2.0.

Best,

Xintong



On Fri, Jul 14, 2023 at 1:06 PM Yu Li  wrote:

> It's great to see the discussion about what we need to improve on
> (completely) switching from DataSet API to DataStream API from the user
> perspective. I feel that these improvements would happen faster (only) when
> we seriously prepare to remove the DataSet APIs with a target release, just
> like what we are doing now. And the same applies to the SinkV1 related
> discussions (smile).
>
> I support Xintong's opinion on keeping "Remove the DataSet APIs" a
> must-have item, meantime I support Yuxia's opinion that we should
> explicitly let our users know how to migrate their existing DataSet API
> based applications afterwards, meaning that the guideline Xintong mentioned
> is a must-have (rather than best efforts) before removing the DataSet APIs.
>
> Best Regards,
> Yu
>
>
> On Wed, 12 Jul 2023 at 14:00, yuxia  wrote:
>
> > Thanks Xintong for clarification. A guideline to help users migrating
> from
> > DataSet to DataStream will definitely be helpful.
> >
> > Best regards,
> > Yuxia
> >
> > - 原始邮件 -
> > 发件人: "Xintong Song" 
> > 收件人: "dev" 
> > 发送时间: 星期三, 2023年 7 月 12日 上午 11:40:12
> > 主题: Re: [VOTE] Release 2.0 must-have work items
> >
> > @Yuxia,
> >
> > We are aware of the issue that you mentioned. Actually, I don't think the
> > DataStream API can cover everything in the DataSet API in exactly the
> same
> > way, because the fundamental model, concepts and primitives of the two
> sets
> > of APIs are completely different. Many of the DataSet APIs, especially
> > those accessing the full data set at once, do not fit in the DataStream
> > concepts at all. I think what's important is that users can achieve the
> > same function, even if they may need to code in a different way.
> >
> > We have gone through all the existing DataSet APIs, and categorized them
> > into 3 kinds:
> > - APIs that are well supported by DataStream API as is. E.g., map, reduce
> > on grouped dataset, etc.
> > - APIs that can be achieved by DataStream API as is, but with a price
> > (programming complexity, or computation efficiency). E.g., reduce on full
> > dataset, sort partition, etc. Admittedly, there is room for improvement
> on
> > these. We may keep improving these for the DataStream API, or we can
> > concentrate on supporting them better in the new ProcessFunction API.
> > Either way, I don't think we should block the retiring of DataSet API on
> > them.
> > - There are also a few APIs that cannot be supported by the DataStream
> API
> > as is, unless users write their custom operators from the ground up. Only
> > left/rightOuterJoin and combineGroup fall into this category. I think
> > combinedGroup is probably not a problem, because this is more like a
> > variant of reduceGroup that allows the framework to execute more
> > efficiently. As for the outer joins, depending on how badly this is
> needed,
> > it can be supported by emitting the non-joined entries upon triggering a
> > window join.
> >
> > We are also planning to draft a guideline to help users migrating from
> > DataSet to DataStream, which should demonstrate how users can achieve
> > things like sort-partition with DataStream API.
> >
> > Last but not least, I'd like to point out that the decision to deprecate
> > and eventually remove the DataSet API was approved in FLIP-131, and all
> the
> > prerequisites mentioned in the FLIP have been completed.
> >
> > Best,
> >
> > Xintong
> >
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866741
> >
> >
> >
> > On Wed, Jul 12, 2023 at 10:20 AM Jingsong Li 
> > wrote:
> >
> > > +1 to Leonard and Galen and Jing.
> > >
> > > About Source and Sink.
> > > We're still missing quite a bit of work, including functionality,
> > > including ease of use, including bug fixes, and I'm not sure we'll be
> > > completely done by 2.0.
> > > Until that's done, we won't be in a position to clean up the old APIs.
> > >
> > > Best,
> > > Jingsong
> > >
> > > On Wed, Jul 12, 2023 at 9:41 AM yuxia 
> > wrote:
> > > >
> > > > Hi，Xintong.
> > > > Sorry to disturb the voting. I just found an email[1] about DataSet
> API
> > > from flink-user-zh channel. And I think it's not just a single case
> > > according to my observation.
> > > >
> > > > Remove DataSet is a must have item in release-2.0. But as the user
> > email
> > > said, if we remove DataSet, how users can implement Sort/PartitionBy,
> etc
> > > as they did with DataSet?
> > > > Do we will also provide similar api in datastream or some other thing
> > > before we remove DataSet?
> > > > Btw, as far as I see, with 

Re: [VOTE] Release 2.0 must-have work items

[Yu Li]

It's great to see the discussion about what we need to improve on
(completely) switching from DataSet API to DataStream API from the user
perspective. I feel that these improvements would happen faster (only) when
we seriously prepare to remove the DataSet APIs with a target release, just
like what we are doing now. And the same applies to the SinkV1 related
discussions (smile).

I support Xintong's opinion on keeping "Remove the DataSet APIs" a
must-have item, meantime I support Yuxia's opinion that we should
explicitly let our users know how to migrate their existing DataSet API
based applications afterwards, meaning that the guideline Xintong mentioned
is a must-have (rather than best efforts) before removing the DataSet APIs.

Best Regards,
Yu


On Wed, 12 Jul 2023 at 14:00, yuxia  wrote:

> Thanks Xintong for clarification. A guideline to help users migrating from
> DataSet to DataStream will definitely be helpful.
>
> Best regards,
> Yuxia
>
> - 原始邮件 -
> 发件人: "Xintong Song" 
> 收件人: "dev" 
> 发送时间: 星期三, 2023年 7 月 12日 上午 11:40:12
> 主题: Re: [VOTE] Release 2.0 must-have work items
>
> @Yuxia,
>
> We are aware of the issue that you mentioned. Actually, I don't think the
> DataStream API can cover everything in the DataSet API in exactly the same
> way, because the fundamental model, concepts and primitives of the two sets
> of APIs are completely different. Many of the DataSet APIs, especially
> those accessing the full data set at once, do not fit in the DataStream
> concepts at all. I think what's important is that users can achieve the
> same function, even if they may need to code in a different way.
>
> We have gone through all the existing DataSet APIs, and categorized them
> into 3 kinds:
> - APIs that are well supported by DataStream API as is. E.g., map, reduce
> on grouped dataset, etc.
> - APIs that can be achieved by DataStream API as is, but with a price
> (programming complexity, or computation efficiency). E.g., reduce on full
> dataset, sort partition, etc. Admittedly, there is room for improvement on
> these. We may keep improving these for the DataStream API, or we can
> concentrate on supporting them better in the new ProcessFunction API.
> Either way, I don't think we should block the retiring of DataSet API on
> them.
> - There are also a few APIs that cannot be supported by the DataStream API
> as is, unless users write their custom operators from the ground up. Only
> left/rightOuterJoin and combineGroup fall into this category. I think
> combinedGroup is probably not a problem, because this is more like a
> variant of reduceGroup that allows the framework to execute more
> efficiently. As for the outer joins, depending on how badly this is needed,
> it can be supported by emitting the non-joined entries upon triggering a
> window join.
>
> We are also planning to draft a guideline to help users migrating from
> DataSet to DataStream, which should demonstrate how users can achieve
> things like sort-partition with DataStream API.
>
> Last but not least, I'd like to point out that the decision to deprecate
> and eventually remove the DataSet API was approved in FLIP-131, and all the
> prerequisites mentioned in the FLIP have been completed.
>
> Best,
>
> Xintong
>
>
> [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866741
>
>
>
> On Wed, Jul 12, 2023 at 10:20 AM Jingsong Li 
> wrote:
>
> > +1 to Leonard and Galen and Jing.
> >
> > About Source and Sink.
> > We're still missing quite a bit of work, including functionality,
> > including ease of use, including bug fixes, and I'm not sure we'll be
> > completely done by 2.0.
> > Until that's done, we won't be in a position to clean up the old APIs.
> >
> > Best,
> > Jingsong
> >
> > On Wed, Jul 12, 2023 at 9:41 AM yuxia 
> wrote:
> > >
> > > Hi，Xintong.
> > > Sorry to disturb the voting. I just found an email[1] about DataSet API
> > from flink-user-zh channel. And I think it's not just a single case
> > according to my observation.
> > >
> > > Remove DataSet is a must have item in release-2.0. But as the user
> email
> > said, if we remove DataSet, how users can implement Sort/PartitionBy, etc
> > as they did with DataSet?
> > > Do we will also provide similar api in datastream or some other thing
> > before we remove DataSet?
> > > Btw, as far as I see, with regarding to replcaing DataSet with
> > Datastream, Datastream are missing many API. I think it may well take
> much
> > effort to fully cover the missing api.
> > >
> > > [1] https://lists.apache.org/thread/syjmt8f74gh8ok3z4lhgt95zl4dzn168
> > >
> > > Best regards,
> > > Yuxia
> > >
> > > - 原始邮件 -
> > > 发件人: "Jing Ge" 
> > > 收件人: "dev" 
> > > 发送时间: 星期三, 2023年 7 月 12日 上午 1:23:40
> > > 主题: Re: [VOTE] Release 2.0 must-have work items
> > >
> > > agree with what Leonard said. There are actually more issues wrt the
> new
> > > Source and SinkV2[1]
> > >
> > > Speaking of must-have vs nice-to-have, I think it depends on the
> > priority.
> > > 

Re: Flink 2.0 connectors, formats, and even user code should also be pluggable.

[Leonard Xu]

Hello Zhiqiang,

Although it is strange to receive such an email on the dev mailing list, I read 
your email carefully. The Pluggable mechanism for Connectors, Formats is indeed 
necessary, and I believe that developers in the community also have this 
consensus. Regarding the relationship between 2.0 and this feature, I would 
like to use two questions to elaborate:

* Can we only develop this feature in 2.0? No, we can also develop in versions 
1.19 and 2.x. This is just a new improvement or mechanism, nothing special.
* Does deprecate API only happen in 2.0? No, we can also deprecate API in 
versions 1.19 and 2.1 as well as other minor versions.

So, I don't see any reason that we have to link this feature to 2.0 version. 
Regarding the feature discussion in the dev mail group, the correct way is to 
prepare a FLIP or at least a FLIP draft, and then open a [DISCUSS] xxx thread 
for discussion.

Btw, some community contributors are no longer active due to personal interest 
or career development direction, usually we should not directly ping the 
corresponding developers, we should just open a [DISCUSS] thread to dev 
mailing-list and discuss the feature.

Best,
Leonard

> On Jul 13, 2023, at 4:22 PM, zhiqiang li  wrote:
> 
> Flink CDC Maintainer
> xbjt...@gmail.com 
> 
> Factory Interface
> Timo Walther mailto:twal...@apache.org>>
>  
> Plugin Interface
> Aleksey Pak mailto:alek...@ververica.com>>
>  
> Connector Base
> Stephan Ewen mailto:se...@apache.org>>
>  
> Flink File System
> Stephan Ewen mailto:se...@apache.org>>
>  
> I have observed that in [1], connectors and formats are pluggable, allowing 
> user code to be easily integrated. The advantages of having pluggable 
> connectors are evident, as it helps avoid conflicts between different 
> versions of jar packages. If classloader isolation is not used, shading 
> becomes necessary to resolve conflicts, resulting in a significant waste of 
> development time. I understand that implementing this change may require 
> numerous API modifications, so I would like to discuss in this email whether 
> it is feasible to make these changes in Flink 2.0.
>  
> > Plugins cannot access classes from other plugins or from Flink that have 
> > not been specifically whitelisted. 
> > This strict isolation allows plugins to contain conflicting versions of the 
> > same library without the need to relocate classes or to converge to common 
> > versions. 
> > Currently, file systems and metric reporters are pluggable but, in the 
> > future, connectors, formats, and even user code should also be pluggable.
>  
> I hope to be able to integrate into a set of APIs, which would require 
> significant changes to the integration API. Is it possible to address this in 
> Flink 2.0? However, due to [2] FLIP-321, Flink 1.18 should deprecate some 
> APIs, and the community is uncertain about the existence of Flink 1.20. This 
> may make it impossible to integrate a set of APIs. @Xintong Song 
> mailto:tonysong...@gmail.com>> suggests continuing to 
> use two APIs.
>  
> I am aware of some API modifications, but considering the extensive changes 
> involved, I am concerned that my understanding may not be comprehensive. I am 
> willing to do the work, but I have not yet found a committer.
>  
>  
> [1] 
> https://nightlies.apache.org/Flink/Flink-docs-master/docs/deployment/filesystems/overview/
>  
> 
> [2] FLIP-321  
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-321%3A+Introduce+an+API+deprecation+process
>  
> 
>  
>  
> Best,
> Zhiqiang

[jira] [Created] (FLINK-32590) Fail to read flink parquet filesystem table stored in hive metastore service.

[Guozhen Yang (Jira)]

Guozhen Yang created FLINK-32590:


 Summary: Fail to read flink parquet filesystem table stored in 
hive metastore service.
 Key: FLINK-32590
 URL: https://issues.apache.org/jira/browse/FLINK-32590
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Hive, Formats (JSON, Avro, Parquet, ORC, 
SequenceFile)
Affects Versions: 1.17.1
Reporter: Guozhen Yang


h2. Summary:

Fail to read flink parquet filesystem table stored in hive metastore service.
h2. The problem:

When I try to read a flink parquet filesystem table stored in hive metastore 
service, I got the following exception.
{noformat}
java.lang.RuntimeException: One or more fetchers have encountered exception
at 
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager.checkErrors(SplitFetcherManager.java:261)
 ~[flink-connector-files-1.17.1.jar:1.17.1]
at 
org.apache.flink.connector.base.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:169)
 ~[flink-connector-files-1.17.1.jar:1.17.1]
at 
org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:131)
 ~[flink-connector-files-1.17.1.jar:1.17.1]
at 
org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:417)
 ~[flink-dist-1.17.1.jar:1.17.1]
at 
org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:68)
 ~[flink-dist-1.17.1.jar:1.17.1]
at 
org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
 ~[flink-dist-1.17.1.jar:1.17.1]
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:550)
 ~[flink-dist-1.17.1.jar:1.17.1]
at 
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
 ~[flink-dist-1.17.1.jar:1.17.1]
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:839)
 ~[flink-dist-1.17.1.jar:1.17.1]
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:788) 
~[flink-dist-1.17.1.jar:1.17.1]
at 
org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:952)
 ~[flink-dist-1.17.1.jar:1.17.1]
at 
org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:931) 
~[flink-dist-1.17.1.jar:1.17.1]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:745) 
~[flink-dist-1.17.1.jar:1.17.1]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562) 
~[flink-dist-1.17.1.jar:1.17.1]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_345]
Caused by: java.lang.NoSuchMethodError: 
shaded.parquet.org.apache.thrift.TBaseHelper.hashCode(J)I
at org.apache.parquet.format.ColumnChunk.hashCode(ColumnChunk.java:812) 
~[flink-sql-parquet-1.17.1.jar:1.17.1]
at java.util.AbstractList.hashCode(AbstractList.java:541) ~[?:1.8.0_345]
at org.apache.parquet.format.RowGroup.hashCode(RowGroup.java:704) 
~[flink-sql-parquet-1.17.1.jar:1.17.1]
at java.util.HashMap.hash(HashMap.java:340) ~[?:1.8.0_345]
at java.util.HashMap.put(HashMap.java:613) ~[?:1.8.0_345]
at 
org.apache.parquet.format.converter.ParquetMetadataConverter.generateRowGroupOffsets(ParquetMetadataConverter.java:1411)
 ~[flink-sql-parquet-1.17.1.jar:1.17.1]
at 
org.apache.parquet.format.converter.ParquetMetadataConverter.access$600(ParquetMetadataConverter.java:144)
 ~[flink-sql-parquet-1.17.1.jar:1.17.1]
at 
org.apache.parquet.format.converter.ParquetMetadataConverter$3.visit(ParquetMetadataConverter.java:1461)
 ~[flink-sql-parquet-1.17.1.jar:1.17.1]
at 
org.apache.parquet.format.converter.ParquetMetadataConverter$3.visit(ParquetMetadataConverter.java:1437)
 ~[flink-sql-parquet-1.17.1.jar:1.17.1]
at 
org.apache.parquet.format.converter.ParquetMetadataConverter$RangeMetadataFilter.accept(ParquetMetadataConverter.java:1207)
 ~[flink-sql-parquet-1.17.1.jar:1.17.1]
at 
org.apache.parquet.format.converter.ParquetMetadataConverter.readParquetMetadata(ParquetMetadataConverter.java:1437)
 ~[flink-sql-parquet-1.17.1.jar:1.17.1]
at 
org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:583)
 ~[flink-sql-parquet-1.17.1.jar:1.17.1]
at 
org.apache.parquet.hadoop.ParquetFileReader.(ParquetFileReader.java:777) 
~[flink-sql-parquet-1.17.1.jar:1.17.1]
at 
org.apache.parquet.hadoop.ParquetFileReader.open(ParquetFileReader.java:658) 
~[flink-sql-parquet-1.17.1.jar:1.17.1]
at 
org.apache.flink.formats.parquet.ParquetVectorizedInputFormat.createReader(ParquetVectorizedInputFormat.java:127)
 ~[flink-sql-parquet-1.17.1.jar:1.17.1]
at 

Re: [DISCUSS] Deprecating Public API for 2.0 requiring FLIPs

[Xintong Song]

Hi Matthias,

I'm not aware of any decision that has already been made by the community
regarding after which 1.x minor release will we ship the 2.0 major release.
I personally don't think it's feasible to avoid 1.19, and even 1.20
depending on the progress.

I also don't think we should push all the deprecation works in 1.18. In the
"Deprecate multiple APIs in 1.18" thread [1], I only listed APIs that
giving the impression already deprecated but are actually not fully (either
in code or in documentation), in order to clarify the status and to
properly deprecate the onse should be. We should not decide when to
deprecate an existing API based on whether we would have a 1.19 or 1.20
minor release. Deciding to deprecate / remove an API definitely deserves
thorough discussions and FLIP, which takes time, and I don't think we
should compromise that for any reason.

I think the potential conflict is between not being able to deprecate APIs
very soon (needs more discussions, the new replacing API is not ready,
etc.) and the wish to ship 2.0 on time. Assuming at some point we want to
ship the 2.0 release, and there are some deprecated APIs that we want to
remove but have not fulfilled the migration period required by FLIP-321
[2], I see 3 options to move forward.
1. Not removing the deprecated APIs in 2.0, carrying them until 3.0. This
might be suitable if there are not a lot of deprecated APIs that we need to
carry and the maintenance overhead is acceptable.
2. Postpone the 2.0 release for another minor release. This might be
suitable if there are a lot of deprecated APIs.
3. Considering such APIs as exceptions of FLIP-321. This might be suitable
if there are only a few of such APIs but have significant maintenance
overhead. As discussed in FLIP-321, the introduction of the migration
period effectively means we need to plan for the deprecation / removal of
APIs early. As it is newly introduced and we haven't given developers the
chance to plan things ahead, it's probably fair to make exceptions for API
removal in the 2.0 version bump.

My options are slightly different from what Chesnay has proposed. But I do
agree that none of these options are great. I guess that's the price for
not having the deprecation process earlier. Trying to deprecate things
early is still helpful, because it reduces the number of APIs we need to
consider when deciding between the options. However, that must not come at
the price of rush decisions. I'd suggest developers to design / discuss /
vote the breaking changes at their pace, and we evaluate the status and
choose between the options later (maybe around the time releasing 1.19).

If there are some contributors who think it makes sense, I will raise it in
> the 1.18 release channel to postpone 1.18 feature freeze again.
>
I don't think postponing 1.18 would help a lot in this case, unless we
postponed it for another couple of months. I don't think all the API
changing plans can be finalized in a couple of weeks.

Best,

Xintong


[1] https://lists.apache.org/thread/3dw4f8frlg8hzlv324ql7n2755bzs9hy
[2] https://lists.apache.org/thread/vmhzv8fcw2b33pqxp43486owrxbkd5x9

On Thu, Jul 13, 2023 at 9:41 PM Jing Ge  wrote:

> Hi,
>
> Thanks Matthias for starting this discussion and thanks Chesnay for the
> clarification.
>
> I don't want to hijack this discussion but I would suggest postponing the
> 1.18 feature freeze over postponing the deprecations to 1.19. If there are
> some contributors who think it makes sense, I will raise it in the 1.18
> release channel to postpone 1.18 feature freeze again.
>
> Speaking of the FLIP rules, if we follow it exactly like it is defined, we
> should also write FLIPs when graduating @PublicEvloving APIs to be @Public,
> especially for those APIs who will replace some deprecated APIs.  Doing
> that is to guarantee that new public APIs will cover all
> functionalities(including the capability that APIs are easy enough to
> implement) that deprecated APIs offer, so that migrations can be executed
> smoothly. With this in mind, we will avoid the big issue we are facing now
> wrt the new Source and Sink API [1].
>
> Best regards,
> Jing
>
> [1] https://lists.apache.org/thread/734zhkvs59w2o4d1rsnozr1bfqlr6rgm
>
> On Thu, Jul 13, 2023 at 3:01 PM Chesnay Schepler 
> wrote:
>
> > The issue isn't avoiding 1.19.
> >
> > The issue is that if things aren't deprecated in 1.18 then for every
> > breaking change we have to start discussing exemptions from the API
> > deprecation process, which stipulates that all Public APIs must be
> > deprecated for at least 2 minor releases before they can be removed
> > (which is now unsurprisingly backfiring on us).
> >
> > So if something isn't deprecated in 1.18 then either:
> > - we delay 2.0 by at 1 release cycle
> > - we effectively ignore the newly agreed upon deprecation process for 2.0
> > - we circumvent the newly agreed upon deprecation process by creating 2
> > minor releases in the same time-frame that we'd usually 

[jira] [Created] (FLINK-32589) Carry over parallelism overrides to prevent users from clearing them on updates

[Maximilian Michels (Jira)]

Maximilian Michels created FLINK-32589:
--

 Summary: Carry over parallelism overrides to prevent users from 
clearing them on updates
 Key: FLINK-32589
 URL: https://issues.apache.org/jira/browse/FLINK-32589
 Project: Flink
  Issue Type: Improvement
  Components: Autoscaler, Kubernetes Operator
Reporter: Maximilian Michels


The autoscaler currently sets the parallelism overrides via the Flink config 
{{pipeline.jobvertex-parallelism-overrides}}. Whenever the user posts specs 
updates, special care needs to be taken in order to carry over existing 
overrides. Otherwise the job will reset to the default parallelism 
configuration. Users shouldn't have to deal with this. Instead, whenever a new 
spec is posted which does not contain the overrides, the operator should 
automatically apply the last-used overrides (if autoscaling is enabled).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Deprecating Public API for 2.0 requiring FLIPs

[Jing Ge]

Hi,

Thanks Matthias for starting this discussion and thanks Chesnay for the
clarification.

I don't want to hijack this discussion but I would suggest postponing the
1.18 feature freeze over postponing the deprecations to 1.19. If there are
some contributors who think it makes sense, I will raise it in the 1.18
release channel to postpone 1.18 feature freeze again.

Speaking of the FLIP rules, if we follow it exactly like it is defined, we
should also write FLIPs when graduating @PublicEvloving APIs to be @Public,
especially for those APIs who will replace some deprecated APIs.  Doing
that is to guarantee that new public APIs will cover all
functionalities(including the capability that APIs are easy enough to
implement) that deprecated APIs offer, so that migrations can be executed
smoothly. With this in mind, we will avoid the big issue we are facing now
wrt the new Source and Sink API [1].

Best regards,
Jing

[1] https://lists.apache.org/thread/734zhkvs59w2o4d1rsnozr1bfqlr6rgm

On Thu, Jul 13, 2023 at 3:01 PM Chesnay Schepler  wrote:

> The issue isn't avoiding 1.19.
>
> The issue is that if things aren't deprecated in 1.18 then for every
> breaking change we have to start discussing exemptions from the API
> deprecation process, which stipulates that all Public APIs must be
> deprecated for at least 2 minor releases before they can be removed
> (which is now unsurprisingly backfiring on us).
>
> So if something isn't deprecated in 1.18 then either:
> - we delay 2.0 by at 1 release cycle
> - we effectively ignore the newly agreed upon deprecation process for 2.0
> - we circumvent the newly agreed upon deprecation process by creating 2
> minor releases in the same time-frame that we'd usually create 1 release
> in.
>
> None of these options are great.
>
> On 13/07/2023 14:03, Matthias Pohl wrote:
> > Jing brought up a question in the FLIP-335 discussion thread [1] which I
> > want to move into a dedicated discussion thread as it's a bit more
> general:
> > How do we handle the deprecation process of Public APIs for Flink 2.0?
> >> I just have a related question: Do we need to create a FLIP each time
> >> when we want to deprecate any classes?
> >
> > The community defined the requirements of a FLIP [2] in the following
> way:
> > - Any major new feature, subsystem, or piece of functionality
> > - Any change that impacts the public interfaces of the project
> >
> > public interfaces in this sense are defined as:
> > - DataStream and DataSet API, including classes related to that, such as
> > StreamExecutionEnvironment
> > - Classes marked with the @Public annotation
> > - On-disk binary formats, such as checkpoints/savepoints
> > - User-facing scripts/command-line tools, i.e. bin/flink, Yarn scripts,
> > Mesos scripts
> > - Configuration settings
> > - Exposed monitoring information
> >
> > I think that this makes sense. There should be a proper discussion on the
> > best approach to change public APIs. Additionally, the FLIP should be a
> way
> > to document the changes in the discussion process towards the change
> > transparently.
> >
> > In contrast, the impression I have is that we're trying to push all the
> > deprecation work into 1.18 (which has its feature freeze date by the end
> of
> > next week) to avoid having an additional 1.19 minor release. (Correct me
> if
> > I'm wrong here but that's the impression I'm getting from the ML threads
> > I've seen so far.)
> >
> > I have some concerns on the Flink 2.0 development in this regard. Zhu Zhu
> > [4] and Chesnay [5] shared similar concerns in the thread about the 2.0
> > must-have work items.
> >
> > Considering that quite a few (or some; I haven't checked in detail to be
> > honest) of the changes for 2.0 should require a FLIP and that there are
> > still some hanging items [6] (Jira issues which are annotated for Flink
> 2.0
> > but have been properly checked, yet): Shouldn't we avoid pushing
> everything
> > into 1.18? Instead, we should follow the required process properly and
> > might plan for another 1.19 minor release, instead?
> >
> > I'm curious how other contributors feel here and sorry in case I have
> > misinterpreted the ongoing discussions.
> >
> > Best,
> > Matthias
> >
> > [1] https://lists.apache.org/thread/48ysrg1rrtl8s1twg9wmx35l201hnc2w
> > [2]
> >
> https://cwiki.apache.org/confluence/display/Flink/Flink+Improvement+Proposals#FlinkImprovementProposals-Whatisconsidereda%22majorchange%22thatneedsaFLIP
> > ?
> > [3] https://lists.apache.org/thread/r0y9syc6k5nmcxvnd0hj33htdpdj9k6m
> > [4] https://lists.apache.org/thread/45xm348jr8n6s89jldntv5z3t13hdbn8
> > [5] https://lists.apache.org/thread/98wgqrx0sycpskvgpydvkywsoxt0fkc6
> > [6] https://lists.apache.org/thread/77hj39ls3kxvx2qd6o09hq1ndtn6hg4y
> >
>
>

[DISCUSS][2.0] FLIP-338: Remove terminationMode query parameter from job cancellation REST endpoint

[Chesnay Schepler]

Hello,

The job cancellation REST endpoint has a terminationMode query 
parameter, which in the past could be set to either CANCEL or STOP, but 
nowadays the job stop endpoint has subsumed the STOP functionality.


Since then the cancel endpoint rejected requests that specified STOP.

I propose to finally remove this parameter, as it currently serves no 
function.


https://cwiki.apache.org/confluence/display/FLINK/FLIP-338%3A+Remove+terminationMode+query+parameter+from+job+cancellation+REST+endpoint 




Regards,

Chesnay

[DISCUSS][2.0] FLIP-337: Remove JarRequestBody#programArgs

[Chesnay Schepler]

Hello,

The request body for the jar run/plan REST endpoints accepts program 
arguments as a string (programArgs) or a list of strings 
(programArgsList). The latter was introduced as kept running into issues 
with splitting the string into individual arguments./

/

We ideally force users to use the list argument, and we can simplify the 
codebase if there'd only be 1 way to pass arguments.


As such I propose to remove the programArgs field from the request body.

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263424796


Regards,

Chesnay

Re: [DISCUSS] FLIP 333 - Redesign Apache Flink website

[Chesnay Schepler]

On 13/07/2023 08:07, Mohan, Deepthi wrote:

However, even these developers when explicitly asked in our conversations often 
comment that the website could do with a redesign


Can you go into more detail as to their specific concerns? Are there 
functional problems with the page, or is this just a matter of "I don't 
like the way it looks"?


What had they trouble with? Which information was 
missing/unnecessary/too hard to find?


The FLIP states that "/we want to modernize the website so that new and 
existing users can easily find information to understand what Flink is, 
the primary use cases where Flink is useful, and clearly understand its 
value proposition/."


From the mock-ups I don't /really/ see how these stated goals are 
achieved. It mostly looks like a fresh coat of paint, with a compressed 
nav bar (which does reduce how much information and links we throw at 
people at once (which isn't necessarily bad)).


Can you go into more detail w.r.t. to the proposed 
text/presentation/diagram changes?


I assume you are not proposing any tooling changes?

Re: [DISCUSS] FLIP-335: Removing Flink's Time classes as part of Flink 2.0

[Jing Ge]

yeah, saw that thread. Thanks for starting the discussion!

Best regards,
Jing

On Thu, Jul 13, 2023 at 2:15 PM Matthias Pohl
 wrote:

> Thanks Jing. About your question regarding FLIPs: The FLIP is necessary
> when these classes touch user-facing API (which should be the case if we're
> preparing it for 2.0). I created a dedicated discussion thread [1] to
> discuss this topic because it's more general and independent of FLIP-335.
>
> Best,
> Matthias
>
> [1] https://lists.apache.org/thread/1007v4f4ms7ftp1qtkjsq25s5lwmk9wo
>
> On Thu, Jul 13, 2023 at 12:21 PM Jing Ge 
> wrote:
>
> > Hi Matthias,
> >
> > Thanks for raising this up! I saw your conversation with Chesnay in the
> PR
> > and after reading the historical discussions, there should be no doubt to
> > deprecate Time classes. +1
> >
> > I just have a related question: Do we need to create a FLIP each time
> when
> > we want to deprecate any classes? (BTW, FLIP-335 contains useful
> > information. It makes sense to write it)
> >
> > Best regards,
> > Jing
> >
> > On Thu, Jul 13, 2023 at 12:09 PM Matthias Pohl
> >  wrote:
> >
> > > The 2.0 feature list includes the removal of Flink's Time classes in
> > favor
> > > of the JDKs java.time.Duration class. There was already a discussion
> > about
> > > it in [1] and FLINK-14068 [2] was created as a consequence of this
> > > discussion.
> > >
> > > I started working on marking the APIs as deprecated in FLINK-32570 [3]
> > > where Chesnay raised a fair point that there isn't a FLIP, yet, to
> > > formalize this public API change. Therefore, I went ahead and created
> > > FLIP-335 [4] to have this change properly documented.
> > >
> > > I'm not 100% sure whether there are better ways of checking whether
> we're
> > > covering everything Public API-related. There are even classes which I
> > > think might be user-facing but are not labeled accordingly (e.g.
> > > flink-cep). But I don't have the proper knowledge in these parts of the
> > > code. Therefore, I would propose marking these methods as deprecated,
> > > anyway, to be on the safe side.
> > >
> > > I'm open to any suggestions on improving the Test Plan of this change.
> > >
> > > I'm looking forward to feedback on this FLIP.
> > >
> > > Best,
> > > Matthias
> > >
> > > [1] https://lists.apache.org/thread/76yywnwf3lk8qn4dby0vz7yoqx7f7pkj
> > > [2] https://issues.apache.org/jira/browse/FLINK-14068
> > > [3] https://issues.apache.org/jira/browse/FLINK-32570
> > > [4]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-335%3A+Removing+Flink%27s+Time+classes
> > >
> >
>

[DISCUSS][2.0] FLIP-336: Remove "now" timestamp field from REST responses

[Chesnay Schepler]

Hello,

Several REST responses contain a timestamp field of the current time

 There is no known use-case for said timestamp, it makes caching of 
responses technically sketchy (since the response differs on each call) 
and  it complicates testing since the timestamp field can't be easily 
predicted.


https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263424789


Regards,

Chesnay

Re: [DISCUSS] Deprecating Public API for 2.0 requiring FLIPs

[Chesnay Schepler]

The issue isn't avoiding 1.19.

The issue is that if things aren't deprecated in 1.18 then for every 
breaking change we have to start discussing exemptions from the API 
deprecation process, which stipulates that all Public APIs must be 
deprecated for at least 2 minor releases before they can be removed 
(which is now unsurprisingly backfiring on us).


So if something isn't deprecated in 1.18 then either:
- we delay 2.0 by at 1 release cycle
- we effectively ignore the newly agreed upon deprecation process for 2.0
- we circumvent the newly agreed upon deprecation process by creating 2 
minor releases in the same time-frame that we'd usually create 1 release in.


None of these options are great.

On 13/07/2023 14:03, Matthias Pohl wrote:

Jing brought up a question in the FLIP-335 discussion thread [1] which I
want to move into a dedicated discussion thread as it's a bit more general:
How do we handle the deprecation process of Public APIs for Flink 2.0?

I just have a related question: Do we need to create a FLIP each time
when we want to deprecate any classes?


The community defined the requirements of a FLIP [2] in the following way:
- Any major new feature, subsystem, or piece of functionality
- Any change that impacts the public interfaces of the project

public interfaces in this sense are defined as:
- DataStream and DataSet API, including classes related to that, such as
StreamExecutionEnvironment
- Classes marked with the @Public annotation
- On-disk binary formats, such as checkpoints/savepoints
- User-facing scripts/command-line tools, i.e. bin/flink, Yarn scripts,
Mesos scripts
- Configuration settings
- Exposed monitoring information

I think that this makes sense. There should be a proper discussion on the
best approach to change public APIs. Additionally, the FLIP should be a way
to document the changes in the discussion process towards the change
transparently.

In contrast, the impression I have is that we're trying to push all the
deprecation work into 1.18 (which has its feature freeze date by the end of
next week) to avoid having an additional 1.19 minor release. (Correct me if
I'm wrong here but that's the impression I'm getting from the ML threads
I've seen so far.)

I have some concerns on the Flink 2.0 development in this regard. Zhu Zhu
[4] and Chesnay [5] shared similar concerns in the thread about the 2.0
must-have work items.

Considering that quite a few (or some; I haven't checked in detail to be
honest) of the changes for 2.0 should require a FLIP and that there are
still some hanging items [6] (Jira issues which are annotated for Flink 2.0
but have been properly checked, yet): Shouldn't we avoid pushing everything
into 1.18? Instead, we should follow the required process properly and
might plan for another 1.19 minor release, instead?

I'm curious how other contributors feel here and sorry in case I have
misinterpreted the ongoing discussions.

Best,
Matthias

[1] https://lists.apache.org/thread/48ysrg1rrtl8s1twg9wmx35l201hnc2w
[2]
https://cwiki.apache.org/confluence/display/Flink/Flink+Improvement+Proposals#FlinkImprovementProposals-Whatisconsidereda%22majorchange%22thatneedsaFLIP
?
[3] https://lists.apache.org/thread/r0y9syc6k5nmcxvnd0hj33htdpdj9k6m
[4] https://lists.apache.org/thread/45xm348jr8n6s89jldntv5z3t13hdbn8
[5] https://lists.apache.org/thread/98wgqrx0sycpskvgpydvkywsoxt0fkc6
[6] https://lists.apache.org/thread/77hj39ls3kxvx2qd6o09hq1ndtn6hg4y

Re: [DISCUSS] FLIP-335: Removing Flink's Time classes as part of Flink 2.0

[Matthias Pohl]

Thanks Jing. About your question regarding FLIPs: The FLIP is necessary
when these classes touch user-facing API (which should be the case if we're
preparing it for 2.0). I created a dedicated discussion thread [1] to
discuss this topic because it's more general and independent of FLIP-335.

Best,
Matthias

[1] https://lists.apache.org/thread/1007v4f4ms7ftp1qtkjsq25s5lwmk9wo

On Thu, Jul 13, 2023 at 12:21 PM Jing Ge  wrote:

> Hi Matthias,
>
> Thanks for raising this up! I saw your conversation with Chesnay in the PR
> and after reading the historical discussions, there should be no doubt to
> deprecate Time classes. +1
>
> I just have a related question: Do we need to create a FLIP each time when
> we want to deprecate any classes? (BTW, FLIP-335 contains useful
> information. It makes sense to write it)
>
> Best regards,
> Jing
>
> On Thu, Jul 13, 2023 at 12:09 PM Matthias Pohl
>  wrote:
>
> > The 2.0 feature list includes the removal of Flink's Time classes in
> favor
> > of the JDKs java.time.Duration class. There was already a discussion
> about
> > it in [1] and FLINK-14068 [2] was created as a consequence of this
> > discussion.
> >
> > I started working on marking the APIs as deprecated in FLINK-32570 [3]
> > where Chesnay raised a fair point that there isn't a FLIP, yet, to
> > formalize this public API change. Therefore, I went ahead and created
> > FLIP-335 [4] to have this change properly documented.
> >
> > I'm not 100% sure whether there are better ways of checking whether we're
> > covering everything Public API-related. There are even classes which I
> > think might be user-facing but are not labeled accordingly (e.g.
> > flink-cep). But I don't have the proper knowledge in these parts of the
> > code. Therefore, I would propose marking these methods as deprecated,
> > anyway, to be on the safe side.
> >
> > I'm open to any suggestions on improving the Test Plan of this change.
> >
> > I'm looking forward to feedback on this FLIP.
> >
> > Best,
> > Matthias
> >
> > [1] https://lists.apache.org/thread/76yywnwf3lk8qn4dby0vz7yoqx7f7pkj
> > [2] https://issues.apache.org/jira/browse/FLINK-14068
> > [3] https://issues.apache.org/jira/browse/FLINK-32570
> > [4]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-335%3A+Removing+Flink%27s+Time+classes
> >
>

[DISCUSS] Deprecating Public API for 2.0 requiring FLIPs

[Matthias Pohl]

Jing brought up a question in the FLIP-335 discussion thread [1] which I
want to move into a dedicated discussion thread as it's a bit more general:
How do we handle the deprecation process of Public APIs for Flink 2.0?
>
> I just have a related question: Do we need to create a FLIP each time
> when we want to deprecate any classes?


The community defined the requirements of a FLIP [2] in the following way:
- Any major new feature, subsystem, or piece of functionality
- Any change that impacts the public interfaces of the project

public interfaces in this sense are defined as:
- DataStream and DataSet API, including classes related to that, such as
StreamExecutionEnvironment
- Classes marked with the @Public annotation
- On-disk binary formats, such as checkpoints/savepoints
- User-facing scripts/command-line tools, i.e. bin/flink, Yarn scripts,
Mesos scripts
- Configuration settings
- Exposed monitoring information

I think that this makes sense. There should be a proper discussion on the
best approach to change public APIs. Additionally, the FLIP should be a way
to document the changes in the discussion process towards the change
transparently.

In contrast, the impression I have is that we're trying to push all the
deprecation work into 1.18 (which has its feature freeze date by the end of
next week) to avoid having an additional 1.19 minor release. (Correct me if
I'm wrong here but that's the impression I'm getting from the ML threads
I've seen so far.)

I have some concerns on the Flink 2.0 development in this regard. Zhu Zhu
[4] and Chesnay [5] shared similar concerns in the thread about the 2.0
must-have work items.

Considering that quite a few (or some; I haven't checked in detail to be
honest) of the changes for 2.0 should require a FLIP and that there are
still some hanging items [6] (Jira issues which are annotated for Flink 2.0
but have been properly checked, yet): Shouldn't we avoid pushing everything
into 1.18? Instead, we should follow the required process properly and
might plan for another 1.19 minor release, instead?

I'm curious how other contributors feel here and sorry in case I have
misinterpreted the ongoing discussions.

Best,
Matthias

[1] https://lists.apache.org/thread/48ysrg1rrtl8s1twg9wmx35l201hnc2w
[2]
https://cwiki.apache.org/confluence/display/Flink/Flink+Improvement+Proposals#FlinkImprovementProposals-Whatisconsidereda%22majorchange%22thatneedsaFLIP
?
[3] https://lists.apache.org/thread/r0y9syc6k5nmcxvnd0hj33htdpdj9k6m
[4] https://lists.apache.org/thread/45xm348jr8n6s89jldntv5z3t13hdbn8
[5] https://lists.apache.org/thread/98wgqrx0sycpskvgpydvkywsoxt0fkc6
[6] https://lists.apache.org/thread/77hj39ls3kxvx2qd6o09hq1ndtn6hg4y

Re: [DISCUSS] FLIP 333 - Redesign Apache Flink website

[Ahmed Hamdy]

+1 the design looks great.
Best Regards
Ahmed Hamdy


On Thu, 13 Jul 2023 at 07:08, Mohan, Deepthi 
wrote:

> Thank you all for the feedback so far.
>
> @Xintong, as you may know, AWS provides a managed Apache Flink service
> called Kinesis Data Analytics. We regularly speak to our customers (these
> range from startups to large enterprises) in various stages of familiarity
> with Apache Flink. Most comments about the current Apache Flink website
> come from developers new to Flink and decision makers (tech leads or
> higher) who look through the website to quickly understand the value
> proposition of Flink and why they should use it. On the other hand, most
> developers already familiar with Flink do not typically use the Flink
> website, rather we see them spend most of their time on Flink
> documentation. However, even these developers when explicitly asked in our
> conversations often comment that the website could do with a redesign.
>
> @Mattias, there are a few more pages in the current website that would
> fall in to the "only design changes" category. I will edit the FLIP to
> include all of them and we can have a discussion about each with the mockup
> once ready. See next step below for details
>
> As far as the time to redesign is concerned, I believe that depends on how
> many engineers with front end experience have capacity to help (any
> volunteers?). We have a few engineers willing and interested to contribute
> from AWS. I am not convinced we need to wait until the 2.0 release or any
> version release for that matter. We can completely decouple the website
> changes so that we are able to execute on the changes quickly once we
> solidify the design for a set of pages.
>
> Next step: I am happy to provide mockups for each of the pages (the ones
> with minor content change and the ones with no content change) and also
> show how the theme will apply to documentation only for consideration. We
> may choose not to make any changes to documentation in this FLIP, but it
> may be useful to visualize before we decide. Given that we aren't really
> making any major content changes at this point, I think a FLIP for each
> page will not be necessary. Rather, we could use this discussion thread for
> specific feedback for each page that will be changed.
>
>
> On 7/12/23, 12:59 PM, "Danny Cranmer"  dannycran...@apache.org>> wrote:
>
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
>
>
>
> Thanks for driving this Deepthi.
>
>
> Looks great to me; a huge improvement over the current website. We had
> agreed in the previous discussion [1] to keep the main website separate
> from the docs since the target audience is different. I am inclined to
> decouple the two and re-evaluate updating the docs in a separate FLIP
> later.
>
>
> Thanks,
> Danny
>
>
> [1] https://lists.apache.org/thread/c3pt00cf77lrtgt242p26lgp9l2z5yc8 <
> https://lists.apache.org/thread/c3pt00cf77lrtgt242p26lgp9l2z5yc8>
>
>
> On Tue, Jul 11, 2023 at 2:38 PM Jingsong Li  > wrote:
>
>
> > It's exciting to finally have someone to refactor the Flink website.
> >
> > Thanks Deepthi.
> >
> > To Xintong,
> >
> > > maintain two sets of website designs at the same time
> >
> > If our website is not so complex, and new webui can be full feature
> > coverage, there would be no need to maintain two sets of web.
> >
> > To Jing
> >
> > +1 to consider Flink doc too.
> >
> > Best,
> > Jingsong
> >
> > On Tue, Jul 11, 2023 at 8:16 PM Jing Ge  lid>
> > wrote:
> > >
> > > Hi,
> > >
> > > +1, the UI design looks good!
> > >
> > > Commonly speaking, there are two parts of the whole website: Flink Web
> > and
> > > Flink doc. Will the dart mode also cover Flink doc?
> > >
> > > Best regards,
> > > Jing
> > >
> > > On Tue, Jul 11, 2023 at 12:40 PM Matthias Pohl
> > > mailto:matthias.p...@aiven.io.inva>lid>
> wrote:
> > >
> > > > I also like the proposed designs. Considering that you want to touch
> > > > individual subpages, there are also some subpages of Flink's website
> > not
> > > > being mentioned in the FLIP (e.g. roadmap [1]). What is the plan with
> > > > those? Are they covered by the "We recommend modifications only to
> the
> > > > design of the following pages" part but are not listed there?
> > > >
> > > > Additionally, it would be nice to get a bit more insight into the
> > feedback
> > > > from users as Xintong pointed out. It would be interesting to
> > understand
> > > > how the new design helps solving certain problems (besides having the
> > > > motivation to modernize the look and feel).
> > > >
> > > > I'm also wondering whether it's doable to do a discussion (FLIP?) per
> > > > subpage on the design (as proposed by Chesnay in the first discussion
> > on
> > > > this topic [2] to have smaller changes rather than a single big one).
> > But I
> 

Re: [DISCUSS] FLIP-335: Removing Flink's Time classes as part of Flink 2.0

[Jing Ge]

Hi Matthias,

Thanks for raising this up! I saw your conversation with Chesnay in the PR
and after reading the historical discussions, there should be no doubt to
deprecate Time classes. +1

I just have a related question: Do we need to create a FLIP each time when
we want to deprecate any classes? (BTW, FLIP-335 contains useful
information. It makes sense to write it)

Best regards,
Jing

On Thu, Jul 13, 2023 at 12:09 PM Matthias Pohl
 wrote:

> The 2.0 feature list includes the removal of Flink's Time classes in favor
> of the JDKs java.time.Duration class. There was already a discussion about
> it in [1] and FLINK-14068 [2] was created as a consequence of this
> discussion.
>
> I started working on marking the APIs as deprecated in FLINK-32570 [3]
> where Chesnay raised a fair point that there isn't a FLIP, yet, to
> formalize this public API change. Therefore, I went ahead and created
> FLIP-335 [4] to have this change properly documented.
>
> I'm not 100% sure whether there are better ways of checking whether we're
> covering everything Public API-related. There are even classes which I
> think might be user-facing but are not labeled accordingly (e.g.
> flink-cep). But I don't have the proper knowledge in these parts of the
> code. Therefore, I would propose marking these methods as deprecated,
> anyway, to be on the safe side.
>
> I'm open to any suggestions on improving the Test Plan of this change.
>
> I'm looking forward to feedback on this FLIP.
>
> Best,
> Matthias
>
> [1] https://lists.apache.org/thread/76yywnwf3lk8qn4dby0vz7yoqx7f7pkj
> [2] https://issues.apache.org/jira/browse/FLINK-14068
> [3] https://issues.apache.org/jira/browse/FLINK-32570
> [4]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-335%3A+Removing+Flink%27s+Time+classes
>

[DISCUSS] FLIP-335: Removing Flink's Time classes as part of Flink 2.0

[Matthias Pohl]

The 2.0 feature list includes the removal of Flink's Time classes in favor
of the JDKs java.time.Duration class. There was already a discussion about
it in [1] and FLINK-14068 [2] was created as a consequence of this
discussion.

I started working on marking the APIs as deprecated in FLINK-32570 [3]
where Chesnay raised a fair point that there isn't a FLIP, yet, to
formalize this public API change. Therefore, I went ahead and created
FLIP-335 [4] to have this change properly documented.

I'm not 100% sure whether there are better ways of checking whether we're
covering everything Public API-related. There are even classes which I
think might be user-facing but are not labeled accordingly (e.g.
flink-cep). But I don't have the proper knowledge in these parts of the
code. Therefore, I would propose marking these methods as deprecated,
anyway, to be on the safe side.

I'm open to any suggestions on improving the Test Plan of this change.

I'm looking forward to feedback on this FLIP.

Best,
Matthias

[1] https://lists.apache.org/thread/76yywnwf3lk8qn4dby0vz7yoqx7f7pkj
[2] https://issues.apache.org/jira/browse/FLINK-14068
[3] https://issues.apache.org/jira/browse/FLINK-32570
[4]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-335%3A+Removing+Flink%27s+Time+classes

[jira] [Created] (FLINK-32588) Flink ML unittest BoundedPerRoundStreamIterationITCase failed

[Jiang Xin (Jira)]

Jiang Xin created FLINK-32588:
-

 Summary: Flink ML unittest BoundedPerRoundStreamIterationITCase 
failed
 Key: FLINK-32588
 URL: https://issues.apache.org/jira/browse/FLINK-32588
 Project: Flink
  Issue Type: Bug
  Components: Library / Machine Learning
Reporter: Jiang Xin
 Fix For: ml-2.4.0


[https://github.com/apache/flink-ml/actions/runs/5306457279/jobs/9604069705]
[https://github.com/apache/flink-ml/actions/runs/5166305530/jobs/9306327867]

 

The error message is as below.
{code:java}
Error:  testPerRoundIterationWithState  Time elapsed: 7.192 s  <<< FAILURE!
620java.lang.AssertionError: expected:<3> but was:<4>
621 at org.junit.Assert.fail(Assert.java:89)
622 at org.junit.Assert.failNotEquals(Assert.java:835)
623 at org.junit.Assert.assertEquals(Assert.java:647)
624 at org.junit.Assert.assertEquals(Assert.java:633)
625 at 
org.apache.flink.test.iteration.BoundedPerRoundStreamIterationITCase.testPerRoundIterationWithState(BoundedPerRoundStreamIterationITCase.java:170)
626 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
627 at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
628 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
629 at java.lang.reflect.Method.invoke(Method.java:498) {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-330: Support specifying record timestamp requirement

[Matt Wang]

Hi Yunfeng,

Thanks for the proposal. The POC showed a performance improvement of 20%, which 
is very exciting. But I have some questions:
1. Is the performance improvement here mainly due to the reduction of 
serialization, or is it due to the judgment consumption caused by tags?
2. Watermark is not needed in some scenarios, but the latency maker is a useful 
function. If the latency maker cannot be used, it will greatly limit the usage 
scenarios. Whether the solution design can retain the capability of the latency 
marker;
3. The data of the POC test is of long type. Here I want to see how much profit 
it will have if it is a string with a length of 100B or 1KB.


--

Best,
Matt Wang


 Replied Message 
| From | Yunfeng Zhou |
| Date | 07/13/2023 14:52 |
| To |  |
| Subject | Re: [DISCUSS] FLIP-330: Support specifying record timestamp 
requirement |
Hi Jing,

Thanks for reviewing this FLIP.

1. I did change the names of some APIs in the FLIP compared with the
original version according to which I implemented the POC. As the core
optimization logic remains the same and the POC's performance can
still reflect the current FLIP's expected improvement, I have not
updated the POC code after that. I'll add a note on the benchmark
section of the FLIP saying that the namings in the POC code might be
outdated, and FLIP is still the source of truth for our proposed
design.

2. This FLIP could bring a fixed reduction on the workload of the
per-record serialization path in Flink, so if the absolute time cost
by non-optimized components could be lower, the performance
improvement of this FLIP would be more obvious. That's why I chose to
enable object-reuse and to transmit Boolean values in serialization.
If it would be more widely regarded as acceptable for a benchmark to
adopt more commonly-applied behavior(for object reuse, I believe
disable is more common), I would be glad to update the benchmark
result to disable object reuse.

Best regards,
Yunfeng


On Thu, Jul 13, 2023 at 6:37 AM Jing Ge  wrote:

Hi Yunfeng,

Thanks for the proposal. It makes sense to offer the optimization. I got
some NIT questions.

1. I guess you changed your thoughts while coding the POC, I found
pipeline.enable-operator-timestamp in the code but  is
pipeline.force-timestamp-support defined in the FLIP
2. about the benchmark example, why did you enable object reuse? Since It
is an optimization of serde, will the benchmark be better if it is
disabled?

Best regards,
Jing

On Mon, Jul 10, 2023 at 11:54 AM Yunfeng Zhou 
wrote:

Hi all,

Dong(cc'ed) and I are opening this thread to discuss our proposal to
support optimizing StreamRecord's serialization performance.

Currently, a StreamRecord would be converted into a 1-byte tag (+
8-byte timestamp) + N-byte serialized value during the serialization
process. In scenarios where timestamps and watermarks are not needed,
and latency tracking is enabled, this process would include
unnecessary information in the serialized byte array. This FLIP aims
to avoid such overhead and increases Flink job's performance during
serialization.

Please refer to the FLIP document for more details about the proposed
design and implementation. We welcome any feedback and opinions on
this proposal.

Best regards, Dong and Yunfeng

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-330%3A+Support+specifying+record+timestamp+requirement

[jira] [Created] (FLINK-32587) The results returned from the CDC sql query are null or the value was changed unexpectly

[jasonliangyc (Jira)]

jasonliangyc created FLINK-32587:


 Summary: The results returned from the CDC sql query are null or 
the value was changed unexpectly
 Key: FLINK-32587
 URL: https://issues.apache.org/jira/browse/FLINK-32587
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Client
Affects Versions: 1.17.1, 1.17.0
Reporter: jasonliangyc
 Attachments: image-2023-07-13-17-35-32-235.png, 
image-2023-07-13-17-37-56-908.png

I created a CDC table as below and then run the query 'select * from so_cdc' 
through sql-client, it gives me the unexpected results.
{code:java}
CREATE TABLE so_cdc (
   REC_ID STRING,
   Create_Date TIMESTAMP(3),
   PRIMARY KEY (REC_ID) NOT ENFORCED
 ) WITH (
    'connector' = 'sqlserver-cdc',
    'hostname' = '',
    'port' = '',
    'username' = 'xxx',
    'password' = '',
    'database-name' = '',
    'schema-name' = '',
    'table-name' = 'xxx',
    'scan.startup.mode' = 'latest-offset'
 ); {code}
Run the query for the first time, the data look normal.

!image-2023-07-13-17-35-32-235.png|width=535,height=141!

 

But after i run the same query multiple times, it gives me the unexpected data, 
and i'm sure that these two columns of my cdc source table don't contain these 
data

!image-2023-07-13-17-37-56-908.png|width=469,height=175!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-32586) Enable input locality in SimpleExecutionSlotAllocator

[xingbe (Jira)]

xingbe created FLINK-32586:
--

 Summary: Enable input locality in SimpleExecutionSlotAllocator
 Key: FLINK-32586
 URL: https://issues.apache.org/jira/browse/FLINK-32586
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Affects Versions: 1.18.0
Reporter: xingbe
 Fix For: 1.18.0


At present, the AdaptiveBatchScheduler uses the `SimpleExecutionSlotAllocator` 
to assign slot to execution, but it currently lacks support for the capability 
of input locality, which may increase unnecessary data transmission overhead. 
In this issue, we aim to enable the `SimpleExecutionSlotAllocator` to support 
the input locality.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Flink 2.0 connectors, formats, and even user code should also be pluggable.

[zhiqiang li]

Flink CDC Maintainer
xbjt...@gmail.com

Factory Interface
Timo Walther 

Plugin Interface
Aleksey Pak 

Connector Base
Stephan Ewen 

Flink File System
Stephan Ewen 

I have observed that in [1], connectors and formats are pluggable, allowing 
user code to be easily integrated. The advantages of having pluggable 
connectors are evident, as it helps avoid conflicts between different versions 
of jar packages. If classloader isolation is not used, shading becomes 
necessary to resolve conflicts, resulting in a significant waste of development 
time. I understand that implementing this change may require numerous API 
modifications, so I would like to discuss in this email whether it is feasible 
to make these changes in Flink 2.0.

> Plugins cannot access classes from other plugins or from Flink that have not 
> been specifically whitelisted.
> This strict isolation allows plugins to contain conflicting versions of the 
> same library without the need to relocate classes or to converge to common 
> versions.
> Currently, file systems and metric reporters are pluggable but, in the 
> future, connectors, formats, and even user code should also be pluggable.

I hope to be able to integrate into a set of APIs, which would require 
significant changes to the integration API. Is it possible to address this in 
Flink 2.0? However, due to [2] FLIP-321, Flink 1.18 should deprecate some APIs, 
and the community is uncertain about the existence of Flink 1.20. This may make 
it impossible to integrate a set of APIs. @Xintong Song  
suggests continuing to use two APIs.

I am aware of some API modifications, but considering the extensive changes 
involved, I am concerned that my understanding may not be comprehensive. I am 
willing to do the work, but I have not yet found a committer.


[1] 
https://nightlies.apache.org/Flink/Flink-docs-master/docs/deployment/filesystems/overview/
[2] FLIP-321  
https://cwiki.apache.org/confluence/display/FLINK/FLIP-321%3A+Introduce+an+API+deprecation+process


Best,
Zhiqiang

Re: [DISCUSS] Update Flink Roadmap

[Jiabao Sun]

Thanks Jark and Martijn for driving this.

There are two suggestions about the Table API:

- Add the JSON type to adapt to the no sql database type.
- Remove changelog normalize operator for upsert stream.


Best,
Jiabao


> 2023年7月13日 下午3:49，Jark Wu  写道：
> 
> Hi all,
> 
> Sorry for taking so long back here.
> 
> Martijn and I have drafted the first version of the updated roadmap,
> including the updated feature radar reflecting the current state of
> different components.
> https://docs.google.com/document/d/12BDiVKEsY-f7HI3suO_IxwzCmR04QcVqLarXgyJAb7c/edit
> 
> Feel free to leave comments in the thread or the document.
> We may miss mentioning something important, so your help in enriching
> the content is greatly appreciated.
> 
> Best,
> Jark & Martijn
> 
> 
> On Fri, 2 Jun 2023 at 00:50, Jing Ge  wrote:
> 
>> Hi Jark,
>> 
>> Fair enough. Let's do it like you suggested. Thanks!
>> 
>> Best regards,
>> Jing
>> 
>> On Thu, Jun 1, 2023 at 6:00 PM Jark Wu  wrote:
>> 
>>> Hi Jing,
>>> 
>>> This thread is for discussing the roadmap for versions 1.18, 2.0, and
>> even
>>> more.
>>> One of the outcomes of this discussion will be an updated version of the
>>> current roadmap.
>>> Let's work together on refining the roadmap in this thread.
>>> 
>>> Best,
>>> Jark
>>> 
>>> On Thu, 1 Jun 2023 at 23:25, Jing Ge  wrote:
>>> 
 Hi Jark,
 
 Thanks for driving it! For point 2, since we are developing 1.18 now,
 does it make sense to update the roadmap this time while we are
>> releasing
 1.18? This discussion thread will be focusing on the Flink 2.0 roadmap,
>>> as
 you mentioned previously. WDYT?
 
 Best regards,
 Jing
 
 On Thu, Jun 1, 2023 at 3:31 PM Jark Wu  wrote:
 
> Hi all,
> 
> Martijn and I would like to initiate a discussion on the Flink
>> roadmap,
> which should cover the project's long-term roadmap and the regular
>>> update
> mechanism.
> 
> Xintong has already started a discussion about Flink 2.0 planning.
>> One
>>> of
> the points raised in that discussion is that we should have a
>>> high-level
> discussion of the roadmap to present where the project is heading
>>> (which
> doesn't necessarily need to block the Flink 2.0 planning). Moreover,
>>> the
> roadmap on the Flink website [1] hasn't been updated for half a year,
>>> and
> the last update was for the feature radar for the 1.15 release. It
>> has
 been
> 2 years since the community discussed Flink's overall roadmap.
> 
> I would like to raise two topics for discussion:
> 
> 1. The new roadmap. This should be an updated version of the current
> roadmap[1].
> 2. A mechanism to regularly discuss and update the roadmap.
> 
> To make the first topic discussion more efficient, Martijn and I
 volunteer
> to summarize the ongoing big things of different components and
>>> present a
> roadmap draft to the community in the next few weeks. This should be
>> a
 good
> starting point for a more detailed discussion.
> 
> Regarding the regular update mechanism, there was a proposal in a
>>> thread
> [2] three years ago to make the release manager responsible for
>>> updating
> the roadmap. However, it appears that this was not documented as a
 release
> management task [3], and the roadmap update wasn't performed for
>>> releases
> 1.16 and 1.17.
> 
> In my opinion, making release managers responsible for keeping the
 roadmap
> up to date is a good idea. Specifically, release managers of release
>> X
 can
> kick off the roadmap update at the beginning of release X, which can
>>> be a
> joint task with collecting a feature list [4]. Additionally, release
> managers of release X-1 can help verify and remove the accomplished
>>> items
> from the roadmap and update the feature radar.
> 
> What do you think? Do you have other ideas?
> 
> Best,
> Jark & Martijn
> 
> [1]: https://flink.apache.org/roadmap.html
> [2]:
>> https://lists.apache.org/thread/o0l3cg6yphxwrww0k7215jgtw3yfoybv
> [3]:
> 
 
>>> 
>> https://cwiki.apache.org/confluence/display/FLINK/Flink+Release+Management
> [4]: https://cwiki.apache.org/confluence/display/FLINK/1.18+Release
> 
 
>>> 
>> 

Re: [DISCUSS] Update Flink Roadmap

[Jark Wu]

Hi all,

Sorry for taking so long back here.

Martijn and I have drafted the first version of the updated roadmap,
including the updated feature radar reflecting the current state of
different components.
https://docs.google.com/document/d/12BDiVKEsY-f7HI3suO_IxwzCmR04QcVqLarXgyJAb7c/edit

Feel free to leave comments in the thread or the document.
We may miss mentioning something important, so your help in enriching
the content is greatly appreciated.

Best,
Jark & Martijn


On Fri, 2 Jun 2023 at 00:50, Jing Ge  wrote:

> Hi Jark,
>
> Fair enough. Let's do it like you suggested. Thanks!
>
> Best regards,
> Jing
>
> On Thu, Jun 1, 2023 at 6:00 PM Jark Wu  wrote:
>
> > Hi Jing,
> >
> > This thread is for discussing the roadmap for versions 1.18, 2.0, and
> even
> > more.
> > One of the outcomes of this discussion will be an updated version of the
> > current roadmap.
> > Let's work together on refining the roadmap in this thread.
> >
> > Best,
> > Jark
> >
> > On Thu, 1 Jun 2023 at 23:25, Jing Ge  wrote:
> >
> > > Hi Jark,
> > >
> > > Thanks for driving it! For point 2, since we are developing 1.18 now,
> > > does it make sense to update the roadmap this time while we are
> releasing
> > > 1.18? This discussion thread will be focusing on the Flink 2.0 roadmap,
> > as
> > > you mentioned previously. WDYT?
> > >
> > > Best regards,
> > > Jing
> > >
> > > On Thu, Jun 1, 2023 at 3:31 PM Jark Wu  wrote:
> > >
> > > > Hi all,
> > > >
> > > > Martijn and I would like to initiate a discussion on the Flink
> roadmap,
> > > > which should cover the project's long-term roadmap and the regular
> > update
> > > > mechanism.
> > > >
> > > > Xintong has already started a discussion about Flink 2.0 planning.
> One
> > of
> > > > the points raised in that discussion is that we should have a
> > high-level
> > > > discussion of the roadmap to present where the project is heading
> > (which
> > > > doesn't necessarily need to block the Flink 2.0 planning). Moreover,
> > the
> > > > roadmap on the Flink website [1] hasn't been updated for half a year,
> > and
> > > > the last update was for the feature radar for the 1.15 release. It
> has
> > > been
> > > > 2 years since the community discussed Flink's overall roadmap.
> > > >
> > > > I would like to raise two topics for discussion:
> > > >
> > > > 1. The new roadmap. This should be an updated version of the current
> > > > roadmap[1].
> > > > 2. A mechanism to regularly discuss and update the roadmap.
> > > >
> > > > To make the first topic discussion more efficient, Martijn and I
> > > volunteer
> > > > to summarize the ongoing big things of different components and
> > present a
> > > > roadmap draft to the community in the next few weeks. This should be
> a
> > > good
> > > > starting point for a more detailed discussion.
> > > >
> > > > Regarding the regular update mechanism, there was a proposal in a
> > thread
> > > > [2] three years ago to make the release manager responsible for
> > updating
> > > > the roadmap. However, it appears that this was not documented as a
> > > release
> > > > management task [3], and the roadmap update wasn't performed for
> > releases
> > > > 1.16 and 1.17.
> > > >
> > > > In my opinion, making release managers responsible for keeping the
> > > roadmap
> > > > up to date is a good idea. Specifically, release managers of release
> X
> > > can
> > > > kick off the roadmap update at the beginning of release X, which can
> > be a
> > > > joint task with collecting a feature list [4]. Additionally, release
> > > > managers of release X-1 can help verify and remove the accomplished
> > items
> > > > from the roadmap and update the feature radar.
> > > >
> > > > What do you think? Do you have other ideas?
> > > >
> > > > Best,
> > > > Jark & Martijn
> > > >
> > > > [1]: https://flink.apache.org/roadmap.html
> > > > [2]:
> https://lists.apache.org/thread/o0l3cg6yphxwrww0k7215jgtw3yfoybv
> > > > [3]:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Release+Management
> > > > [4]: https://cwiki.apache.org/confluence/display/FLINK/1.18+Release
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-330: Support specifying record timestamp requirement

[Yunfeng Zhou]

Hi Jing,

Thanks for reviewing this FLIP.

1. I did change the names of some APIs in the FLIP compared with the
original version according to which I implemented the POC. As the core
optimization logic remains the same and the POC's performance can
still reflect the current FLIP's expected improvement, I have not
updated the POC code after that. I'll add a note on the benchmark
section of the FLIP saying that the namings in the POC code might be
outdated, and FLIP is still the source of truth for our proposed
design.

2. This FLIP could bring a fixed reduction on the workload of the
per-record serialization path in Flink, so if the absolute time cost
by non-optimized components could be lower, the performance
improvement of this FLIP would be more obvious. That's why I chose to
enable object-reuse and to transmit Boolean values in serialization.
If it would be more widely regarded as acceptable for a benchmark to
adopt more commonly-applied behavior(for object reuse, I believe
disable is more common), I would be glad to update the benchmark
result to disable object reuse.

Best regards,
Yunfeng


On Thu, Jul 13, 2023 at 6:37 AM Jing Ge  wrote:
>
> Hi Yunfeng,
>
> Thanks for the proposal. It makes sense to offer the optimization. I got
> some NIT questions.
>
> 1. I guess you changed your thoughts while coding the POC, I found
> pipeline.enable-operator-timestamp in the code but  is
> pipeline.force-timestamp-support defined in the FLIP
> 2. about the benchmark example, why did you enable object reuse? Since It
> is an optimization of serde, will the benchmark be better if it is
> disabled?
>
> Best regards,
> Jing
>
> On Mon, Jul 10, 2023 at 11:54 AM Yunfeng Zhou 
> wrote:
>
> > Hi all,
> >
> > Dong(cc'ed) and I are opening this thread to discuss our proposal to
> > support optimizing StreamRecord's serialization performance.
> >
> > Currently, a StreamRecord would be converted into a 1-byte tag (+
> > 8-byte timestamp) + N-byte serialized value during the serialization
> > process. In scenarios where timestamps and watermarks are not needed,
> > and latency tracking is enabled, this process would include
> > unnecessary information in the serialized byte array. This FLIP aims
> > to avoid such overhead and increases Flink job's performance during
> > serialization.
> >
> > Please refer to the FLIP document for more details about the proposed
> > design and implementation. We welcome any feedback and opinions on
> > this proposal.
> >
> > Best regards, Dong and Yunfeng
> >
> > [1]
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-330%3A+Support+specifying+record+timestamp+requirement
> >

Re: [DISCUSS] FLIP 333 - Redesign Apache Flink website

[Mohan, Deepthi]

Thank you all for the feedback so far.

@Xintong, as you may know, AWS provides a managed Apache Flink service called 
Kinesis Data Analytics. We regularly speak to our customers (these range from 
startups to large enterprises) in various stages of familiarity with Apache 
Flink. Most comments about the current Apache Flink website come from 
developers new to Flink and decision makers (tech leads or higher) who look 
through the website to quickly understand the value proposition of Flink and 
why they should use it. On the other hand, most developers already familiar 
with Flink do not typically use the Flink website, rather we see them spend 
most of their time on Flink documentation. However, even these developers when 
explicitly asked in our conversations often comment that the website could do 
with a redesign.

@Mattias, there are a few more pages in the current website that would fall in 
to the "only design changes" category. I will edit the FLIP to include all of 
them and we can have a discussion about each with the mockup once ready. See 
next step below for details

As far as the time to redesign is concerned, I believe that depends on how many 
engineers with front end experience have capacity to help (any volunteers?). We 
have a few engineers willing and interested to contribute from AWS. I am not 
convinced we need to wait until the 2.0 release or any version release for that 
matter. We can completely decouple the website changes so that we are able to 
execute on the changes quickly once we solidify the design for a set of pages.

Next step: I am happy to provide mockups for each of the pages (the ones with 
minor content change and the ones with no content change) and also show how the 
theme will apply to documentation only for consideration. We may choose not to 
make any changes to documentation in this FLIP, but it may be useful to 
visualize before we decide. Given that we aren't really making any major 
content changes at this point, I think a FLIP for each page will not be 
necessary. Rather, we could use this discussion thread for specific feedback 
for each page that will be changed.


On 7/12/23, 12:59 PM, "Danny Cranmer" mailto:dannycran...@apache.org>> wrote:


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.






Thanks for driving this Deepthi.


Looks great to me; a huge improvement over the current website. We had
agreed in the previous discussion [1] to keep the main website separate
from the docs since the target audience is different. I am inclined to
decouple the two and re-evaluate updating the docs in a separate FLIP later.


Thanks,
Danny


[1] https://lists.apache.org/thread/c3pt00cf77lrtgt242p26lgp9l2z5yc8 



On Tue, Jul 11, 2023 at 2:38 PM Jingsong Li mailto:jingsongl...@gmail.com>> wrote:


> It's exciting to finally have someone to refactor the Flink website.
>
> Thanks Deepthi.
>
> To Xintong,
>
> > maintain two sets of website designs at the same time
>
> If our website is not so complex, and new webui can be full feature
> coverage, there would be no need to maintain two sets of web.
>
> To Jing
>
> +1 to consider Flink doc too.
>
> Best,
> Jingsong
>
> On Tue, Jul 11, 2023 at 8:16 PM Jing Ge  lid>
> wrote:
> >
> > Hi,
> >
> > +1, the UI design looks good!
> >
> > Commonly speaking, there are two parts of the whole website: Flink Web
> and
> > Flink doc. Will the dart mode also cover Flink doc?
> >
> > Best regards,
> > Jing
> >
> > On Tue, Jul 11, 2023 at 12:40 PM Matthias Pohl
> > mailto:matthias.p...@aiven.io.inva>lid> wrote:
> >
> > > I also like the proposed designs. Considering that you want to touch
> > > individual subpages, there are also some subpages of Flink's website
> not
> > > being mentioned in the FLIP (e.g. roadmap [1]). What is the plan with
> > > those? Are they covered by the "We recommend modifications only to the
> > > design of the following pages" part but are not listed there?
> > >
> > > Additionally, it would be nice to get a bit more insight into the
> feedback
> > > from users as Xintong pointed out. It would be interesting to
> understand
> > > how the new design helps solving certain problems (besides having the
> > > motivation to modernize the look and feel).
> > >
> > > I'm also wondering whether it's doable to do a discussion (FLIP?) per
> > > subpage on the design (as proposed by Chesnay in the first discussion
> on
> > > this topic [2] to have smaller changes rather than a single big one).
> But I
> > > could imagine this being quite tedious because different people might
> > > have different opinions on how something should be done.
> > >
> > > I don't have any experience with frontend design. I'm wondering how
> much
> > > time such a redesign takes. Could it be linked to the 2.0 release?
> > >
>