2BCalls%2BWhile%2BConstant-folding=gmail-imap=171573131400=AOvVaw3sVTK3M3Qs45haptzQbUmo
> >>
> >> [2]
> >> https://www.google.com/url?q=https://lists.apache.org/thread/ko5ndv5kr87nm011psll2hzzd0nn3ztz=gmail-imap=171573131400=AOvVaw3YKYwhLhbgWkX5hbzHRW31
> >>
> >> Thanks,
> >> Alan
> >
>
--
Best,
Benchao Li
chnical vision of Flink SQL and
> >> participated in plenty of discussions in the dev mailing list. Besides
> >> that,
> >> he is community-minded, such as being the release manager of 1.19,
> >> verifying releases, managing release syncs, writing the release
> >> announcement etc.
> >>
> >> Congratulations and welcome Lincoln!
> >>
> >> Best,
> >> Jark (on behalf of the Flink PMC)
>
--
Best,
Benchao Li
> >>> Congratulations, Jing!
> >>>
> >>> Best,
> >>> Ron
> >>>
> >>> Junrui Lee jrlee@gmail.com 于2024年4月12日周五 18:54写道：
> >>>
> >>>> Congratulations, Jing!
> >>>>
> >>>> Best,
> >>>> Junrui
> >>>>
> >>>> Aleksandr Pilipenko z3d...@gmail.com 于2024年4月12日周五 18:28写道：
> >>>>
> >>>>> Congratulations, Jing!
> >>>>>
> >>>>> Best Regards,
> >>>>> Aleksandr
> >>
>
--
Best,
Benchao Li
t;>
> > > >> > > >> I just created an issue to track the things we need to modify
> > > [2],
> > > >> > > >> please comment on it if you feel that something is missing. You
> > > can
> > > >> > > >> refer to apache documentation [1] too.
> > > >> > > >>
> > > >> > > >> And, we already completed the GitHub repo migration [3], please
> > > update
> > > >> > > >> your local git repo to track the new repo [4].
> > > >> > > >>
> > > >> > > >> You can run the following command to complete the remote repo
> > > tracking
> > > >> > > >> migration.
> > > >> > > >>
> > > >> > > >> git remote set-url origin https://github.com/apache/paimon.git
> > > >> > > >>
> > > >> > > >> If you have a different name, please change the 'origin' to
> > your
> > > remote
> > > >> > > name.
> > > >> > > >>
> > > >> > > >> Please join me in celebrating!
> > > >> > > >>
> > > >> > > >> [1]
> > > >> > >
> > >
> > https://incubator.apache.org/guides/transferring.html#life_after_graduation
> > > >> > > >> [2] https://github.com/apache/paimon/issues/3091
> > > >> > > >> [3] https://issues.apache.org/jira/browse/INFRA-25630
> > > >> > > >> [4] https://github.com/apache/paimon
> > > >> > > >>
> > > >> > > >> Best,
> > > >> > > >> Jingsong Lee
> > > >> > >
> > > >> > >
> > >
> >
--
Best,
Benchao Li
For example:
>
> - pre-partitioned columns and aggregate columns are the same but have
> different order (e.g., source pre-partitioned w.r.t. a,b and aggregate has
> a GROUP BY b,a)
> - columns in the Exchange operator is a list-prefix of pre-partitoned
> columns of source (e.g.,
gt; > > > Welcome to explore the new repository and documentation. Your feedback
> > and
> > > > contributions are invaluable as we continue to improve Flink CDC.
> > > >
> > > > Thanks everyone for your support and happy exploring Flink CDC!
> > > >
> > > > Best，
> > > > Leonard
> > > > [1] https://lists.apache.org/thread/cw29fhsp99243yfo95xrkw82s5s418ob
> > > >
> > > >
> > >
> > > --
> > > Best
> > >
> > > ConradJam
> >
--
Best,
Benchao Li
FLINK/FLIP-436%3A+Introduce+Catalog-related+Syntax
> > [2] https://lists.apache.org/thread/10k1bjb4sngyjwhmfqfky28lyoo7sv0z
> >
> > Best regards,
> > Yubin
> >
--
Best,
Benchao Li
lable, and accurate data
> > > > streaming
> > > > >> applications.
> > > > >>>>
> > > > >>>> The release is available for download at:
> > > > >>>> https://flink.apache.org/downloads.html
> > > > >>>>
> > > > >>>> Please check out the release blog post for an overview of the
> > > > >> improvements for this bugfix release:
> > > > >>>>
> > > > >>
> > > >
> > > https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/
> > > > >>>>
> > > > >>>> The full release notes are available in Jira:
> > > > >>>>
> > > > >>
> > > >
> > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282
> > > > >>>>
> > > > >>>> We would like to thank all contributors of the Apache Flink
> > > community
> > > > >> who made this release possible!
> > > > >>>>
> > > > >>>>
> > > > >>>> Best,
> > > > >>>> Yun, Jing, Martijn and Lincoln
> > > > >>
> > > >
> > > >
> > >
> >
> >
> > --
> > Best,
> > Hangxiang.
--
Best,
Benchao Li
urces. For Flink, with FLIP-434, the proposed planner rule
> > > > could remove the shuffle.
> > > >
> > > > Definitely a fun idea; I look forward to hearing more!
> > > >
> > > > Cheers,
> > > >
> > > > Jim
> > > >
> > > >
> > > > 1.
> > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-376%3A+Add+DISTRIBUTED+BY+clause
> > > > 2.
> > > >
> > > >
> > >
> > https://docs.ksqldb.io/en/latest/developer-guide/joins/partition-data/#co-partitioning-requirements
> > > >
> > > > On Sun, Mar 10, 2024 at 3:38 PM Jeyhun Karimov
> > > > wrote:
> > > >
> > > > > Hi devs,
> > > > >
> > > > > I’d like to start a discussion on FLIP-434: Support optimizations for
> > > > > pre-partitioned data sources [1].
> > > > >
> > > > > The FLIP introduces taking advantage of pre-partitioned data sources
> > > for
> > > > > SQL/Table API (it is already supported as experimental feature in
> > > > > DataStream API [2]).
> > > > >
> > > > >
> > > > > Please find more details in the FLIP wiki document [1].
> > > > > Looking forward to your feedback.
> > > > >
> > > > > Regards,
> > > > > Jeyhun
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-434%3A+Support+optimizations+for+pre-partitioned+data+sources
> > > > > [2]
> > > > >
> > > > >
> > > >
> > >
> > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/experimental/
> > > > >
> > > >
> > >
> >
--
Best,
Benchao Li
> > > > has been introduced in FLIP-295 [3], we could use this component
> > to
> > > > > > > implement such a long-awaited
> > > > > > > feature, Please refer to the document [1] for implementation
> > > details.
> > > > > > >
> > > > > > > examples as follows:
> > > > > > >
> > > > > > > Flink SQL> create catalog cat2 WITH ('type'='generic_in_memory',
> > > > > > > > 'default-database'='db');
> > > > > > > > [INFO] Execute statement succeeded.
> > > > > > > > Flink SQL> show create catalog cat2;
> > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > >
> > ++
> > > > > > > > | result |
> > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > >
> > ++
> > > > > > > > | CREATE CATALOG `cat2` WITH (
> > > > > > > > 'default-database' = 'db',
> > > > > > > > 'type' = 'generic_in_memory'
> > > > > > > > )
> > > > > > > > |
> > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > >
> > ++
> > > > > > > > 1 row in set
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Looking forward to hearing from you, thanks!
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Yubin
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > >
> > > > >
> > >
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=296290756
> > > > > > > [2] https://issues.apache.org/jira/browse/FLINK-24939
> > > > > > > [3]
> > > > > > >
> > > > > >
> > > > >
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > > > >
> > > > >
> > >
> >
--
Best,
Benchao Li
; > > > > > > > +--+
> > > > > > > > > > > 1 row in set
> > > > > > > > > > >
> > > > > > > > > > > Flink SQL> DESCRIBE DATABASE default_database;
> > > > > > > > > > > [ERROR] Could not execute SQL statement. Reason:
> > > > > > > > > > > org.apache.calcite.sql.validate.SqlValidatorException:
> > > Column
> > > > > > > > > > > 'default_database' not found in
> > > > > > > > > > > any table
> > > > > > > > > > > ```
> > > > > > > > > > >
> > > > > > > > > > > Is this an error in the release notes, or my mistake in
> > > > > > > interpreting
> > > > > > > > > > them?
> > > > > > > > > > >
> > > > > > > > > > > thanks, Robin.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > [1]
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282
> > > > > > > > > > >
> > > > > > > > > > > On Thu, 7 Mar 2024 at 10:01, Lincoln Lee <
> > > > > lincoln.8...@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > >
> > > > > > > > > > > > Please review and vote on the release candidate #2 for
> > > the
> > > > > > > version
> > > > > > > > > > > 1.19.0,
> > > > > > > > > > > > as follows:
> > > > > > > > > > > > [ ] +1, Approve the release
> > > > > > > > > > > > [ ] -1, Do not approve the release (please provide
> > > specific
> > > > > > > > comments)
> > > > > > > > > > > >
> > > > > > > > > > > > The complete staging area is available for your review,
> > > which
> > > > > > > > > includes:
> > > > > > > > > > > >
> > > > > > > > > > > > * JIRA release notes [1], and the pull request adding
> > > release
> > > > > > > note
> > > > > > > > > for
> > > > > > > > > > > > users [2]
> > > > > > > > > > > > * the official Apache source release and binary
> > > convenience
> > > > > > > > releases
> > > > > > > > > to
> > > > > > > > > > > be
> > > > > > > > > > > > deployed to dist.apache.org [3], which are signed with
> > > the
> > > > > key
> > > > > > > > with
> > > > > > > > > > > > fingerprint E57D30ABEE75CA06 [4],
> > > > > > > > > > > > * all artifacts to be deployed to the Maven Central
> > > > > Repository
> > > > > > > [5],
> > > > > > > > > > > > * source code tag "release-1.19.0-rc2" [6],
> > > > > > > > > > > > * website pull request listing the new release and
> > adding
> > > > > > > > > announcement
> > > > > > > > > > > blog
> > > > > > > > > > > > post [7].
> > > > > > > > > > > >
> > > > > > > > > > > > The vote will be open for at least 72 hours. It is
> > > adopted by
> > > > > > > > > majority
> > > > > > > > > > > > approval, with at least 3 PMC affirmative votes.
> > > > > > > > > > > >
> > > > > > > > > > > > [1]
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282
> > > > > > > > > > > > [2] https://github.com/apache/flink/pull/24394
> > > > > > > > > > > > [3]
> > > > > > > https://dist.apache.org/repos/dist/dev/flink/flink-1.19.0-rc2/
> > > > > > > > > > > > [4]
> > > https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > > > > > > > > > [5]
> > > > > > > > > > >
> > > > > > > >
> > > > >
> > https://repository.apache.org/content/repositories/orgapacheflink-1709
> > > > > > > > > > > > [6]
> > > > > > > >
> > https://github.com/apache/flink/releases/tag/release-1.19.0-rc2
> > > > > > > > > > > > [7] https://github.com/apache/flink-web/pull/721
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Yun, Jing, Martijn and Lincoln
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
--
Best,
Benchao Li
s technical contributions, Jiabao is an active member of our
> community, participating in the mailing list and consistently volunteering
> for release verifications and code reviews with enthusiasm.
>
> Please join me in congratulating Jiabao for becoming an Apache Flink
> committer!
>
> Best,
> Qingsheng (on behalf of the Flink PMC)
--
Best,
Benchao Li
> > > > ASF Confluence site since a recent change. One of the results of
> > > this
> > > > > > > decision is that users can't signup and access Confluence, so
> > only
> > > > > > > committers+ can create FLIPs.
> > > > > > >
> > > > > > > ASF Infra hopes to improve this situation when they move to the
> > > Cloud
> > > > > > > shortly (as in: some months), but they haven't committed on an
> > > actual
> > > > > > > date. The idea would be that we find a temporary solution until
> > > > anyone
> > > > > > > can request access to Confluence.
> > > > > > >
> > > > > > > There are a couple of ways we could resolve this situation:
> > > > > > > 1. Contributors create a Google Doc and make that view-only, and
> > > post
> > > > > > > that Google Doc to the mailing list for a discussion thread. When
> > > the
> > > > > > > discussions have been resolved, the contributor ask on the Dev
> > > > mailing
> > > > > > > list to a committer/PMC to copy the contents from the Google Doc,
> > > and
> > > > > > > create a FLIP number for them. The contributor can then use that
> > > FLIP
> > > > > > > to actually have a VOTE thread.
> > > > > > > 2. We could consider moving FLIPs to "Discussions" on Github,
> > like
> > > > > > > Airflow does at https://github.com/apache/airflow/discussions
> > > > > > > 3. Perhaps someone else has another good idea.
> > > > > > >
> > > > > > > Looking forward to your thoughts.
> > > > > > >
> > > > > > > Best regards,
> > > > > > >
> > > > > > > Martijn
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > Best regards,
> > Sergey
> >
--
Best,
Benchao Li
> >>>> From: David Morávek mailto:d...@apache.org>>
> >>>> Sent: Wednesday, January 10, 2024 6:26
> >>>> To: dev@flink.apache.org<mailto:dev@flink.apache.org> <
> >>> dev@flink.apache.org<mailto:dev@flink.apache.org>>; Piotr Nowojski <
> >>> piotr.nowoj...@gmail.com<mailto:piotr.nowoj...@gmail.com>>
> >>>> Subject: Re: [DISCUSS] FLIP 411: Chaining-agnostic Operator ID
> >>> generation for improved state compatibility on parallelism change
> >>>> Hi Zhanghao,
> >>>>
> >>>> Thanks for the FLIP. What you're proposing makes a lot of sense +1
> >>>>
> >>>> Have you thought about how this works with unaligned checkpoints in case
> >>>> you go from unchained to chained? I think it should be fine because this
> >>>> scenario should only apply to forward/rebalance scenarios where we, as
> >>> far
> >>>> as I recall, force alignment anyway, so there should be no exchanges to
> >>>> snapshot. It might just work, but something to double-check. Maybe @Piotr
> >>>> Nowojski mailto:piotr.nowoj...@gmail.com>>
> >>> could confirm it.
> >>>> Best,
> >>>> D.
> >>>>
> >>>> On Wed, Jan 3, 2024 at 7:10 AM Zhanghao Chen >>> <mailto:zhanghao.c...@outlook.com>>
> >>>> wrote:
> >>>>
> >>>> Dear Flink devs,
> >>>>
> >>>> I'd like to start a discussion on FLIP 411: Chaining-agnostic Operator ID
> >>>> generation for improved state compatibility on parallelism change [1].
> >>>>
> >>>> Currently, when user does not explicitly set operator UIDs, the chaining
> >>>> behavior will still affect state compatibility, as the generation of the
> >>>> Operator ID is dependent on its chained output nodes. For example, a
> >>> simple
> >>>> source->sink DAG with source and sink chained together is state
> >>>> incompatible with an otherwise identical DAG with source and sink
> >>> unchained
> >>>> (either because the parallelisms of the two ops are changed to be unequal
> >>>> or chaining is disabled). This greatly limits the flexibility to perform
> >>>> chain-breaking/building for performance tuning.
> >>>>
> >>>> The dependency on chained output nodes for Operator ID generation can be
> >>>> traced back to Flink 1.2. It is unclear at this point on why chained
> >>> output
> >>>> nodes are involved in the algorithm, but the following history background
> >>>> might be related: prior to Flink 1.3, Flink runtime takes the snapshots
> >>> by
> >>>> the operator ID of the first vertex in a chain, so it somewhat makes
> >>> sense
> >>>> to include chained output nodes into the algorithm as
> >>>> chain-breaking/building is expected to break state-compatibility anyway.
> >>>>
> >>>> Given that operator-level state recovery within a chain has long been
> >>>> supported since Flink 1.3, I propose to introduce StreamGraphHasherV3
> >>> that
> >>>> is agnostic of the chaining behavior of operators, so that users are free
> >>>> to tune the parallelism of individual operators without worrying about
> >>>> state incompatibility. We can make the V3 hasher an optional choice in
> >>>> Flink 1.19, and make it the default hasher in 2.0 for backwards
> >>>> compatibility.
> >>>>
> >>>> Looking forward to your suggestions on it, thanks~
> >>>>
> >>>> [1]
> >>>>
> >>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-411%3A+Chaining-agnostic+Operator+ID+generation+for+improved+state+compatibility+on+parallelism+change
> >>>> Best,
> >>>> Zhanghao Chen
> >>>>
> >>>
>
--
Best,
Benchao Li
Benchao Li created FLINK-34403:
--
Summary: GHA misc test cannot pass due to OOM
Key: FLINK-34403
URL: https://issues.apache.org/jira/browse/FLINK-34403
Project: Flink
Issue Type: Bug
it after CI
passes (it may take a few hours).
Sorry for the inconvenience.
[1] https://github.com/apache/flink/pull/24128
[2] https://github.com/apache/flink/pull/24133
--
Best,
Benchao Li
t; [image: favicon.ico]
> <https://lists.apache.org/thread/pd9gzslq20dtzzfphxqvwhc43hrzo2y1>
> <https://lists.apache.org/thread/pd9gzslq20dtzzfphxqvwhc43hrzo2y1>
>
>
--
Best,
Benchao Li
g feature which deserves a FLIP to describe it in detail.
> I'm forward to your feedback and suggestions before I start to do it.
>
> Best,
> Yisha
>
> [1]https://docs.snowflake.com/en/sql-reference/functions/infer_schema
> <https://docs.snowflake.com/en/sql-reference/functions/infer_schema>
--
Best,
Benchao Li
parameter,
> > after the user upgrades the version, some update details may be lost due to
> > the mini-batch connection being enabled by default, resulting in distorted
> > audit results.
> >
> >> 2024年1月11日 16:19，Benchao Li 写道：
> >>
> >>> th
remains consistent, and there is minor
> modification to the computational logic.
>
> Best,
> Xu Shuai
>
> > 2024年1月10日 22:56，Benchao Li 写道：
> >
> > Thanks shuai for driving this, mini-batch Join is a very useful
> > optimization, +1 for the general idea.
> >
> &
option is added to control the operator to avoid influencing existing
> jobs.
>
> Please find more details in the FLIP wiki document [1]. Looking
> forward to your feedback.
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-415%3A+Introduce+a+new+join+operator+to+support+minibatch
>
> Best,
> Xu Shuai
--
Best,
Benchao Li
personal email to the remaining binding voters
> > in the
> > voting mail thread for at least 2 attempts with at least 7 days between
> > two attempts.
> > > 3. If the binding voter being contacted still failed to respond after
> > all the attempts,
> > the binding voter will be considered as inactive for the purpose of this
> > particular voting.
> >
> > Welcome voting !
> >
> > Best,
> > Leonard
> > [1] https://lists.apache.org/thread/o7klnbsotmmql999bnwmdgo56b6kxx9l
> > [2]
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120731026
--
Best,
Benchao Li
d/bto7mpjvcx7d7k86owb00dwrm65jx8cn
>
>
> Best,
> Feng Jin
--
Best,
Benchao Li
g+options+for+administrator+JVM+options
> [2] https://lists.apache.org/thread/cflonyrfd1ftmyrpztzj3ywckbq41jzg
>
> Best,
> Zhanghao Chen
--
Best,
Benchao Li
Y%20resolved%20DESC%2C%20created%20DESC
> >[3] https://lists.apache.org/list?dev@flink.apache.org:lte=100M:Fedulov
> >[4] https://lists.apache.org/list?u...@flink.apache.org:lte=100M:Fedulov
> >[5]
> >https://flink.apache.org/2020/01/15/advanced-flink-application-patterns-vol.1-case-study-of-a-fraud-detection-system/
> >[6]
> >https://www.ververica.com/blog/presenting-our-streaming-concepts-introduction-to-flink-video-series
--
Best,
Benchao Li
gt; Looking forward to hearing from you.
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-397%3A+Add+config+options+for+administrator+JVM+options
> [2] https://issues.apache.org/jira/browse/SPARK-23472
>
> Best,
> Zhanghao Chen
--
Best,
Benchao Li
gt; https://github.com/apache/flink/compare/master...hackergin:flink:poc_named_argument
>
>
>
> --
>
> Best！
> Xuyang
>
>
>
>
>
> 在 2023-12-20 13:31:26，"Benchao Li" 写道：
> >I didn't see your POC code, so I assumed that you'll need to add
&g
)`, I guess this
is not what we want to support right? If so, I would suggest to throw
proper errors for these unexpected usages.
Benchao Li 于2023年12月20日周三 13:16写道：
>
> Thanks Feng for driving this, it's a very useful feature.
>
> In the FLIP, you mentioned that
> > During POC
l = true)
> > > @ArgumentHint(name = "in3", isOptional = true)})
> > > public String[] call(ProcedureContext procedureContext, String
> > > arg1, String arg2, String arg3) {
> > > return new String[]{arg1 + ", " + arg2 + "," + arg3};
> > > }
> > > }
> > > ```
> > >
> > >
> > > Currently, we offer support for two scenarios when calling a function
> > > or procedure:
> > >
> > > 1. The corresponding parameters can be specified using the parameter
> > > name, without a specific order.
> > > 2. Unnecessary parameters can be omitted.
> > >
> > >
> > > There are still some limitations when using Named parameters:
> > > 1. Named parameters do not support variable arguments.
> > > 2. UDX or procedure classes that support named parameters can only
> > > have one eval method.
> > > 3. Due to the current limitations of Calcite-947[2], we cannot specify
> > > a default value for omitted parameters, which is Null by default.
> > >
> > >
> > >
> > > Also, thanks very much for the suggestions and help provided by Zelin
> > > and Lincoln.
> > >
> > >
> > >
> > >
> > > 1.
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-387%3A+Support+named+parameters+for+functions+and+call+procedures
> > .
> > >
> > > 2. https://issues.apache.org/jira/browse/CALCITE-947
> > >
> > >
> > >
> > > Best,
> > >
> > > Feng
> > >
> >
> >
--
Best,
Benchao Li
> +++-----+----+-+-+-+-+
> >>
> >> | +I |10.10.10.10 | 2023-12-15 17:38:05.169 |
> >> 10.10.10.10 | 2 | 40 | 90 |
> >> 110 |
> >>
> >> | +I |10.10.10.10 | 2023-12-15 17:38:05.169 |
> >> 10.10.10.10 | 2 | 50 | 80 |
> >> 120 |
> >>
> >> | +I |20.20.20.20 | 2023-12-15 17:38:05.170 |
> >> | | | |
> >> |
> >>
> >> | +I |30.30.30.30 | 2023-12-15 17:38:05.172 |
> >> | | | |
> >> |
> >>
> >> Unless otherwise stated above:
> >>
> >> IBM United Kingdom Limited
> >> Registered in England and Wales with number 741598
> >> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> >>
> >
--
Best,
Benchao Li
1.19). Azure CI will still serve as the
> >> project's
> >>>>>> ground
> >>>>>>>> of truth until the community decides in a final vote to switch
> >> to
> >>>> GHA
> >>>>>> or
> >>>>>>>> stick to Azure CI.
> >>>>>>>>
> >>>>>>>> The related discussion thread can be found in [2].
> >>>>>>>>
> >>>>>>>> The vote will remain open for at least 72 hours and only
> >> concluded
> >>>> if
> >>>>>>> there
> >>>>>>>> are no objections and enough (i.e. at least 3) binding votes.
> >>>>>>>>
> >>>>>>>> Matthias
> >>>>>>>>
> >>>>>>>> [1]
> >>>>>>>>
> >>>>>>>>
> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-396%3A+Trial+to+test+GitHub+Actions+as+an+alternative+for+Flink%27s+current+Azure+CI+infrastructure
> >>>>>>>> [2]
> >>>> https://lists.apache.org/thread/h4cmv7l3y8mxx2t435dmq4ltco4sbrgb
> >>>>>>>> --
> >>>>>>>>
> >>>>>>>> [image: Aiven]<https://www.aiven.io>
> >>>>>>>>
> >>>>>>>> *Matthias Pohl*
> >>>>>>>> Opensource Software Engineer, *Aiven*
> >>>>>>>> matthias.p...@aiven.io | +49 170 9869525
> >>>>>>>> aiven.io<https://www.aiven.io>| <
> >>>>>>> https://www.facebook.com/aivencloud
> >>>>>>>><https://www.linkedin.com/company/aiven/><
> >>>>>>>> https://twitter.com/aiven_io>
> >>>>>>>> *Aiven Deutschland GmbH*
> >>>>>>>> Alexanderufer 3-7, 10117 Berlin
> >>>>>>>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >>>>>>>> Amtsgericht Charlottenburg, HRB 209739 B
> >>>>>>>>
--
Best,
Benchao Li
gt; +1
> >
> > also +1 to include FLINK-33313 mentioned by Benchao Li
> >
> > On Fri, Dec 8, 2023 at 9:17 AM Benchao Li wrote:
> >
> > > Thanks Jing for driving 1.18.1 releasing.
> > >
> > > I would like to include FLINK-33313[1] in 1.18.
//lists.apache.org/thread/m4c879y8mb7hbn2kkjh9h3d8g1jphh3j
> [6] https://issues.apache.org/jira/projects/FLINK/versions/12353640
> Thanks,
--
Best,
Benchao Li
;
>
> Really looking forward to hear what you think!
>
>
> Best,
> Leonard (on behalf of the Flink CDC Connectors project maintainers)
>
> [1] https://github.com/ververica/flink-cdc-connectors
> [2]
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
> [3] https://debezium.io
> [4]
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
--
Best,
Benchao Li
]
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-395%3A+Deprecate+Global+Aggregator+Manager
> > [2] https://issues.apache.org/jira/browse/FLINK-10886
> > [3] https://issues.apache.org/jira/browse/FLINK-31245
> >
> > Best,
> > Zhanghao Chen
> >
--
Best,
Benchao Li
be open for at least 72
> hours unless there is an objection or not enough votes.
>
> [1] https://cwiki.apache.org/confluence/x/vQ2ZE
> [2] https://lists.apache.org/thread/ztyk68brsbmwwo66o1nvk3f6fqqhdxgk
--
Best,
Benchao Li
tring on flink ui. It would be easy for us to
> > finish this feature if we can get serialized sql from QueryOperation
> > directly.
> >
> > So +1 for the overall proposal.
> >
> > Regards,
> > Xiangyu
> >
> > Benchao Li 于2023年11月17日周五 19:07写道
n
> details. I mentioned Expressions, because this requires changing
> semi-public interfaces in BuiltinFunctionDefinitions.
>
> Hope this makes it clearer.
>
> Regards,
> Dawid
>
> On Thu, 16 Nov 2023 at 12:12, Benchao Li wrote:
>
> > Sorry that I wasn't expressi
a4bfe596fc580f8280750fa3bfa22007671d9/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/QueryOperation.java#L46
>
> On Wed, 15 Nov 2023 at 16:36, Benchao Li wrote:
>
> > +1 for the idea of choosing SQL as the serialization format for
> > QueryOp
PI. I posted the proposed
> changes in the FLIP-393. I'd like to get to know what others think of
> choosing SQL as the serialization format for QueryOperations.
> Regards,
> Dawid Wysakowicz
>
> [1] https://cwiki.apache.org/confluence/x/vQ2ZE
--
Best,
Benchao Li
Benchao Li created FLINK-33533:
--
Summary: Add documentation about performance for huge ProtoBuf
definations
Key: FLINK-33533
URL: https://issues.apache.org/jira/browse/FLINK-33533
Project: Flink
e is an objection
> > > or not enough votes.
> > >
> > > [1]
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-376%3A+Add+DISTRIBUTED+BY+clause
> > > [2] https://lists.apache.org/thread/z1p4f38x9ohv29y4nlq8g1wdxwfjwxd1
> > >
> > > Cheers,
> > > Timo
> >
--
Best,
Benchao Li
; > Here is an example:
> > > >
> > > > CREATE TABLE MyTable
> > > >(
> > > > uid BIGINT,
> > > > name STRING
> > > >)
> > > >DISTRIBUTED BY (uid) INTO 6 BUCKETS
> > > >WITH (
> > > > 'connector' = 'kafka'
> > > >)
> > > >
> > > > The full syntax specification can be found in the document. The clause
> > > > should be optional and fully backwards compatible.
> > > >
> > > > Regards,
> > > > Timo
> > > >
> > > > [1]
> > > >
> > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-376%3A+Add+DISTRIBUTED+BY+clause
> > >
> >
--
Best,
Benchao Li
applications.
> > >>>>>>
> > >>>>>> The release is available for download at:
> > >>>>>> https://flink.apache.org/downloads.html
> > >>>>>>
> > >>>>>> Please check out the release blog post for an overview of the
> > >>>>> improvements
> > >>>>>> for this release:
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> > https://flink.apache.org/2023/10/24/announcing-the-release-of-apache-flink-1.18/
> > >>>>>>
> > >>>>>> The full release notes are available in Jira:
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12352885
> > >>>>>>
> > >>>>>> We would like to thank all contributors of the Apache Flink
> > >> community
> > >>>> who
> > >>>>>> made this release possible!
> > >>>>>>
> > >>>>>> Best regards,
> > >>>>>> Konstantin, Qingsheng, Sergey, and Jing
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> >
--
Best,
Benchao Li
down is not allowed for the planner.
> >> Regardless of the value of the "scan.filter-push-down.enabled"
> >> configuration, filter pushdown is disabled.
> >> In this scenario, the behavior remains consistent with the old version as
> >> well.
> >>
> >>
t; >> to
> > >>>> disable filter pushdown for Table/SQL Sources[1].
> > >>>>>
> > >>>>> Currently, Flink Table/SQL does not expose fine-grained control for
> > >>>> users to enable or disable filter pushdown.
> > >>>>> However, filter pushdown has some side effects, such as additional
> > >>>> computational pressure on external systems.
> > >>>>> Moreover, Improper queries can lead to issues such as full table
> > scans,
> > >>>> which in turn can impact the stability of external systems.
> > >>>>>
> > >>>>> Suppose we have an SQL query with two sources: Kafka and a database.
> > >>>>> The database is sensitive to pressure, and we want to configure it to
> > >>>> not perform filter pushdown to the database source.
> > >>>>> However, we still want to perform filter pushdown to the Kafka source
> > >> to
> > >>>> decrease network IO.
> > >>>>>
> > >>>>> I propose to support configuration to disable filter push down for
> > >>>> Table/SQL sources to let user decide whether to perform filter
> > pushdown.
> > >>>>>
> > >>>>> Looking forward to your feedback.
> > >>>>>
> > >>>>> [1]
> > >>>>
> > >>
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=276105768
> > >>>>>
> > >>>>> Best,
> > >>>>> Jiabao
> > >>>>
> > >>>>
> > >>
> > >>
> >
> >
--
Best,
Benchao Li
gt; >[1]
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-373%3A+Support+Configuring+Different+State+TTLs+using+SQL+Hint
> > >[2] https://lists.apache.org/thread/3s69dhv3rp4s0kysnslqbvyqo3qf7zq5
> > >
> > >Best,
> > >Jane
> >
--
Best,
Benchao Li
23-10-10 18:23:55, "Jane Chan"
> > > > wrote:
> > > > > > > > > >Hi Feng,
> > > > > > > > > >
> > > > > > > > > >Thank you for your valuable comments. The reason for not
> > > > including
> > > > > > the
> > > > > > > > > >scenarios above is as follows:
> > > > > > > > > >
> > > > > > > > > >For <1>, the automatically inferred stateful operators are
> > not
> > > > > > easily
> > > > > > > > > >expressible in SQL. This issue was discussed in FLIP-292,
> > and
> > > > > > besides
> > > > > > > > > >ChangelogNormalize, SinkUpsertMateralizer also faces the
> > same
> > > > > > problem.
> > > > > > > > > >
> > > > > > > > > >For <2> and <3>, the challenge lies in internal
> > implementation.
> > > > > > > During the
> > > > > > > > > >default_rewrite phase, the row_number expression in
> > > > LogicalProject
> > > > > > is
> > > > > > > > > >transformed into LogicalWindow by Calcite's
> > > > > > > > > >CoreRules#PROJECT_TO_LOGICAL_PROJECT_AND_WINDOW. However,
> > > > > > > CalcRelSplitter
> > > > > > > > > >does not pass the hints as an input argument when creating
> > > > > > > LogicalWindow,
> > > > > > > > > >resulting in the loss of the hint at this step. To support
> > > > this, we
> > > > > > > may
> > > > > > > > > >need to rewrite some optimization rules in Calcite, which
> > could
> > > > be a
> > > > > > > > > >follow-up work if required.
> > > > > > > > > >
> > > > > > > > > >Best,
> > > > > > > > > >Jane
> > > > > > > > > >
> > > > > > > > > >On Tue, Oct 10, 2023 at 1:40 AM Feng Jin <
> > jinfeng1...@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Hi Jane,
> > > > > > > > > >>
> > > > > > > > > >> Thank you for proposing this FLIP.
> > > > > > > > > >>
> > > > > > > > > >> I believe that this FLIP will greatly enhance the
> > flexibility
> > > > of
> > > > > > > setting
> > > > > > > > > >> state, and by setting different operators' TTL, it can
> > also
> > > > > > > increase job
> > > > > > > > > >> stability, especially in regular join scenarios.
> > > > > > > > > >> The parameter design is very concise, big +1 for this,
> > and it
> > > > is
> > > > > > > also
> > > > > > > > > >> relatively easy to use for users.
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >> I have a small question: in the FLIP, it only mentions
> > join
> > > > and
> > > > > > > group.
> > > > > > > > > >> Should we also consider other scenarios?
> > > > > > > > > >>
> > > > > > > > > >> 1. the auto generated deduplicate operator[1].
> > > > > > > > > >> 2. the deduplicate query[2].
> > > > > > > > > >> 3. the topN query[3].
> > > > > > > > > >>
> > > > > > > > > >> [1]
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/config/#table-exec-source-cdc-events-duplicate
> > > > > > > > > >> [2]
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/deduplication/
> > > > > > > > > >> [3]
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/topn/
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >> Best,
> > > > > > > > > >> Feng
> > > > > > > > > >>
> > > > > > > > > >> On Sun, Oct 8, 2023 at 5:53 PM Jane Chan <
> > > > qingyue@gmail.com>
> > > > > > > wrote:
> > > > > > > > > >>
> > > > > > > > > >> > Hi devs,
> > > > > > > > > >> >
> > > > > > > > > >> > I'd like to initiate a discussion on FLIP-373: Support
> > > > > > Configuring
> > > > > > > > > >> > Different State TTLs using SQL Hint [1]. This proposal
> > is
> > > > on top
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > >> > FLIP-292 [2] to address typical scenarios with
> > unambiguous
> > > > > > > semantics
> > > > > > > > > and
> > > > > > > > > >> > hint propagation.
> > > > > > > > > >> >
> > > > > > > > > >> > I'm looking forward to your opinions!
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> > [1]
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-373%3A+Support+Configuring+Different+State+TTLs+using+SQL+Hint
> > > > > > > > > >> > [2]
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-292%3A+Enhance+COMPILED+PLAN+to+support+operator-level+state+TTL+configuration
> > > > > > > > > >> >
> > > > > > > > > >> > Best,
> > > > > > > > > >> > Jane
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best
> > > > > >
> > > > > > ConradJam
> > > > > >
> > > >
> >
--
Best,
Benchao Li
; From: yu zelin
> >>>> Sent: Monday, October 16, 2023 10:16
> >>>> To: dev@flink.apache.org
> >>>> Cc: ron9@gmail.com
> >>>> Subject: Re: [ANNOUNCE] New Apache Flink Committer - Ron Liu
> >>>>
> >>>> Congratulations!
> >>>>
> >>>> Best,
> >>>> Yu Zelin
> >>>>
> >>>> 2023年10月16日 09:56，Jark Wu 写道：
> >>>>
> >>>> Hi, everyone
> >>>>
> >>>> On behalf of the PMC, I'm very happy to announce Ron Liu as a new
> >>>> Flink
> >>>> Committer.
> >>>>
> >>>> Ron has been continuously contributing to the Flink project for
> >>>> many
> >>>> years,
> >>>> authored and reviewed a lot of codes. He mainly works on Flink SQL
> >>>> parts
> >>>> and drove several important FLIPs, e.g., USING JAR (FLIP-214),
> >>>> Operator
> >>>> Fusion CodeGen (FLIP-315), Runtime Filter (FLIP-324). He has a
> >>>> great
> >>>> knowledge of the Batch SQL and improved a lot of batch performance
> >>>> in
> >>>> the
> >>>> past several releases. He is also quite active in mailing lists,
> >>>> participating in discussions and answering user questions.
> >>>>
> >>>> Please join me in congratulating Ron Liu for becoming a Flink
> >>>> Committer!
> >>>>
> >>>> Best,
> >>>> Jark Wu (on behalf of the Flink PMC)
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
>
--
Best,
Benchao Li
cipated deeply in development
> >> discussions and also helped answer user question emails. Jane was also a
> >> core contributor of Flink Table Store (now Paimon) when the project was in
> >> the early days.
> >>
> >> Please join me in congratulating Jane Chan for becoming a Flink Committer!
> >>
> >> Best,
> >> Jark Wu (on behalf of the Flink PMC)
> >>
>
--
Best,
Benchao Li
display/FLINK/FLIP-374%3A+Adding+a+separate+configuration+for+specifying+Java+Options+of+the+SQL+Gateway
> > [2] https://lists.apache.org/thread/g4vl8mgnwgl7vjyvjy6zrc8w54b2lthv
> >
> > Best,
> > Yangze Guo
> >
--
Best,
Benchao Li
ss there is an objection or insufficient votes.
>
> [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263429150
> [2] https://lists.apache.org/thread/gtpswl42jzv0c9o3clwqskpllnw8rh87
>
> Best,
> Zhanghao Chen
--
Best,
Benchao Li
une
> > > the memory settings, garbage collection behavior, and other relevant
> > > Java parameters specific to the SQL Gateway, ensuring optimal
> > > performance and stability in production environments.
> > >
> > > Looking forward to your feedback.
> > >
> > > [1]
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-374%3A+Adding+a+separate+configuration+for+specifying+Java+Options+of+the+SQL+Gateway
> > >
> > > Best,
> > > Yangze Guo
> > >
> >
--
Best,
Benchao Li
s of whether the source parallelism is configured or not.
> > If I
> > > > > understand correctly, currently, except for the sink exec node (which
> > > has
> > > > > the ability to configure its own parallelism), the rest of the exec
> > > nodes
eId=263429150>
> [2] FLIP-146: Improve new TableSource and TableSink interfaces - Apache Flink
> - Apache Software
> Foundation<https://cwiki.apache.org/confluence/display/FLINK/FLIP-146%3A+Improve+new+TableSource+and+TableSink+interfaces>
>
> Best,
> Zhanghao Chen
--
Best,
Benchao Li
thread/zon967w7synby8z6m1s7dj71dhkh9ccy
> >
> > Cheers,
> > Timo
> >
--
Best,
Benchao Li
rbitrary queries. That "_proctime" may
> conflict
> >>>>> with
> >>>>>> the proposed
> >>>>>> pseudo-column concept.
> >>>>>>
> >>>>>> Did you consider making it as a built-in defined pseudo-column
>
> > > > > > > Best,
> > > > > > > Junrui
> > > > > > >
> > > > > > > Yun Tang 于2023年8月7日周一 15:19写道：
> > > > > > >
> > > > > > > Congratulations, Yanfei!
> > > > > > >
> > > > > > > Best
> > > > > > > Yun Tang
> > > > > > >
> > > > > > > From: Danny Cranmer
> > > > > > > Sent: Monday, August 7, 2023 15:10
> > > > > > > To: dev
> > > > > > > Subject: Re: [ANNOUNCE] New Apache Flink Committer - Yanfei Lei
> > > > > > >
> > > > > > > Congrats Yanfei! Welcome to the team.
> > > > > > >
> > > > > > > Danny
> > > > > > >
> > > > > > > On Mon, 7 Aug 2023, 08:03 Rui Fan, <1996fan...@gmail.com>
> wrote:
> > > > > > >
> > > > > > > Congratulations Yanfei!
> > > > > > >
> > > > > > > Best,
> > > > > > > Rui
> > > > > > >
> > > > > > > On Mon, Aug 7, 2023 at 2:56 PM Yuan Mei <
> yuanmei.w...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > On behalf of the PMC, I'm happy to announce Yanfei Lei as a new
> > > > > > > Flink
> > > > > > > Committer.
> > > > > > >
> > > > > > > Yanfei has been active in the Flink community for almost two
> > > > > > > years
> > > > > > > and
> > > > > > > has
> > > > > > > played an important role in developing and maintaining State
> > > > > > > and
> > > > > > > Checkpoint
> > > > > > > related features/components, including RocksDB Rescaling
> > > > > > > Performance
> > > > > > > Improvement and Generic Incremental Checkpoints.
> > > > > > >
> > > > > > > Yanfei also helps improve community infrastructure in many
> > > > > > > ways,
> > > > > > > including
> > > > > > > migrating the Flink Daily performance benchmark to the Apache
> > > > > > > Flink
> > > > > > > slack
> > > > > > > channel. She is the maintainer of the benchmark and has
> > > > > > > improved
> > > > > > > its
> > > > > > > detection stability significantly. She is also one of the major
> > > > > > > maintainers
> > > > > > > of the FrocksDB Repo and released FRocksDB 6.20.3 (part of
> > > > > > > Flink
> > > > > > > 1.17
> > > > > > > release). Yanfei is a very active community member, supporting
> > > > > > > users
> > > > > > > and
> > > > > > > participating
> > > > > > > in tons of discussions on the mailing lists.
> > > > > > >
> > > > > > > Please join me in congratulating Yanfei for becoming a Flink
> > > > > > > Committer!
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Yuan Mei (on behalf of the Flink PMC)
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > >
> >
>
--
Best,
Benchao Li
feature prod-ready).
> > Hangxiang
> > > > >is also the main driver of the FLIP-263: Resolving schema
> > compatibility.
> > > > >
> > > > >Hangxiang is passionate about the Flink community. Besides the
> > technical
> > > > >contribution above, he is also actively promoting Flink: talks about
> > > > Generic
> > > > >Incremental Checkpoints in Flink Forward and Meet-up. Hangxiang also
> > spent
> > > > >a good amount of time supporting users, participating in
> Jira/mailing
> > list
> > > > >discussions, and reviewing code.
> > > > >
> > > > >Please join me in congratulating Hangxiang for becoming a Flink
> > Committer!
> > > > >
> > > > >Thanks,
> > > > >Yuan Mei (on behalf of the Flink PMC)
> > > >
> >
>
--
Best,
Benchao Li
> > >
> > > > > Best,
> > > > > Wencong Liu
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > At 2023-08-04 11:18:00, "Xintong Song"
> > > > > wrote:
> > > > > Hi everyone,
> > > > >
> > > > > On behalf of the PMC, I'm very happy to announce that Matthias Pohl
> > > > > has
> > > > > joined the Flink PMC!
> > > > >
> > > > > Matthias has been consistently contributing to the project since
> > > > > Sep
> > > > > 2020,
> > > > > and became a committer in Dec 2021. He mainly works in Flink's
> > > > > distributed
> > > > > coordination and high availability areas. He has worked on many
> > > > > FLIPs
> > > > > including FLIP195/270/285. He helped a lot with the release
> > > > > management,
> > > > > being one of the Flink 1.17 release managers and also very active
> > > > > in
> > > > > Flink
> > > > > 1.18 / 2.0 efforts. He also contributed a lot to improving the
> > > > > build
> > > > > stability.
> > > > >
> > > > > Please join me in congratulating Matthias!
> > > > >
> > > > > Best,
> > > > >
> > > > > Xintong (on behalf of the Apache Flink PMC)
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
>
--
Best,
Benchao Li
t; distributed
> > > > > > coordination and high availability areas. He has worked on many
> > FLIPs
> > > > > > including FLIP195/270/285. He helped a lot with the release
> > management,
> > > > > > being one of the Flink 1.17 release managers and also very active
> > in
> > > > > Flink
> > > > > > 1.18 / 2.0 efforts. He also contributed a lot to improving the
> > build
> > > > > > stability.
> > > > > >
> > > > > > Please join me in congratulating Matthias!
> > > > > >
> > > > > > Best,
> > > > > >
> > > > > > Xintong (on behalf of the Apache Flink PMC)
> > > > >
> > > > >
> > > >
> >
>
--
Best,
Benchao Li
ale job
> > scheduling and improvements. He is also quite active in mailing lists,
> > participating discussions and answering user questions.
> >
> > Please join me in congratulating Weihua!
> >
> > Best,
> >
> > Xintong (on behalf of the Apache Flink PMC)
>
>
--
Best,
Benchao Li
ore recently, improvements to the Flink REST API.
> > Additionally,
> > >> Hong is a very active community member, supporting users and
> > participating
> > >> in discussions on the mailing lists, Flink slack channels and speaking
> > at
> > >> conferences.
> > >>
> > >> Please join me in congratulating Hong for becoming a Flink Committer!
> > >>
> > >> Thanks,
> > >> Danny Cranmer (on behalf of the Flink PMC)
> > >
> >
> >
>
--
Best,
Benchao Li
problem of metadata columns currently is that a catalog
> > > implementation can't provide them by default because they would affect
> > > `SELECT *` when adding another one.
> > >
> > > Looking forward to your feedback on FLIP-348:
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-348%3A+Support+System+Columns+in+SQL+and+Table+API
> > >
> > > Thanks,
> > > Timo
> > >
> >
> >
> > --
> > https://twitter.com/snntrable
> > https://github.com/knaufk
> >
>
--
Best,
Benchao Li
y rule instead of big bang.
>
> I would volunteer to update the FLIP page with usage of a custom factory if
> there are no objections and come back with a discussion thread to revive
> the work on it.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-154%3A+SQL+Implicit+Type+Coercion
> --
> Best regards,
> Sergey
>
--
Best,
Benchao Li
gt;
> > [1]
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/overview/
> > [2]
> >
> https://docs.google.com/document/d/1XP2fBpcntK0YIdQ_Ax7JV2MhNdebvkFxSiNJRp6WQ24/edit?usp=sharing
> >
> >
> > Best,
> > Zhiqiang
> >
> >
>
--
Best,
Benchao Li
t; 抄送: Shammon FY
> > > > > 主题: Re: [ANNOUNCE] New Apache Flink Committer - Yong Fang
> > > > >
> > > > > Congratulations!
> > > > >
> > > > > Best,
> > > > > Weihua
> > > > >
> > > > >
> > > > > On Mon, Jul 24, 2023 at 11:04 AM Paul Lam
> > > wrote:
> > > > >
> > > > > > Congrats, Shammon!
> > > > > >
> > > > > > Best,
> > > > > > Paul Lam
> > > > > >
> > > > > > > 2023年7月24日 10:56，Jingsong Li 写道：
> > > > > > >
> > > > > > > Shammon
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
--
Best,
Benchao Li
Benchao Li created FLINK-32547:
--
Summary: Add missing doc for Timestamp support in ProtoBuf format
Key: FLINK-32547
URL: https://issues.apache.org/jira/browse/FLINK-32547
Project: Flink
Issue
r any or all of these feature requests? We're
> contemplating forking flink-protobuf internally, but it would be better if
> we could just upstream the relevant changes. (To my mind, 1, 2, and 4 are
> broadly applicable features that are definitely worthy of upstream support.
> 3 and 5 may be somewhat more specific to our use case.)
>
> Thanks,
> Adam Richardson
>
--
Best,
Benchao Li
t;>> 收件人: "Xintong Song" > > > > > > > > > >> tonysong...@gmail.com>>
> > > > > > > > > > >> >>> 抄送: "dev" > > > > > dev@flink.apache.org>>,
> > > > > > > > > > >> "User" > > u...@flink.apache.org
> > > > >>
> > > > > > > > > > >> >>> 发送时间: 星期一, 2023年 7 月 03日 下午 8:36:30
> > > > > > > > > > >> >>> 主题: Re: [ANNOUNCE] Apache Flink has won the 2023
> > > SIGMOD
> > > > > > Systems
> > > > > > > > > > Award
> > > > > > > > > > >> >>>
> > > > > > > > > > >> >>> Congratulations \uD83E\uDD73
> > > > > > > > > > >> >>>
> > > > > > > > > > >> >>> On 03-Jul-2023, at 3:30 PM, Xintong Song <
> > > > > > tonysong...@gmail.com
> > > > > > > > > > >> <mailto:tonysong...@gmail.com>> wrote:
> > > > > > > > > > >> >>>
> > > > > > > > > > >> >>>
> > > > > > > > > > >> >>> Dear Community,
> > > > > > > > > > >> >>>
> > > > > > > > > > >> >>> I'm pleased to share this good news with everyone.
> > As
> > > > some
> > > > > > of
> > > > > > > > you
> > > > > > > > > > may
> > > > > > > > > > >> have already heard, Apache Flink has won the 2023
> SIGMOD
> > > > Systems
> > > > > > > > Award
> > > > > > > > > > [1].
> > > > > > > > > > >> >>>
> > > > > > > > > > >> >>> "Apache Flink greatly expanded the use of stream
> > > > > > > > data-processing."
> > > > > > > > > > --
> > > > > > > > > > >> SIGMOD Awards Committee
> > > > > > > > > > >> >>>
> > > > > > > > > > >> >>> SIGMOD is one of the most influential data
> > management
> > > > > > research
> > > > > > > > > > >> conferences in the world. The Systems Award is awarded
> > to
> > > an
> > > > > > > > individual
> > > > > > > > > > or
> > > > > > > > > > >> set of individuals to recognize the development of a
> > > > software or
> > > > > > > > > > hardware
> > > > > > > > > > >> system whose technical contributions have had
> > significant
> > > > > > impact on
> > > > > > > > the
> > > > > > > > > > >> theory or practice of large-scale data management
> > systems.
> > > > > > Winning
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > >> award indicates the high recognition of Flink's
> > > > technological
> > > > > > > > > > advancement
> > > > > > > > > > >> and industry influence from academia.
> > > > > > > > > > >> >>>
> > > > > > > > > > >> >>> As an open-source project, Flink wouldn't have
> come
> > > > this far
> > > > > > > > without
> > > > > > > > > > >> the wide, active and supportive community behind it.
> > Kudos
> > > > to
> > > > > > all
> > > > > > > > of us
> > > > > > > > > > who
> > > > > > > > > > >> helped make this happen, including the over 1,400
> > > > contributors
> > > > > > and
> > > > > > > > many
> > > > > > > > > > >> others who contributed in ways beyond code.
> > > > > > > > > > >> >>>
> > > > > > > > > > >> >>> Best,
> > > > > > > > > > >> >>> Xintong (on behalf of the Flink PMC)
> > > > > > > > > > >> >>>
> > > > > > > > > > >> >>> [1] https://sigmod.org/2023-sigmod-systems-award/
> > > > > > > > > > >> >>>
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > >
> >
>
--
Best,
Benchao Li
; >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> At 2023-06-23 23:49:53, "Lijie Wang"
> > wrote:
> > > >>> Hi all,
> > > >>>
> > > >>> Thanks for all the feedback about the FLIP-324: Introduce Runtime
> > Filter
> > > >>> for Flink Batch Jobs[1]. This FLIP was discussed in [2].
> > > >>>
> > > >>> I'd like to start a vote for it. The vote will be open for at least
> > 72
> > > >>> hours (until June 29th 12:00 GMT) unless there is an objection or
> > > >>> insufficient votes.
> > > >>>
> > > >>> [1]
> > > >>>
> > > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-324%3A+Introduce+Runtime+Filter+for+Flink+Batch+Jobs
> > > >>> [2]
> https://lists.apache.org/thread/mm0o8fv7x7k13z11htt88zhy7lo8npmg
> > > >>>
> > > >>> Best,
> > > >>> Lijie
> > > >>
> > >
> >
>
--
Best,
Benchao Li
gt; [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-308%3A+Support+Time+Travel
> > [2]https://lists.apache.org/thread/rpozdlf7469jmc7q7vc0s08pjnmscz00
> >
> >
> > Best,
> > Feng
> >
>
--
Best,
Benchao Li
start a discussion about
> FLIP-324:
> >> > Introduce Runtime Filter for Flink Batch Jobs[1]
> >> >
> >> > Runtime Filter is a common optimization to improve join performance.
> It
> >> is
> >> > designed to dynamically generate filter conditions for certain Join
> >> queries
> >> > at runtime to reduce the amount of scanned or shuffled data, avoid
> >> > unnecessary I/O and network transmission, and speed up the query. Its
> >> > working principle is building a filter(e.g. bloom filter) based on the
> >> data
> >> > on the small table side(build side) first, then pass this filter to
> the
> >> > large table side(probe side) to filter the irrelevant data on it, this
> >> can
> >> > reduce the data reaching the join and improve performance.
> >> >
> >> > You can find more details in the FLIP-324[1]. Looking forward to your
> >> > feedback.
> >> >
> >> > [1]
> >> >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-324%3A+Introduce+Runtime+Filter+for+Flink+Batch+Jobs
> >> >
> >> > Best,
> >> > Ron & Gen & Lijie
> >> >
> >>
> >
>
--
Best,
Benchao Li
ener
> [2] https://lists.apache.org/thread/185mbcwnpokfop4xcb22r9bgfp2m68fx
>
>
> Best,
> Shammon FY
>
--
Best,
Benchao Li
gt; Best regards,
> Yuxia
>
--
Best,
Benchao Li
3f2f795f5bc5/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TimeTravelSpec.scala#L56
> [2]
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/timezone/#daylight-saving-time-support
>
>
>
>
> > On Jun 9, 2023, at 1:13 PM, Benchao Li wrote:
> >
>
able(ObjectPath tablePath, long
> timestamp)`. Maybe something like "timestamp of the table snapt, which is
> millseconds since 1970-01-01 00:00:00 UTC".
>
> Thank you for the suggestions regarding the document. I will add them to
> FLIP.
>
>
> Best,
> Feng
>
//cwiki.apache.org/confluence/display/FLINK/FLIP-315+Support+Operator+Fusion+Codegen+for+Flink+SQL
> > [2]: https://lists.apache.org/thread/9cnqhsld4nzdr77s2fwf00o9cb2g9fmw
> >
> > Best,
> > Ron
>
>
--
Best,
Benchao Li
y property, its should
> >> not
> >> >>> be
> >> >>>> coupled with the Catalog or table definition. In my opinion this is
> >> >>>> something that DynamicTableFactory only has to know about. I would
> >> >> rather
> >> >
gt; As main memory grows, query performance is more and more
> > > > determined
> > > > > > by
> > > > > > > > the
> > > > > > > > > raw CPU costs of query processing itself, this is due to
> the
> > > > query
> > > > > > > > > processing techniques based on interpreted execution shows
> poor
> > > > > > > > performance
> > > > > > > > > on modern CPUs due to lack of locality and frequent
> instruction
> > > > > > > > > mis-prediction. Therefore, the industry is also
> researching how
> > > > to
> > > > > > > > improve
> > > > > > > > > engine performance by increasing operator execution
> efficiency.
> > > > In
> > > > > > > > > addition, during the process of optimizing Flink's
> performance
> > > > for
> > > > > > TPC-DS
> > > > > > > > > queries, we found that a significant amount of CPU time was
> > > spent
> > > > > on
> > > > > > > > > virtual function calls, framework collector calls, and
> invalid
> > > > > > > > > calculations, which can be optimized to improve the overall
> > > > engine
> > > > > > > > > performance. After some investigation, we found Operator
> Fusion
> > > > > > Codegen
> > > > > > > > > which is proposed by Thomas Neumann in the paper[2] can
> address
> > > > > these
> > > > > > > > > problems. I have finished a PoC[3] to verify its
> feasibility
> > > and
> > > > > > > > validity.
> > > > > > > > >
> > > > > > > > > Looking forward to your feedback.
> > > > > > > > >
> > > > > > > > > [1]:
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-315+Support+Operator+Fusion+Codegen+for+Flink+SQL
> > > > > > > > > [2]: http://www.vldb.org/pvldb/vol4/p539-neumann.pdf
> > > > > > > > > [3]: https://github.com/lsyldliu/flink/tree/OFCG
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Ron
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
--
Best,
Benchao Li
nt` to custom a procedure, we can then add a
> method like `getTableEnvironment()` in `ProcedureContext`.
>
> Best regards,
> Yuxia
>
> - 原始邮件 -
> 发件人: "Benchao Li"
> 收件人: "dev"
> 发送时间: 星期四, 2023年 6 月 01日 下午 12:58:08
> 主题: Re: [D
popular
> compute engines like Trino for it's convenience. Therefore, we propose
> adding support for call stored procedure in Flink to enable better
> integration with external storage systems.
> > >
> > > With this FLIP, Flink will allow connector developers to develop their
> own built-in stored procedures, and then enables users to call these
> predefiend stored procedures.
> > >
> > > Looking forward to your feedbacks.
> > >
> > > [1]:
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-311%3A+Support+Call+Stored+Procedure
> > >
> > > Best regards,
> > > Yuxia
>
--
Best,
Benchao Li
> the following statement
> > SELECT * FROM t VERSION AS OF 1
> >
> > Could we support this in Flink too?
> >
> > Best,
> > Shammon FY
> >
> > On Fri, May 26, 2023 at 1:20 PM Benchao Li wrote:
> >
> > > Regarding the implementation, did y
a/org/apache/flink/table/planner/delegation/hive/HiveParser.java#L348
> [2]:https://issues.apache.org/jira/browse/FLINK-26681
> [3]:https://issues.apache.org/jira/browse/FLINK-31413
> [4]:https://issues.apache.org/jira/browse/FLINK-30064
>
>
>
> Best regards,
> Yuxia
>
--
Best,
Benchao Li
epos/dist/release/flink/KEYS
> [4]
> https://repository.apache.org/content/repositories/orgapacheflink-1636/
> [5] https://github.com/apache/flink-connector-jdbc/releases/tag/v3.1.1-rc1
> [6] https://github.com/apache/flink-web/pull/654
>
--
Best,
Benchao Li
ms the syntax in Calcite should be `FOR SYSTEM_TIME AS OF`,
> > right?
> > But the the syntax part in this FLIP, it seems to be `AS OF TIMESTAMP`
> > instead of `FOR SYSTEM_TIME AS OF`. Is it just a mistake or by design?
> >
> >
> > Best regards,
> > Yuxi
rs to specify a point in time and retrieve the data and
> >> schema of
> >> a table as it appeared at that time. With time travel, users can easily
> >> analyze and compare historical versions of data.
> >>
> >>
> >> With the widespread use of data lake systems such as Paimon, Iceberg,
> and
> >> Hudi, time travel can provide more convenience for users' data analysis.
> >>
> >>
> >> Looking forward to your opinions, any suggestions are welcomed.
> >>
> >>
> >>
> >> 1.
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-308%3A+Support+Time+Travel+In+Batch+Mode
> >>
> >>
> >>
> >> Best.
> >>
> >> Feng
> >>
> >
>
>
--
Best,
Benchao Li
t; to
> > > > > be
> > > > > >
> > > > > > deployed to dist.apache.org [2], which are signed with the key
> > with
> > > > > >
> > > > > > fingerprint 8D56AE6E7082699A4870750EA4E8C4C05EE6861F [3],
> > > > > >
> > > > > > * all artifacts to be deployed to the Maven Central Repository
> [4],
> > > > > >
> > > > > > * source code tag "release-1.17.1-rc1" [5],
> > > > > >
> > > > > > * website pull request listing the new release and adding
> > > announcement
> > > > > blog
> > > > > >
> > > > > > post [6].
> > > > > >
> > > > > >
> > > > > > The vote will be open for at least 72 hours. It is adopted by
> > > majority
> > > > > >
> > > > > > approval, with at least 3 PMC affirmative votes.
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Release Manager
> > > > > >
> > > > > >
> > > > > > [1]
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12352886
> > > > > >
> > > > > > [2]
> https://dist.apache.org/repos/dist/dev/flink/flink-1.17.1-rc1/
> > > > > >
> > > > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > > >
> > > > > > [4]
> > > > > >
> > > >
> > https://repository.apache.org/content/repositories/orgapacheflink-1635/
> > > > > >
> > > > > > [5]
> > https://github.com/apache/flink/releases/tag/release-1.17.1-rc1
> > > > > >
> > > > > > [6] https://github.com/apache/flink-web/pull/650
> > > > > >
> > > > >
> > > >
> > >
> >
>
--
Best,
Benchao Li
> > for
> > > > > > many
> > > > > > > > >years and became a committer in Nov 2021. He has been
> involved
> > > in
> > > > > > > various
> > > > > > > > >areas of the project, from code contributions to community
> > > > building.
> > > > > > His
> > > > > > > > >contributions are mainly focused on Flink SQL and
> connectors,
> > > > > > especially
> > > > > > > > >leading the flink-cdc-connectors project to receive 3.8+K
> > GitHub
> > > > > > stars.
> > > > > > > He
> > > > > > > > >authored 150+ PRs, and reviewed 250+ PRs, and drove several
> > > FLIPs
> > > > > > (e.g.,
> > > > > > > > >FLIP-132, FLIP-162). He has participated in plenty of
> > > discussions
> > > > in
> > > > > > the
> > > > > > > > >dev mailing list, answering questions about 500+ threads in
> > the
> > > > > > > > >user/user-zh mailing list. Besides that, he is community
> > minded,
> > > > > such
> > > > > > as
> > > > > > > > >being the release manager of 1.17, verifying releases,
> > managing
> > > > > > release
> > > > > > > > >syncs, etc.
> > > > > > > > >
> > > > > > > > >Congratulations and welcome Leonard!
> > > > > > > > >
> > > > > > > > >Best,
> > > > > > > > >Jark (on behalf of the Flink PMC)
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
--
Best,
Benchao Li
> > > > is
> > > > > > the
> > > > > > > >core contributor and maintainer of the Kafka connector and
> > > > > > > >flink-cdc-connectors, bringing users stability and ease of use
> > in
> > > > both
> > > > > > > >projects. He drove discussions and implementations in
> FLIP-221,
> > > > > > FLIP-288,
> > > > > > > >and the connector testing framework. He is continuously
> helping
> > > with
> > > > > the
> > > > > > > >expansion of the Flink community and has given several talks
> > about
> > > > > Flink
> > > > > > > >connectors at many conferences, such as Flink Forward Global
> and
> > > > Flink
> > > > > > > >Forward Asia. Besides that, he is willing to help a lot in the
> > > > > community
> > > > > > > >work, such as being the release manager for both 1.17 and
> 1.18,
> > > > > > verifying
> > > > > > > >releases, and answering questions on the mailing list.
> > > > > > > >
> > > > > > > >Congratulations and welcome Qingsheng!
> > > > > > > >
> > > > > > > >Best,
> > > > > > > >Jark (on behalf of the Flink PMC)
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
--
Best,
Benchao Li
n [2], we have come to a consensus, so
> I
> > > > would like
> > > > > > > > to start a vote.
> > > > > > > >
> > > > > > > > The vote will last for at least 72 hours unless there is an
> > > > objection or
> > > > > > > > insufficient votes.
> > > > > > > >
> > > > > > > > [1]:
> > > > > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-302%3A+Support+TRUNCATE+TABLE+statement+in+batch+mode
> > > > > > > > [2]: [
> > > > https://lists.apache.org/thread/m4r3wrd7p96wdst3nz3ncqzog6kf51cf |
> > > > > > > >
> > https://lists.apache.org/thread/m4r3wrd7p96wdst3nz3ncqzog6kf51cf
> > > ]
> > > > > > > >
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Yuxia
> > > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best,
> > > > > > Ron
> > > >
> > >
> >
>
--
Best,
Benchao Li
And I've filed a hotfix PR[1] to fix the copyright issue.
[1] https://github.com/apache/flink-connector-jdbc/pull/41
Benchao Li 于2023年4月16日周日 17:51写道：
> * Verified hashes, signatures (OK)
> * Download, build and test source code (OK)
> * Checked files in Nexus staging (OK)
> * S
s.
> >>
> >> Thanks,
> >> Danny
> >>
> >> [1]
> >>
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12352678
> >> [2]
> >>
> https://dist.apache.org/repos/dist/dev/flink/flink-connector-jdbc-3.1.0-rc1
> >> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> >> [4]
> >> https://repository.apache.org/content/repositories/orgapacheflink-1614/
> >> [5]
> https://github.com/apache/flink-connector-jdbc/releases/tag/v3.1.0-rc1
> >> [6] https://github.com/apache/flink-web/pull/638
> >>
>
>
--
Best,
Benchao Li
fficient votes.
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240883951
> > [2] https://lists.apache.org/thread/ffmc96gv8ofoskbxlhtm7w8oxv8nqzct
> >
> > Best,
> > Jane Chan
> >
>
--
Best,
Benchao Li
Benchao Li created FLINK-31673:
--
Summary: Add E2E tests for flink jdbc driver
Key: FLINK-31673
URL: https://issues.apache.org/jira/browse/FLINK-31673
Project: Flink
Issue Type: Sub-task
and `json plan`
> > to
> > > > > configure `TTL on Operator` better. What do you think? Thanks
> > > > >
> > > > >
> > > > > Best,
> > > > > Shammon FY
> > > > >
> > > > >
> > > > > On Thu, Mar 23, 2023 at 9:58 PM Shuo Cheng
> > wrote:
> > > > >
> > > > > > Correction: “users can set 'scan.startup.mode' for kafka
> connector”
> > > ->
> > > > > > “users
> > > > > > can set 'scan.startup.mode' for kafka connector by dynamic table
> > > > option”
> > > > > >
> > > > > > Shuo Cheng 于2023年3月23日 周四21:50写道：
> > > > > >
> > > > > > > Hi Jane,
> > > > > > > Thanks for driving this, operator level state ttl is
> absolutely a
> > > > > desired
> > > > > > > feature. I would share my opinion as following:
> > > > > > >
> > > > > > > If the scope of this proposal is limited as an enhancement for
> > > > compiled
> > > > > > > json plan, it makes sense. I think it does not conflict with
> > > > > configuring
> > > > > > > state ttl
> > > > > > > in other ways, e.g., SQL HINT or something else, because they
> > just
> > > > work
> > > > > > in
> > > > > > > different level, SQL Hint works in the exact entrance of SQL
> API,
> > > > while
> > > > > > > compiled json plan is the intermediate results for SQL.
> > > > > > > I think the final shape of state ttl configuring may like the
> > that,
> > > > > users
> > > > > > > can define operator state ttl using SQL HINT (assumption...),
> but
> > > it
> > > > > may
> > > > > > > affects more than one stateful operators inside the same query
> > > block,
> > > > > > then
> > > > > > > users can further configure a specific one by modifying the
> > > compiled
> > > > > json
> > > > > > > plan...
> > > > > > >
> > > > > > > In a word, this proposal is in good shape as an enhancement for
> > > > > compiled
> > > > > > > json plan, and it's orthogonal with other ways like SQL Hint
> > which
> > > > > works
> > > > > > in
> > > > > > > a higher level.
> > > > > > >
> > > > > > >
> > > > > > > Nips:
> > > > > > >
> > > > > > > > "From the SQL semantic perspective, hints cannot intervene in
> > the
> > > > > > > calculation of data results."
> > > > > > > I think it's more properly to say that hint does not affect the
> > > > > > > equivalence of execution plans (hash agg vs sort agg), not the
> > > > > > equivalence
> > > > > > > of execution results, e.g., users can set 'scan.startup.mode'
> for
> > > > kafka
> > > > > > > connector, which also "intervene in the calculation of data
> > > results".
> > > > > > >
> > > > > > > Sincerely,
> > > > > > > Shuo
> > > > > > >
> > > > > > > On Tue, Mar 21, 2023 at 7:52 PM Jane Chan <
> qingyue@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > >> Hi devs,
> > > > > > >>
> > > > > > >> I'd like to start a discussion on FLIP-292: Support
> configuring
> > > > state
> > > > > > TTL
> > > > > > >> at operator level for Table API & SQL programs [1].
> > > > > > >>
> > > > > > >> Currently, we only support job-level state TTL configuration
> via
> > > > > > >> 'table.exec.state.ttl'. However, users may expect a
> fine-grained
> > > > state
> > > > > > TTL
> > > > > > >> control to optimize state usage. Hence we propose to
> > > > > > serialize/deserialize
> > > > > > >> the state TTL as metadata of the operator's state to/from the
> > > > compiled
> > > > > > >> JSON
> > > > > > >> plan, to achieve the goal that specifying different state TTL
> > when
> > > > > > >> transforming the exec node to stateful operators.
> > > > > > >>
> > > > > > >> Look forward to your opinions!
> > > > > > >>
> > > > > > >> [1]
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240883951
> > > > > > >>
> > > > > > >> Best Regards,
> > > > > > >> Jane Chan
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
--
Best,
Benchao Li
s://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240885081
> > [2] https://lists.apache.org/thread/bk8x0nqg4oc62jqryj9ntzzlpj062wd9
> >
> >
> > Best,
> > Lincoln Lee
> >
>
--
Best,
Benchao Li
read/d1owrg8zh77v0xygcpb93fxt0jpjdkb3
>
>
> Best,
> Shammon.FY
>
--
Best,
Benchao Li
5% compatibility and small
> > file
> > > compaction.
> > > In addition, Yuxia driven FLIP-282 (DELETE & UPDATE API) to better
> > > integrate
> > > Flink with data lakes. He actively participated in dev discussions and
> > > answered
> > > many questions on the user mailing list.
> > >
> > > Please join me in congratulating Yuxia Luo for becoming a Flink
> > Committer!
> > >
> > > Best,
> > > Jark Wu (on behalf of the Flink PMC)
> > >
> >
>
--
Best,
Benchao Li
an
> > > > > option such as `table.result.changelog-mode`, users can set it
> > through
> > > > > connection properties or set dynamic parameters. The option value
> can
> > > be
> > > > > `insert-only`, `upset` or `all` and the default value
1 - 100 of 285 matches
Mail list logo
